Hacker News Personal Blogs 2016 | Top 100 Blogs

Walking through a basic Racket web service

Notes on software development — Thu, 29 Dec 2016 02:00:00 GMT

Racket is an impressive language and ecosystem. Compared to Python, Racket (an evolution of Scheme R5RS is three years younger. It is as concise and expressive as Python but with much more reasonable syntax and semantics. Racket is also faster in many cases due in part to:

JIT compilation on x86 platforms
support for both concurrency and parallelism
support for optimizing statically-typed code

Furthermore, the built-in web server libraries and database drivers for MySQL and PostgreSQL are fully asynchronous. This last bit drove me here from Play / Akka. (But strong reservations about the complexity of Scala and the ugliness of Play in Java helped too.)

With this motivation in mind, I'm going to break down the simple web service example provided in the Racket manuals. If you don't see the following code in the linked page immediately, scroll down a bit.

#lang web-server

(require web-server/http)

(provide interface-version stuffer start)

(define interface-version 'stateless)

(define stuffer
  (stuffer-chain
   serialize-stuffer
   (md5-stuffer (build-path (find-system-path 'home-dir) ".urls"))))

(define (start req)
  (response/xexpr
   `(html (body (h2 "Look ma, no state!")))))

First we notice the #lang declaration. Racket libraries love to make new "languages". These languages can include some entirely new syntax (like the Algol language implementation) or can simply include a summary collection of libraries and alternative program entrypoints (such as this web-server language provides). So the first thing we'll do to really understand this code is to throw out the custom language. And while we're at it, we'll throw out all typical imports provided by the default racket language and use the racket/base language instead. This will help us get a better understanding of the Racket libraries and the functions we're using from these libraries.

While we're throwing the language away, we notice the paragraphs just below that original example in the manual. It mentions that the web-server language also imports a bunch of modules. We can discover which of these modules we actually need by searching in the Racket manual for functions we've used. For instance, searching for "response/xexpr" tells us it's in the web-server/http/xexpr module. We'll import the modules we need using the "prefix-in" form to make function-module connections explicit.

#lang racket/base

(require (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in hash: web-server/stuffers/hash)
         (prefix-in stuffer: web-server/stuffers/stuffer)
         (prefix-in serialize: web-server/stuffers/serialize))

(provide interface-version stuffer start)

(define interface-version 'stateless)

(define stuffer
  (stuffer:stuffer-chain
   serialize:serialize-stuffer
   (hash:md5-stuffer (build-path (find-system-path 'home-dir) ".urls"))))

(define (start req)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!")))))

Now we've got something that is a little less magical. We can run this file by calling it: "racket server.rkt". But nothing happens. This is because the web-server language would start the service itself using the exported variables we provided. So we're going to have to figure out what underlying function calls "start" and call it ourselves. Unfortunately searching for "start" in the manual search field yields nothing relevant. So we Google "racket web server start". Down the page on the second search result we notice an example using the serve/servlet function to register the start function. This is our in.

#lang racket/base

(require (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in hash: web-server/stuffers/hash)
         (prefix-in stuffer: web-server/stuffers/stuffer)
         (prefix-in serialize: web-server/stuffers/serialize)
         (prefix-in servlet-env: web-server/servlet-env))

(provide interface-version stuffer start)

(define interface-version 'stateless)

(define stuffer
  (stuffer:stuffer-chain
   serialize:serialize-stuffer
   (hash:md5-stuffer (build-path (find-system-path 'home-dir) ".urls"))))

(define (start req)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!")))))

(servlet-env:serve/servlet start)

Run this version and it works! We are directed to a browser with our HTML. But we should clean this code up a bit. We no longer need to export anything so we'll drop the provide line. We aren't even using the interface-version and stuffer code. Things seem to be fine without them, so we'll drop those too. Also, looking at the serve/servlet documentation we notice some other nice arguments we can tack on.

#lang racket/base

(require (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in servlet-env: web-server/servlet-env))

(define (start req)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!")))))

(servlet-env:serve/servlet
 start
 #:servlet-path "/"
 #:servlet-regexp rx""
 #:stateless? #t)

Ah, that's much cleaner. When you run this code, you will no longer be directed to the /servlets/standalone.rkt path but to the site root -- set by the #:servlet-path optional variable. Also, every other path you try to reach such as /foobar will successfully map to the start function -- set by the #:servlet-regexp optional variable. Finally, we also found the configuration to set the servlet stateless -- set by the optional variable #:stateless?.

But this is missing two things we could really use out of a simple web service. The first is routing. We do that by looking up the documentation for the web-server/dispatch module. We'll use this module to define some routes -- adding a 404 route to demonstrate the usage.

#lang racket/base

(require (prefix-in dispatch: web-server/dispatch)
         (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in servlet: web-server/servlet-env))

(define (not-found-route request)
  (xexpr:response/xexpr
   `(html (body (h2 "Uh-oh! Page not found.")))))

(define (home-route request)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!!!!!!!!!")))))

(define-values (route-dispatch route-url)
  (dispatch:dispatch-rules
   [("") home-route]
   [else not-found-route]))

(servlet:serve/servlet
 route-dispatch
 #:servlet-path "/"
 #:servlet-regexp #rx""
 #:stateless? #t)

Run this version and check out the server root. Then try any other path. Looks good. The final missing piece to this simple web service is logging. Thankfully, the web-server/dispatch-log module has us covered with some request formatting functions. So we'll wrap the route-dispatch function and we'll print out the formatted request.

#lang racket/base

(require (prefix-in dispatch: web-server/dispatch)
         (prefix-in dispatch-log: web-server/dispatchers/dispatch-log)
         (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in servlet: web-server/servlet-env))

(define (not-found-route request)
  (xexpr:response/xexpr
   `(html (body (h2 "Uh-oh! Page not found.")))))

(define (home-route request)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!!!!!!!!!")))))

(define-values (route-dispatch route-url)
  (dispatch:dispatch-rules
   [("") home-route]
   [else not-found-route]))

(define (route-dispatch/log-middleware req)
  (display (dispatch-log:apache-default-format req))
  (flush-output)
  (route-dispatch req))

(servlet:serve/servlet
 route-dispatch/log-middleware
 #:servlet-path "/"
 #:servlet-regexp #rx""
 #:stateless? #t)

Run this version and notice the logs displayed for each request. Now you've got a simple web service with routing and logging! I hope this gives you a taste for how easy it is to build simple web services in Racket without downloading any third-party libraries. Database drivers and HTML template libraries are also included and similarly well-documented. In the future I hope to add an example of a slightly more advanced web service.

I have had huge difficulty discovering the source of Racket libraries. These library sources are nearly impossible to Google and search on Github is insane. Best scenario, the official racket.org docs would link directly to the source of a function when the function is documented. Of course I could just download the Racket source and start grepping... but I'm only so interested.

Walking through a basic Racket web service https://t.co/J3us48kzga @racketlang
— Phil Eaton (@phil_eaton) December 30, 2016

Falsehoods Programmers Believe About CSVs

Donat Studios — Wed, 28 Dec 2016 03:41:04 GMT

Much of my professional work for the last 10+ years has revolved around handling, importing and exporting CSV files. CSV files are frustratingly misunderstood, abused, and most of all underspecified. While RFC4180 exists, it is far from definitive and goes largely ignored.

Partially as a companion piece to my recent post about how CSV is an encoding nightmare, and partially an expression of frustration, I've decided to make a list of falsehoods programmers believe about CSVs. I recommend my previous post for a more in-depth coverage on the pains of CSVs encodings and how the default tooling (Excel) will ruin your day.

Everything on this list is a false assumption that developers make.

All CSVs are ASCII
All CSVs are Win1252
All CSVs are in 8-bit encodings
All CSVs are UTF-8
All CSVs are UTF-16
All CSVs contains a single consistent encoding
All records contain a single consistent encoding
All fields contain a single consistent encoding
All CSVs contain records
All records contain fields
Fields never contain record separators
Fields never contain delimiters
Fields never contain control characters
Delimiters are escaped with a \
All fields are enclosed by double quotes
All records are a single line
All lines contain a single record
All records contain the same number of fields
All records contain the same number of fields as the header
All records contain the same number of fields or fewer than the header
All CSVs contain a header
All record separators are CRLF
All record separators are LF
All record separators are a single byte
All record separators are a single rune
All newlines are a single byte
All CSVs are delimited with a comma
All CSVs are delimited with a comma, tab or semicolon
TSV isn't CSV
All delimiters are a single byte
All commas are a single byte
All CSVs are readable with Excel
Excel is a good tool for working with CSVs
Excel is an OK tool for working with CSVs
Excel can losslessly save CSVs it opens
Using ="{value}" is a good way to get around Excel auto-formatting
The first line will never be a poorly supported instruction header
Using sep={char} is a good way to get Excel to accept your delimiter
Prepending a BOM is a good way to get Excel to read your encoding
You can safely name your first column "ID"
All CSVs follow RFC4180
Most CSVs follow RFC4180
All CSVs follow the same defined standard
All CSVs follow a defined standard
All CSVs have a .csv extension
All CSV is human readable

Please take these into consideration next time you find yourself working with CSV. If you can think of anything I may have missed I'd be happy to add it.

As a suggested further reading, "The Art of Unix Programming" http://www.catb.org/esr/writings/taoup/html/ch05s02.html#id2901882 section on DSV style which notably says "CSV is a textbook example of how not to design a textual file format"

Updates:

Thanks to Max for pointing out the Excel supported sep={value} header I was strangely entirely unaware of.
Thanks to Don Hopkins for the note about not being able to start a header with ID

Vertically striping images for fun

infrequent oscillations — Sat, 24 Dec 2016 10:46:10 GMT

In which we create a montage.

MH-Z19 CO2 sensor reader, logger and visualiser

smcleod.net — Wed, 21 Dec 2016 07:37:38 GMT

MH-Z19 CO2 sensor reader, logger and visualiser

Reads data from UART(serial)-connected MH-Z19 (or MH-Z14) sensor using python 3.
If you dare to install nodejs you can visualise the logged data (using html and plotly.js library).

Repository: sammcj/CO2-Logger

Usage

Note this post is from 2016, in 2021 I replaced my custom Co2 loggers with an Aranet4. While very expensive, is and excellent off-the-shelf solution, with many features.

Connection

Sensor can be queried using 3.3v UART at 9600 bps. Sensor main feed voltage is 5v.

Annoying CSV and the JSON solution

infrequent oscillations — Wed, 07 Dec 2016 00:44:30 GMT

In which we skirt a CSV problem by using JSON.

Belated ElixirConf 2016 Recap

Brightball Articles — Wed, 30 Nov 2016 02:00:00 GMT

This past September I took vacation time and paid out of pocket to drive to Orlando and attend ElixirConf with a few other programmers from Greenville who did the same thing. We weren't the only ones. Here is a belated recap from our combined notes and experiences.

Vignettes

infrequent oscillations — Tue, 29 Nov 2016 15:11:20 GMT

In which we summarise through subtitles.

Repeating History...on Purpose...with Elixir

Brightball Articles — Tue, 29 Nov 2016 02:00:00 GMT

A dive into the highlights of Elixir that make it the ideal platform for the web...and how all these questions were answered figured out 30 years ago. Presented to Upstate Elixir in Greenville, SC on Nov 16.

The many methods of Decim8

infrequent oscillations — Fri, 25 Nov 2016 19:16:51 GMT

In which we try to provide a visual reference for Decim8.

Syncing from Postgres to Salesforce - Data Mappings

CRAIG KERSTIENS — Wed, 23 Nov 2016 22:55:56 GMT

For the second time now I’ve had to implement a system that syncs from my system of record into Salesforce.com, the first at Heroku and now at Citus Data. The case here is pretty simple, I have a software-as-a-service, B2B product. It’s a homegrown application in these cases in Ruby, but could be Python, .Net, any language of your choosing. The problem is I don’t want to have to be rebuilding my own CRM, reporting, etc. on top of all of my internal database. And as soon as you’re at some reasonable size (sales guy of 1 or more) you need to be able to provide insights on what’s in that system of record database to others.

While my tooling isn’t a full fledged product by any means, here’s a bit of how I’ve developed this process a few times over and some of the annoying bits of code to help get you started. In this post I’ll walk through some of the basic datatypes, then we’ll follow-up with the overall architecture and tweaks you need to make to Salesforce, and finally we’ll provide some example code to help you create this setup yourself.

Leads, Contacts, Accounts oh my

Despite being some of the largest as-a-service vendors in the world, Salesforce is still primarily setup for traditional high touch sales. What this means is some of the data you’ll commonly have, or in this case not have, can make it difficult to figure out what maps from your internal system to Salesforce. Within Salesforce there’s really 4 key data models you’re going to care about.

Lead vs. Contact

In every as a service product you’ll have some user that creates and account which usually has an email address tied to it. This seems simple enough to load up to Salesforce as there is a clear email field. Within Salesforce there are two key data types which have a default field for this lead and contact, in Salesforce terms a lead is someone considering doing business with you, a contact someone who more so is doing business with you. If you have a freemium or timed trial model you might think to start classifying everyone that they’re a lead. Then, when they convert to a paying customer you turn them into a contact.

If you’re anything like me, in running your SaaS business, you want a sign-up process that’s frictionless. This means give me an email address, password, and you’re off and running. Salesforce immediately starts to breakdown a bit in this regard. First you’re required for both lead and contact to provide a first and last name. In my case I do ask for name, and do a little bit of work on the code side to get values into both. You’ll see later that our process does result in some regular cleanup work needing to happen, but in our case we’re optimizing to get them signed up more than capturing every detail perfectly about them from the start.

Leads are even more broken than contacts though. Leads require you to enter a company. While you may be able to just drop a company form field onto your sign-up page you’re likely to end up with junk data at least, if not actually driving some sign-ups away. Some of my favorite pieces of junk data I’ve seen users enter for company name: “pissed off developer”, “Acme Inc.”, and the all too common “Test Co.”. In reality these are often real developers, with real problems, and real budget, they just don’t want to share details before they’re ready.

So in this case the TLDR; is that leads require:

First name
Last name
Email
Company name

This results in contacts being a more favorable datatype because it only requires:

First name
Last name
Email

Accounts vs. Opportunities

We have in some ways a similar but different dichotomy with Accounts and Opportunities as we did Leads and Contacts. Though this one can often map a bit more cleanly than we saw with leads. From a pretty straight forward definition:

Account - A business entity. Contacts work for Accounts.
Opportunities - Sales events related to an Account and one or more Contacts.

This again can become problematic if you have no notion of Accounts at all in your system of record. Though if you are building a B2B application there is a good chance you may have something that makes sense. If you let uses free-form enter this instead of AT&T they may put “interactive team”, but you at least have some logical team that in their mind they roll up to.

Opportunities is a much harder one in the SaaS world. In traditional marketing you have your standard stages of MQL (Marketing Qualified Lead), progressing to SQL (Sales Qualified Lead), etc. that you expect these potential customers to flow through. In the as-a-service world you may have people look from afar for weeks, then suddenly sign-up and give you a credit card and start paying within minutes. While there is still steps the customer may go through before buying you often have less insight into these. How you decide to structure your opportunities flow is entirely up to you. In my case I tend to opt to still have htem, but they’re an exception basis where a salesperson is actively engaged vs. the other 90%+ of fully self-service customers.

Shifting back a little bit on accounts. The key with accounts is that if you have some notion of an team or org within your system of record then it makes sense to have that same structure setup in Salesforce. The most basic of this might be an idea of “Account owner” and “Team members”. You may have a person in there just for billing, an admin, and then users. Even if you don’t want to recreate the entire structure at least having all the contacts tied to the account is critical. I can’t count the number of times I’ve seen teams setup a “billing@mycompany.com” email, seen people try to interact with that email when in reality they wanted to be talking to “jane@mycompany.com” who logged in yesterday.

In summary

For the most part Salesforce doesn’t quite let you map to what many of you’ll want to do in terms of mapping your data from your system of record to Salesforce. Expect to have to contort a bit and likely pump Salesforce with some garbage data. In general you’ll want to skip leads and go straight for contacts as contacts don’t require the same restrictions. Tying contacts to an account is the right level anyway, and from there up to you on how you’ll more manage the opportunities.

From Developer to DevOps: My Story

Ben Overmyer's Site — Sun, 20 Nov 2016 02:00:00 GMT

In late 2014, the sole system administrator for the Star Tribune Digital department (let's call him John) turned in his two-weeks' notice. Thus began a panic as we realized we had no idea how our infrastructure worked. At the time, I was one of two senior software engineers there. I was in the middle of a large refactoring project for the mobile website. John had, thankfully, kept up an internal wiki that at least roughly documented how some of the legacy infrastructure worked. The new infra on AWS was very new and very undocumented, but at least we had a place to start. John's disciple, an eager but more junior developer who had originated as a copy-editor, did what he could to absorb as much knowledge as he could from John before he left. But his heart just wasn't in operations, and he didn't want to fall on that sword.

In a one-on-one with my boss, we talked about the dilemma facing us. I mentioned that I had some experience with Linux in production environments. He gently suggested that I take on the system administration role, at least until they could hire someone to replace John. I blanched, knowing enough to know that I knew little about the world of operations. I was reluctant, but in the end he convinced me to at least try and hold down the fort until help arrived. I set the condition that I be immediately removed from all development projects I was assigned to. My boss agreed. At this point, we were about six months from launch of a scratch-built web platform that would replace our proprietary hosted CMS. None of the developers had experience in operational concerns at that scale, including myself. John had written some Chef cookbooks to provision the few things living on the new infrastructure, but only an outside contractor brought in to help in the interim with operations actually knew Chef. We struggled for a short period of time to maintain it. I learned enough of Ruby and Chef to get by, but it was always a battle to get the tool to do what I expected whenever I needed to do something slightly different from the existing process.

I forget who originally suggested it, but we started to switch our orchestration code to Ansible. Ansible was far easier for me to wrap my head around, and at a time when I needed to learn as much as possible as fast as possible, that was a lifesaver. At this point, I'd been “the operations guy” for about a month. The hiring ad for a “devops engineer” had been open for a month and a half. We had had only a few interested parties, and none of them were anything close to what we needed. I saw the writing on the wall, and asked my boss for a meeting. I told him I was willing to permanently become the operations engineer for the team. His relief was obvious. Shortly after that, the engineer opening changed from operations to developer, and within a short space of time we had an awesome new junior developer on the team. Meanwhile, I dove headlong into the world of system administration and operations. No longer held back by my own thought of “this is only temporary,” I devoured anything and everything I could find on the subject of what Google would later call Site Reliability Engineering.

After a couple short months, all of the Chef code was gone, a lot of our infrastructure had been deployed onto AWS, and we had hired a junior operations engineer to help out. Our on-call rotation was still a manually-maintained forwarded phone line, though, and fighting fires was always a panic moment… since usually, it was a problem that had been happening for hours, and we didn't even know about it until someone from the news floor informed us. But that's a story for another time. It was the launch of our web platform that completely changed operations for us, and for me. In early May 2015, the new website launched. And immediately crashed. And stayed down. We reverted to the old CMS. The next ten days saw both development and operations working feverishly to try and figure out what had gone wrong and how to fix it.

We switched from using a clever-but-misleading traffic simulation to using a tool called siege to do load testing, and finally got to a point where we thought we could keep the new site online. At 3:00 AM ten days after the first attempt, the new platform went online again and stayed online. For the next three hours, the launch team (including myself as the sole operations engineer on the team) built and deployed multiple iterations of the new site, and finally at six in the morning, we all left the office. I greeted the post-launch engineering team as we passed each other in the office entrance. After a couple hours' sleep in a downtown hotel, the launch team returned to the office to assess the situation and help out with any problems that might have occurred.

Let me be clear. When I was a developer, I pulled my share of sixteen-hour days to meet deadlines. But I never before this launch had experienced a shift this long or this rewarding. Before then, I had never cheered and exchanged high fives with the team as we saw the site working as expected in the wee hours of the morning. At this point, I was invested in operations. It had acquired my interest and not just my attention.

Sphinx Real Time Index How to Distribute and Hidden Gotcha

Ben E. C. Boyter — Fri, 11 Nov 2016 11:34:42 GMT

I have been working on real time indexes with Sphinx recently for the next version of searchcode.com and ran into a few things that were either difficult to search for or just not covered anywhere.

The first is how to implement a distributed search using real time indexes. It’s actually done the same way you would normally create an index. Say you had a single server with 4 index shards on it and you wanted to run queries against it. You could use the following,

Death's End

Krishna's blog — Thu, 10 Nov 2016 02:00:00 GMT

Liu Cixin's Death's End is a special set of books that puts as much Science as it can in Science Fiction.

Don’t build cockpits, become a coach

Bytepawn - Marton Trencseni — Wed, 09 Nov 2016 01:00:00 GMT

I used to think that a good analogy for using data is the instrumentation of a cockpit in an airliner. Lots of instruments, and if they fail, the pilot can’t fly the plane and bad things happen. There’s no autopilot for companies. The problem with this analogy is that planes aren’t built in mid-air. Product teams and companies constantly need to build and ship new products.

Noisevember 2016

infrequent oscillations — Mon, 31 Oct 2016 11:20:07 GMT

Having singularly failed with all of Juneathon, Inktober and Blogtober, I shall be picking myself up, dusting myself down, and having another crack at Noisevember.

Explaining VarnishHist – What Does it Tell Us

Ben E. C. Boyter — Thu, 27 Oct 2016 02:56:01 GMT

The varnishhist tool is one of the most underused varnish tools that come with your standard varnish install. Probably because of how it appears at first glance.

In short, you want as many | symbols as possible and you want everything far toward the left hand side. The closer to the left the faster the responses are regardless if they are cached or not. The more | symbols then more items were served from cache.

MySQL Dump Without Impacting Queries

Ben E. C. Boyter — Wed, 26 Oct 2016 04:11:01 GMT

Posted more for my personal use (I have to look it up every time) but here is how to run a mysqldump without impacting performance on the box. It sets the ionice and nice values to be as low as possible (but still run) and uses a single transaction and ups the max packet size for MySQL.

ionice -c2 -n7 nice -n19 mysqldump -u root -p DATABASE --single-transaction --max_allowed_packet=512M > FILENAME

Open DNS for when DNS outages occur

CRAIG KERSTIENS — Fri, 21 Oct 2016 23:55:56 GMT

Open DNS is a DNS resolver that caches records beyond their TTL if the upstream DNS server cannot be found. In cases like today’s major outage it can be handy to swap your DNS settings out for this, or it may be worth using as a standard default. Resolution may be a bit slow as it will try to see if the upstream server cannot be found, but it at least can get you back to a working state.

If you know what you’re doing then all you need to do is configure your DNS settings to: 208.67.220.220 and 208.67.222.222.

If you need a little more guidance you can go into your System Preferences on Mac, select Network, then Advanced and finally the DNS tab. You should set it up to look as follows:

Embedded Linux Conference Europe 2016

Linux Engineer's random thoughts — Fri, 21 Oct 2016 01:00:00 GMT

I was in Berlin last week for ELCE, and it was great. It was a nice mix of talks on many different subjects, and as always you come back with lots of new ideas and improved motivation.

As you know, I took some notes for Kernel Recipes 2016, and lots …

Elixir ETS + Mnesia vs Redis

Brightball Articles — Sun, 09 Oct 2016 03:00:00 GMT

Learning Elixir has a way of challenging everything you know about programming. Redis is becoming an assumed part of many web stacks, in the same breath as your database. But with Elixir, do you need it? Do you even need a database?

Blogtober 2016, #2

infrequent oscillations — Sun, 02 Oct 2016 23:39:33 GMT

Today’s #inktober prompt was “NOISY”. Not a great effort but at least he looks human. Also had a decent Cathedral win on BoI:A even though the very first item was Dr Fetus - made for some tense dodging and shimmying! Still haven’t managed a win streak of more than one but the loss streaks are slowly getting shorter…

Blogtober 2016, #1

infrequent oscillations — Sun, 02 Oct 2016 00:53:43 GMT

Made some musical output using a couple of samples taken from a Tidal performance by Alex (with permission) - “Deconstructing Yaxu” Mostly just a droney throbby glitchy noisescape thrown together using Samplr in a coffee shop. Also did day 1, “fast”, of #inktober.

Kernel Recipes 2016 notes

Linux Engineer's random thoughts — Wed, 28 Sep 2016 01:00:00 GMT

Update 2016-10-21: I've added links to the videos and LWN articles, which are of much higher quality than these live notes.

This year I'm trying a live blog of Kernel Recipes 2016, live from Paris, at Mozilla's headquarters. You can watch the live stream here.

The kernel report

by Jonathan …

A tour of Postgres' Foreign Data Wrappers

CRAIG KERSTIENS — Sun, 11 Sep 2016 23:55:56 GMT

SQL can be a powerful language for reporting. Whether you’re just exploring some data, or generating reports that show month over month revenue growth it’s the lingua franca for data analysis. But, your data isn’t always in a SQL database, even then if you’re using Postgres you can still likely use SQL to analyze, query, even joing with that data. Foreign data wrappers have been around for years in Postgres, but are continuing to mature and be a great option for joining disparate systems.

Overview of foreign data wrappers

If you’re unfamiliar, foreign data wrappers, or FDW, allow you to connect from within Postgres to a remote system. Then you can query them from directly within Postgres. While there is an official Postgres FDW that ships with Postgres itself, that allows you to connect from one Postgres DB to another, there’s also a broad community of others.

At the core of it Postgres provides certain APIs under the covers which each FDW extension can implement. This can include the ability to map SQL to whatever makes sense for a given system, push down various operators like where clauses, and as of Postgres 9.3 can even write data.

To setup a FDW you first would install the extension, then provide the connection to the remote system, setup your schema/tables, and then you’re off to the races–or well ready to query. If you’ve got more than 2-3 databases or systems in your infrastructure, you’ll often benefit from FDWs as opposed to introducing a heavyweight ETL pipeline. Don’t mistake FDWs as the most performant method for joining data, but they are often the developer time efficient means of joining these data sets.

Let’s look at just a few of the more popular and interesting ones.

Postgres FDW

The Postgres one is the easiest to get started with. First you’ll just enable it with CREATE EXTENSION, then you’ll setup your remote server:

CREATE EXTENSION postgres_fdw;
CREATE SERVER core_db
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host 'foo', dbname 'core_db', port '5432');

Then you’ll create the user that has access to that database:

CREATE USER MAPPING FOR bi SERVER core OPTIONS (user 'bi', password 'secret');

Finally, create your foreign table:

CREATE FOREIGN TABLE core_users (
id integer NOT NULL,
username varchar(255),
password varchar(255),
last_login timestamptz
)
SERVER core;

Now you’ll see a new table in the database you created this in called core_users. You can query this table just like you’d expect:

SELECT *
FROM core_users
WHERE last_login >= now() - '1 day'::interval;

You can also join against local tables, such as getting all the invoices for users that have logged in within the last month:

SELECT *
FROM invoices, core_users
WHERE core_users.last_login >= now() - '1 month::interval'
AND invoices.user_id = core_users.id

Hopefully this is all straight forward enough, but let’s also take a quick look at some of the other interesting ones:

MySQL FDW

For MySQL you’ll also have to download it and install it as well since it doesn’t ship directly with Postgres. This should be fairly straight forward:

$ export PATH=/usr/local/pgsql/bin/:$PATH
$ export PATH=/usr/local/mysql/bin/:$PATH
$ make USE_PGXS=1
$ make USE_PGXS=1 install

Now that you’ve built it you’d follow a very similar path to setting it up as we did for Postgres:

CREATE EXTENSION mysql_fdw;
CREATE SERVER mysql_server
FOREIGN DATA WRAPPER mysql_fdw
OPTIONS (host '127.0.0.1', port '3306');
CREATE USER MAPPING FOR postgres
SERVER mysql_server
OPTIONS (username 'foo', password 'bar');
CREATE FOREIGN TABLE core_users (
id integer NOT NULL,
username varchar(255),
password varchar(255),
last_login timestamptz
)
SERVER mysql_server;

But MySQL while different than Postgres is also more similar in SQL support than say a more exotic NoSQL store. How well do they work as a foreign data wrapper? Let’s look at our next one:

MongoDB

First you’ll go through much of the same setup as you did for MySQL. The one major difference though is in the final step to setup the table. Since a table doesn’t quite map in the same way with Mongo you have the ability to set two items: 1. the database and 2. the collection name.

CREATE FOREIGN TABLE core_users(
_id NAME,
user_id int,
user_username text,
user_last_login timestamptz)
SERVER mongo_server
OPTIONS (database 'db', collection 'users');

With this you can do some basic level of filtering as well:

SELECT *
FROM core_users
WHERE user_last_login >= now() - '1 day'::interval;

You can also write and delete data as well now just using SQL:

DELETE FROM core_users
WHERE user_id = 100;

Of course just putting SQL on top of Mongo doesn’t mean you get all the flexibility of analysis that you’d have directly within Postgres, this does go a long way towards allowing you to analyze data that lives across two different systems.

Many more

A few years ago there were some key ones which already made FDWs useful. Now there’s a rich list covering probably every system you could want. Whether it’s Redis, a simple CSV one, or something newer like MonetDB chances are you can find an FDW for the system you need that makes your life easier.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Making a Twitter bot that looks for hashes

Linux Engineer's random thoughts — Fri, 09 Sep 2016 19:00:00 GMT

This is a followup to What do you find when you search Twitter for hashes ?

Why ?

I'm not sure I remember how it started.

It all started four years ago. Jon Oberheide was still an independent security researcher and not yet CTO of a successful product company. He posted some …

What do you find when you search Twitter for hashes ?

Linux Engineer's random thoughts — Fri, 09 Sep 2016 18:00:00 GMT

This image:

This is what I found with hashbot, a twitter bot that looks for hashes.

What is this image ?

Posted with the hash "2f404a288d1b564fadee944827a39a14" by japanese accounts (of which @furueru_zekkei used to be the top poster, now suspended).

After a bit of research on google images and more, I …

To all Companies Currently Recruiting

Ben E. C. Boyter — Fri, 09 Sep 2016 01:29:19 GMT

I am writing this on behalf of all developers/engineers out there. Please stop with the take home coding challenge questions. Really. Just stop it. They are a lazy and frankly an unprofessional way of sorting the wheat from the chaff. Before closing your browser in disgust hear me out on this one and hopefully I can convince you of the error of your ways.

There has become an alarming trend these days of companies during the hiring process to issue lengthy coding challenges in order to prove that the individual they are hiring knows their stuff. I totally understand why you might be doing this but frankly its flawed. Lets go through several reasons why.

Guess the city

infrequent oscillations — Tue, 06 Sep 2016 11:23:06 GMT

In which the future is predicted.

GPL Time-bomb an interesting approach to #FOSS licensing

Ben E. C. Boyter — Mon, 29 Aug 2016 04:47:39 GMT

UPDATES Following some feedback I am going to rename my usage of “Time-Bomb” due to potential negative connotation on the words. I am going to call it “Eventually Open”. Also a few other things need mentioning. I am not looking for code submissions back into the source at this time. This was a move to show that there are no back-doors in the code sending source code back to a master server.

Test-Defective

The Cranky Developer on Crater Moon Development — Sat, 27 Aug 2016 03:00:00 GMT

If you have automated tests, believe them or discard them.

searchcode server under fair source

Ben E. C. Boyter — Thu, 25 Aug 2016 02:38:33 GMT

A very quick blog today. I have released searchcode server under the fair source licence. This means that as of a few days ago you can view the source, change it modify it and run it as you see fit so long as you have less than 5 users.

The source is hosted on github (I may move this to GitLab sometime in the future) and you can view it here.

So what does this mean? Well the community edition still exists (run searchcode with as many users as you want) as do the paid versions with support and all the full features. The real advantage however is that you can now vet the source code to ensure that searchcode server is not secretly sending your most valuable asset to some hidden server somewhere. In addition it means I can now talk about the source openly and will be writing some posts about how I ran into some CPU branching issues which slowed down some code.

The Worst Individual I Ever Worked With

Ben E. C. Boyter — Tue, 23 Aug 2016 05:55:20 GMT

Taken from a comment I posted on HN in a thread about a Soccer Con Man.

Not actually a programmer. The guy was hired to be a project manager.

After joining things were as expected but after a few weeks we noticed that he was rarely around after lunch and never around after lunch on a Friday.

We would email him at those times deliberately to catch him out and I recall starting to put sticky notes on his laptop “Came to see you a X time”. He would come back and just dump all the notes in the rubbish and claim he never got them. He would often claim to be working from home, despite his laptop being on his desk and usually closed. He would also never responding during those times to email or IM.

Types of Testing in Software Engineering

Ben E. C. Boyter — Thu, 18 Aug 2016 01:42:37 GMT

There are many different types of testing which exist in software engineering. They should not be confused with the test levels, unit testing, integration testing, component interface testing, and system testing. However the different test levels may be used by each type as a way of checking for software quality.

The following are all different types of tests in software engineering.

A/B

A/B testing is testing the comparison of two outputs where a single unit has changed. It is commonly used when trying to increase conversion rates for online websites. A real genius in this space is Patrick McKenzie and a few very worthwhile articles to read about it are How Stripe and AB Made me A Small Fortune and AB Testing

Castle Hill Winter Trip 2016 Photos

smcleod.net — Sat, 06 Aug 2016 03:00:00 GMT

Castle Hill Winter Trip 2016 [Photos]

Email as a Microservice

Brightball Articles — Sat, 06 Aug 2016 03:00:00 GMT

Email might be one of the most often overlooked pieces of any web application. Usually the biggest discussion around it in a project begins and ends with “and we’ll send them an email when this happens…”. A little thought and some minor adjustments can help us avoid some problems that will grow as your project does. Let’s talk about email as a microservice.

Syncing Stash/BitBucket with searchcode server

Ben E. C. Boyter — Thu, 04 Aug 2016 08:14:19 GMT

Recently it came up to perform a slight integration piece between a on premises Stash/BitBucket install and a searchcode server install. Thankfully both have an API and very thankfully there is a nice Python library for talking to Stash/BitBucket.

Below is the code used. It pulls out all of the repositories from every project, checks if it exists in searchcode and if not adds it as a repository to be indexed. You need to install stashy (pip install stashy) and run it whenever you have new repositories. One idea is to set it as a cron task and ensure everything is in sync.

Python Fabric: Getting File from Host as String

Ben E. C. Boyter — Tue, 02 Aug 2016 01:52:39 GMT

When using fabric for deployments you will sometimes want check an existing file for the presence of a value before applying an update. A common example I run into is checking if an apt-source has already been added before adding it again. This is a little clunky in fabric, but thankfully you can write a simple helper which takes case of it for you.

def _get_remote(fileloc):
    '''Pulls back a file contents from connection as string'''
    from StringIO import StringIO

    fd = StringIO()
    get(fileloc, fd)
    content = fd.getvalue()
    return content

Usage is fairly simple. Say we want to install the latest version of Varnish Cache on an Ubuntu server. Usage like so works,

On Check Constraints and Whole Value

The Cranky Developer on Crater Moon Development — Mon, 01 Aug 2016 03:00:00 GMT

A business object with setters for attributes that are interdependent, will need check constraints on the setters. But there is a better way, using the Whole Value pattern.

searchcode.com: The Architecture – migration 3.0

Ben E. C. Boyter — Thu, 28 Jul 2016 04:09:56 GMT

27th July 2016 at about 9:30pm my local time (GMT +10) I updated the A records for searchcode’s nameservers to point at a new stack that has been several months in the making. As with most posts of this sort of nature a quick recap of where things were and where they are now.

The previous searchcode stack consisted of two dedicated servers hosted by Hetzner. I have previously discussed this about two years ago when discussing searchcode next. The first server was a reasonably powerful machine running pretty much all of code required to deliver searchcode.com itself with the exception of the actual index. searchcode is a Django application and was using nginx to serve results directly out of memcached where possible to avoid consuming a running Gunicorn process. The move to have another server for the index came pretty quickly after searchcode was released as it was just not performant enough with everything on a single box which was the situation for a short time.

Counting Pizza with Python

Tom Forbes — Thu, 28 Jul 2016 03:03:32 GMT

I’m a full time nerd, even when I’m ordering pizza online I can’t stop myself from investigating how the websites I’m ordering from work. My latest investigation was Dominoes where I found a neat way to count the number of orders that they process throughout the day. This post is supposed to highlig...

PostgreSQL functions with Elixir Ecto

Brightball Articles — Wed, 27 Jul 2016 03:00:00 GMT

Functions within PostgreSQL can be setup to return rows and included in queries just like any other table. Continuing with our theme of trying to push Elixir and Phoenix a little on this site rebuild, we will move our site search inside of a database function and experiment with different ways to call it from Ecto.

Syntax highlighting and CSS support added to wordinserter

Tom Forbes — Tue, 26 Jul 2016 04:22:28 GMT

I recently added syntax highlighting and support for CSS stylesheets to wordinserter , and the implementation was satisfying enough that I thought I would blog about it. Wordinserter is a library I maintain that lets you insert HTML documents/snippets into Word documents: It’s primary use case is wh...

How to Hide Methods From Fabric Task Listing

Ben E. C. Boyter — Fri, 22 Jul 2016 01:34:49 GMT

Occasionally you may want to hide a method from appearing inside the fabric listing of available tasks. Usually its some sort of helper method you have created that is shared by multiple tasks. So how to hide it? Simply prefix with _

For example,

def _apt_get(packages):
    '''Makes installing packages easier'''
    sudo('apt-get update')
    sudo('apt-get -y --force-yes install %s' % packages)

When listing the fabric tasks this method will no longer appear in the results.

Python Fabric How to Show or List All Available Tasks

Ben E. C. Boyter — Thu, 21 Jul 2016 01:30:51 GMT

Showing or displaying the available tasks inside a fabric fabfile is one of those things that almost everyone wants to do at some point and usually works out you can just request a task you know will not exist (usually found through a typo). However there is a way to list them built into fabric itself.

The below are all methods which can be used to display the currently defined tasks.

Set Ubuntu Linux Swapfile Using Python Fabric

Ben E. C. Boyter — Wed, 20 Jul 2016 01:53:52 GMT

Annoyingly most cloud providers have an irritating habit of not adding any swap memory to any instance you spin up. Probably because if they added swap to the instance the disk size would appear to be smaller then it is or if they had a dedicated swap partition they would have to bear the cost or again use some of your disk space.

Thankfully adding swap to your Ubuntu linux instance is fairly easy. The following task when run will check if a swapfile already exists on the host and if not create one, mount it and set it to be remounted when the instance is rebooted. It takes in a parameter which specifies the size of the swap in gigabytes.

Python Fabric Set Host List at Runtime

Ben E. C. Boyter — Tue, 19 Jul 2016 01:40:13 GMT

With the advent of cloud computing where you spin up and tear down servers at will it becomes extremely useful to pick the hosts you want fabric to run on at runtime rather then through the usual env.hosts setting. This allows you to query your servers through your cloud providers API without having to maintain a list. This can be a more powerful and flexible technique then using roles and in a devops world can save you a lot of time.

The State of Android in 2016 & The OnePlus 3 Phone

smcleod.net — Mon, 11 Jul 2016 03:00:00 GMT

I wanted to try Android for a couple of weeks, I like staying on top of technology, gadgets and making sure I never become a blind ‘zealot’ for any platform or brand.

The OnePlus 3

I did a lot of research and decided to try the “Oneplus 3” as it was good bang-for-buck, ran the latest software had plenty of grunt with the latest 8 core, high clock speed Qualcomm processor coupled with 6GB of DDR4 - the specs really are very impressive, especially for a $400USD phone.

Insanity with Elixir + Phoenix + PostgreSQL

Brightball Articles — Wed, 06 Jul 2016 03:00:00 GMT

I'm at the borderline of obsessed with Elixir and Phoenix lately. I've avoided writing about it so far because it feels a bit too good to be true. In an effort to test my own enthusiam, I decided to rebuild this site with them in the most ridiculous way possible just to try to test some limits. Because I already have an unhealthy obsession with PostgreSQL, we're getting crazy with it too. DISCLAIMER: This is not a "how to build a blog" article. If you do what I'm about to do, people will look at you funny (and probably should).

Update Delayed Serial STONITH Design

smcleod.net — Mon, 04 Jul 2016 03:00:00 GMT

note: This is a follow up post from 2015-07-21-rcd-stonith

A Linux Cluster Base STONITH provider for use with modern Pacemaker clusters

This has since been accepted and merged into Fedora’s code base and as such will make it’s way to RHEL.

Source Code: Github
Diptrace CAD Design: Github
I have open sourced the CAD circuit design and made this available within this repo under CAD Design and Schematics
Related RedHat Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1240868

`v1` vs `v2/v3` versions of the `rcd_serial` STONITH system

The v2/v3 cables include the following improvements:

What is Chaos Testing / Engineering

Ben E. C. Boyter — Sat, 02 Jul 2016 10:48:29 GMT

A blog post by the excellent technical people at Netflix about Chaos Engineering and further posts about the subject by Microsoft in Azure Search prompted me to ask the question, What is chaos engineering and how can chaos testing be applied to help me?

What is Chaos Testing?

First coined by the afore-mentioned Netflix blog post, chaos engineering takes the approach that regardless how encompassing your test suite is, once your code is running on enough machines and reaches enough complexity errors are going to happen. Since failure is unavoidable, why not deliberately introduce it to ensure your systems and processes can deal with the failure?

CSV: An Encoding Nightmare

Donat Studios — Tue, 21 Jun 2016 03:17:57 GMT

UPDATE 2019-01-16: In the three years since this article was written, parts of the article, in particular talking about UTF-8 are thankfully no longer accurate.

It would appear in a recent update Microsoft has added support for safely reading and writing UTF-8 CSVs to Excel. There is a new format in the save dialog CSV UTF-8 (Comma delimited) which is distinct from Comma Separated Values which is also still in there.

~~In my testing it appears to safely both load and save. Best of all it appears if there is a BOM, it leaves it, if there isn't it doesn't add one. Very nice handling indeed.~~

UPDATE 2022-07-22:

It would appear that the stricken above was incorrect. Current versions of Excel will only reliably load a UTF-8 CSV if a UTF-8 BOM is in place.

The company I work for manages a self-service data import system, handling information from school districts “Student Information Systems”. There are hundreds if not thousands of SIS’s out there, but what the vast majority have in common is the ability to export CSV. It is the lingua franca, everyone can get us CSVs.

CSV to the casual observer seems a simple portable format, but its looks are deceiving. If the data is pure ASCII (bytes 0-127) you’ll be fine. However, if there is a need for any sort of non-ASCII character, there is some work ahead.

In our case, handling school data from around the world, correctly handling non-ASCII characters is of the utmost importance.

Excel: A World of Hurt

The biggest problem is not CSV itself, but that the primary tool used to interact with it is Excel. Excel handles CSV encodings badly.

Creating a new document in Excel and saving as ”Comma Separated Values (.csv)” it uses your locale’s Windows or Mac codepage. Win-1252 and MacRoman respectively in the United States. If your codepage doesn’t support a character in your document, it will be silently replaced with an underscore _ character.

Because it uses codepages and not a Unicode encoding, it makes processing a painful chore. There is no way to tell the difference between different 8-bit codepages programatically.

One can use heuristics to sort them into an order of likelihood, but there is no way to ever know for sure. We chose to present the user with previews of the most likely codepages, and let them pick the correct one.

Excel for Mac: Broken Beyond Belief

The Macintosh version of Microsoft Excel is particularly harrowing.

As I mentioned above, Excel saves your CSV in your locales codepage. One should note that the Mac codepages fell out of use with OS 9, almost 15 years ago. Microsoft did not get that memo.

While that by definition makes the CSVs the Mac version of Excel exports unusable on Windows, the problem is more unfortunate than that. The Mac version can only read the locales Windows codepage.

That means the Mac version of Excel cannot read CSVs it wrote. That's pathetic.

If you had any extended characters when you saved, they are scrambled when you reopen it. This problem has persisted in every version of Mac Excel up to the current Excel 2016.

One simply cannot safely open a CSV created with the Mac version of Excel, on any platform, anywhere.

UTF-8? What’s that?

UTF-8 is the encoding of the 21st Century. It was quick in overtaking every other encoding. Almost every modern application supports it. This will be easy and we’ll be fine? Not so fast.

If one attempts to open a CSV file encoded as UTF-8 without a Byte Order Mark (BOM) as recommended, any non-ASCII characters are again scrambled.

This is the CSV format Apple’s Numbers exports by default, UTF-8 sans BOM.

If we try it again with a UTF-8 BOM prepended to the file and Excel will read it. This is deceptive because once saved the text will remain correctly encoded UTF-8, but bizarrely the BOM will be stripped causing the file to no longer be correctly readable.

Many naïve application output UTF-8 + BOM CSVs, and they read correctly but do not write correctly. I’ve seen this cause all sorts of headaches because it appears to work but does not.

The Right Way Functional Workaround

As of this writing, there exists a single usable CSV format that Microsoft Excel can both read and write safely across platforms. Tab delimited UTF-16LE with leading Byte Order Mark.

Using Excel to output into this format to begin with, you have to use the Save As dialog and choose “UTF-16 Unicode Text (.txt)”. Frustratingly, it includes a .txt extension by default which can be confusing for non-technical users. However, if you provide clear instructions to Windows users to save their filename in quotes with a CSV extension such as "example.csv" it is workable.

If you operate an application with a CSV exporter please use this as your default export format. It will save us all many headaches in the long run.

Starting a Snap site with Stack and Persistent

Dan Palmer — Sun, 19 Jun 2016 02:00:00 GMT

Following on from my previous post about Haskell web frameworks, I’ve dived into making a non-trivial web application, with type-safe database access.

Five mistakes beginners make when working with databases

CRAIG KERSTIENS — Tue, 07 Jun 2016 23:55:56 GMT

When you start out as a developer there’s an overwhelming amount of things to grasp. First there’s the language itself, then all the quirks of the specific framework you’re using,and after that (or maybe before) we’ll throw front-end development into the mix, and somewhere along the line you have to decide to store your data somewhere.

Early on, with so many things to quickly master, the database tends to be an after-though in application design (perhaps because it doesn’t make an impact to end user experience). As a result there’s a number of bad practices that tend to get picked up when working with databases, here’s a rundown of just a few.

1. Storing images

Images don’t belong in your database. Just because you can do something, it doesn’t mean you should.Images take up a massive amount of space in databases, and slow applications down by unnecessarily eating your database’s IO resources. The most common way this mistake occurs is when new developers base64 encode an image and store it in a database large text/blob field.

The better approach is to upload your images directly to a service like Amazon S3, then store the image URL (hosted by Amazon) in your database as a text field. This way, each time you need to load an image, you need to simply output the image URL into a valid <img> tag. This will greatly improve website responsiveness, and generally help scale web applications.

2. Limit/Offset

Pagination is extremely common in a number of applications.As soon as you start to learn SQL, the most straight-forward way to handle pagination is to ORDER BY some column then LIMIT the number of results returned, and for each extra page you’ll OFFSET by so many records. This all seems entirely logical, until you realize at any moderate scale:

The load this exerts on your database will be painful.
It isn’t deterministic, should records change as the user flips between pages.

The unfortunate part is: pagination is quite complex, and there isn’t a one-size-fits-all solution. For more information on solving pagination problems, you can check out numerous options

3. Integer primary keys

The default for almost all ORMs when creating a primary key is to create a serial field. This is a sequence that auto-increments and then you use that number as your primary key. This seems straight forward as an admin, because you can browse from /users/1 to /users/2, etc. And for most applications this can often be fine. And for most applications, this is fine. But, you’ll soon realize as you start to scale that integers primary keys can be exhausted, and are not ideal for large-scale systems. Further you’re reliant on that single system generating your keys. If a time comes when you have to scale the pain here will be huge. The better approach is to start taking advantage of UUIDs from the start.

There’s also the bonus advantage of not secretly showcasing how many users/listings/whatever the key references directly to users on accident.

4. Default values on new columns

No matter how long you’ve been at it you won’t get the perfect schema on day 1. It’s better to think of database schemas as continuously evolving documents. Fortunately, it’s easy to add a column to your database, but: it’s also easy to do this in a horrific way. By default, if you just add a column it’ll generally allow NULL values. This operation is fast, but most applications don’t truly want null values in their data, instead they want to set the default value.

If you do add a column with a default value on the table, this will trigger a full re-write of your table. Note: this is very bad for any sizable table on an application. Instead, it’s far better to allow null values at first so the operation is instant, then set your default, and then, with a background process go and retroactively update the data.

This is more complicated than it should be, but fortunately there are some handy guides to help.

5. Over normalization

As you start to learn about normalization it feels like the right thing to do. You create a posts table, which contains authors, and each post belongs in a category. So you create a categories table, and then you create a join table post_categories. At the real root of it there’s not anything fundamentally wrong with normalizing your data, but at a certain point there are diminishing returns.

In the above case categories could very easily just be an array of varchar fields on a post. Normalization makes plenty of sense, but thinking through it a bit more every time you have a many to many table and wondering if you really need a full table on both sides is worth giving a second thought.

Edit: It’s probably worth saying that under-normalization is also a problem as well. There isn’t a one size fits all here. In general there are times where it does make sense to have a completely de-normalized and a completely normalized approach. As @fuzzychef described: “use an appropriate amount of normalization i.e. The goldilocks principle”

Conclusion

When I asked about this on twitter I got a pretty great responses, but they were all over the map. From the basics of never looking at the queries the ORM is generating, to much more advanced topics such as isolation levels. The one I didn’t hit on that does seem to be a worthwhile one for anyone building a real world app is indexing. Knowing how indexing works, and understanding what indexes you need to create is a critical part of getting good database performance. There’s a number of posts on indexing that teach the basics, as well as practical steps for analyzing performance with Postgres.

In general, I encourage you to treat the database as another tool in your chest as opposed to a necessary evil, but hopefully, the above tips will at least prevent you from making some initial mistakes as you dig in as a beginner.

Special thanks to @mdeggies and @rdegges for the initial conversation to spark the post at PyCon.

On Maps

infrequent oscillations — Mon, 06 Jun 2016 13:33:05 GMT

In which we criticise Apple Maps.

First week of Juneathon

infrequent oscillations — Mon, 06 Jun 2016 11:44:33 GMT

Being lazy and repurposing the Gyroscope report.

Beautiful A/B testing

Bytepawn - Marton Trencseni — Sun, 05 Jun 2016 01:00:00 GMT

I gave this talk at the O’Reilly Strata Conference London in 2016 June, mostly based on what I learned at Prezi from 2012-2016.

Haskell Web Frameworks

Dan Palmer — Sat, 04 Jun 2016 02:00:00 GMT

A quick overview of 5 of the popular Haskell web frameworks, and what they can do to improve the state of web development.

May's Gyroscope Report

infrequent oscillations — Fri, 03 Jun 2016 21:45:37 GMT

In which we look at some graphs.

Fixing the Ansible 2.1 Temporary File Permissions Issue

Ben Overmyer's Site — Thu, 02 Jun 2016 03:00:00 GMT

In Ansible 2.1 and later, Ansible will not allow the creation of world-readable temporary files. It does this for a good reason, but it's a change from how it was before. If you're experiencing this problem, you'll see an error when trying to become_user other than root, and it'll look like this:

fatal: [12.34.567.8]: FAILED! => {"failed": true, "msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user. For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}

The documentation offers solutions. The one that worked for me was this: add the installation (and enablement) of ACL as part of your common tasks for a given playbook. Everything else will then work behind the scenes to make sure those temporary files are handled securely and silently.

The disappointment of OutRun

infrequent oscillations — Wed, 01 Jun 2016 06:31:54 GMT

In which we fail a goal but that's ok.

A couple of ambient tracks from the noise of a train ticket

infrequent oscillations — Tue, 24 May 2016 10:55:26 GMT

In which we make something from nothing.

Monitoring SystemD Units With Nagios

smcleod.net — Mon, 23 May 2016 03:00:00 GMT

Ever forgotten to add a critical service to monitoring?

Want to know if a service or process fails without explicitly monitoring every service on a host?

…Then why not use SystemD’s existing knowledge of all the enabled services? Thanks to ‘Kbyte’ who made a simple Nagios plugin to do just this!

Requirements

Python3 (For RHEL/CentOS 7 yum install python34)
python-nagiosplugin My pre-built RPMs or pip3 install nagiosplugin
PyNagSystemD

Online Conversion from SQL_ASCII to UTF8 in PostgreSQL

smcleod.net — Mon, 23 May 2016 03:00:00 GMT

Scripts and source available here: sql_ascii_to_utf8

The Goal

To be able to take a Postgres Database which is in SQL_ASCII encoding, and import it into a UTF8 encoded database.

Requirements:

Python3 (For RHEL/CentOS 7 yum install python34)
python-nagiosplugin My pre-built RPMs or pip3 install nagiosplugin
PyNagSystemD

The Problem

PostreSQL will generate errors like this if it encounters any non-UTF8 byte-sequences during a database restore:

# pg_dump -Fc test_badchar | pg_restore -d test_badchar_utf8
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 2839; 0 26852 TABLE DATA table101 postgres
pg_restore: [archiver (db)] COPY failed for table "table101": ERROR:  invalid byte sequence for encoding "UTF8": 0x91
CONTEXT:  COPY table101, line 1
WARNING: errors ignored on restore: 1

And the corresponding data will be omitted from the database (in this case, the whole table, even the rows which did not have a problem):

More Victoria Line output

infrequent oscillations — Fri, 20 May 2016 12:16:52 GMT

More ambient drones from the Victoria Line samples provided by TFL/Matt Rogers.

Hack, HHVM and avoiding the Second-system effect

Bytepawn - Marton Trencseni — Sat, 14 May 2016 01:00:00 GMT

I read this book on my first vacation after I started working at Facebook and thus became a semi-regular Hack/HHVM user. I highly recommend reading (parts of) it. But not to learn Hack/PHP, which is irrelevant to most people. Instead, it’s to learn about how Facebook improved it’s www codebase and performance without rewriting the old PHP code in one big effort, and thus avoided the famous Second-system effect.

TFL and the Victoria Line samples

infrequent oscillations — Tue, 10 May 2016 11:29:08 GMT

In which we borrow some samples to make some tunes.

Speeding Up rsync

smcleod.net — Tue, 03 May 2016 03:00:00 GMT

The most common way to use rsync is probably as such:

rsync -avr user@<source>:<source_dir> <dest_dir>

Resulting in 30-35MB/s depending on file sizes

This can be improved by using a more efficient, less secure encryption algorithm, disabling compression and telling the SSH client to disable some unneeded features that slow things down.

With the settings below I have achieved 100MB/s (at work between VMs) and over 300MB/s at home between SSD drives.

Benchmarking IO with FIO

smcleod.net — Fri, 29 Apr 2016 03:00:00 GMT

This is a quick tldr there are many other situations and options you could consider

FIO man page
IOP/s = Input or Output operations per second
Throughput = How many MB/s can you read/write continuously

Variables worth tuning based on your situation

--iodepth

The iodepth is very dependant on your hardware.

Rotational drives without much cache and high latency (i.e. desktop SATA drives) will not benefit from a large iodepth, Values between 16 to 64 could be sensible.

3 Reasons why I gave up the Ghost

Ben Overmyer's Site — Mon, 25 Apr 2016 03:00:00 GMT

This's about why I quit paying $10/month for a hosted Ghost blog and migrated everything back to a self-hosted WordPress blog. It wasn't easy, and I lost all images in the process, but here's why I made that leap.

Getting my finances in order

$10.00 USD per month isn't much for someone in my socioeconomic class. It covers two deluxe lattés. However, I recently got hit with the Hammer of Reality, and I'm culling all unnecessary expenses. My definition of "unnecessary" got a big revision. As someone who makes roughly six figures a year, I can afford to spend money on things that have no real bearing on my happiness but satisfy temporary impulses. But I don't want to do that anymore. My family (id est, my dad) has always done that, and I inherited that behavior. But I'm not cool with it anymore. Not when over 30% of my take-home income is spent on debt. So, with the teachings of Dave Ramsey once again firmly in my mind, and the experience of life recently burned in there, I'm improving the efficiency of my expenses. In this case, that means self-hosting my blog, since I already pay for a Digital Ocean droplet to host other sites I manage.

WordPress is more mature

Over the past year, not much has changed with Ghost. It's still the same basic Markdown-based blog engine that it was two years ago. Its market share (and, more importantly, developer mind share) has not significantly increased. By comparison, WordPress has seen thousands of new plugins, hundreds of new themes, and unmeasurable numbers of code contributions to plugins, themes, and WordPress core over the same time period. I could hate WordPress. It's written in PHP and follows code practices that are a decade out of date. However, I must recognize the stability of the platform, and the passion and strength of its community. There are a number of people much smarter and more experienced than myself who've made significant contributions to WordPress just over the past year. When it comes to maintaining a system on your own, the predictability and resilience of that system are very important metrics to consider.

I am in full control of my content

This is not really so much a dig on Ghost as it is a dig on self-hosting versus hosted blogs. My reason for going with a self-hosted WordPress blog is the same reason I discounted going with WordPress.com hosting. I run the database behind the blog. I am in full control of every byte stored there. Whether it's backed up or not, where it's backed up, how it's backed up, that's all in my realm of control. I determine the version of PHP, the version of MySQL, and the version of Linux that hosts it all. I alone control access to the server, and how incoming (and outgoing) connections are treated. As someone who's spent the last year and a half as a site reliability engineer/manager, that has suddenly become of vital concern to me. And renting a VPS (Virtual Private Server) is much different than trusting someone else to manage your environment for you. The next step up, fully controlling the environment from bare metal on up, is too expensive for someone like me to handle on my own.

So, to sum up (or TL/DR)...

I changed back to self-hosted WordPress from Ghost because A) I spend less money, B) I believe that WordPress is more likely to be supported in the future, and C) I like being able to control (almost) every aspect of how my blog exists.

A Collection of O'rly book Covers

Ben E. C. Boyter — Fri, 22 Apr 2016 03:48:41 GMT

A quick collection of all the fake Orly book covers I find around on the internet.

I am seriously considering writing content for the “Expert Excuses for Not Writing Unit Tests” one.

EDIT - I actually did write content and publish it as a book! Expert Excuses for Not Writing Unit Tests

Moving back to WordPress.

Ben Overmyer's Site — Fri, 22 Apr 2016 03:00:00 GMT

I'm transitioning away from hosted Ghost back to WordPress. More about this later.

Switching away from iPhone

Ben Overmyer's Site — Fri, 15 Apr 2016 03:00:00 GMT

Yesterday I went on the Google Fi site and signed up for a Nexus 6P and a data plan roughly equivalent to what I spend at Verizon. My iPhone 6 Plus, despite not (yet) being paid off, is going to sit in a drawer collecting dust starting late next week when the Nexus arrives. Am I displeased with the iPhone? No, not especially. Am I unhappy with Verizon? Well yeah, that's kind of a given for any Verizon customer. I have personal reasons for needing a new phone on a new carrier, but I also have a professional interest in Google Fi and the way it handles mobile phones. We'll see where this goes.

searchcode server released

Ben E. C. Boyter — Fri, 01 Apr 2016 00:19:33 GMT

searchcode server the downloadable self hosted version of searchcode.com is now available. A large amount of work went into the release with a variety of improvements based on feedback from the general beta releases.

searchcode server has a number of advantages over searchcode.com that will eventually be back-ported in. The full list of things to check out is included below,

New Single Page Application UI for smooth search experience
Ability to split on terms so a search for “url signer” will match “UrlSigner”
Massively improved performance 3x in the worst case and 20x in the best
Configurable through UI and configuration
Spelling suggestion that learns from your code

A few things of note,

Weekend Panoramas

infrequent oscillations — Wed, 16 Mar 2016 13:52:00 GMT

Some panoramas from my weekend wandering.

So, You Want to Leave America

Ben Overmyer's Site — Mon, 14 Mar 2016 02:00:00 GMT

Many people (more people than ever before, apparently) are declaring their intention to move away from America permanently if Candidate X wins the election for President of the United States in November.

I won't lie, I was one of those. But then I started digging into what that meant.

See, I've done it before. In 2000, my parents, sister, and I all moved together to New Zealand when George W. Bush won the election. It was more than a little bit of a political decision.

Don't get me wrong, New Zealand was amazing. Most of the best memories of my entire life happened in the four years I lived there. But I also acutely remember the cost.

New Zealand was a relatively safe choice. The primary language was English. The legal system was fairly similar to the United States'. Most of the culture was familiar, if a little different. But it was really expensive, and not just in the material cost of goods. New Zealand is a remote island nation at the end of the world. Internet usage at the time was far more expensive than in the USA. My family lost touch with almost everyone we had known. I experienced my first real sense of profound loss as a result.

If you choose to leave the United States for political reasons, you need to seriously consider a few things. All of the following assumes that you and yours are leaving because you're appalled at the political situation, and not in immediate physical danger because of the election's result. If you fall into the latter category, this post doesn't apply to you, and you should get your family to safety as soon as possible.

If you are leaving because you disagree with the country's choice of leader, then you need to make damn sure that the country you're choosing to emigrate to follows your own political beliefs. If you pick a country that's MORE extreme than the president apparent, then you may as well have just saved yourself the trouble and stayed home, because you're in for a world of hurt.

On the other hand, if your host country of choice is politically favorable, you have a ton of other considerations to keep in mind. If you've spent your entire life in one part of the USA, then you're basically f***ed, in terms of culture shock. It will hurt, and it will hurt bad, to discover that everything you're familiar with is either A) extremely different, B) absent, or worse, C) slightly different. Let me elaborate.

My sister was brought to tears by all pickles in New Zealand being basically sweet miniature-as-in-quark-sized pickles. The straw that shattered the camel's back, as it were. I saw the same kind of reaction by most of the American ex-pats I met in NZ. Not to pickles, mind you, but to the little things.

A relatively small portion of the world has English as its mother tongue. Sure, it's spoken just about everywhere, and you'll get on OK in most developed or even developing nations. But there is a marked difference in communicating with someone in identical preferred languages and in your preferred language but the other person's second language. This is particularly true in countries where English is recognized as a business necessity but not a cultural one, such as eastern Asian countries.

Let's say you're able to adapt to the language barrier, and to the availability barrier. Congratulations on defying the two biggest hurdles. Now how are you going to deal with the fact that people in your host country don't understand your pet peeves, or your quirks that you thought were common, or your personal passions that have no support locally? How about the prejudice that the locals feel towards Americans, which is basically universal in all countries that are aware of the USA? Are you ready for that? Are you willing to subject your family to that?

OK, so you're still ready to go. You've got your passports, and you've secured your visas. You understand the financial and cultural pitfalls. You still think that leaving the USA is the best choice, given the alternative. OK, that's fine. Before you commit to a particular country, then you better damn well visit each of your prospective host countries for at least a month each (you DO have multiple options, right?) before making the big move.

Because I will tell you, as someone who has done it and is considering doing it again, leaving your home country is one of the hardest goddamn things you can do to yourself or your family.

On the other hand, you might be of the YOLO type, in which case ignore everything I've warned you about above. It might be the most amazing adventure of your entire life.

Segfaulting Python with afl-fuzz

Tom Forbes — Mon, 07 Mar 2016 00:31:37 GMT

American Fuzzy Lop is both a really cool tool for fuzzing programs and an adorable breed of bunny. In this post I’m going to show you how to get the the tool (rather than the rabbit) up and running and find some crashes in the cPython interpreter. Fuzzing? Explaining in detail what fuzzing is would ...

I Am Done With the Mac App Store

Donat Studios — Fri, 04 Mar 2016 07:44:46 GMT

tl;dr Do not depend on App Store Apps, Apple can and will pull the carpet out from under you.

Reading Hacker News today there is an article about how OS X Installers stopped working on February 14th, 2016. This is due to an expired certificate Apple used to sign the binaries with.

In that same time range I had a handful of Apps purchased from the App store suddenly and mysteriously stop working. They just quit. No dialog, no error mesasage, nothing at all. Apps I paid for and depend on just stopped working.

The thing they all these Apps have in common is they are no longer available from the App store and have not been updated in several years. I am nearly certain that this is the same signing issue.

If this is the case, this means all Apps from the app store are signed built-in self destruct date. One which only Apple or the original author can fix.

I have been careful to keep backups of the .app’s. Apple dropped the ability to re-download App's removed from the App store.

This gives way too much power over my work to third parties. Software I paid for should not stop working because someone else stopped supporting it. How can someone build a reliable work-flow when any program stop working at any time? You can't, that’s simply insanity.

One of the apps I used daily, the author took down a few years ago because it “flopped and [was] plagued with technical issues”. I have encountered no technical issues with it until this point, and it an important part of my work-flow.

I, as well as others have asked the author to open source it on many occasions. [1], [2], [3], [4] … There’s more

Her only response to our cries has been a cringe-worthy LOL. I wish I were joking.

The moral of my story comes down to the following life lessons.

Don’t depend on things that others can take from you.
Be able to scratch your own itch; seek open source when at all possible.

I will no longer be purchasing things from the Mac App Store. The convenience is not worth the incredible down sides.

Richard Stallman has been warning us for years, I should have listened.

CRAIG KERSTIENS — Sun, 28 Feb 2016 22:55:56 GMT

Notice: Much of this post still applies, but now applies more directly to Citus. Since this post originally published, pg_shard is now deprecated. You can find some further guidance for sharding on the Citus blog and docs

Back in 2012 I wrote an overview of database sharding. Since then I’ve had a few questions about it, which have really increased in frequency over the last two months. As a result I thought I’d do a deeper dive with some actual hands on for sharding. Though for this hands on, because I do value my time I’m going to take advantage of pg_shard rather than creating mechanisms from scratch.

For those unfamiliar pg_shard is an open source extension from Citus data who has a commerical product that you can think of is pg_shard++ (and probably much more). Pg_shard adds a little extra to let data automatically distribute to other Postgres tables (logical shards) and Postgres databases/instances (physical shards) thus letting you outgrow a single Postgres node pretty simply.

Alright, enough talk about it, let’s get things up and running.

Build, install

The rest assume you have Postgres.app, version 9.5 setup and are on a Mac, much of these steps could be easily adapted for other Postgres installs or OSes.

PATH=/Applications/Postgres.app/Contents/Versions/latest/bin/:$PATH make

sudo PATH=/Applications/Postgres.app/Contents/Versions/latest/bin/:$PATH make install

cp /Applications/Postgres.app/Contents/Versions/9.5/share/postgresql/postgresql.conf.sample /Applications/Postgres.app/Contents/Versions/9.5/share/postgresql/postgresql.conf.sample

Edit your postgresql.conf:

#shared_preload_libraries = ''

TO:

shared_preload_libraries = 'pg_shard'

Then create a file in /Users/craig/Library/Application\ Support/Postgres/var-9.5/pg_worker_list.conf where craig is your username:

# hostname port-number
localhost 5432
localhost 5433

You’ll also need to create a new Postgres instance:

initdb -D /Users/craig/Library/Application\ Support/Postgres/var-9.5-2

Then edit that postgresql.conf inside that newly created folder with two main edits:

port = 5432

port = 5433

Finally setup our database then start it up:

createdb instagram
postgres -D /Users/craig/Library/Application\ Support/Postgres/var-9.5-2

Setup

Now you should have two running instances of Postgres, now let’s finally turn on the pg_shard extension, create some tables and see what we have. First connect to your main running Postgres instance, so in this case the the instagram database we first created psql instagram, then let’s set things up:

CREATE EXTENSION pg_shard;
CREATE TABLE customer_reviews (customer_id TEXT NOT NULL, review_date DATE, review_rating INTEGER, product_id CHAR(10));
CREATE TABLE
Time: 4.734 ms
SELECT master_create_distributed_table(table_name := 'customer_reviews', partition_column := 'customer_id');
master_create_distributed_table
---------------------------------
(1 row)
SELECT master_create_worker_shards(table_name := 'customer_reviews', shard_count := 16, replication_factor := 2);
master_create_worker_shards
-----------------------------
(1 row)

Understanding and using

So that was a lot of initial setup. But now we have an application that could in theory scale to a shared application across 16 instances. If you want a refresher, there’s a difference between physical and logical shards. In this case above we have 16 logical ones and it’s replicated across 2 physical Postgres instances albeit on the same instance.

Alright so a little more poking under the covers to see what happened before we actually start doing something with our data. If you’re still connected go ahead and run \d, and you should see:

 List of relations
Schema | Name | Type | Owner
--------+------------------------+-------+-------
public | customer_reviews | table | craig
public | customer_reviews_10000 | table | craig
public | customer_reviews_10001 | table | craig
public | customer_reviews_10002 | table | craig
public | customer_reviews_10003 | table | craig
public | customer_reviews_10004 | table | craig
public | customer_reviews_10005 | table | craig
public | customer_reviews_10006 | table | craig
public | customer_reviews_10007 | table | craig
public | customer_reviews_10008 | table | craig
public | customer_reviews_10009 | table | craig
public | customer_reviews_10010 | table | craig
public | customer_reviews_10011 | table | craig
public | customer_reviews_10012 | table | craig
public | customer_reviews_10013 | table | craig
public | customer_reviews_10014 | table | craig
public | customer_reviews_10015 | table | craig
(17 rows)

You can see that under the cover there’s a lot more customer_reviews tables, in reality you don’t have to think about these or do anything with them. But just for reference they’re just plain ole Postgres tables under the cover. You can query them and poke at the data. The now mystical customer_reviews will actually roll up the data across all your logical shards (tables) and physical shards (spanning across machines).

It’s also of note that in production you might not actually use your primary DB as a worker, we did this more for expediency in setting it up on a local Mac. More typically you’d have 2 or more workers which are not the same a the primary, these were the ports we setup in our pg_worker_list.conf. A common setup would look something more like:

So now start inserting away:

INSERT INTO customer_reviews (customer_id, review_rating) VALUES ('HN802', 5);
INSERT INTO customer_reviews (customer_id, review_rating) VALUES ('FA2K1', 10);

For extra homework on your own you can now go and poke at where the underlying data actually surfaced.

Conclusion

Yes, there’s a number of limitations that you can learn a bit more about over on the github repo for pg_shard. Though even with those it’s very usable as is, and let’s you get quite far in prepping an app for sharding. While I will say that all apps think they’ll need sharding and few actually do, given pg_shard it’s minimal extra effort now to plan for such scaling should you need it.

Up next we’ll look at how it’d work with a few languages, so you can get an idea of the end to end experience.

$ curl poptart.spinda.net

spinda's dreamwidth — Sun, 28 Feb 2016 02:17:14 GMT

comments

Unofficial witty cloud module documentation with nodemcu firmware

Linux Engineer's random thoughts — Thu, 25 Feb 2016 01:00:00 GMT

I wanted to try my hand with ESP8266 modules, so I got a witty cloud development board. It's running a proprietary firmware from gizwits which I backed up if anyone wants to look at it.

The board is in two parts: programming board("cape") with ch340g usb serial and 3 …

A better rich text editor for Dreamwidth

spinda's dreamwidth — Tue, 23 Feb 2016 03:39:29 GMT

For whatever reason, Dreamwidth’s default rich text editor (built on an old version of FCKEditor) hides a lot of the available toolbar buttons. I’ve written a quick userscript that enables all available buttons in the editor, and also increases the height a bit to make writing more pleasant. I’ve tested it on Firefox with GreaseMonkey; presumably it should work on Chrome/Chromium as well.

// ==UserScript==
// @name        Better Dreamwidth Editor
// @namespace   betterdweditor
// @include     https://www.dreamwidth.org/update?usejournal=spindas
// @include     https://www.dreamwidth.org/editjournal*
// @version     1
// @grant       none
// ==/UserScript==

window.onload = function () {
  var frame = document.getElementById('draft___Frame');
  frame.onload = function () {
    document.getElementById('draft___Frame').style.height = '600px';
  };
  frame.src = 'https://www.dreamwidth.org/stc/fck/editor/fckeditor.html?InstanceName=draft&Toolbar=Default';
};

comments

Reverse engineering the Firewatch.camera API (and pranking r/Firewatch)

spinda's dreamwidth — Fri, 19 Feb 2016 01:22:05 GMT

Firewatch is a story-centric game by Campo Santo and Panic. I played it all the way through this weekend and had a really great time with it. Besides the top-notch voice acting, gorgeous visuals, and mystery-filled plot, one feature that’s been getting a lot of attention is the disposable camera the player carries through most of the game. Starting with 18 unused shots on the roll, you can fill it up with snaps of trees and trails and plenty of sunsets peaking over the hills. At the end they’re all uploaded to a personal page on Firewatch.camera, where you can share them or order prints from the "Fotodome".

At the end of my own playthrough, I uploaded my shots and logged onto the site to see them.

But one of them came out… weird.( Read more... )

comments

Identifying days with exercise automagically

infrequent oscillations — Wed, 17 Feb 2016 19:00:17 GMT

In which we use heart rate to detect exercise.

Einstein's amazing theory

Bytepawn - Marton Trencseni — Tue, 16 Feb 2016 01:00:00 GMT

This post is about the amazing success of Einstein's general theory of relativity. The theory predicts, among other things the accelerating Universe, black holes, gravitational lensing and gravitational waves. The real shocker is to remember that Einstein didn't invent general relativity to explain these. He didn’t know about these, they didn't exist at that time!

2015 in Review

Nemo's Home — Mon, 15 Feb 2016 02:00:00 GMT

2015 was a good year for me. I accomplished a lot of things, and was happy for most part of the year. In no particular order, here are the things that stand out for me:

Went to HillHacks, and made lots of friends. Helped organize the conference.
Joined Razorpay in June
Moved to Bangalore
Announced my book, The Joy of Software Development
Did lots of speaker things:
- Joy of Software Development talk at IIT Roorkee
- Flash Talks on SDSLabs, CTFs and a few more things at HillHacks
- A full-length talk on the book itself at HillHacks
- Did the HillHacks and HackBeach quizes
- Announced my Homeopathy Bug Bounty Program at hackbeach.
- Gave a talk on Fun with HTTP at Barcamp Bangalore.
Learnt some slacklining at hackbeach. Helped with conference scheduling as well.
Made lots of new friends at Bangalore.
Started playing board games, including being a DM at several Dungeons and Dragons sessions.
Started quizzing in bangalore as well. Mostly at Cluesday, Vapors.

Reading

I started with a goal of 20 books for this year, and ended up reading about 28. Tried experimenting with Audiobooks near the year end, and failed. Bought a Kindle paperwhite as well. I tend to read a lot of Fantasy and SF, and this continued in 2015 as well. The best books I read this year (in order):

Lions of Al Rassan [review]
The Goblin Emperor [review]
The Martian [review]

A complete list is on Goodreads

Tech

Launched hackercouch as a personal project.
Shifted to Arch Linux + i3wm setup for my laptop. See Setup page for more details.
Left facebook, mostly because of their stance on Net Neutrality and the FreeBasics debacle in India.

Heisengames and the importance of patience in business

Bytepawn - Marton Trencseni — Mon, 08 Feb 2016 01:00:00 GMT

Most bets businesses take, be it hiring, features, products or strategy don't work out. Still, many businesses are successful despite setbacks. A negative attitude---even when the analysis of the situation is in fact correct---may be missing the bigger picture.

Modified Flapjacks

infrequent oscillations — Mon, 08 Feb 2016 00:04:09 GMT

In which we experiment with flapjacks.

Cloud9: Cloud coding that actually works

Bytepawn - Marton Trencseni — Sun, 07 Feb 2016 01:00:00 GMT

For the past 2 months I've been using Cloud9 for writing code in the cloud, and I can wholeheartedly recommend it: it just works for me. It's basically Docker plus an IDE: you get a Docker container running Ubuntu that you can access over a web IDE.

Luigi vs Airflow vs Pinball

Bytepawn - Marton Trencseni — Sat, 06 Feb 2016 01:00:00 GMT

A spreadsheet comparing the three opensource workflow tools for ETL.

Pinball review

Bytepawn - Marton Trencseni — Sat, 06 Feb 2016 01:00:00 GMT

Pinball is an ETL tool written by Pinterest. Like Airflow, it supports defining tasks and dependencies as Python code, executing and scheduling them, and distributing tasks across worker nodes. It supports calendar scheduling (hourly/daily jobs, also visualized on the web dashboard). Unfortunately, I found Pinball has very little documentation, very few recent commits in the Github repo and few meaningful answers to Github issues by maintainers, while it's architecture is complicated and undocumented.

Mirroring a Gitlab project to Github

smcleod.net — Thu, 04 Feb 2016 02:00:00 GMT

Let’s pretend you have a project on Gitlab called ask-izzy and you want to mirror it up to Gitlab which is located at https://github.com/ask-izzy/ask-izzy

Assuming you’re running Gitlab as the default user of git and that your repositories are stored in /mnt/repositories you can following something similar to the following instructions:

Grant write access to Github

Get your Gitlab install’s pubkey from the git user

cat /home/git/.ssh/id_rsa.pub

On Github add this pubkey as deploy key on the repo, make sure you tick the option to allow write access.

How to parse this?

infrequent oscillations — Tue, 02 Feb 2016 12:23:57 GMT

In which we wonder about Jim Henson.

Jamathon 2016 #summary

infrequent oscillations — Mon, 01 Feb 2016 17:02:31 GMT

Almost a success - 27 tracks uploaded for 31 days. All of them are in my Jamathon playlist. One (#17) was a cheat because it didn’t involve any apps but was a recording using binaural microphones. Favourites are marked thusly: ★ Jamathon #1: Created from a one minute recording of my bathroom fan Jamathon #2: Made with some previously recorded-with-Fieldscaper samples Jamathon #3: Samplr with various samples Jamathon #4: ★ Created from three iPhone Fieldscaper recordings of the DLR this morning Jamathon #5: I think this was mucking around with Dedalus.

Janathon 2016 #summary

infrequent oscillations — Sun, 31 Jan 2016 23:38:58 GMT

(All numbers from VeloViewer - a couple of days are missing from here because I only recorded them on the Fitbit, not the Garmin, which means they don’t end up in Strava.) 37 activities, 251.97km, 40:20:14 elapsed, 36:52:10 moving, 2215m elevation (421m max) 3 rides, 29.38km, 2:15:23 elapsed, 1:57:43 moving, 108m elevation (54m max) 10 runs, 103.5km, 12:21:18 elapsed, 11:46:35 moving, 1077m elevation (421m max) 24 walks, 119.08km, 25:43:32 elapsed, 23:07:36 moving, 1030m elevation (201m max)

Janathon 2016 #31

infrequent oscillations — Sun, 31 Jan 2016 22:58:45 GMT

Yes! Survived! Finished! 31/31! Today was recovery from Cakeathon by way of cycling from St. Pancras to Peckham Rye (I got lost again) and then walking home from there. [Cycling] [Walking] (Janathon summary in the next post)

Janathon 2016 #30

infrequent oscillations — Sat, 30 Jan 2016 23:04:32 GMT

A (hard) small (tiring) jaunt (hilly) around a country park. With an excellent goody bag… [Winter Cakeathon 2016]

Janathon 2016 #29

infrequent oscillations — Fri, 29 Jan 2016 21:21:08 GMT

First bike ride of 2016. Obviously I got lost even though it was almost a straight line between Peckham and St. Pancras. But at least now I have some cycle miles for the year. [A jaunt on a Brompy]

AskIzzy

smcleod.net — Fri, 29 Jan 2016 02:00:00 GMT

Today we launched a mobile website for homeless people

… and it was launched by one of Australia’s many recent Prime Ministers

Today alone we served up over 87,000 requests

As many of you know, I work with Infoxchange as the operations lead.

When I first heard the idea of a website or app for people that have found or are worried about finding themselves homeless in Australia I really didn’t think it made sense - until I saw the stats showing how many homeless people in Australia have regular access to a smart phone and data either via a cellular provider or free WiFi.

Janathon 2016 #28

infrequent oscillations — Thu, 28 Jan 2016 23:49:24 GMT

Three walks. Because I’m worth it. And also I didn’t plan my day off very well. [Picking up new art] [Thames wander] [Picking up new shoes]

What being a PM is really like - Software is easy, People are hard

CRAIG KERSTIENS — Thu, 28 Jan 2016 22:55:56 GMT

In recent months I’ve had the question nearly once a week about advice/tips for becoming a Product Manager or more commonly referred to as PM. These are generally coming from people that are either currently engineers, or previously were and are in some engineer/customer role such as a sales engineer or solution architect. There’s a number of high level pieces talking about PM and it often feels glorious, I mean you get to make product decisions right? You get to call some shots. Well that sometimes may be true, but don’t assume it’s all rainbows and sparkles.

Especially as a first time PM what your day to day will look like won’t be debating strategy all day long. Here’s a few of the good and the bad sides of being a PM.

Plenty of grunt work

While you may get to make a decision or two, the bulk of your time will not be thinking about grandiose visions, instead you’ll be doing a lot to gather data. There’s a lot of means for gathering data across lots of sources, the more you use the better you’ll be. Knowing the ones you steer towards, as well as ones you steer away from is useful so you can balance a bias more fairly. For myself SQL is a go-to, then customer interactions both qualitative and quantitative such as surveys, following what media is saying about your space is important as well. And while user studies are often relegated to design and UX, as a PM you need to make sure it at least happens (Invision App is a favorite for lightweight tests).

In a given week I probably spend 10 hrs interacting with customers, looking at data, and sadly that’s probably not enough.

A few practical examples of this

Each morning I send emails to 10-20 users who used the product for the first time, yes this is automated but carving out 30 minutes of my day to actually follow-up with each of them is less automated.

Another example is keeping a health of business dashboard up to date. Personally I use google sheets for this. Within one spreadsheet I have monthly and weekly targets as well as how we’re tracking against them. These are all updated on actual real time data, powered by Heroku’s dataclips with a simple =importCSV(‘http://dataclips.heroku.com/abcdefghij….csv’). In total my google spreadsheets has 1 high level overview, with about 20 underlying sheets that do all of the computations. In any given month 1 of my key 4-5 goals may be missed, which then spawns digging in deeper to figure out why and what we can do about it.

Dictating vs. consensus

From a product decision making perspective you can force alignment by explicitly making every decision, or you can allow decisions to be made as a group voting if needed. Expect to use some balance of both of these among the team, and neither is never perfect. When it comes to outside the team you may still use both, but steer more strongly one way or the other. For example with the executive team it may be more consensus, with marketing it may be dictating your product roadmap which they can help support.

Even within the team there will be times a decision must be made and there will be some people that don’t align. It’s key that you make the decision clearly and explicitly. Even though some individuals don’t like it, they won’t fight against it… unless you make a habit of taking the input, then discarding it and going along your ‘intuition’. Even when there’s a strong case based on the data it may not be as clear as you think.

In contrast, decision making by consensus most people will feel happier that they provided input into the product. If you take everyone’s input expect to end up with a product that feels like 10 people designed it, needless to say incoherent.

As a PM expect to do a lot of listening, a good bit of convincing, and some occasional big decision making.

The pain you feel inside the building doesn’t matter

You may think you’re solving a problem that exists for users, when in reality there was no problem at all. This is just a reminder that you need to keep empathy in mind above so much else.

As an example, once a data team put in place a tool, supposedly for me. I looked at the tool and more or less didn’t understand why it was in place. They proceeded to explain that my problem was it took me too long to write SQL, so this tool will help me get the reports I need without SQL. At that point I proceeded to actually list off all the issues I did have, none of which were SQL.

Prescribing a solution without knowing clearly from customers what the problem is will leave you in a bad spot. All this means you have to set aside building the tool you want to use, and make sure you know what the customer wants.

Marketing is your job

Not external marketing, though often that work may still fall to you, but rather internal marketing.

It’s important that you internally market the wins for your team. These wins should very much be for the team, and not for your own benefit. The best PMs seem to disappear into the background, this is because you’re more surfacing all the work your team is doing than any of the details you helped coordinate. This is often counter to our natural instinct to tout our own accomplishments. This is only exaggerated in PM role, one where things can still ship if you’re not there, so there can often be a tendency to try to highlight the roles value. Fight that urge to self market.

Rest assured though–getting the team focused on solving the right problems, and then surfacing their wins will only help you go faster.

On leading

At the end of the day ensuring the product is advancing is your job, so be prepared to do what you need to whether it’s leading or not to accomplish that.

It’s not all rainbows, but it is fun

The range of things you’ll have to focus on can be diverse and complex. In the end if you get a rush out of shipping and launching products, then all the work that goes into it can make it all worthwhile. It’s as much about figuring out what customers want and then getting your team building the right thing. For a first time PM it can be summed up by the notion that software is easy, people are hard.

Special thanks to Lukas Fittl and Arun Thampi for reviews and feedback on this post.

Janathon 2016 #27

infrequent oscillations — Thu, 28 Jan 2016 01:32:19 GMT

Not quite as restful but the Jubilee Line was buggered which meant walking to Stratford instead made sense. Yes. [Just about 7.5km to Stratford]

Janathon 2016 #26

infrequent oscillations — Tue, 26 Jan 2016 12:58:47 GMT

Everything from now until Saturday is basically rest. Because Saturday is a biggie - Winter Cakeathon. [Short walk commute]

Janathon 2016 #25

infrequent oscillations — Mon, 25 Jan 2016 23:11:29 GMT

A heavy-legged recovery run to the local garage for some Red Bull and a KitKat Chunky. Bleugh. [Heavy Legs Make Slow Running]

Garmin vs Fitbit

infrequent oscillations — Mon, 25 Jan 2016 08:33:09 GMT

Preamble Garmin data from my TCX files via tapiriik sync Fitbit data downloaded using API with “1sec” resolution Only data points coincident on seconds are used Tools used Modified version of my TCX dumper FitBit.py Ploticus Various shell commands (awk, sort, etc.) Results 48 minutes of hill repeats; Garmin using smart recording; Fitbit in exercise mode. 39 minutes of walking; Garmin using 1sec recording; Fitbit in passive mode. 39 minutes of walking; Garmin using 1sec recording; Fitbit in passive mode.

Janathon 2016 #24

infrequent oscillations — Mon, 25 Jan 2016 01:25:40 GMT

A year ago, I did a short session of hill repeats (3 down, 4 up) on Greenwich Park hill and it was hard work. By coincidence I ended up doing the same today and, oddly, almost exactly the same duration but this year I managed 6 down, 5 up on an empty stomach. (If I can train my body to burn fat efficiently, I should be able to run non-stop for, oh, about 8000 miles.

Moving Apps to the External SD Card on Android

Beetle Space — Sun, 24 Jan 2016 10:00:00 GMT

Some months ago I bought a new Android phone. The internal storage was only 16 GB, so I put in an external SD card of 32 GB.

As I started installing apps, I began to move the larger ones to the SD card using the Android Apps menu.

And I …

Janathon 2016 #23

infrequent oscillations — Sat, 23 Jan 2016 21:33:28 GMT

Another rest-ish day with a small jaunt down the road in the Feelmax Kuuva 3 before picking up coffee and salad for tea. THE EXCITEMENT IS NON-STOP. NON. STOP. [5km to coffee]

Janathon 2016 #22

infrequent oscillations — Sat, 23 Jan 2016 00:54:46 GMT

Another brisk 10km but with a twist - this time I recorded the entire binaural soundscape using the PCM-M10 and the CS-10EM. Here’s a snippet from Narrow Street - obviously you need stereo headphones to get the full effect. Managed the 10km marker in 1:50:45 which isn’t bad but then forgot to stop the watch when I got to Charing Cross (because I’d lost one of the windshields for the CS-10EMs and was flustered - turned out to be attached to one of my gloves somehow!

Janathon 2016 #21

infrequent oscillations — Fri, 22 Jan 2016 00:22:33 GMT

A rest day which involved going to the Co-op to pick up a windshield (very effective but doesn’t really fit the PCM-M10 if you have headphones plugged in) and then home via Point Hill. Which is Quite Steep in places. Windshield vs Muji fan Muji fan, full speed, pointed at the PCM-M10 mics from 4". 10s with windshield; 10s without; 10s without on low sensitivity.

Janathon 2016 #20

infrequent oscillations — Thu, 21 Jan 2016 01:55:42 GMT

Into the last stretch! Just as the weather seems to pivoting out of “COR BLIMEY MY KNACKERS HAVE VANISHED” cold into “DON’T NEED A COAT MUM” mild as well. Today was a rest day and if it hadn’t have been for those surprise Creme Eggs, it would have been a successful one. Still, a walk up a hill in “brisk” conditions can’t hurt.

Janathon 2016 #19

infrequent oscillations — Wed, 20 Jan 2016 01:07:34 GMT

First run for 8 days and it was hard. Not because of the running but because Muji base layer + dare2b shell was FAR TOO HEAT EFFICIENT. I was burning up after half an hour. Then got some twangy calf pre-cramps at 8km and it was an easy decision to abort at just over 10km by Deptford ASDA for some recovery food. That included a banana - the eating of which made me feel proper ill.

Code Monkey or CAD Monkey?

Beetle Space — Tue, 19 Jan 2016 10:00:00 GMT

I recently came across this comment:

I can’t tell you how many times I’ve heard “oh, I just COULDN’T stare at a computer all day lol”.

If you want a decent job, it’s probably going to be an office job staring at a computer all day …

Internships 2016

spinda's dreamwidth — Tue, 19 Jan 2016 08:29:51 GMT

After doing Google Summer of Code last year, this is my first year of applying for internships. So I’m nervous-yet-excited, and am really hoping that resumes actually get looked at as I don’t have any “employers” to put down so far.

Places I’ve applied to:

Mozilla in Mountain View – open source, makers of Firefox and Rust! This is kind of my dream job.
Adobe in San Jose – they’re really easy to get to from home, which is great since I don’t have my driver’s license yet.
Yahoo in Sunnyvale – also very reachable.
Facebook in Mountain View – one at WhatsApp, where they use Erlang! And two elsewhere in the company.
Lesson learned: if you have a Facebook account, make sure you log in before applying, ~~and use Chrome – the “Skills” field is currently broken on Firefox~~ (this seems to be fixed now). Also, some fields have character caps that aren't apparent unless you proceed to the next page and then go back.

Places I’ve looked at:

Jane Street – it’d be fun to work in a functional language like OCaml, but they only have positions in NYC, London, and Hong Kong.
Galois – another FP (Haskell) shop that’s too far away (Portland).
Google – they’re all closed up for the summer internships. I’ll have to apply earlier next year.
Nest – lots of positions open, but the closest office is in Palo Alto, which lacks good public transportation to and from San Jose.
Twitter – again, the nearest office is in-state but not close enough (San Francisco).

As it turns out, location is a major limiting factor for me. I definitely need to start driving soon.

comments

Janathon 2016 #18

infrequent oscillations — Tue, 19 Jan 2016 00:51:08 GMT

Recovery seems to be progressing - knocked out a brisk walk in decent cold without dying (average of 10:03 min/km is good going for me, even when fully fit.) [Brisk as a monkey up a vicar’s noodle]

Midpoint Football

infrequent oscillations — Mon, 18 Jan 2016 19:04:47 GMT

In which we try to predict football results.

Fix XenServer SR with corrupt or invalid metadata

smcleod.net — Mon, 18 Jan 2016 02:00:00 GMT

If a disk / VDI is orphaned or only partially deleted you’ll notice that under the SR it’s not assigned to any VM.

This can cause issues that look like metadata corruption resulting in the inability to migrate VMs or edit storage.

For example:

[root@xenserver-host ~]# xe vdi-destroy uuid=6c2cd848-ac0e-441c-9cd6-9865fca7fe8b
Error code: SR_BACKEND_FAILURE_181
Error parameters: , Error in Metadata volume operation for SR. [opterr=VDI delete operation failed for parameters:
  /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/MGT, 6c2cd848-ac0e-441c-9cd6-9865fca7fe8b.
  Error: Failed to write file with params [3, 0, 512, 512]. Error: 5],

Removing stale VDIs

To fix this, you need to remove those VDIs from the SR after first deleting the logical volume:

Marketing definitions for developers

CRAIG KERSTIENS — Sun, 17 Jan 2016 22:55:56 GMT

Marketing often feels like a dirty-icky thing to many developers. Well until you feel like you have a great product, but no one using it then you have to get a crash course in all of that. And while I might cover some of the actual basics in the future, just knowing what marketing people actually mean when they’re talking can be a huge jump start. Here’s a guide that distills many of the acronyms and terms down to what they actually mean in reality.

SEO - Search engine optimization. There’s two sides to this, one where you’re attempting to game the system known known as black hat. The other is simply creating good content.

Tip: Now, unlike several years ago social sharing helps impact this.

SEM - Search engine marketing. The short of this is adwords, but broadly it’s any search engine.

Tip: Be wary here, you can spend a lot of money quickly. Properly managing it takes time and effort otherwise you’re wasting money.

Display ads - Banner ads on websites. There’s a few common form factors in this world, so you’ll create a few then reuse them across lots of properties.

Tip: Results may vary here, there are some hidden gems when advertising on various long tail sites.

Retargeting - This is where you’re serving an ad (most commonly display) to someone that’s previously visited your site. The process happens due to you ‘pixeling’ them, and a cookie being set so the ad server knows they’ve seen you.

Tip: Generally good bang for the buck here, but you still need initial visitors to even retarget to.

Funnel - The process of someone going from finding you to paying to paying more. Generally a process will look something like: Anonymous visitor by referral, sign ups, low money bucket, big money bucket.

Top of the funnel - Hopefully clear from the previous one, top of the funnel would be the max of users you reach out to or get to your site, usually down to getting them to sign up.

Bottom of the funnel - This is usually going from time you have a user to customer and then growing that customer via cross-selling and upselling.

Drip marketing - This is the process of gradually sending emails/notifications to your customers to get them to engage and learn about the product. Think of it as a welcome email on day 1, an intro on day 3, and on day 5 a different email based on what they’ve done so far. Really good drip marketing will create a different email for the user based on what they have or haven’t done.

Attribution (last/first/multi) - Attribution relates to how you got the user or customer (via web referral). There’s a few different ways of looking at this, last touch is the last website they visited before signing up, first touch is the first referral they were sent from, and multi touch (often more complicated to put in place) attributes something to all referrals they’ve come from.

AR - Analyst relations. Analysts cover particular products or areas in an industry, write reports, and often consult with large enterprises when making buying decisions. Analyst relations or AR is the common term for interacting with them, you can learn more here

PR - Public relations. This is generally the press/media side. It often involves launches, press releases, pitching media etc. You can read more of a guide on it here

Briefing - This is normally with an analyst or press, and is typically a quick 30 minute call occasionally a demo of an upcoming launch.Usually

Inquiry - This refers more to the analyst side. Where a briefing is often more one sided for you to pitch/update them on what you’ve been doing, an inquiry is more a back and forth where you can ask what they’re seeing in the market and for input on direction.

Campaign - A collection of activities that go on around a certain thing focused on specific keywords or theme. This can be as little as a search engine marketing campaign which is the most common, or much larger and coordinated with billboards, webinars, etc.

Lead Gen - The sales funnel usually goes from just getting an email, to talking to them, to getting them to try a demo or run a POC, to eventually buying. Lead gen is the activity of just getting that initial contact so you can then further engage with them. In practice this can often involve giving something away, like a t-shirt in exchange for an email.

Conclusion

While this doesn’t cover every marketing activity under the sun, hopefully it’s a good primer on things you may have heard but been confused by. If there’s important ones I’ve missed please feel free to let me know @craigkerstiens

Janathon Halfway Update

infrequent oscillations — Sun, 17 Jan 2016 19:01:12 GMT

19 activities: 6 runs; 13 walks All Time: 21:04:44 elapsed; 18:56:31 moving Speed: 6.4km/h avg. Distance: 118.88km total; 6.26km avg. Mile: 16:41 avg. Running Time: 6:39:47 elapsed; 6:12:02 moving Speed: 9.4km/h avg. Distance: 55.04km total; 9.17km avg. Mile: 9:37 avg. Walking Time: 14:24:57 elapsed; 12:44:29 moving Speed: 5.0km/h avg. Distance: 63.85km total; 4.96km avg. Mile: 19:57 avg.

Janathon 2016 #17

infrequent oscillations — Sun, 17 Jan 2016 18:39:48 GMT

Having to fetch a new sink waste trap from B&Q and a new gadget from the Co-op locker is a good way to force yourself out into the world. Knocked out 5 miles of mostly OK walking with the proper walking boots on (there was a mild threat of rain/sleet/snow necessitating waterproof footwear). Hopefully another week of rest* should quell the back pain sufficiently that Cakeathon remains a sensible option.

Janathon 2016 #16

infrequent oscillations — Sun, 17 Jan 2016 01:32:32 GMT

Bleurgh. [Just over 2 miles to the shops]

Janathon 2016 #15

infrequent oscillations — Fri, 15 Jan 2016 23:31:00 GMT

The triple sledgehammer of illness, tiredness, and misery arrived in a conjunction of blather to the ankle of 2016, hobbling it. Which means the Janathon is reduced to a slow trundle from Greenwich to home via the Sainsbury’s to fetch some kitchen roll and cake.

Janathon 2016 #14

infrequent oscillations — Fri, 15 Jan 2016 00:48:17 GMT

Today was the day everything hit in spades and laid me up in bed. Apart from two slow and mildly painful walks, I’ve accomplished little except learning that MovieBob loves Ghostbusters and that Dedalus is definitely more fun with a nanoKONTROL2. Maybe tomorrow, I’ll want to settle … no, wait, wrong hobo. [Walk from hot chocolate] [Walk to home]

Janathon 2016 #13

infrequent oscillations — Thu, 14 Jan 2016 00:37:00 GMT

The excesses of Christmas, the indolent misery of a new year, the weekend’s over-exertions, consequent excesses caused by misery, and the abysmal weather have all started to take a grinding toll on my physical (back, stomach) and mental (it’s not pretty) states. My emotional state remains, as ever, deep in the toilet and has been unaffected by 2016 thus far. Yay? A chilly 10km with 2008-vintage Rich Conaty for company did little to alleviate matters.

golang Learnings

infrequent oscillations — Wed, 13 Jan 2016 17:08:38 GMT

Things in golang this week (Mostly falling out of the diggymoo rewrite into godiggy.) [How to include version information] [Changing the usage text from flag] [Twitter library - anaconda] [Simple but useful ORM - crud] [Calling a method by name and returning a string] - I’m experimenting with post-statement hooks for crud in order to enable ON CONFLICT REPLACE or INSERT OR REPLACE support for SQLite.

Janathon 2016 #12

infrequent oscillations — Wed, 13 Jan 2016 01:46:13 GMT

Finally a post that’s not late. Today was a “mildly twinging back really tired why did I sign up for this bleugh” kind of day. Made the best of a bad situation and dug the weights out for 20kg of 2x(10x 10x 10x) arm exercises in about 9 minutes. Obviously there’s no point putting that on Strava - you’ll have to make do with the heart rate graph from ConnectStats instead.

Janathon 2016 #11

infrequent oscillations — Tue, 12 Jan 2016 16:19:07 GMT

Another delayed update - sorry everyone. Yesterday was very much recovery from the weekend and, consequently, was limited to two short walks. [Walk to Starbucks] [Walk home]

Janathon 2016 #10

infrequent oscillations — Mon, 11 Jan 2016 19:41:20 GMT

(Slightly delayed because I’m only now just recovering from the weekend.) After Saturday’s insane hills (660ft in a single climb!), a recovery run was in order (and also because I wasn’t going to be in any mood for exercise by the time I got back to London.) Except I thought it would be a good idea to try Twentywell Lane. Now I am sorry. For everything. And anything. But at least it was only 80m of climbing this time instead of 200m.

Janathon 2016 #9

infrequent oscillations — Sat, 09 Jan 2016 23:52:41 GMT

Crikey. Was planned to be 20 miles but only managed just over 16 miles. BECAUSE OF THE HILLS. LOOK AT THIS BUGGER. That’s about 660ft of climbing in 2 miles. “It’s levelling off” said my LYING running LIAR companion LIAR every 200 LYING yards. No wonder I’m ruined. [26km up Sheffield]

Writing more legible SQL

CRAIG KERSTIENS — Fri, 08 Jan 2016 22:55:56 GMT

A number of times in a crowd I’ve asked how many people enjoy writing SQL, and often there’s a person or two. The follow up is how many people enjoy reading other people’s SQL and that’s unanimously 0. The reason for this is that so many people write bad SQL. It’s not that it doesn’t do the job, it’s just that people don’t tend to treat SQL the same as other languages and don’t follow strong code formatting guidelines. So, of course here’s some of my own recommendations on how to make SQL more readable.

One thing per line

Only put a single column/table/join per line. This is going to make for slightly more verbose SQL, but it will be easier to read and edit.. Here’s a basic example:

SELECT foo,
bar
FROM baz

Align your projections and conditions

You can somewhat see this in the above with foo and bar being on the same line. This is reasonably common for columns you’re selecting, but it’s not applied as often in AND or GROUP BY clauses. As you can see there is a difference though between:

SELECT foo,
bar
FROM baz
WHERE foo > 3
AND bar = 'craig.kerstiens@gmail.com'

And a cleaner version:

SELECT foo,
bar
FROM baz
WHERE foo > 3
AND bar = 'craig.kerstiens@gmail.com'

Use column names when grouping/ordering

This is personally an awful habit of mine, but it is extremely convenient to just order by the column number. In the above query we could just ORDER BY 1. This is especially easy when column 1 may be something like SUM(foo). However, ensuring you explicitly ORDER BY SUM(foo) will help limit any misunderstanding of the data.

Comments

You comment your code all the time, yet so few seem to comment their queries. A simple -- allows you to inline a comment, perhaps where there’s some oddities to what you’re joining or just anywhere it may need clarification. You can of course go much further, but at least some basic level of commenting should be required.

Casing

As highlighted in these examples, having a standard for how you case your queries is especially handy. Sticking with all SQL keywords in caps allows you to easily parse what is SQL and what are columns or literals that you’re using in queries.

CTEs

First, yes they can be an optimisation boundary. But they can also make your query much more read-able and prevent you from doing the wrong thing because you couldn’t reason about a query.

For those unfamiliar CTEs are like a view that exist just for the duration of that query being executed. You can have them reference previous CTEs so you can gradually build on them, much like you would code blocks. I won’t repeat too much of what I’ve already written about them, but if you’re unfamiliar with them or not using them they are a must. CTEs are easily one of the few pieces of SQL that I use on a daily basis.

Conclusion

Of course this isn’t the only way to make your SQL more readable and this isn’t an exhaustive list. But hopefully you find these tips helpful, and for your favorite tip that I missed… let me know about it @craigkerstiens.

A special thanks to @Case for reviewing.

Janathon 2016 #8 - Sitting on trains doesn't count

infrequent oscillations — Fri, 08 Jan 2016 14:11:08 GMT

Since I’ll be doing many hours of that later and the insomnia prevents me from getting enough sleep to be able to get up sufficiently early to knock out a quick morning run before work and it’s cold and I’m tired and not feeling very well and WAH WAH WAH WAH WAH. Anyway. Walking to Starbucks will have to do for today. Tomorrow will make up for it… [Just over 2km]

Janathon 2016 #7 - Insomnia strikes

infrequent oscillations — Fri, 08 Jan 2016 00:08:48 GMT

One of my (annoyingly frequent) bouts of insomnia struck leaving today a thorough write-off. No point trying to force through today’s run when it might risk the the weekend’s fun*. * Not fun. [Barely 2.5km from DLR to home]

Janathon 2016 #6 - Slow Boat To Home

infrequent oscillations — Thu, 07 Jan 2016 01:06:28 GMT

Well, not a boat, but Shanks’s Pony (look it up.) With the world’s tiniest game of football (ok, I kicked a ball back to someone by Island Gardens DLR) and a very brief stop in Greenwich Superdrug for some protein snacks. All whilst carrying 2kg of biltong. I tell you, it’s 24/7 non-stop literally all out party time excitement. [Biltong carrying trundle]

How to make a blog like this

Bytepawn - Marton Trencseni — Thu, 07 Jan 2016 01:00:00 GMT

Make a simple blog with Github Pages and Pelican.

Airflow review

Bytepawn - Marton Trencseni — Wed, 06 Jan 2016 01:00:00 GMT

Airflow is a workflow scheduler written by Airbnb. It supports defining tasks and dependencies as Python code, executing and scheduling them, and distributing tasks across worker nodes. It supports calendar scheduling (hourly/daily jobs, also visualized on the web dashboard), so it can be used as a starting point for traditional ETL. It has a nice web dashboard for seeing current and past task state, querying the history and making changes to metadata such as connection strings.

Systems thinking and system traps

Bytepawn - Marton Trencseni — Wed, 06 Jan 2016 01:00:00 GMT

Thinking in Systems, written by the late Donella Meadows, is a book about how to think about systems, how to control systems and how systems change and control themselves. A system can be anything from a heating furnace to a social system. The gem of the book is the part about system traps. System traps are ways a system can go wrong; examples are drift to low performance, seeking the wrong goals, shifting the burden, etc.

Janathon 2016 #5 - A trundle t't Curp

infrequent oscillations — Wed, 06 Jan 2016 00:12:16 GMT

From work t't Curp I ran With the sounds of a labouring man My knees creaked like biscuits And I leaked like a whippet Until I resembled peach flan About 12km t’t Curp

Janathon 2016 #4 - Furshernchurps

infrequent oscillations — Mon, 04 Jan 2016 21:55:45 GMT

A while back I decided I was going to have fish and chips on Fridays. Obviously, being old and senile, that fell by the wayside pretty quickly. Until I remembered earlier and designated today “Honorary (if I walk to Blackheath Station chippy) Friday”. No heart strap today which means no heart rate data from the Garmin but here’s the data from the Fitbit. (In theory, I could merge the TCX files from both but…)

Scraping websites with Cyborg

Tom Forbes — Mon, 04 Jan 2016 02:13:02 GMT

I often find myself creating one-off scripts to scrape data off websites for various reasons. My go-to approach for this is to hack something together with Requests and BeautifulSoup , but this was getting tiring. Enter Cyborg , my library that makes writing web scrapers quick and easy. Cyborg is an...

Janathon 2016 #3 - two runs, one cup (of coffee)

infrequent oscillations — Sun, 03 Jan 2016 20:36:20 GMT

Had to go and pick up some extra drone batteries (which may turn out to be a mistake - the 500mAh one it came with hadn’t died after 12 minutes and these are 680mAh - giving me at least an hour of cumulative flight) and I decided that running there and back would make up for lazing about watching Elementary all day. (Plus it would earn me a coffee and croissant.

How Necessary Are Those Expenses?

Tilting at Windmills — Sun, 03 Jan 2016 02:00:00 GMT

One of the key elements of personal finance is separating "needs" from "wants". Every guide to controlling your budget usually starts with…

2015 Year in Review

Ben Overmyer's Site — Sun, 03 Jan 2016 02:00:00 GMT

2015 was a year of big changes for me. The biggest was starting on the path to divorce and moving into my own apartment. For the first time in my life, I'm living completely on my own. That feels very different, and very exciting. I also had my first anxiety attack earlier in the year. Since then, while I haven't had any more, I've definitely felt more anxious and vulnerable than before. This is strange, since prior to the attack, I always felt like I didn't worry about anything. I'm still mystified as to why it happened.

Continuing the health-related theme, in December I noticed that I had lost ten pounds since the last time I'd been weighed. I chalked it up to my new method of commuting since moving into the apartment - walking to work. I'm not convinced that was completely the case anymore, but it was probably a big contributor. The holidays erased half of that loss, but eh, I can recover from that. I established a habit of doing push-ups every morning after waking, I'm working on my nutrition, and I joined Bodyspace, a fitness social networking run by Bodybuilding.com.

At work, I went from being sole operations engineer on a legacy platform to lead operations engineer on a team that helped launched a brand-new platform to, in October, digital operations manager. Granted, I'm a “working manager,” and still spend a lot of time on the engineering side of things. Starting in November, I began budgeting in earnest. I'm much more aware of where my money is going and for the first time in well over a decade I feel in control of my finances. I've kept it up, and by this point I think budgeting has become a daily habit. 2015 was very stressful. 2016 seems full of promise and I'm very optimistic about the future. Here's to a year of health and personal growth!

Janathon 2016 #2 - even slower and even easier

infrequent oscillations — Sun, 03 Jan 2016 00:19:51 GMT

Didn’t feel like parkrun (too tired, upset stomach) but eventually stirred for what was going to be a 5km coffee jaunt but which turned into a hilly* 10km coffee wander and then a 6km wet wander home for a total of just over 16km (10 miles for imperialistic loons.) Elevation graph for the hilly jaunt, a la Buko, via Strava Out to Woolwich… And back

Janathon 2016 #1 - Slow but easy

infrequent oscillations — Fri, 01 Jan 2016 20:35:46 GMT

I was hoping to do a double-parkrun this morning but the start times were inconvenient - whilst I could get to (not-)Greenwich parkrun for 09:00 (Sunday bus service notwithstanding), getting from there to the other options (Hilly Fields, 10:00; Southwark, 10:30) wasn’t possible unless I cycled. And who wants to cycle 20km+ just for two parkruns? Southwark won by dint of being easier to get to (and later). Southwark parkrun, 29:05, Strava

Hacker News Personal Blogs 2016 | Top 100 Blogs

Walking through a basic Racket web service

Falsehoods Programmers Believe About CSVs

Updates:

Vertically striping images for fun

MH-Z19 CO2 sensor reader, logger and visualiser

MH-Z19 CO2 sensor reader, logger and visualiser

Usage

Connection

Annoying CSV and the JSON solution

Belated ElixirConf 2016 Recap

Vignettes

Repeating History...on Purpose...with Elixir

The many methods of Decim8

Syncing from Postgres to Salesforce - Data Mappings

Leads, Contacts, Accounts oh my

Lead vs. Contact

Accounts vs. Opportunities

In summary

From Developer to DevOps: My Story

Sphinx Real Time Index How to Distribute and Hidden Gotcha

Death's End

Don’t build cockpits, become a coach

Noisevember 2016

Explaining VarnishHist – What Does it Tell Us

MySQL Dump Without Impacting Queries

Open DNS for when DNS outages occur

Embedded Linux Conference Europe 2016

Elixir ETS + Mnesia vs Redis

Blogtober 2016, #2

Blogtober 2016, #1

Kernel Recipes 2016 notes

The kernel report

A tour of Postgres' Foreign Data Wrappers

Overview of foreign data wrappers

Postgres FDW

MySQL FDW

MongoDB

Many more

Making a Twitter bot that looks for hashes

Why ?

What do you find when you search Twitter for hashes ?

What is this image ?

To all Companies Currently Recruiting

Guess the city

GPL Time-bomb an interesting approach to #FOSS licensing

Test-Defective

searchcode server under fair source

The Worst Individual I Ever Worked With

Types of Testing in Software Engineering

Castle Hill Winter Trip 2016 Photos

Email as a Microservice

Syncing Stash/BitBucket with searchcode server

Python Fabric: Getting File from Host as String

On Check Constraints and Whole Value

searchcode.com: The Architecture – migration 3.0

Counting Pizza with Python

PostgreSQL functions with Elixir Ecto

Syntax highlighting and CSS support added to wordinserter

How to Hide Methods From Fabric Task Listing

Python Fabric How to Show or List All Available Tasks

Set Ubuntu Linux Swapfile Using Python Fabric

Python Fabric Set Host List at Runtime

The State of Android in 2016 & The OnePlus 3 Phone

The OnePlus 3

Insanity with Elixir + Phoenix + PostgreSQL

Update Delayed Serial STONITH Design

A Linux Cluster Base STONITH provider for use with modern Pacemaker clusters

v1 vs v2/v3 versions of the rcd_serial STONITH system

What is Chaos Testing / Engineering

What is Chaos Testing?

CSV: An Encoding Nightmare

Excel: A World of Hurt

Excel for Mac: Broken Beyond Belief

UTF-8? What’s that?

The Right Way Functional Workaround

Starting a Snap site with Stack and Persistent

Five mistakes beginners make when working with databases

1. Storing images

2. Limit/Offset

`v1` vs `v2/v3` versions of the `rcd_serial` STONITH system