Hacker News Personal Blogs 2017 | All Blog Posts

Fees are a metric worth looking into but I would say that the story they tell is more of how…

🦉 olshansky 🦁 — Sun, 31 Dec 2017 12:38:45 GMT

They’re actually growing at a much higher rate than I had expected! I think this is healthy given how BTC will probably be a settlement…

Forget Net Neutrality. Think personal clouds instead

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 31 Dec 2017 09:00:00 GMT

The FCC Net Neutrality decision on Dec. 14th, 2017 (*) has rightly caused a lot of outrage, and concrete reactions, both political (e.g. petitions) and technical ones

December Begins

Avy Faingezicht — Sun, 31 Dec 2017 02:00:00 GMT

December was a busy month. A lot of our friends hosted events, and it went by super fast. Posting _December begins_ on the 31st has its own kind of irony, I guess.

📸 Seoul

bpev.me — Sun, 31 Dec 2017 02:00:00 GMT

Migrating from Google (and more)

Nemo's Home — Sun, 31 Dec 2017 02:00:00 GMT

Email
Google Play Music
Google Keep
Phone
LastPass
GitHub

As part of working on my home-server setup, I wanted to move off few online services to ones that I manage. This is a list of what all services I used and what I’ve migrated to .

Why: I got frustrated with Google Play Music a few times. Synced songs would not show up across all clients immediately (I had to refresh,uninstall,reinstall), and I hated the client limits it would impose. Decided to try microG on my phone at the same time, and it slowly came together.

Email

I’ve been using email on my own domain for quite some time (captnemo.in), but it was managed by a Outlook+Google combination that I didn’t like very much.

I switched to Migadu sometime last year, and have been quite happy with the service. Their Privacy Policy, and pro/cons section on the website is a pleasure to read.

Why: Email is the central-point of your online digital identity. Use your own-domain, at the very least. That way, you’re atleast protected if Google decides to suspend your account. Self-hosting email is a big responsibility that requires critical uptime, and I didn’t want to manage that, so went with migadu.

Why Migadu: You should read their HN thread.

Caviats: They don’t yet offer 2FA, but hopefully that should be fixed soon. Their spam filters aren’t the best either. Migadu even has a Drawbacks section on their website that you must read before signing up.

Alternatives: RiseUp, FastMail.

Google Play Music

I quite liked Google Play Music. While their subscription offering is horrible in India, I was a happy user of their “bring-your-own-music” plan. In fact, the most used Google service on my phone happened to be Google Play Music! I switched to a nice subsonic fork called [AirSonic][airsonic], which gives me the ability to:

Listen on as many devices as I want (Google has some limits)
Listen using multiple clients at the same time
Stream at whatever bandwidth I pick (I stream at 64kbps over 2G networks!)

I’m currently using Clementine on the Desktop (which unfortunately, doesn’t cache music), and UltraSonic on the phone. Airsonic even supports bookmarks, so listening to audiobooks becomes much more simpler.

Why: I didn’t like Google Play Music limits, plus I wanted to try the “phone-without-google” experiment.

Why AirSonic: Subsonic is now closed source, and the Libresonic developers forked off to AirSonic, which is under active development. It is supports across all devices that I use, while Ampache has spotty Android support.

Google Keep

I switched across to WorkFlowy, which has both a Android and a Linux app (both based on webviews). I’ve used it for years, and it is a great tool. Moreover, I’m also using DAVDroid sync for Open Tasks app on my phone. Both of these work well enough offline.

Why: I didn’t use Keep much, and WorkFlowy is a far better tool anyway.

Why WorkFlowy: It is the best note-taking/list tool I’ve used.

Phone

I switched over to the microG fork of lineageOS which offers a reverse-engineered implementation of the Google Play Services modules. It includes:

microG Core

Which talks to Google for Sync, Account purposes.

Why: Saves me a lot of battery. I can uninstall this, unlike Google Play Services.

Cons: Not all google services are supported very well. Push notifications have some issues on my phone. See the Wiki for Implementation Status.

UnifiedLP

Instead of the Google Location Provider. I use the Mozilla Location Services, along with Mozilla Stumbler to help improve their data.

Why: Google doesn’t need to know where I am.

Caviats: GALP (Google Assisted Location Provider) does GPS locks much faster in comparision. However, I’ve found the Mozilla Location Services coverage in Bangalore to be pretty good.

Maps

Stil looking for decent alternatives.

Uber

microG comes with a Google Maps shim that talks to Open Street Maps. The maps feature on Uber worked fine with that shim, however booking cabs was not always possible. I switched over to m.uber.com which worked quite well for some time.

Uber doesn’t really spend resources on their mobile site though, and it would ocassionaly stop working. Especially with regards to payments. I’ve switched over to the Ola mobile website, which works pretty well. I keep the OlaMoney app for recharging the OlaMoney wallet alongside.

Uber->Ola switch was also partially motivated by how-badly-run Uber is.

Calendar/Contacts

Most implementations support caldav/carddav for calendar/contacts sync. I’m using DAVDroid for syncing to a self-hosted Radicale Server.

Why: I’ve always had contacts synced to Google, so it was always my-single-source-of-truth for contacts. But since I’m on a different email provider now, it makes sense to move off those contacts as well. Radicale also lets me manage multiple addressbooks very easily.

Why Radicale: I looked around at alternatives, and 2 looked promising: Sabre.io, and Radicale. Sabre is no longer under development, so I picked Radicale, which also happened to have a regularly updated docker image.

Google Play Store

Switch to FDroid - It has some apps that Google doesn’t like, and some more. Moreover, you can use YALP Store to download any applications from the Play Store. You can even run a FDroid repository for the apps you use from Play Store, as an alternative. See this excellent guide on the various options.

Why: Play Store is tightly linked to Google Play Services, and doesn’t play nice with microG.

Why FDroid: FDroid has publicly verifiable builds, and tons of open-source applications.

Why Yalp: Was easy enough to setup.

If you’re looking to migrate to MicroG, I’d recommend going through the entire NO Gapps Setup Guide by shadow53 before proceeding.

LastPass

I’ve switched to pass along with a sync to keybase.

Why: LastPass has had multiple breaches, and a plethora of security issues (including 2 RCE vulnerabilities). Their fingerprint authentication on Android could be bypassed til recently. I just can’t trust them any more

Why pass: It is built on strong crypto primitives, is open-source, and has good integration with both i3 and firefox. There is also a LastPass migration script that I used.

Caviats: Website names are used as filenames in pass, so even though passwords are encrypted, you don’t want to push it to a public Git server (since that would expose the list of services you are using). I’m using my own git server, along with keybase git(which keeps it end-to-end encrypted, even branch names). You also need to be careful about your GPG keys, instead of a single master password.

GitHub

For bonus, I setup a Gitea server hosted at git.captnemo.in. Gitea is a fork of gogs, and is a single-binary go application that you can run easiy.

Just running it for fun, since I’m pretty happy with my GitHub setup. However, I might move some of my sensitive repos (such as this) to my own host.

Why Gitea: The other alternatives were gogs, and GitLab. There have been concerns about gogs development model, and GitLab was just too overpowered/heavy for my use case. (I’m using the home server for gaming as well, so it matters)

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Writing first ray tracer

Whacky Labs — Sat, 30 Dec 2017 13:02:00 GMT

One of the many goals I had for 2017 were:

Writing a ray tracer.
Restarting the blog.

Writing a ray tracer

Sometime when the trees were starting to shed their leaves for the winters, I was worried that I was falling behind my goals for the year. I finally decided to order the book I wanted to read for many years now. The following day I placed an order for Physically Based Rendering, which arrived within a long and sleepless month.

As I was close to finishing the first chapter, I realized I had never written a ray tracer. Although, many people tend to claim that writing a ray tracer is the most natural and natural first step in the beautiful world of computer graphics, or like how easy it is to write a ray tracing algorithm, as it requires no prior knowledge of any graphics API or how the graphics pipeline functions, for that matter. Still, I had never written it.

The closest I had ever been to understanding a ray tracer was when following the dissection of the mysterious and amusing ray tracer on a business card. As a follow up, I started following the articles on https://www.scratchapixel.com about ray tracing, to get a more in depth understanding of how would one go in order to write a ray tracer from scratch.

Although, these blogs were tempting enough for me to start diving into writing my first ray tracer, but still I was too busy (read lazy) to actually do it. Then, one day I ran across the minibook Ray tracing in one weekend. Although the book was probably designed to be finished within a weekend, and rightfully so, still I took it really slow and maybe finished in about a month.

For every chapter, I used to do my own side research as well. For example, when working on the camera, I would often go side track and read about the innards of a real life camera. How do things like appertures, depth of field are supposed to work? What is the role of a lens in camera, and so on. Also, while coding, I tried to write a bit modern-ish C++ and explore the Apple simd.h library for all my math needs, which interestingly has no or very less online presence. For example, there is no proper documentation except for the header files, and searching any thing with keywords simd would return results from anything related to the SSE instruction sets to the architecture itself.

Eventually, today, just when the year is about to end, I was able to render the final target image from the book!

https://github.com/chunkyguy/SimpleRayTracer

Restarting the blog

Another thing I had planned to do for a very long time was restarting this blog. The death of this blog started with some miscommunication between me and my hosting service Yahoo Small business India. Unfortunately my service renew date coincided with their transition to Aabaco. So every time I tried to renew my service I received a generic and unhelpful try again later error. At the same time, I was also moving continents, literally speaking. Later when I was finally settled and tried to renew the service again to my horror I received the message from them that due my failure to renew they had wiped my data from the server.

Eventually, I transferred the domain name and built the blog up from whatever scrap I could find. This entire process had left me a bit wretched to say the least. Also, at the same time I was having hard time finding something interesting to focus on that I could write about.

Finally, it was the goal of 2017 to restart the blog again, and if this post is published before the dawn of 2018 I believe 2017 was a great year after all for me!

I hope you had a great year too! Let’s carry the positive spirit to 2018!

Transportation as a service needs better cities, not smarter cars

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 30 Dec 2017 09:00:00 GMT

I just discovered some declarations from a former vice chairman of General Motors and a Bay Area think thank that confirm what I recently proposed about driverless cars.

Favorite Books Read in 2017

dominik.net — Sat, 30 Dec 2017 02:00:00 GMT

My favorite books out of the 102 books (~38,000 pages) that I read in 2017:

Sci-Fi, Fantasy, and Speculative Fiction

Revelation Space series (Revelation Space, Redemption Ark, Absolution Gap) — Hard scifi space opera. Greatly enjoyable, with memorable characters, backstory, science, history, and of course the requisite galactic-scale threats.

House of Suns — Set outside the Revelation Space universe, Alastair Reynolds manages to craft a compelling tale that spans galaxies and offers new mysteries of its own.

Stories of Your Life and Others — Succinct almost to the point of austerity, Ted Chiang’s hauntingly written tales waste not a word to weave a web of wonder, delight, and, at times, echoing mystery.

Invisible Planets — Translated from Chinese by Ken Liu (who translated The Three Body Problem), this collection of science fiction short stories offers new and intriguing perspectives on both classic and novel themes. Hao Jingfang’s Folding Beijing struck me as particularly moving and poignant; an imaginative meditation on inequality, class, and power.

Reamde — A fun, rollicking romp across worlds both virtual and real, told with Stephenson’s usual brilliant flair for detailed history and similarly detailed prognostication.

Modern Fiction

The Neapolitan Novels/L’amica geniale (My Brilliant Friend, The Story of a New Name, Those Who Leave and Those Who Stay, The Story of the Lost Child) — Told in an evocative, warm, almost conversational style, this quadrilogy compellingly tells the story of the life of two girls who grew up in Naples, and all that life brought them to bear, and all they brought to bear to life. Moving and meaningful.

Classics

The Old Man and the Sea — Somehow I hadn’t read this classic until this year. Hemingway at his finest: a simple tale, beautifully told.

History

Dreamland — Powerful exposition on the modern history of the opiate drug trade in the United States: how it came about, how it prospered, how its tale interweaves with that of prescription painkillers, and how it rages on today, with more Americans dying each year to drug overdoses than die in car crashes.

The Way of Strangers — Why would someone join the Islamic State? This book examines that question both at the personal and global level.

Dynasty — Caesar is dead. Long live Caesar. This book explores the immediate history of Rome after the murder of Julius Caesar, narrating the machinations and consolidations of power that led to the reign of the emperors for the next several hundred years.

Economics

Seeing Like a State — An amazing book. Introduces the powerful lenses of legibility and metis; illustrates, with examples from scientific forestry to communist Russia, the dangers of hubris and model-blindness. Thought provoking, particularly considering the increasing legibility and inherent reductionism of a future mediated by ever-more-prevalent technology.

The Rise and Fall of American Growth — Argues that 1870 to 1970 comprised the most productive hundred years of history and that future growth will not reach those heights of productivity any time soon. Strongest in its economic history, offering insights into the everyday and illustrating the impact of now-commonplace inventions such as the car, lightbulb, indoor plumbing, television, and flight.

Leadership and Self-Development

Radical Candor — Preaches the importance of radical candor: a combination of caring personally and directly challenging. Contrasted, in sometimes humorous tales from the field, with the failure modes of ruinous empathy and obnoxious aggression. Clearly and concisely written, with lessons applicable to all, whether manager or individual contributor.

Principles — How to live and work effectively? This book explores, in great detail, Ray Dalio’s own approach, how he arrived at it, how he put it in practice, and why he believes it works. Not all lessons are every where applicable, but there’s something to learn from here.

Business

The Attention Merchants — A comprehensive history of advertising, thoughtfully told, from the penny press to the native ad.

The Upstarts — Much like Brad Stone’s earlier work The Everything Store explored the history and rise of Amazon, this book examines the rise of Uber and Airbnb, along with a few other companies that didn’t quite rise so far. Published in early 2017, so it doesn’t take into account the most recent upheavals at Uber.

WTF?: What’s the Future — Tim O’Reilly counters the zeitgeist’s prevailing pessimism about the future of tech and offers an inspiring vision of how things could/should be. Written with learning, humility, and love.

Design

Visual Explanations — A classic that long rested on my bookshelves but that I hadn’t properly read, cover to cover. Dated in parts, stunningly farsighted in others, the book is treat to enter into and read.

The Shape of Design — Delightful meditations upon the nature of design, and indeed, of this wonderful world we find ourselves in.

Politics

The Righteous Mind — Each of us arrives at our most deeply held convictions through rigorous application of reason, right? Wrong. This book illustrates how and why.

Philosophy

When Breath Becomes Air — Moving, meaningful, lyrically written. A powerful reflection on what it means to be human.

The Subtle Art of Not Giving a F*ck — Stoicism for the digital age. Entertainingly written, with profound lessons hidden beneath seemingly casual prose.

Science

Seven Brief Lessons on Physics — Short (86 pages!) and sweet. Evocatively explores seven aspects of modern physics and communicates the author’s passion for his field.

Machine, Platform, Crowd — A splendid book, replete with riches. Vital for understanding where we, as a society and an economy, are and where we’ll be heading in the decades ahead.

Soonish — You know that sparkle in your eye when you learn something new? This book is full of those sparkles.

Links - December 30, 2017

Avy Faingezicht — Sat, 30 Dec 2017 02:00:00 GMT

Hello, and welcome to my last set of links of 2017. Let's get right into it:

Earthed by UA Fanthorpe

ho.dges.online — Sat, 30 Dec 2017 02:00:00 GMT

Not precisely, like a pylon or
A pop-up toaster, but in a general
Way, stuck in the mud.

Not budding out of it like gipsies,
Laundry lashed to a signpost, dieting on
Nettles and hedgehogs,

Not lodged in its layers like badgers,
Tuned to the runes of its home-made walls, wearing
Its shape like a skin,

Not even securely rooted, like
Tribesmen tied to the same allotment, sure of
The local buses,

2017 in review

Jason Eckert's Website and Blog — Sat, 30 Dec 2017 02:00:00 GMT

So what happened in 2017?

Last year I thought I completed my basement arcade by getting Donkey Kong (see my blog post entitled “2016 in review”), but I was wrong! I actually got two new arcade games this year. The first one was an arcade machine I never thought I’d find in good working condition. It’s also the absolute coolest arcade game of the 1980s with its amazing vector graphics (as well as my all-time favourite arcade game)…….Atari’s Tempest!

Write Your Leadership Philosophy

allan.reyes.sh — Fri, 29 Dec 2017 02:00:00 GMT

At best, it represents who you are and what you can deliver as a leader, and at worst, it’s a contract for what you aspire to be.

I first wrote down my leadership philosophy about a decade ago by writing down all the values and practices I embraced. At the time, every great leader and commander I served with had a written philosophy—so, I thought I should craft one myself. It changed a lot since then, and I’ve changed a lot.

Exit strategy for the new year resolution

D13V — Fri, 29 Dec 2017 02:00:00 GMT

As the year is almost over, more and more people start reflecting on everything that happened through the year, and start making plans for the new year.

But the last day of the calendar year is just an arbitrary day. There is nothing special about it. It’s all in our heads. We assign that day importance, but it’s arbitrary. The earth rotates around the sun, and there is nothing special about its position on this date. It just happens to be somewhere within the winter season in Europe. The Chinese New Year is in mid February. What gives.

The calendar is a convenience. It makes it possible to refer to a point in time in a clear, unambiguous way.

But nature doesn’t give a fuck about new years eve. Animals are minding their own hunting and hiding. Rains, storms and earthquakes don’t check the calendar to see if it is a good day to wreak havoc. Our bodies’ growth, health and aging don’t change in discrete steps, once per year. They keep coming at us all the time!

So, what’s the point of end of year summaries and new year resolutions? It’s as if at the end of the year someone comes and shakes us “Wake up, wake up!”. “Huh. What’s happening. What happened”… “Ah, OK, this year I learned how to cook pot roast, next year I should loose my belly fat.” By end of January, its sweet dreams again. We’ve forgotten all about new years resolutions. We keep dreaming, and December comes, and we are again surprised, and not ready for it. We didn’t lose the belly fat.

It’s because new years resolutions are bullshit. They are lousy and lazy. If you want to make something happen, you make a resolution. Not a new years resolution, but emotional fortitude and commitment. Not just setting the label on the 365 days as “belly-fat-be-gone”, but actually deciding to go for it, as long as it takes, until you accomplish it, or consciously resolve not to do it.

And, in my opinion, that emotional attachment makes the difference. Not everything needs a whole year. It might take three months, or two years. It might take ten years or even a lifetime.

The end of the calendar year is really dumb exit strategy for a goal.

Instead, emotional attachment to our resolution is an effective exit strategy. That is, we are emotionally committed to our resolution, which means that we don’t abandon it, unless another, stronger emotion comes against it, or we achieve it, or we re-assess our emotion and decide it is not that strong.

Emotions work because they’re more real, than the calendar year’s beginning and end. Our emotions affect our hormones, and biochemistry, and even if it is placebo, placebo sometimes works.

My current resolutions are to write a lot and to figure out a long term career path. Deciding on vocation whether it’s a software engineer, or running my own business, or something else. These resolutions are not new years resolutions. They took a long time of deliberation and thinking to arrive at. It took a bunch of effort to commit to them. My emotional resolve built up over that time, and are what drives me to make sustainable progress towards these goals.

These goals are active. I have to do something to achieve them. They are not avoidance goals. An avoidance goal is “Stop drinking” or “Stop smoking” or “Don’t be distracted by the Internet”. Those are lousy goals. They are just as much bullshit as the new years resolution. The underlying objectives are good and commendable, and something to strive towards, but the way of measuring success is lousy. They can never be achieved in the abstract.

I’ve had a little bit of a success with the drinking aspect so let me elaborate. If your goal is the absolute “Never drink again” then that’s destined to fail. It is just easy to slip up sometimes and forget about the resolution not to drink, and out of habit have a sip whiskey sour at a birthday party, until you realize “Oh shit!”.

In that case, it’s an immediate failure. The statement “Never drink again” is false. You can reset the timer on it, but that’s a slippery slope. Every time it happens, it chips away from the emotional resolve. It’s almost impossible to muster so much emotional resolve to last forever. But I think it is possible to muster enough resolve to last for some period of time.

I think a better resolution in this case is to set a period such as thirty or ninety days and resolve to accomplish one or multiple such “dry” periods. With no end date for completing them. If you’re not done by the end of the year, why, keep going. You’ve presumably made some progress, so you haven’t failed. Personally, I’ve done a 30 day no drinking every year for ten years in a row and over time it really decreased my overall alcohol consumption to a point where I rarely drink.

I have a much more healthy relationship with alcohol now. I don’t think I’ll ever fully eliminate it from my life, but I have very little thirst for it now.

My first such period happened when I was hungover. It started in May, and not as a new year resolution. I was disgusted with myself for drinking too much the night before, and decided to avoid drinking for month and a half, until my birthday. I only made it twenty six days then, but I still regarded it as a success. My personal record. If I had the lousy goal of never drinking, I would’ve relapsed much sooner and the goal would have felt much more like a failure. My emotion would’ve been :(, instead of the :) that I got from accomplishing twenty six days without drinking!

Taking a defensive goal of “never drink again” and turning it into the offensive “accomplish 30 days of no drinking” made the difference emotion-wise. It embedded the exit strategy within the goal. I had enough emotional fortitude for achieving that goal, at creation time. I didn’t, and still don’t have enough emotional fortitude to give up alcohol completely.

Goals and resolutions without exit strategy are lousy goals.

How to set up rsnapshot instead of Time Machine

bastibe.de — Fri, 29 Dec 2017 02:00:00 GMT

(This blog post was changed since my initial strategy of disabling the lockfile didn't work. Turns out, the lockfile is required, and backups have to be stacked.)

Yesterday, I wrote about how Time Machine has failed me. Time Machine keeps regular backups, going back as far as your hard drive space permits. In theory. In practice, every year or so it messes up somehow and has to start over, thereby deleting all your older backups. A backup that is not reliable is not a backup.

Luckily, there are alternatives. Probably the easiest is rsync¹, a very cool tool that copies files and directories from one place to another. You could simply run this once a day, and have a new backup every day. You can even configure rsync so it doesn't need to copy unchanged files, and instead hard-links them from an older backup. rsnapshot automates this process to keep a number of tiered copies, for example ten hourly backups, seven daily backups, four weekly backups, and a hundred monthly backups. Each backup is then simply a directory that contains your files. No fancy starfield-GUI, but utterly reliable and trivial to understand ².

Setting up rsnapshot on macOS is not quite as straight-forward as I'd like, and I couldn't find a great guide online. So, without further ado, here's how to configure rsnapshot on macOS:

Install rsnapshot

brew install rsnapshot
Write the config file

You can copy a template from homebrew: cp /usr/local/Cellar/rsnapshot/1.4.2/etc/rsnapshot.conf.default /usr/local/etc/rsnapshot.conf And then configure the new configuration file to your liking (preserve the tabs!):

  config_version	1.2 # default
  verbose		2   # default
  loglevel	3   # default

  # this is where your backups are stored:
  snapshot_root	/Volumes/BBackup/Backups.rsnapshot/ # make sure this is writeable
  # prevent accidental backup corruption:
  lockfile	/Users/bb/.rsnapshot.pid
  # use this if you back up to an external drive:
  no_create_root	1   # don't back up if the external drive is not connected

  # configure how many tiers of backups are created:
  retain	hourly	10
  retain	daily	7   # dailies will only be created once 10 hourlies exist
  retain	weekly	4   # weeklies will only be created once 7 dailies exist
  retain	monthly	100 # monthlies will only be created once 4 weeklies exist

  # the list of directories you want to back up:
  backup	/Users/bb/Documents		localhost/
  backup	/Users/bb/eBooks		localhost/
  backup	/Users/bb/Movies		localhost/
  backup	/Users/bb/Music		localhost/
  backup	/Users/bb/Pictures		localhost/
  backup	/Users/bb/Projects		localhost/
  backup	/Users/bb/Projects-Archive		localhost/

Instead of localhost, you can use remote machines as well. Check man rsync for details

Make sure it works and create initial backup

rsnapshot -c /usr/local/etc/rsnapshot.conf hourly The first backup will take a while, but subsequent backups will be fast. A normal backup on my machine takes about two minutes and runs unnoticeably in the background.
Write launchd Agent

Next, we have to tell macOS to run the backups in regular intervals. Conceptually, you do this by writing a launchd agent script³, which tells launchd when and how to run your backups. In my case, I create four files in /Users/bb/Library/LaunchAgents/, called rsnapshot.{hourly,daily,weekly,monthly}.plist. Apple's documentation for these files is only mildly useful (as usual), but man launchd.plist and man plist should give you an idea how this works.

Here is my hourly launchd agent (I'll explain the bash/sleep thing later):

<?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
      <key>Label</key>
      <string>rsnapshot.hourly</string>
      <key>ProgramArguments</key>
      <array>
          <string>/bin/bash</string>
          <string>-c</string>
          <string>sleep 0 && /usr/local/bin/rsnapshot -c /usr/local/etc/rsnapshot.conf hourly</string>
      </array>
      <key>StartCalendarInterval</key>
      <dict>
          <key>Minute</key>
          <integer>0</integer>
      </dict>
  </dict>
  </plist>

For the other four scripts, change the two occurrences of hourly to {daily,weekly,monthly} and change the <dict> portion at the end to

daily:

<key>Minute</key>
    <integer>0</integer>
    <key>Hour</key>
    <integer>0</integer>

weekly:

<key>Minute</key>
    <integer>0</integer>
    <key>Hour</key>
    <integer>0</integer>
    <key>Weekday</key>
    <integer>1</integer>

monthly:

<key>Minute</key>
    <integer>0</integer>
    <key>Hour</key>
    <integer>0</integer>
    <key>Day</key>
    <integer>1</integer>

However, rsnapshot can only ever run one backup at a time without stepping on its own toes. This is a problem when the computer wakes up, and more than one backup was scheduled during its sleep, since launchd will then happily launch all missed backups at the same time. But only one of them will succeed.

To fix this, I delay the later backup tiers using the sleep 0 directive. I use sleep 900 (15 minutes later) for daily, sleep 1800 (30 minutes), and sleep 2700 (45 minutes) for the lower tiers⁴. It seems that there should be a more elegant solution than this, but I haven't found one.

From the documentation, you might think that <key>Program</key> would be more succinct than supplying the binary as the first argument of <key>ProgramArguments</key>, but this apparently uses a different syntax and does not in fact work as expected.

Load launchd agents

launchctl load ~/Library/LaunchAgents/rsnapshot.*
Test launchd agent

launchctl start rsnapshot.hourly If it doesn't work, Console.app might show a relevant error message.
Remove backup directory from Spotlight

Go to System Preferences → Spotlight → Privacy → Add your snapshot_root directory from earlier
Disable TimeMachine and delete your existing backup (if you want)

Start Time Machine, right-click any directory you want to delete, and select "delete all backups of $dir"

Caveats

The configuration file of rsnapshot says that you might experience data corruption if you run several copies of rsnapshot at the same time (and you can use the lockfile to prevent this). This is a problem if your computer is asleep while rsnapshot is scheduled to run, since launchd will then re-schedule all missed tasks at once when the computer wakes up. If you enable the lockfile, only one of them will run.

On the other hand, only the hourly task will actually create a new backup. All higher-level backup tiers merely copy existing backups around, so in theory, they shouldn't step on each other's toes when run concurrently. I have opened an issue asking about this.

There are other possible solutions: ① You could modify the launchd entry such that backups only trigger after a few minutes or, better yet, only once all other instances of rsnapshot have finished. I am not sure if launchd supports this, though. ② You could schedule the hourly task using cron instead of launchd, since cron will not reschedule missed tasks. This would only work for two tiers of backups, though. ③ You could just ignore the issue and hope for the best. After all, if a daily or hourly backup gets corrupted every now and then, you still have enough working backups...

rsync is one of those reliable tools I talked about. It is rock solid, incredibly versatile, and unapologetically single-minded. A true gem!↩
This works great for local backups. If you need encrypted backups or compressed backups (maybe on an untrusted remote machine), this post recommends Borg instead of rsnapshot, but you will lose the simplicity of simple directories.↩
I use launchd instead of cron since launchd will re-schedule missed backups if the computer was asleep.↩
This will fail if the hourly backup takes longer than 15 minutes. This is rather unlikely, though, or at least should not happen often enough to be of concern.↩

Sourcing developer marketing content

CRAIG KERSTIENS — Thu, 28 Dec 2017 22:55:56 GMT

I spend a lot of time with dev tool and data companies. I think I’ve more or less banished myself to a life of working in the space, no consumer products for me. In that world a common topic that comes up amongst marketing teams is how do I get my team to contribute to content? Sometimes the person already has an idea of how they want the team to jump onto the bandwagon of their plan, sometimes they’re entirely open minded. I won’t get into pros and cons of various approaches here, rather after sharing some of my approaches in one on one settings I thought it could be useful to share more broadly here.

Turn emails into blogs

I’m a big fan of high quality content when it comes to developer focused products. Yes, you can publish x vs. y with FUD and get some traction with it. You can publish high level customer stories that don’t get down into the details. But publishing high quality deep technical content that other engineers appreciate gives you a huge head start in getting buy in from your customers. The best thing about much of this type of content is you likely already have it sitting within your organization.

Engineers spend a lot of time being thorough and articulate in their emails to their peers. If you see a well written email from one engineer to the engineers@ list and then others chime in either with questions, follow-up, or praise then it’s likely a great candidate for a blog post. We recently had one of these at Citus where another engineer replied with “Nice blog post :)”, a few weeks later we had a post for the rest of the world to read.

The thing about emails though is the engineers won’t usually take and turn them into a blog post. Get a technically minded person that likes to write and put some framing around it, pull out relevant details, and collaborate with the engineer. I’ve often found going from one of these emails to a fully published blog post can be anywhere from 2-8 hrs (including reviews and edits).

Beginner mindset with tickets

As your product people and engineers spend more and more time with the product they become numb to the cool feature from 2 years ago. That initial awe is just expected. Yes there are new advanced features and tech that is being built… but most of your users aren’t the advanced power users. You still need to speak to the people just now onboarding and discovering you.

The best way to maintain this beginner mindset is to capture the interactions with those that are newer to your product. Over the years I’ve made the practice of cataloging support tickets and the type of issue encountered. Any time the same issue is seen several times it becomes a candidate for a blog post or at the very least being clearly documented.

Make sales engineering obsolete

Your sales engineers spend weeks with customers helping them implement complex solutions with your product. In a lot of cases they get really good at helping re-implement a similar solution over and over. Similar to your process with support tickets they’re a great source to take what they’ve done a few times over and turn it into a blog post.

Usually after the first time implementing they have some ideas but there are rough edges. The second or third time they’ve helped a customer implement a particular approach it becomes a candidate for a post.

Training and Documentation don’t have to be siloed

I’m a big fan of not doing anything for one reason. You should absolutely document your product, and if you need to do training to support customers go for it. But that doesn’t mean those bits of content can’t be re-used. Documentation is a great place to take the factual how something works and then give it some narration of the end to end experience in a blog post.

The process

If you’re asking the question how do I get more good content out there, then you’re not leveraging the content you’re already sitting on top of. Though perhaps equal is getting a process in place. A few tips for that:

Don’t blanket ask to a list for volunteers, ask individuals and about specific posts. If you see a great email, respond and ask if you can work with them to get it turned into a blog post
You’ll need to navigate the process, and make sure to have a process
Separate messaging/flow from grammar
My typical process is: outline -> draft -> feedback from 2-3 on draft -> finalize draft -> feedback from 4-5 (often some external parties) -> publish

What’d I miss?

Have other tips that are key to the process? Know of other places where content may already exist but isn’t being leveraged? Would love to hear your thoughts.

Dropbox deleted my pictures and Time Machine didn't backup

bastibe.de — Thu, 28 Dec 2017 02:00:00 GMT

Dropbox deleted some of my favorite photos. Have you looked at all your old pictures lately and checked if they are still there? I have, and they were not. Of course Dropbox denies it is their fault, but no other program routinely accessed my pictures. I am not alone with this problem. It must have happened some time between the summer of 2015, when I put my pictures on Dropbox, and the summer of 2016, when Time Machine last corrupted its backups and had to start over, thereby deleting my last chance of recovering my pictures. The pictures are gone for good.

So, what have I learned? Dropbox loses your data, and Time Machine can't restore it. These programs are obviously no good for backups. Let me repeat this: Dropbox and Time Machine are not a backup! A true backup needs to be reliable, keep an infinite history, and never, never, never accidentally delete files.

From now on, I will use rsnapshot for backups. Here's a tutorial on how to set it up on a Mac. I have used rsnapshot for years at work, and it has never let me down. For syncronizing things between computers, I now use syncthing. Both of these programs are not as user-friendly as Dropbox or Time Machine, but that is a small price to pay for a working backup.

A few years ago, I had high hopes that Apple and Dropbox and Google and Amazon would lead us to a bright future of computers that "just work", and could solve our daily chores ever more conveniently and reliably. But I was proven wrong. So. Many. Times. It seems that for-profit software inevitably becomes less dependable as it adds ever more features to attract ever more users. In contrast, free software can focus on incremental improvements and steadily increasing reliability.

Top album picks for 2017

smcleod.net — Thu, 28 Dec 2017 02:00:00 GMT

Here are my top album pics for 2017 (in no particular order)

Note: This is by no means an exhaustive list, it’s just the top albums that really stood out to me and in all fairness, I’ve thought of several others since so there may be a follow up post (or two).

Father John Misty - Pure Comedy

As a bonus, a short film was released with the album:

From Wikipedia:

Bitcoin's price isn't a bubble, it's an adoption curve

🦉 olshansky 🦁 — Wed, 27 Dec 2017 21:35:51 GMT

It's difficult to objectively quantify Bitcoin's adoption rate since there are too many factors to consider. Should it be the number of…

HP 4951C Protocol Analyser

smcleod.net — Wed, 27 Dec 2017 02:00:00 GMT

My good friend Joel Shea received a most unlikely gift this Christmas - A vintage HP 4951 Protocol Analyser.

According to the HP Computer Museum:

Original Price: $3595 The 4951B was replaced by the 4951C and 4952A in 1986. Both new models handled Async, BSC, SDLC, HDLC, X.25 and SNA protocols. The 4951C also handled DDCMP, while the 4952A did not. The 4952A handled X.21 while the 4951C did not. Both new analysers used a floppy dive (618 KB) for removable media.

2017-12-27/01

ho.dges.online — Wed, 27 Dec 2017 02:00:00 GMT

2017-12-27/02

ho.dges.online — Wed, 27 Dec 2017 02:00:00 GMT

2017-12-27/03

ho.dges.online — Wed, 27 Dec 2017 02:00:00 GMT

Guidance on performing retrospectives

CRAIG KERSTIENS — Tue, 26 Dec 2017 22:55:56 GMT

In my career I’ve had to conduct a number of retrospectives. Ahead of them it already sucked, there was an outage at some point, customers were impacted, and it was our fault. Never was it solely on our underlying infrastructure provider (AWS or Heroku), nope the blame was on us and we’d failed in some way. And as soon as the incident was resolved, it wasn’t time to go home and decompress with a beer, it was time start the process of a retrospective.

Finding the motivation to get right back to work is tough, but not losing time is important. There is probably a lot out there on retrospectives, and in general I was well rehearsed at them. But since I’d not performed a large scale one in a few years I found myself rusty and thought it’d be good to share some of our process.

Capture details immediately

It may not be clear if you’ve not been involved in many, but a retrospective is more than just a meeting to discuss what happened and how to fix it. It’s an overall process, it begins with capturing thorough details of what happened. The start is a timeline. The best thing to do is capture the details while they’re fresh. Start with a google doc and simply document the timeline of everything. Capture chat logs that are relevant while they’re fresh in history and easy to find.

The start of an outage likely wasn’t the start of the timeline, there may have been something that happened days, weeks, or even years ago. Don’t just start from the time things went offline, go back to the causes as much as possible. If code was committed a year ago that was the offender make sure to note that.

Running the retrospective (the meeting part)

There are a number of various good practices for running the retrospective itself. There are also a lot of different formats, all valid each with their own pros and cons. You can do with a basic timeline, what went well/didn’t, do a five whys analysis. I tend to prefer a clean and dry analysis of timeline, what went well and what didn’t and what we’re doing about it.

Some key tips to help a retrospective meeting be productive:

Explicitly set time bounds for each activity ahead of time, more so than maybe any other meeting it’s important to get through all your agenda. Hard time limits on the planned items is how you accomplish this.
Spend at least some time on what went well, retrospectives aren’t fun. Spending some time on the good parts of your process and response isn’t wasted, just don’t be overly self-back-patting (that’s not a word but you get it).
Don’t discuss people, or rather don’t point fingers. Yes, people will come up, but it’s about the technical and process errors not the person that performed them.

The important part

Every step in the retrospective is important, but the goal of them all is to get to how you can improve. With any retrospective there are likely two categories of improvements that will surface. The first is bugs that caused the issue or engineering that could go in place to help with the specific issue. The second are process improvements. If you don’t have improvements in both areas then spend more time thinking on the one you’re missing.

Improvements shouldn’t be isolated to the exact issue you saw. Yes you may see the exact same issue again, but there is also a lot more you can draw out that helps improve overall quality. It’s inevitable you’ll see different issues in the future, thinking of how you can improve your systems and processes to catch those future issues is time well spent.

You’ve done the hard work in the above, but it’s still good to share publicly and transparently. Within your public retrospective I tend to follow:

Apologize, and mean it
Show a firm understanding of your systems, and communicate the problem. Don’t try to be fancy technically, but don’t be too highlevel. Think goldilocks.
Share what you’re doing to improve

Credit to Mark for being the first person I’m aware of to lay it out like the above

Thanks to Mark Imbriaco, Blake Gentry, Daniel Farina, Lukas Fittl, Will Leinweber for input and feedback along the way.

Looking for more resources on the topic? Make sure to check out these two talks:

Editing Gameplay Videos without Re-encoding using FFmpeg

Varun Ramesh's Blog — Tue, 26 Dec 2017 20:04:26 GMT

Editing gameplay videos in Premiere requires that you re-encode, thus losing quality. Instead, you can use FFmpeg to edit losslessly.

My terminal setup

Josh Thompson — Tue, 26 Dec 2017 15:00:00 GMT

note: this is a draft. Please ping me in slack/email with questions, spots where this is unclear. I’ll answer your question, and update this post.

Here’s some quick notes on how I have my terminal setup.

First, I use Zsh. If you’re on a new Macbook Pro, you also are using Zsh.

Install iTerm

If you’re using Terminal.app, head to https://iterm2.com/index.html and click the “download” button. Install the app.

Install `oh my zsh`

Install oh my zsh:

Head to https://ohmyz.sh/ and follow their installation instructions, which is

Run the following command in your terminal:

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Edit ~/.zshrc`

Open up ~/.zshrc in Atom. We’ll use it a few times.

Line 10 or 11, change the theme value to agnoster

github.com/agnoster/agnoster-zsh-theme

Head to the repo and follow the installation instructions, which touch on:

Install the Patched Solarized Fonts:

github.com/powerline/fonts

# clone
git clone https://github.com/powerline/fonts.git --depth=1
# install
cd fonts
./install.sh
# clean-up a bit
cd ..
rm -rf fonts

Update iTerm to use one of the new fonts

Profiles > Text > Font (Meslo LG M DZ for Powerline)

open up this file: ~/.oh-my-zsh/themes/agnoster.zsh-theme

or do atom ~/.oh-my-zsh

Atom open up the themes directory, and open up agnoster.zsh-theme

comment out the existing line that says PROMPT on line 230, and paste in:

PROMPT='$(build_prompt) 
> '

Should look like this:

Resources

A (mostly) automated release process

www.mootoday.com - RSS Feed — Tue, 26 Dec 2017 02:00:00 GMT

Automate whatever you can automate and share the responsibility for the remaining tasks.

Cultivate Curiosity, or 'Reasons to be More Childlike'

Josh Thompson — Mon, 25 Dec 2017 15:00:00 GMT

I’ve had an idea rolling around my head.

I suspect that “being curious” will correlate well with positive outcomes in my life, on pretty much any time horizon, be it days, weeks, or decades. Curiosity feels like a tolerable antidote to boredom, though boredom in and of itself is something to celebrate and embrace.

The goal isn’t to not be bored, it is to not be jaded and closed.

What is “jaded and closed”? I cannot quite define it, but I know it when I see it.

I enjoy spending time with people who are both older than me, and people who are just plain old. When they display traits or attitudes that I either want to emulate or want to avoid, I try to reverse engineer what led to that thing I do or don’t want.

The presence/absence of curiosity has correlated well the presence/absence of other traits I want to emulate.

This habit seems amenable to practice. I.E. the more you look for (and expect to find) objects, remarks, incidents, thoughts, comments that spark curiosity, the more you find these exact curiosity-sparking… things.

I’ve encountered objects and phenomena recently that sparked my curiosity. For example:

an old bridge: How was it constructed? What was it’s general strategy for staying up? (Was it held up by tension or compression? I couldn’t tell)
Metal tags nailed to a sidewalk: Were they marking property lines? Underground pipes?
a pipe in a basement: it was warm. Obviously it had warm water inside, but I couldn’t tell why. It might have been a return for a radiant heater, but I wasn’t sure.
A phrase someone said. Of all the ways they could have expressed a similar thought, why’d they use that particular phrase?
many many more things. (The topic of programming alone could fill books with just questions. Everything from “how the heck do wifi signals not interfere with each other” to “what’s the physics of the millions and billions of calculations a computer does every second to render my favorite cat video?”)

Here’s the thing. It would be very, very easy to not find them curious.

I mean, come on. This is 21st century America! We’ve got real things to worry about, plus smart phones, plus plenty of outrage to marinate in all day. Why be curious about a stupid bridge?

In a small way, curiosity is a potent antidote to all the mind-numbing attention-stealing crap shoved at us from every corner of the average environment.

Once I started looking for things to find curious, I found them everywhere. I’m glad I started looking.

This extends to people, too. I’ve been apologizing regularly to people for what will feel like an interrogation, as I start grilling them. My go-to questions:

What’s a good book you’ve read recently? Why? (If not a good book, a good piece of media. Movie, TV show, documentary, comic book, cartoon, magazine, blog)
If you could go back to {current_age - (age * .25)} and give yourself any piece of advice, knowing it was coming from current you, what would it be? Before you tell me the advice, tell me the context. What was going on in your life at the time, etc.
What’s something the world has too much of?
What’s something you look forward to about aging? What’s something you fear about aging?
Are you spiritual, or religious? Why/why not?
What’s the hardest thing you’ve faced in the last year?

I’ve asked one or more of these questions of probably more than 100 people. I’ve learned so much from these questions, and follow-up questions. I’ll probably be asking some variation of these questions until the day I die, and that doesn’t trouble me one bit.

My programming history (part 2)

Negative Feedback — Mon, 25 Dec 2017 10:40:48 GMT

This post overlaps a little bit with the the previous one, recapping my programming history.

We ended the previous part in 2002 when I finished my Masters and started teaching, but I want to go into a little more detail on my history with scripting languages.

I encountered Matlab for the first time in my second year (in 1998) in a course called CRV220. I still have the study guide so I can say with confidence that we were taught a combination of Turbo Pascal and Matlab. At the time, the Turbo Pascal environment was much nicer than the Matlab one. Even though the Borland IDE was text-based, it had syntax highlighting and step-by-step debugging. It looked like this:

By contrast, in 1998 Matlab was in version 5.2 and looked like this:

We typed Matlab code into Notepad and executed it from this command window. The most impressive part of Matlab was that it could produce graphical output very easily. But the development environment was non-existent. It was, however, very clear that you could solve engineering problems using much less Matlab code than Turbo Pascal. Matlab was also the first language I encountered which had a REPL - you could type commands into the command window one at a time and see the result. I really liked this concept.

In 1998 also bought an HP 48 GL calculator and spent a lot of time programming it do solve problems like balancing chemical equations and so on. We were often allowed to use our calculators in tests, but they were often cleared before we wrote. The HP introduced me to the idea of stack-based languages.

In my final year (in 2000), we did our final year project calculations using a spreadsheet called Quattro Pro, since Excel had not yet achieved the dominant position it currently enjoys. We did not do much programming in our final year.

By the time I started my Masters in 2001, Matlab had just undergone a major rework in the release of version 6, moving to a Java-based workspace and for the first time Matlab was an IDE with an integrated code editor, debugging, and all the features from the Turbo Pascal IDE that I had missed when I first encountered Matlab. I found this new environment quite comfortable and used it happily throughout my Masters work. I used it to do the actual simulation and to produce all the graphs in my dissertation. But I had also bumped up against one of the things that Matlab was not very good at when I was writing the code to make the call graph I posted in the previous part: Matlab was really terrible at text processing.

By 2002 I was completely immersing myself in the Unix way. I had started to see the power of the "small tools doing one thing well" philosophy and spend a great deal of time reading the GNU Info pages for all the various unix tools while also learning about Richard Stalman and the Free Software movement. I was using Matlab on my Linux machine and learning bash. This is when I started learning Emacs, after reading In the beginning was the commandline where Neal Stephenson says the following about it:

"I use emacs, which might be thought of as a thermonuclear word processor. It was created by Richard Stallman; enough said. It is written in Lisp, which is the only computer language that is beautiful. It is colossal, and yet it only edits straight ASCII text files, which is to say, no fonts, no boldface, no underlining. In other words, the engineer-hours that, in the case of Microsoft Word, were devoted to features like mail merge, and the ability to embed feature-length motion pictures in corporate memoranda, were, in the case of emacs, focused with maniacal intensity on the deceptively simple-seeming problem of editing text."

Teaching

In the first semester of 2002 I was the teaching assistant for CRV210 (the same subject where I had learned Matlab, now moved to the first semester). In the second semester I took over CMO320 (Mass transfer) from a lecturer who had moved to New Zealand. At this point I was done with my Masters and thinking of but had not yet enrolled for my PhD. I barely kept my head above water in that first semester, having to revise the material even as I taught it. In the second go around I started using more computer methods in my teaching, using Excel and Matlab to solve mass transfer problems.

As I did this, I started realising that our students were not well prepared to deal with computer solutions of these problems. I eventually moved over to teaching our computer course.

Other languages

I started evaluating different scripting languages which would be better at the kinds of string processing I had done for the call graph program and came across Perl and Python around this time. The earliest proper Python program I could find in my records was from 2004 and was for creating the same kind of file I used to do the call graph (the GraphViz package). I also learned the classic grep, sed and awk combination for text processing in Linux, motivated largely by reading "Unix Text Processing"

I picked up Fortran around 2005 as I started looking for ways to speed up my simulations. In the process I learned that modern Fortran was not at all like the FORTRAN 77 that people love to mock. It is a pretty decent language for numerical work. I learned R in 2007 while I was looking for better ways to handle statistical graphics. At that time R was the best language to learn for statistical work. Some might say it still is, even though Python has the big momentum for data science as of 2017.

Teaching programming

I started teaching CRV210 in 2006. Before I did, I went on a deep dive to understand the best ways to teach programming. I found out about the MIT programming course and read The Structure and Interpretation of Computer Programs, picking up a deep and abiding love for Lisp in the process. I read up on SmallTalk, various graphical languages for teaching programming and all the different models which can be used to understand programming.

I settled on teaching Matlab with a strong focus on functional programming. I taught students to avoid scripts and write everything as functions, building functionality up slowly. I taught recursion and scoping before I even handled basic iteration (for loops). I think this made students understand functions a lot better, but it was an unpopular choice. My students tend to view programming more as a necessary evil than the transformative experience I found it to be. They want to solve problems and learn as little programming on the way to those solutions. This made my course feel hopelessly obtuse as I went over problems like computability and logic before simple things like solving a set of equations using the tools.

Moving from Matlab to open source and Python

Not long after I started teaching programming, it became clear to me that students were pirating Matlab at an alarming rate. Part of the problem was that the cheap student version could only handle small matrices. The problem actually became more pronounced after the Mathworks moved over to a more capable student version which required you to send in your proof of registration to get the student discount. Most students found it easier to crack a downloaded version than to deal with the paperwork of getting the student version. The university was also paying a large sum of money to retain Matlab licenses for the lab computers, and we calculated large costs for he research license which we would need to do actual research with Matlab instead of only using it for teaching.

I started gathering support for a move to Python. The first phase of this was to stop renewing our license for Matlab and move to Octave, an open-source reimplementation of Matlab. This allowed lecturers to retain their Matlab code and get used to a less featureful environment (we were back to a single window interface). CRV was taught for the last time in 2008. Our students started attending the MPR212 classes in 2009, which are now still taught to all the engineering students at UP. They moved to Python in 2013 and from Python scripts to Jupyter Notebooks in 2016.

The notebook revolution

I started hearing about IPython notebooks around 2011, but I only started using them at the end of 2012. It took a while for me to warm to them, but I now feel they are excellent as an exploration tool as well as a teaching tool. I have committed deeply to producing notebooks for the subjects that I teach. There is a great deal of momentum behind the Jupyter project right now and I think they will continue to be one of the best ways to interact with your code for a couple of years yet. My one big issue with notebooks is that bundling the results of calculations together with the code makes it quite hard to handle them in source code management tools. But I am also sure that a future incarnation will fix this issue. It's not a big enough problem to cancel the benefit.

Final word

Right now I feel like we made the right bet when we moved away from Matlab. Python is a much more general purpose language and has a large installed base. I think it is a better first language than Matlab and is something that you can see yourself using even when you go to industry. Of course Matlab hasn't stood still, and is still one of the fastest ways to solve a problem if a toolbox exists for the field you're in, but I haven't felt the need to open much else than Python for a very long time.

I'm keeping an eye on Julia as a competitor to Python in the computational space, but so far it seems to lack critical mass. I'm also testing the waters on Rust as a replacement for Fortran for my high speed code, but it also has too little support for the well-established numerical libraries.

I think the last bit of my recap on history will be about simulation environments, moving from Simulink to Modelica, but this post is probably already too long.

Books I read in 2017

Krishna's blog — Mon, 25 Dec 2017 02:00:00 GMT

I share my top book recommendations for 2017, including Trevor Noah's Born a Crime and The Dictator's Handbook by Bruce Bueno de Mesquita and Alastair Smith.

Smoking in fashion as a promise of bodily experience

Blogs on Michail Rybakov — Sun, 24 Dec 2017 16:07:46 GMT

Diseases have always been fashionable. Tuberculosis brought us the fad of looking pale and skinny. Syphilis and it’s symptomatic light sensitivity and rotten noses brought us sunglasses and fake noses that were worn even by those not afflicted by the illness.
I won’t go into detail as to why disease becomes fashionable - first, I don’t really know, and then there’s a whole conference on it.

One point of interest for me is how a fashionable disease is a promise of a physical experience. Similar to a certain subset of internet memes I wrote about before, where a human or a non-human stand-in reminds the viewer of a bodily experience from everyday life.
Smoking in film and fashion can be interpreted in the same way - as a powerful, addictive, self-inflicted bodily experience, as a promise of being highly attentive to the body, the fingers, the mouth and the inside of the lungs.

As smoking becomes much less acceptable socially - and to advertisers - the fashion photography looks for new ways to make the signifier more abstract - the models shown here are not-smoking emptied non-cigarettes.

It would be interesting to test which other bodily experiences could be promised by fashion - besides the obvious promises of eternal youth, sex and drugs. How about being uncomfortable? Being fat using a fabulous fat-suit? Freezing in the winter while wearing a down jacket with huge holes? Not being able to pee, as the process of unwrapping the clothes takes hours? Never being thirsty with a 12-liter water pack hidden in the garment? Feeling unprotected with sleeves that don’t allow your hands to touch your body, combined with an exposed belly?

Include some promise of bodily experience within your work. It changes things.

What is not in this text but could be: - golden teeth and broken nose as a criminal status symbol - movement restricting feathers in male peacocks as a sign of fitness - jewellery made of wearer’s hair for cancer patients - fidget jewellery for autists as described in a previous text

Man's Search for Meaning

Krishna's blog — Sun, 24 Dec 2017 02:00:00 GMT

Man's Search for Meaning offers a unique perspective on the Holocaust and introduces the theory of logotherapy.

Installing Arch Linux on my Intel NUC

utf9k — Sat, 23 Dec 2017 13:00:00 GMT

My self help guide to installing Arch

ELBO Surgery

Rachit Singh — Sat, 23 Dec 2017 02:00:00 GMT

tldr: The ubiquitous isotropic Gaussian prior for generative models doesn't make sense / doesn't work, which motivates work on priors.

At NIPS, Dawen Liang mentioned Hoffman & Johnson's ELBO surgery paper offhand while talking about tuning KL divergences, and it's very interesting, so I thought I'd go over it. It's very clearly written, so I won't go into any of the derivations, but instead offer my interpretation.

Motivation

I worked in the past on applying variational inference and comparing it to models trained via MAP/MLE inference. I decomposed the evidence lower bound (ELBO) as:

$$\mathcal{L} = \frac{1}{N}\sum_{i = 1}^N\left(\underbrace{\mathbb{E}_{q(z_n | x_n)}[\log p(x_n | z_n)]}_{\text{log-likelihood}} - \underbrace{\operatorname{KL}(q(z_n | x_n) || p(z_n))}_{\text{KL divergence}}\right)$$

This is, I think, the most common interpretation: split the ELBO into a reconstruction term and a KL divergence term. The first encourages the model to reconstruct the data, and the second regularizes the model, asking the posterior distribution over $z_n$ to have a certain shape, like a Gaussian. For example, in a VAE the second term is what prevents the model from just learning a Dirac delta-like posterior $q(z_n | x_n) \sim \mathcal{N}(x_n, 0.001)$ around the original value¹.

In the NLP world, people have seen some problems though - when we have a very powerful generative model (e.g., an RNN), the KL divergence can vanish. This means the posterior $q(z_n | x_n) \approx p(z_n)$ learns nothing about the data, so the generative model $p(x_n | z_n)$ becomes like a language model. The usual trick is to anneal the KL divergence term in, so that that inference can be useful. A lot of people are unhappy with this because it adds extra hyperparameters and it feels really non-Bayesian.

Contribution

The contribution of this paper is the following observation: the KL divergence above measures the distance from the posterior for a single $z_n$ to the prior, but we really care about the KL divergence from the average posterior over all data points to the prior. So they define

$$q(z) = \frac{1}{N}\sum_{n = 1}^Nq(z_n | x_n)$$

which is the average posterior we see. The intuition here is that when we're trying to do inference, we shouldn't exactly be penalized for being very confident in $q(z_n | x_n)$. However, we want the average distribution to be close to the prior, so this term can go to 0 safely without worrying about whether the posterior has learned something. In fact, at the cost of a lot of extra computation, we can even safely set the prior to be this distribution, or let $p(z) \triangleq q(z)$!

Then, they view $n$, the index variable, as a random variable, where the interpretation is that our generative model samples $n \sim \operatorname{Unif}[N]$, and then picks a $z_n$ from $p(z)$. This isn't totally intuitive, but it makes more sense on the inference side, which we'll see below. Finally, they decompose the second term further as follows:

$$\mathcal{L} = \underbrace{\frac{1}{N}\sum_{i = 1}^N E_{q(z_n | x_n)}[\log p(x_n | z_n)]}_{\text{log-likelihood}} - \underbrace{\vphantom{\sum_{i = 1}^N}E_{q(z)}[\operatorname{KL}(q(n | z) || p(n)))]}_{\text{index-code mutual information}} - \underbrace{\vphantom{\sum_{i = 1}^N}\operatorname{KL}(q(z) || p(z))}_{\text{marginal KL}}$$

(this is not an obvious derivation, but the math checks out). Here, $q(n | z)$ (which we can decompose using Bayes' law) can be interpreted as 'the distribution over which datapoint this $z$ belongs to'. The description 'index-code mutual information' comes from an alternative way to write the expression, but I like this one more. Also, they upper bound this value by $\log N$, a not insignificant quantity! This is 11 nats on MNIST.

Experiments

Finally, the most interesting section, which is the quantitative analysis: they apply the model to a set of the usual VAEs with an isotropic Gaussian prior used for binarized MNIST, and get the following results:

	ELBO	Average KL	Mutual info.	Marginal KL
2D latents	-129.63	7.41	7.20	0.21
10D latents	-88.95	19.17	10.82	8.35
20D latents	-87.45	20.2	10.67	9.53

So, what's going on is that as we increase the number of latent dimensions to 10/20, the marginal KL gets large! Which means that the Gaussian prior is not good enough anymore. At least, that's my interpretation. This is interesting food for thought, since that gives a lot of evidence for a hunch that people have had for a while (and motivates work on new prior distributions, like our paper).

Setting up for 'SQL Queries for Mere Mortals'

Josh Thompson — Fri, 22 Dec 2017 15:00:00 GMT

This tweet is from… a while ago. Turns out I didn’t dig into this book, because the pace at Turing didn’t allow for a few weeks of thinking just about SQL.

yes, I'm digging into sql to better my AR skills, and ultimately whatever I need to use next. pic.twitter.com/UhjyGKv1FQ
— Josh Thompson (@josh_works) April 24, 2017

I’ve got more time in my life now, and the company I work for makes heavy use of SQL queries, so I’m spending a little time each day working through this book. The following started as a gist, and I’m putting here, so I can more easily share it with others.

Why Study SQL

Pretty much everything on the internet lives in a database. I figure any boost to my SQL skills will provide outsized returns down the road.

For example, just today I used some basic SQL queries to validate an assumption I had about the frequencies of null values in certain places in our database. I wouldn’t have even thought to try that, if I didn’t know in advance what I could do with a basic SQL query.

If you buy this book, and want to get set up with it’s accompanying data on your own computer, so you can practice yourself and follow along with the book… read on.

The friction of getting set up kept me from getting started with this book, and I don’t want that to happen to you too. :)

I’m working on wrapping my head around SQL. Be it MySql, SQLite, PostgreSQL, etc - these are all relational database management tools, and (I assume) they have much in common with each other.

I’ve got a copy of SQL Queries for Mere Mortals, and I plan on spending some time each day over the next ~month working through the lessons.

Lesson 1 is “setting the damn thing up”. I struggled so much getting it set up, I wrote this gist and gave it to a former coworker/awesome developer when he offered to help.

Turns out, he helped me set it up! I got it working!

And then Turing took over my life, and I stopped studying, then I got a job, and moved machines, and had to start the install from scratch.

So, I’m writing this guide as if you, dear reader, wanted to follow along through the book, and needed help getting the “dev environment” set up locally.

Step 1 - install MySQL on your machine

note: I ended up wiping my entire MySQL local install and instead setting it on Docker. My company uses MySQL 5.6, which is a generation behind the current 5.7, and navigating between the two locally was a massive headache. The rest of this will still work, if you’re good running MySQL 5.7.

I don’t have much help for you here. I struggled hard on this. Turns out I was using Brew to manage MySQL, and that wasn’t playing nice with other tools, so I threw my hands in the air, and did a clean uninstall/re-install of MySQL on my machine.

I followed these instructions to remove/reinstall.

And, the last setup command in the instructions is deprecated. Where it says to use:

mysql_install_db --verbose --user=`whoami` --basedir="$(brew --prefix mysql)" --datadir=/usr/local/var/mysql --tmpdir=/tmp

instead use:

mysqld -initialize --verbose --user=whoami --basedir="$(brew --prefix mysql)" --datadir=/usr/local/var/mysql --tmpdir=/tmp

(The latter came from a comment in the article.)

In your command line, try to run mysql. If it lets you in, great! if not… read on

Step 1.1 - creating a user for MySQL

I followed the instructions in this StackOverflow post on creating MySQL users and got it working:

$ mysql -u root

mysql> CREATE USER 'joshthompson'@'localhost' IDENTIFIED BY 'super_secret_password';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON * . * to 'joshthompson'@'localhost';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

and now I can connect to the DB by running $ mysql -p. It asks for my password, I enter it, and I’m good to go.

Step 2 - import sample date on your machine

I’m using Sequel Pro at the moment. I think you could do this all from the command line, but since I use Sequel Pro for my real job, I want to build familiarity with the tool. (Despite “pro” in the name, the software is free.)

In Sequel Pro:

name: Local Development
host: 127.0.0.1
username: your_username
password: your_password

click connect

You should now be connected to a database in Sequel Pro, and you’re ready to import some data.

The book includes companion data so the reader can practice the queries mentioned in the book. Here’s the readme for the data. The data itself is in a zip available under the “downloads” section here

In Sequel Pro, go to File > Import, and navigate to the data you downloaded and unzipped.

I started small, and imported everything prefixed with 00. As far as I can tell, all the files prefixed with 00 relate to the schema, but not the actual data. If you import one or all of them, you’ll see a very robust schema, with many tables and keys, but no actual content.

After importing the nine structure files, you might see something like this:

And, after importing the data files:

OK! I’m big on getting friction out of the way, and this is helpful. I’ll not touch the 02 files related to views yet, because i have no idea what those do.

Lets spot check this to make sure it works.

I’ve got the Third Edition of SQL for Mere Mortals, published in 2014. I can pop over to page 98, in chapter 4, and run an example query on a given database, to see if it works.

Here we go…

The query is (in plain english) “Show me the names of all our vendors”.

In sql:

SELECT VendName FROM Vendors;

Easy enough. The only thing to be mindful of seems to be selecting the right database. If the database doesn’t have the Vendors table, the query comes up blank. So I jump into one of the databases with a vendors table, run the above query, and have results! (I ran it against both SalesOrdersExample and SalesOrderModify, and got the same thing.

OK, that does it for now. I’m gonna carry on with my learnings elsewhere, and will update this gist occasionally, as needed.

The rest of my thinking on MySQL will hopefully just be on the actual building and executing queries, both as the relate to this book, and things I’m learning/wrestling with on the job.

Just for fun, and to familiarize myself with the databases and tables, I quickly worked through everything in chapter 4. It’s super basic select stuff from single tables, no joins or groupings or anything.

Onward!

Sleep is for backpropagation

D13V — Fri, 22 Dec 2017 02:00:00 GMT

Diving deeper and deeper into machine learning and neural networks I can’t help but notice how some of the techniques there are metaphors for what happens in our brains. After all, our brains a just a lot more sophisticated and larger versions of the deep computer networks that do computer vision, or predict clicks on ads.

When I first grokked backpropagation, I was in awe about how elegant it is. An iterative approach to computing the partial derivative of multi-dimensional, complex probability density function. Beauty, in the mathematical sense of the word. Then, as I learned about the more practical aspects of it, I learned about the less intuitive concept of “batch_size”.

Every round of backpropagation makes the neural network slightly better at its task, by updating the individual neurons based on the errors they make. The batch size comes from the algorighm known as SGD which will only look at small amount of examples before updating. It makes sense to look at the data one example at a time, because humans can learn from single examples. Yet, things tend to work better in, when the network updates by average amount over a batch of examples.

(Image credit: https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/)

The computer will look at multiple examples, then it will update its brain to be better at its task. A human will have a day full of work and conversations, and then they will go to sleep. Almost as if… the human is updating their brain connections during that time. Doing a backpropagation over the batch of experiences during the day.

Here’s how I imagine the human brain. As we go with our days, our neurons collect feedback signals about all the things they did wrong. Those signals manifest themselves as “error chemicals” at each neuron, which temporary update its function. If a neuron gathers too much error chemicals, it gets tired. When too many neurons get tired, we need to sleep. Sleep will remove the error chemicals, and update the underlying neurons in the direction the error chemicals suggest. We can help this process of learning by getting enough sleep, but also by taking naps when needed and meditating.

Even if you think this “error chemical” model is nuts, I don’t disagree with you. But I think that the model is still more useful than nothing. At the least, I’ve prioritized getting good sleep recently, as I’m trying to learn a whole bunch of things, and I think sleep becomes necessary to “save” the current learnings before starting into others. I’ve also been pacing my learning through the day, realizing that taking breaks of mental rest but physical activity do help to recharge me for more learning.

And I’ve heard and read other things to suggest that “the error chemical” model is not too different from reality. For example, the amount of sleep necessary increases with age. Babies need more sleep and more often because their brains have so much to adapt to. And old people don’t need too much sleep. Furthermore, at end of every day, it often happens that our dreams reference things that happened to us, such as eating spaghetti, or playing with a dog.

And sometimes we have nightmares about trauma. PTSD is at least partially about having learned about a version of reality that doesn’t fit our normality assumptions.

Additionally, cramming for exams doesn’t work. Not if you want to get a deep and lasting understanding. Instead, by engaging in deliberate and consistent practice, it is possible to master a skill over a long time.

And not just for advanced skills. Sleep is universal among animals. Even fish sleep. Which means that sleep must have a universal and simple purpose. And updating the error chemicals makes sense even if you’re a dumb fish.

But even if you’re not a dumb fish. If you a smart human. There’s a bunch of research on the hippocampus. It’s where the hippopotamumi go to college :P. It’s the area of brain responsible for memories. And it can get full of hippos during a day of learning.

A good night of sleep helps those hippos graduate and go around the brain, looking for a place to settle down. Not getting enough sleep is like failing all the hippos, so they remain in the hippocampus for longer. They can’t graduate, but they’ve filled the hippocampus. So no new hippos can arrive. No new memories can be made until the old hippos graduate, or drop out.

Getting a good sleep later on won’t fully reverse the trouble of lost sleep. The error chemicals will saturate and the neurons will stop adapting to the new information, or they’ll start forgetting the old information.

Again, this model is approximation at best, and reflects only a small amount of the things sleep is for. But I’ll use it for my reasoning base.

Why Net Neutrality Matters

Dan Stroot — Fri, 22 Dec 2017 02:00:00 GMT

Internet access is so fundamentally important that many people believe that Internet access should be a legal right. The British government has said that it will guarantee a legal right to high-speed internet for all homes and businesses in the U.K. by 2020 under the Digital Economy Act of 2017. Regardless of your belief in the Internet as a legal right, we should all be able to agree that access to the Internet promotes the digital economy and improves the lives of all citizens.

This post Why Net Neutrality Matters first appeared on Dan Stroot's Blog

A simple GUI spreadsheet in less than 100 lines of Python

Negative Feedback — Thu, 21 Dec 2017 16:14:32 GMT

Original challenge: Standard library only

I set myself a little challenge to write a simple spreadsheet-like program in Python in less than 100 lines, using only the standard library.

The code in this repository manages it in 100 non-blank lines, including comments. 88 if you ignore the comments. The full code is also at the end of this post. Note: I have also redone the code in PyQT5. This is discussed after the tkinter code.

This is what it the program looks like when you run it:

You can click on cells to edit their formulae and move around with the arrow keys. Formulae are evaluated once you press Return or navigate way from a cell. Every cell is evaluated as a Python expression. This means you don't do the Excel thing of having formulae start with =. Everything is a fomula. So the formula in A2 in the screenshot above is valid and will evaluate to 1 once focus moves away from the cell. The math module is imported by default, so you can use sin and cos and so on. I use a very cheap and nasty call of eval to make it all work, which you should never do for production code.

Other than using eval, I also play a little fast and loose with the way objects are allowed to touch each other in the code. I allow each cell to access all their siblings directly instead of going through an intermediary. I probably also break a couple of other OO rules along the way with the way the cells propagate their calculations when they are changed. But the whole code only took a couple of hours to write and it actually works.

And now, here is the full code:

#!/usr/bin/env python

import tkinter as tk
import math
import re
from collections import ChainMap

Nrows = 5
Ncols = 5

cellre = re.compile(r'\b[A-Z][0-9]\b')


def cellname(i, j):
    return f'{chr(ord("A")+j)}{i+1}'


class Cell():
    def __init__(self, i, j, siblings, parent):
        self.row = i
        self.col = j
        self.siblings = siblings
        self.name = cellname(i, j)
        self.formula = '0'
        self.value = 0
        # Dependencies - must be updated if this cell changes
        self.deps = set()
        # Requirements - values required for this cell to calculate
        self.reqs = set()

        self.var = tk.StringVar()
        entry = self.widget = tk.Entry(parent,
                                       textvariable=self.var,
                                       justify='right')
        entry.bind('<FocusIn>', self.edit)
        entry.bind('<FocusOut>', self.update)
        entry.bind('<Return>', self.update)
        entry.bind('<Up>', self.move(-1, 0))
        entry.bind('<Down>', self.move(+1, 0))
        entry.bind('<Left>', self.move(0, -1))
        entry.bind('<Right>', self.move(0, 1))

        self.var.set(self.value)

    def move(self, rowadvance, coladvance):
        targetrow = (self.row + rowadvance) % Nrows
        targetcol = (self.col + coladvance) % Ncols

        def focus(event):
            targetwidget = self.siblings[cellname(targetrow, targetcol)].widget
            targetwidget.focus()

        return focus

    def calculate(self):
        currentreqs = set(cellre.findall(self.formula))

        # Add this cell to the new requirement's dependents
        for r in currentreqs - self.reqs:
            self.siblings[r].deps.add(self.name)
        # Add remove this cell from dependents no longer referenced
        for r in self.reqs - currentreqs:
            self.siblings[r].deps.remove(self.name)

        # Look up the values of our required cells
        reqvalues = {r: self.siblings[r].value for r in currentreqs}
        # Build an environment with these values and basic math functions
        environment = ChainMap(math.__dict__, reqvalues)
        # Note that eval is DANGEROUS and should not be used in production
        self.value = eval(self.formula, {}, environment)

        self.reqs = currentreqs
        self.var.set(self.value)

    def propagate(self):
        for d in self.deps:
            self.siblings[d].calculate()
            self.siblings[d].propagate()

    def edit(self, event):
        self.var.set(self.formula)
        self.widget.select_range(0, tk.END)

    def update(self, event):
        self.formula = self.var.get()
        self.calculate()
        self.propagate()
        # If this was after pressing Return, keep showing the formula
        if hasattr(event, 'keysym') and event.keysym == "Return":
            self.var.set(self.formula)


class SpreadSheet(tk.Frame):
    def __init__(self, rows, cols, master=None):
        super().__init__(master)
        self.rows = rows
        self.cols = cols
        self.cells = {}

        self.pack()
        self.create_widgets()

    def create_widgets(self):
        # Frame for all the cells
        self.cellframe = tk.Frame(self)
        self.cellframe.pack(side='top')

        # Column labels
        blank = tk.Label(self.cellframe)
        blank.grid(row=0, column=0)
        for j in range(self.cols):
            label = tk.Label(self.cellframe, text=chr(ord('A')+j))
            label.grid(row=0, column=j+1)

        # Fill in the rows
        for i in range(self.rows):
            rowlabel = tk.Label(self.cellframe, text=str(i + 1))
            rowlabel.grid(row=1+i, column=0)
            for j in range(self.cols):
                cell = Cell(i, j, self.cells, self.cellframe)
                self.cells[cell.name] = cell
                cell.widget.grid(row=1+i, column=1+j)


root = tk.Tk()
app = SpreadSheet(Nrows, Ncols, master=root)
app.mainloop()

Part 2: PyQT5

I posted this blog to Reddit and one of the commenters mentioned that PyQT5 has a nice table widget which is more spreadsheet-like. So I re-did the spreadsheet in PyQT5. The code is functionally similar at the moment and also has approximately the same number of lines (87 non-comment code lines). The main differences are that I could use the QTableWidget to handle all the navigation. I had to do some interesting things with delegates to enable to editing mode I wanted, though, which cost me all the lines I saved from the navigation. I think my next step is going to be to extract the calculation logic into a common class which both of these can use - there is way too much duplication going on here right now!

So, this is what the PyQT5 one looks like. It's a little more compact, but otherwise very similar. The interaction is slightly more spreadsheet-like though, and you get automatic highlighting and selection of multiple cells for free. Very nice.

And here is the code. You will notice a great deal of similarity. The key to getting the edit mode thing to work properly is the piece of magic with SpreadSheetDelegate. I didn't think of that myself - I stole it from this PyQT5 example, which has a much nicer spreadsheet (you can change colours, fonts, print etc) with much worse formula logic.

#!/usr/bin/env python

import re
import sys
from collections import ChainMap
import math

from PyQt5.QtCore import Qt
from PyQt5.QtWidgets import (QApplication, QMainWindow, QTableWidget,
        QTableWidgetItem, QItemDelegate, QLineEdit)


cellre = re.compile(r'\b[A-Z][0-9]\b')


def cellname(i, j):
    return f'{chr(ord("A")+j)}{i+1}'


class SpreadSheetDelegate(QItemDelegate):
    def __init__(self, parent=None):
        super(SpreadSheetDelegate, self).__init__(parent)

    def createEditor(self, parent, styleOption, index):
        editor = QLineEdit(parent)
        editor.editingFinished.connect(self.commitAndCloseEditor)
        return editor

    def commitAndCloseEditor(self):
        editor = self.sender()
        self.commitData.emit(editor)
        self.closeEditor.emit(editor, QItemDelegate.NoHint)

    def setEditorData(self, editor, index):
        editor.setText(index.model().data(index, Qt.EditRole))

    def setModelData(self, editor, model, index):
        model.setData(index, editor.text())


class SpreadSheetItem(QTableWidgetItem):
    def __init__(self, siblings):
        super(SpreadSheetItem, self).__init__()
        self.siblings = siblings
        self.value = 0
        self.deps = set()
        self.reqs = set()

    def formula(self):
        return super().data(Qt.DisplayRole)

    def data(self, role):
        if role == Qt.EditRole:
            return self.formula()
        if role == Qt.DisplayRole:
            return self.display()

        return super(SpreadSheetItem, self).data(role)

    def calculate(self):
        formula = self.formula()

        if formula is None or formula == '':
            self.value = 0
            return

        currentreqs = set(cellre.findall(formula))

        name = cellname(self.row(), self.column())

        # Add this cell to the new requirement's dependents
        for r in currentreqs - self.reqs:
            self.siblings[r].deps.add(name)
        # Add remove this cell from dependents no longer referenced
        for r in self.reqs - currentreqs:
            self.siblings[r].deps.remove(name)

        # Look up the values of our required cells
        reqvalues = {r: self.siblings[r].value for r in currentreqs}
        # Build an environment with these values and basic math functions
        environment = ChainMap(math.__dict__, reqvalues)
        # Note that eval is DANGEROUS and should not be used in production
        self.value = eval(formula, {}, environment)

        self.reqs = currentreqs

    def propagate(self):
        for d in self.deps:
            self.siblings[d].calculate()
            self.siblings[d].propagate()

    def display(self):
        self.calculate()
        self.propagate()
        return str(self.value)


class SpreadSheet(QMainWindow):
    def __init__(self, rows, cols, parent=None):
        super(SpreadSheet, self).__init__(parent)

        self.rows = rows
        self.cols = cols

        self.cells = {}

        self.create_widgets()

    def create_widgets(self):
        table = self.table = QTableWidget(self.rows, self.cols, self)

        headers = [chr(ord('A') + j) for j in range(self.cols)]
        table.setHorizontalHeaderLabels(headers)

        table.setItemDelegate(SpreadSheetDelegate(self))

        for i in range(self.rows):
            for j in range(self.cols):
                cell = SpreadSheetItem(self.cells)
                self.cells[cellname(i, j)] = cell
                self.table.setItem(i, j, cell)

        self.setCentralWidget(table)


if __name__ == '__main__':
    app = QApplication(sys.argv)
    sheet = SpreadSheet(5, 5)
    sheet.resize(520, 200)
    sheet.show()
    sys.exit(app.exec_())

On Friction

Josh Thompson — Wed, 20 Dec 2017 15:00:00 GMT

warning. self-indulgeant diatribe coming. I generally try to avoid these, but it’s my website, and I can write what I want.

We’re rapidly approaching the end of the year, and I’ve got a few dozen ideas rolling around my head that I want to solidify my thoughts on.

One of the reasons I write these words on this little corner of the internet is simply to clarify my own thinking on a given topic. I try to not overthink them

This is known as rubber duck debugging and I’ve found it to be enormously effective.

I’ve found myself to be often giving the advice of “write more” to people. One of my favorite past-times is helping people get new/better jobs, and a common thread throughout that process is “write something” or “write more”. But in the same breath as I recommend writing as a means to a better job, I underscore that writing has endless value beyond the fairly transactional purpose of demonstrating competence.

This act (of pen to paper/fingers to keyboard) clarifies your thinking. On anything.

I’ve been resisting writing because much of my attention is unwillingly drawn to things that frustrate me. (Please reference: politics, marketing). The internet certainly doesn’t need another screed about the injustices of {unjust thing}, and I don’t plan to add another one.

I’m trying to choose to spend my attention entirely within the zone of things I can control. I’m trying to reduce my area of awareness down to that which I can control. When my awareness expands far beyond that which I can control, I spend my day angry and frustrated.

Me being angry and upset at things happening in a realm I cannot control (like politics) wears on those those around me. (Ask my wife about the last time I launched into a monologue about racist zoning policies.)

The low-friction path in life seems to be outrage, loss of attention, and consumption. (Ads, commercials, billboards, clickbait articles, etc.)

I don’t want to live that way, though, so I have to go to great effort to reduce the friction I face in doing more valuable things. If I’m confronted with a choice, in some ways, I’ve already lost. I don’t want to have to make decisions about important and good things. If they are really important and valuable, I’ll find a way to automate them, or fit them into a system.

(This is why regular automatic deductions of money from a checking account to an investment account are so powerful. If it’s important, automate it)

OK, I get it Josh. Why are you still going on and on about friction, then? Get to the point.

Ah, great point, provocative voice inside my own head. Good point.

I’m writing here about reducing friction… to get back in the groove of writing things I want to write about. Writing (and publishing) sometimes feels similar to replying to a long-neglected email in an inbox.

Image caption: “I would be honored, but I know I don’t belong in your network. The person you invited was someone who had not yet inflicted this two-year ordeal upon you. I’m no longer that person.”

I think my these words have helped shepherd my brain towards my point:

I’ve recently spent great effort getting other people to think certain things about me. I was actively job hunting (but not anymore!) AND I was running for city council in Golden. I didn’t realize how much self-censure I was doing throughout that process, so I only wrote a little tiny bit, and at that, it was mostly technical posts.

I’m employed, and am not on Golden’s city council, so I can go back to treating these posts as a simple indulgence.

Ahhh… that feels great.

rust dependency hell with features

Pete's Points — Wed, 20 Dec 2017 02:00:37 GMT

I've encountered my first bit of rust dependency hell. I'm trying to use a combination of nom v3.2.1 (a parsing combinator framework) plus nom_pem v1.0.3 (which uses nom to parse the PEM file format) plus der-parser v0.4.4 (which uses nom to parse DER data). This would almost work EXCEPT der-parser has the following line in it's Cargo.toml:

nom = {version = "^3.1", features = ["verbose-errors"]}

I haven't used feature toggling at all yet, and my initial reaction is OMG whose ideas was this and how did they think this would ever work?. In this case, the feature verbose-errors enables a totally different and incompatible Error API in nom, so it's very likely this only works if all of your nom-based deps agree on whether to use or not use this feature. One out of line and it's a hung jury: your code doesn't compile.

So at the moment my thought is: only top-level application projects should use cargo feature toggles and library crates should never use them. But as I read more maybe it'll get refined to something like "feature toggles are OK only within the bounds defined by the semver version, but if a feature toggle changes the semver API conventions, it's verboten."

Off to read a bit more on this topic.

Books of 2017

bastibe.de — Wed, 20 Dec 2017 02:00:00 GMT

In late 2016, I took a short ferry flight to a small island in the area, and rekindled my love for aviation. Shortly afterwards, I started training for a pilot's license, and reading about aviation. From a literary perspective, aviation exists in the perfect goldilocks time frame of being just old enough to be thoroughly romanticized, but young enough for first-hand reports and thorough documentation to be available. What is more, powered flight has provided human observers with an unprecedented view of our world and our struggles, and is often as philosophical as it is exhilarating.

Aviation Reading List

Out of a long list of fascinating books on aviation I have read over the last two years, my favorites are:

Fate is the Hunter by Ernest K. Gann, a gripping memoir of the early days of aviation. It has been only a little more than a hundred years since humans first took to the skies in Kitty Hawk in 1905, yet today aviation feels as mundane as horse-drawn carriages must have felt to the Wright Brothers. Gann lived through these early days, and tells his tales from a time when aviation was still young, dangerous, and perhaps more interesting. If you want to read more like this, Flight of Passage by Rinker Buck, and The Spirit of St. Louis by Charles Lindbergh are easy recommendations as well.

Carrying the Fire by Michael Collins, one of the few first-hand accounts of an Apollo astronaut's voyage to the Moon. Most astronauts have published books later on, but most had them ghost-written, and none are as visceral and engaging as Michael Collins' journey on Apollo 8. It is humbling that Apollo's achievements have not been surpassed, despite our much more advanced technology and science. Other accounts worth reading are The Last Man on the Moon by Gene Cernan on Apollo 17, and How Apollo Flew to the Moon by W. David Woods, for a more technical view.

Skyfaring by Mark Vanhoenacker is a more modern, and more philosophical account of how aviation has changed our perception of the world. If you yearn to fly like I do, then this book is a balm for the soul. The almost spiritual feeling of cutting your bonds with the ground is what this book is about, despite being written in today's un-romantic days of routine commercial airliners. I love it dearly. A more grounded account of aviation's history is Turbulent Skies by T. A. Heppenheimer, and maybe Slide Rule by Nevil Chute for some history on airships.

Fiction Reading List

But as much as I love aviation, my first love is still Science Fiction. We live in strange times of unprecedented prosperity, and yet we are strangely unsatisfied, as if the future didn't turn out to be the utopia it was meant to be. Or is this just a reflection of ourselves, how we do not live up to the future we built?

A Closed and Common Orbit by Becky Chambers describes a more distant future, when space travel is as mundane as airliners are today, and humanity is just one of several alien species. Yet, with all its technological marvels, we still yearn for meaning and love, regardless of what strange world we live in. A Closed and Common Orbit is the second book in the series, and the first book I read. I think I prefer it in this order.

The Laundry Files Series by Charles Stross is closer to home. Did you ever notice how computer programming is eerily similar to the arcane incantations we use to describe magic in fiction? Where is the difference between invoking a function that effects a robot, and invoking a magic spell that affects a demon? According to Charles Stross, this difference is really only playing with semantics, and we should be very careful with our incantations, lest the ghost in the machine really does have our demise in mind. These books have made me laugh out loud so many times, like when a weaponized PowerPoint turned people into zombies, or when a structured cabling project turned out to create an inadvertent summoning grid for an elder horror.

Thoughts on Crypto Currency Mania

Martin Rue — Wed, 20 Dec 2017 02:00:00 GMT

It's the end of 2017 and right now there's a huge amount of interest in buying [insert any crypto currency coin here]. Everyone wants to buy everything, and as a result we're very much in a bull market at the moment.

And Then There Were None

Krishna's blog — Wed, 20 Dec 2017 02:00:00 GMT

A review of Agatha Christie's 'And Then There Were None' highlighting the absence of her typical detectives and the prevalence of casual racism in the 1930s.

Simply encrypt or decrypt a string using Boto3 Python and AWS KMS

Ben E. C. Boyter — Mon, 18 Dec 2017 04:54:59 GMT

Another one of those things I need to look up every now and then. Below is a snippet of how to encrypt and decrypt a string using Python and KMS in AWS. The interesting thing is that you don’t need to supply the KMS key alias in the decryption portion. So long as whatever role or key you are using can access the key it should work. For the encryption you can either supply the full ARN of the key or the alias so long as you prefix it with alias/

Celia Celia

ho.dges.online — Mon, 18 Dec 2017 02:00:00 GMT

When I am sad and weary
When I think all hope has gone
When I walk along High Holborn
I think of you with nothing on.

Adrian Mitchell

Learnings from building my own home server

Nemo's Home — Mon, 18 Dec 2017 02:00:00 GMT

Learnings

I forgot to do this on the last blog post, so here is the list:

archlinux has official packages for intel-microcode-updates.
wireguard is almost there. I’m running openvpn for now, waiting for the stable release.
While traefik is great, I’m concerned about the security model it has for connecting to Docker (uses the docker unix socket over a docker mounted volume, which gives it root access on the host). Scary stuff.
Docker Labels are a great signalling mechanism. Update: After seeing multiple bugs with how traefik uses docker labels, they have limited use-cases but work great in those. Don’t try to over-architect them for all your metadata.
Terraform still needs a lot of work on their docker provider. A lot of updates destroy containers, which should be applied without needing a destroy.
I can’t proxy gitea’s SSH authentication easily, since traefik doesn’t support TCP proxying yet.
The docker_volume resource in terraform is useless, since it doesn’t give you any control over the volume location on the host. (This might be a docker limitation.)
The upload block inside a docker_container resource is a great idea. Lets you push configuration straight inside a container. This is how I push configuration straight inside the traefik container for eg:
```
upload {
  content = "${file("${path.module}/conf/traefik.toml")}"
  file    = "/etc/traefik/traefik.toml"
}
```

Advice

This section is if you’re venturing into a docker-heavy terraform setup:

Use traefik. Will save you a lot of pain with proxying requests.
Repeat the ports section for every IP you want to listen on. CIDRs don’t work.

If you want to run the container on boot, you want the following:

 restart = "unless-stopped"
 destroy_grace_seconds = 10
 must_run = true

If you have a single docker_registry_image resource in your state, you can’t run terraform without internet access.
Breaking your docker module into images.tf, volumes.tf, and data.tf (for registry_images) works quite well.
Memory limits on docker containers can be too contrained. Keep an eye on logs to see if anything is getting killed.
Setup Docker TLS auth first. I tried proxying Docker over apache with basic auth, but it didn’t work out well.

MongoDB with forceful server restarts

Since my server gets a forceful restart every few days due to power-cuts (I’m still working on a backup power supply), I faced some issues with MongoDB being unable to recover cleanly. The lockfile would indicate a ungraceful shutdown, and it would require manual repairs, which sometimes failed.

As a weird-hacky-fix, since most of the errors were from the MongoDB WiredTiger engine itself, I hypothesized that switching to a more robust engine might save me from these manual repairs. I switched to MongoRocks, and while it has stopped the issue with repairs, the wiki stil doesn’t like it, and I’m facing this issue: https://github.com/Requarks/wiki/issues/313

However, I don’t have to repair the DB manually, which is a win.

SSHD on specific Interface

My proxy server has the following

eth0 139.59.22.234

And an associated Anchor IP for static IP usecases via Digital Ocean. (10.47.0.5, doesn’t show up in ifconfig).

I wanted to run the following setup:

eth0:22 -> sshd
Anchor-IP:22 -> simpleproxy -> gitea:ssh

where gitea is the git server hosting git.captnemo.in. This way:

I could SSH to the proxy server over 22
And directly SSH to the Gitea server over 22 using a different IP address.

Unfortunately, sshd doesn’t allow you to listen on a specific interface, and since the eth0 IP is non-static I can’t rely on it.

As a result, I’ve resorted to just using 2 separate ports:

22 -> simpleproxy -> gitea:ssh 222 -> sshd

There are some hacky ways around this by creating a new service that boots SSHD after network connectivity, but I thought this was much more stable.

Wiki.js public pages

I’m using wiki.js setup at https://wiki.bb8.fun. A specific requirement I had was public pages, so that I could give links to people for specific resources that could be browser without a login.

However, I wanted the default to be authenticated, and only certain pages to be public. The config for this was surprisingly simple:

YAML config

You need to ensure that defaultReadAccess is false:

auth:
  defaultReadAccess: false
  local:
    enabled: true

Guest Access

The following configuration is set for the guest user:

Now any pages created under the /public directory are now browseable by anyone.

Here is a sample page: https://wiki.bb8.fun/public/nebula

Docker CA Cert Authentication

I wrote a script that goes with the docker TLS guide to help you setup TLS authentication

OpenVPN default gateway client side configuration

I’m running a OpenVPN configuration on my proxy server. Howver, I don’t always want to use my VPN as the default route, only when I’m in an untrusted network. I still however, want to be able to connect to the VPN and use it to connect to other clients.

The solution is two-fold:

Server Side

Make sure you do not have the following in your OpenVPN server.conf:

push "redirect-gateway def1 bypass-dhcp"

Client Side

I created 2 copies of the VPN configuration files. Both of the them have identical config, except for this one line:

redirect-gateway def1

If I connect to the VPN config using this configuration, all my traffic is forwarded over the VPN. If you’re using Arch Linux, this is as simple as creating 2 config files:

/etc/openvpn/client/one.conf
/etc/openvpn/client/two.conf

And running systemctl start openvpn-client@one. I’ve enabled my non-defaut-route VPN service, so it automatically connects to on boot.

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Some thoughts about running a wiki

utf9k — Sun, 17 Dec 2017 13:00:00 GMT

I ran a Wiki for a little while and here's what I learned

Your first robot: A beginner's guide to ROS and Ubuntu Core [1/5]

kyrofa's blog — Sun, 17 Dec 2017 02:00:00 GMT

UPDATE: I’m leaving this series up for historical purposes, but please note that I no longer recommend Ubuntu Core or snaps for use in robotics. Some time ago I created a blog/video series that walked the reader through creating a prototype using the Robot Operating System (ROS) and taking it to production using Ubuntu Core. However, that series was intended more for robotics professionals; it assumed quite a bit of ROS knowledge, and required some costly equipment (the robot was about $1k).

rust converting bytes chars and strings

Pete's Points — Sat, 16 Dec 2017 20:57:12 GMT

I found converting from many representations of essentially the same data really inconsistent and hard to memorize in rust. I was doing a lot of work that frequently switched between text and binary and couldn't find a good quick reference cheat sheet, so I sat down to pair with Jared McDonald at Recurse Center and code one up.

Whether you've got some bytes, chars, Strings, strs, arrays, or vecs, we'll get you from A to B!

use std::str;

fn main() {
    // This is "hello world" as an array of bytes.
    // You can also start from a byte string b"hello world" and debug print that to get the
    // utf8 encoded decimal values
    println!("hello byte string: {:?}", b"hello world");

    // OK so let's say you have an array of u8s
    let array_of_u8 = [104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100];

    // [u8] to String (lossy)
    // Any invalid bytes that are not utf8 will be replaced with
    // the unicode replacement character '\u{FFFD}'
    // You get a Cow (Clone on Write) not exactly a String
    let string_utf8_lossy = String::from_utf8_lossy(&array_of_u8);
    println!("string_utf8_lossy: {}", string_utf8_lossy);

    // [u8] to String (result)
    // The non-lossy version needs a vec not an array
    let mut vec_of_u8 = vec![];
    vec_of_u8.extend_from_slice(&array_of_u8);
    let string_utf8_result = String::from_utf8(vec_of_u8).unwrap();
    println!("string_utf8_result: {}", string_utf8_result);

    // [u8] to str (result)
    let str_utf8_result = str::from_utf8(&array_of_u8).unwrap();
    println!("str_utf8_result: {}", str_utf8_result);

    // [u8] to str (lossy)
    // There is no str::from_utf8_lossy. Have to use String::from_utf8_lossy

    // [u8] to Vec<char>
    let vec_of_chars: Vec<char> = array_of_u8.iter().map(|byte| *byte as char).collect();
    println!("vec_of_chars: {:?}", vec_of_chars);

    // Vec<char> to Vec<u8>
    let vec_of_u8s: Vec<u8> = vec_of_chars.iter().map(|c| *c as u8).collect();
    println!("vec_of_u8s: {:?}", vec_of_u8s);

    // Vec<char> to String
    let mut string_of_collected_chars: String = vec_of_chars.iter().collect();
    println!("string_of_collected_chars: {}", string_of_collected_chars);

    // Now we have a mutable String. We can push chars
    string_of_collected_chars.push('!');

    // and we can push a str
    string_of_collected_chars.push_str("!!");

    // String to str
    let str_slice = &string_of_collected_chars[..5];
    println!("str_slice: {}", &str_slice);

    // String to [u8]
    let array_of_u8_from_string = string_of_collected_chars.as_bytes();
    println!("array_of_u8_from_string: {:?}", array_of_u8_from_string);

    // String to Vec<char>
    let vec_of_chars_to_string: Vec<char> = string_of_collected_chars.chars().collect();
    println!("vec_of_chars: {:?}", vec_of_chars_to_string);

    // String from several Strings
    let concat_strings = vec!["abc".to_string(), "def".to_string()].concat();
    println!("concat_strings: {}", concat_strings);
    let joined_strings = vec!["abc".to_string(), "def".to_string()].join("---");
    println!("joined_strings: {}", joined_strings);
}

Magit amend commit author

xenodium.com @alvaro — Sat, 16 Dec 2017 02:00:00 GMT

Rarely use it, but handy. Use Magit to amend git commit author.

Rebase interactively (r, i).
Move point to commit to ammend.
Execute command (x).

git commit --amend --author="name <email>"

Commit (c, c).

Get scrollbar width in JavaScript

Muffin Man — Fri, 15 Dec 2017 02:00:00 GMT

Another one-liner I love, that returns body scrollbar width. If scrollbar is not shown it will return zero (including mobile devices).

function getScrollbarWidth() {
  return window.innerWidth - document.documentElement.clientWidth;
}

Click on the button to try it yourself:

Browser support #

Tested on:

Windows: IE9, IE10, IE11, Edge
MacOS: Safari, Firefox, Chrome, Opera
iOS: Safari, Chrome
Android: Chrome

Homebrew install from cache

xenodium.com @alvaro — Wed, 13 Dec 2017 02:00:00 GMT

Came across a 404 while installing graphviz-2.40.1.tar.gz via homebrew. If you can find the package elsewhere, copy over to homebrew's cache directory.

brew --cache

Recurse Center 23: SQL window functions

Pete's Points — Wed, 13 Dec 2017 01:01:30 GMT

Today was the first day where I focused primarily on my SQL project, which was my second priority overall. I did most of the pgexercises on aggregation. There were quite a few of them and by the end they were really challenging. I learned bit about common table expressions which seem awesome and windowing functions which seem great for the business rollup type reports which are otherwise frustrating in regular SQL. I met with the study group today and we lamented the lack of good material that actually teaches this stuff clearly. We are looking at 3-4 books to potentially select.

I managed to get into code flow this afternoon on tealeaves and I expanded my arsenal of test key files to include the special -o openssh-key-v1 format for the rsa, dsa, and ecdsa algorithms. ed25519 keys always use this format, but the other algorithms can be saved in 2 formats, this one or ASN.1.

The good news is that some of my nom code to parse this format in the public key code can be re-used for private key code as well. I'm excited to refactor to that tomorrow.

How to resolve 'X defined in resolvers, but not in schema' with babel-plugin-inline-import

Max Rozen — Tue, 12 Dec 2017 20:52:00 GMT

Showing how to resolve the dreaded 'Query.X defined in resolvers, but not in schema' issue when your X is **definitely** defined in both

Recurse Center 22: tealeaves refactored

Pete's Points — Tue, 12 Dec 2017 03:17:07 GMT

Today I finished a big push of work on tealeaves and got the code fully cleaned up and merged to master and all the tests passing again. When I introduced nom_pem last week things destabilized and it took a while to shake all that out. I'm about ready to start adding some linter style rules like making sure the filesystem permissions on your private keys are restrictive.

I also worked a bit more on pgexercises this afternoon but they really are just exercises with nothing to actually teach you the SQL you need, so I switched back to some Khan Academy tutorials I had worked on about 2 years ago.

Few things I learned as an engineer at Twitter

D13V — Tue, 12 Dec 2017 02:00:00 GMT

A month ago I left a job as a software engineer and data scientist at Twitter.

I was there 4.4 years and I learned a great deal out of it. I not only got significantly better at the job-description part of my job - contributing to code and data analysis - but I also picked up a few bits about what makes larger efforts and collaboration successful.

So… without pretense of completeness or correctness, and without further ado, here is a sample of things I learned.

It’s better to get a relatively smaller win now, than a larger one

later.

Shipping a successful feature brings more resources and people towards your project and will actually accelerate the larger, future win. Work is like a Hydra. Kill one head “successfully” and three more will grow out of it. You’ll have more work to do. When a project is done successfully, that usually opens up other possibilities, on top if it. It also brings momentum to that work stream. Exciting people like to join exciting projects, and successful projects are more exciting.

I was on the Timelines team when we were moving from reverse-chron timeline to a ranked one. This project was successful and helped our team grow, and attracted awesome people to join the team. Shipping a win made the team better and able to ship further wins faster. If we were to wait until we’ve done some of the follow up work, it would have taken a lot longer to get it done without the new team members. MVPs rule.

Working together is not a zero-sum game.

We mutually help each other, but it’s counter-productive to keep score of how much each person has contributed. Keep helping and supporting others and it will pay back, in unpredictable ways.

Most times I’ve taken time to help with other’s brainstorm, or work through some issue with them, or pay extra effort in code reviews and design reviews, I’ve felt an increase in comradeship from the other person, and I’ve received more unsolicited help and advice from them in the future. Not only this helped us do better work, it made it more of a pleasure. Teamwork matters. Can’t do it alone. Teams will beat geniuses about 99.99% of the time in the short term, and 100% in the long term.  What we’ve been able to accomplish as a team is a lot more than what each of us can do by themselves.

In a typical project there are just too many details to get right for a single person. Even for the smallest code changes, we do code reviews and having a different person with a detached perspective on the problem take a look often finds ways to improve it. This is even more apparent when there’s a larger project which requires different roles. No matter how smart I may think I am, it is pretty clear that some aspects I’d never think about, even if given unlimited time.

Collaboration amplifies. Or if you like TPS reports, you might want to call it synergy :)

Doing a lot of code reviews for others the fastest ways to learn

It helps to keep up with what all the teammates are doing, builds up karma, and is the best way to discover the code patterns to use. It also gets a lot easier to do with time - economies of scale.

As a relatively junior engineer to begin with, there were a lot of systems and patterns that I didn’t know and understand. Doing a lot of code reviews for my more senior teammates showed me the way they write code, the considerations they make and helped me learn about the different systems we were using as a team. There were times during which I tried to review almost all code by teammates. This really helped me keep up with what each of them is doing, and gave me enough context so I can provide more adequate and specific suggestions. It also made it easier to write my code which interacted, or built upon their code, which increased my productivity - at least measured quantitatively. I believe there was a qualitative increase as well, as code reviews enhance collaboration, and not merely cooperation.

If you are a junior engineer, or new to a team or company - doing a lot of code reviews is the hands down best way to get better. It’s also my favorite example of collaboration done right.

Make BigData small.

When analyzing large amounts of data, it’s important to aggregate and summarize as much as possible. As much as we don’t like to admit it, we as humans are not good at pattern recognition. We can recognize straight lines, and some high level features, but that’s about it.

Often, I had to analyze terabytes of data. I had Scalding, Hadoop and MapReduce as tools, but still, I had to find ways to summarize that data to only a handfull of dimensions and metrics and create plots for those, that make sense for humans. For data-mining it was a two-stage process. First, create a small dimensions and metrics dataset that would fit in memory or in database. Then, it’s a lot easier to mine this smaller dataset for insights, than having to go back to Hadoop. Each iteration and new idea takes seconds, not hours to investigate.

I really like the way Tableau approaches it, by explicitly calling out “Dimensions” and “Metrics” for the small data table. Dimensions are how you can slice and dice the data, and metrics are statistics for each possible slice. To anyone new to data mining, this would be my first advice.

Visualization matters.

How we communicate our findings is as important, and as hard to get right as deriving the findings in first place. The most common bias, myself included, is to overestimate the importance of the findings, and underestimate the importance of the communication.

I took a one-day course by Edward Tufte on how to present quantitative data and it was well worth it. It taught me how it is important to show comparisons, causality, proper documentation, and not to underestimate the communicative power of text labels. Text labels rule. I’ve learned the hard way that it takes just as much work to communicate a finding effectively, than to do the work for the finding. Sometimes even many times more.

If something it isn’t tested, it WILL break.

If it’s tested, it might still break. This so far has been true 100% of the time.

Unit tests have saved my butt many times. I don’t know if anyone can write bug-free code, but almost every time I’ve written unit tests I’ve caught bugs and code design issues. I learned to associate writing tests with finding bugs so much that I do it all the time, and avoid giving a “Shipit!” to other people’s code if they don’t have tests.

One vivid example is from the day of launching the ranked timeline. We had to make some last day changes the day before, which came from high level. It was also impossible to push the launch date further until we’ve done more testing, due to press commitments. We had added new code, and added a whole bunch of unit tests for it. But we forgot to unit test one case and that did break during the launch. I felt embarrassed. Exceptions in production, and delaying the launch by a few hours. At time though, we were not panicked and calmly proceeded to write some more unit tests to reproduce the issue, as to be certain when it’s fixed. Our manager read the situation correctly and tried to give us the calm and isolation so we can focus on the issue. He offered to bring us coffee.

Clean data models are worth the investment.

It’s worth the time to make sure data model and structs are right for the case, and have clear semantics. If you are clobbering and reusing fields that will cost you more in the future.

Me and my teammates were working with a result structure representing tweets. A field in the structure was used for multiple purposes, which were most likely mutually exclusive at the time the data structure was designed. Or maybe it was designed with only one case in mind and the other one sneaked in there, in the same field, as it is a PITA to update data format once live. Anyhow, all was rainbow and sunshines, except not. There was a bunch of work required to parse out the different cases into a less ambiguous format, and everything broke when we were migrating, rebuilding one of the services and a different person had to re-do the parsing in a different language, and the quirks of the fields were only documented in some old JIRA tickets.

We spent much more time working around a clumsy data format that would have been necessary to do it right the first time. We also had tricky to fix production bugs that remained undiagnosed for a while. This kind of tech debt comes with a steep interest rate and unpleasant payment schedule.

Names stick.

My professor in college used to say that naming is very powerful. He’s so right. So when coming up with a name for some feature or behavior, find a good one. Chances are, that everyone will start using the name and it will be impossible to change later.

I’ve had my fun of observing that process in vivo, when I started using the term “headless reply” for one of my project, and in the following months I started hearing it in other contexts and other projects. Perhaps I didn’t coin that term, but sure enough I hadn’t heard it before. I’ve seen other suboptimal terms start to get used so much that it just becomes impossible to start calling them something else.

Except in very rare cases, names don’t change. Once they’re in people’s minds you can’t press Fn+F6 and do a “Refactor -> Rename” on them in IntelliJ.

Every metric will become a vanity metric.

Monthly active users, User active minutes, etc. These usually are great to optimize in some cases, but will stop being what correlates well with user satisfaction and product health in the medium-long term.

Of course, it’s a lot easier to see when other teams are over optimizing a given metric. It’s a lot harder to be self-critical. Over the long term, the product built is the best fit to the targeted metrics. Oh. Key. Ars. If these metrics don’t evolve, the product will overfit.

Estimate time generously. And then pad it some more.

Work will always take longer than you thought, even when you adjust for that. Nobody will complain if you get it done sooner, instead. There’s a very asymmetrical cost reward to any engineer to whether their estimate is too long, or too short.

When I’ve been asked to estimate how long it will take me to do a certain work, I’ve always had the inclination to imagine the steps of the work and imagine how long they’ll take. But this doesn’t account for meetings, off-sites, missed edge cases, time spent helping other team-mates, and other delays beyond my control. And also…

After the first 90% of each project, it’s time for the other 90%.

For example, if building product, no matter how hard you think about it there will always be edge cases which users are doing, that are broken and need further work.

We’ve always, always found issues when dogfooding new features, and many times we’ve had to iterated in A/B tests. Every rare, or weird, use pattern will happen in very small percentage of the time. But that’s still too many times when the product is used by hundreds of millions of users every day. The question I learned to ask myself is not whether any unexpected behaviour would happen, the question is what will the code do when it inevitably happens.

Volunteer for more projects than you can do.

It’s better to drop a less important, and less exciting project than to be stuck on it as the only project.

I think this tends to work because more important projects are usually more exciting. It becomes possible to communicate the benefits of the more important and exciting project to the manager and convince them that you should focus more on it, and not the less important one.

2017 Notre Dame CSE 20133/20232 Final Project Presentations

CS Crunch — Mon, 11 Dec 2017 21:45:09 GMT

Here are a few videos that were created by the students of the 2017 CSE 20133/20232 programming class, to demonstrate their end-of-semester projects. This class consisted of EE and other non-CS engineering students, and taught programming using C and C++.

Videos included here are done with the permission of the students, and I will add additional videos to the playlist if and when more students decide to make their project videos public.

I am very proud of what the students accomplished, and with the variety of ideas and inginuity of the students. They all did great!

How to style react-select with styled-components or emotion

Max Rozen — Mon, 11 Dec 2017 20:52:00 GMT

Styling react-select can be annoying. Here's a quick guide on how to do it with popular CSS-in-JS libraries.

Should I buy Bitcoin?

Krishna's blog — Mon, 11 Dec 2017 02:00:00 GMT

This article explores the question of whether or not to invest in Bitcoin, examining its value as a currency and asset, as well as its potential for future growth and stability.

The Mathematics of 2048: Counting States by Exhaustive Enumeration

jdlm.info — Sun, 10 Dec 2017 02:00:00 GMT

Updates

2017-12-11 This post was discussed on Hacker News.

This post is the third in a series. Next: Optimal Play with Markov Decision Processes.

So far in this series on the mathematics of 2048, we’ve seen that it takes at least 938.8 moves on average to win, and we’ve obtained some rough estimates on the number of possible states using combinatorics.

In this post, we will try to refine those estimates by simply counting every reachable state by brute force enumeration. There are many states, so this will require some computer science as well as mathematics. With efficient processing and storage of states, we’ll see that it is possible to enumerate all reachable states for 2x2 and 3x3 boards.

Enumeration of all states for the full game on a 4x4 board remains an open problem. I ran the code on an OVH HG-120 instance with 32 cores at 3.1GHz and 120GB RAM for one month, during which it enumerated 1.3 trillion states, which is a respectable half a million states per second, but that turns out to be nowhere near enough. At present, the best we we’ll be able to do on the 4x4 board is to enumerate all states for the game played up to the 64 tile.

Overall, the results show that the combinatorial estimates from the last post were substantial overestimates, as suspected. However, the state space for the full game is still very large. If a large nation state or tech company decided that 2048 was their top priority, which may not be the craziest thing that’s happened this year, I think they could probably finish the job.

The (research quality) code behind this article is open source, mainly in ruby and C++.

Counting States

Here a state captures a complete configuration of the board by specifying the value of the tile, if any, in each of the board’s cells. Our overall goal is to count all of the states that can actually occur in the game and no more. In the previous post, the estimates from (very basic) combinatorics counted many states that can’t actually occur in the game. By enumerating states systematically from each of the possible start states, we ensure that the states we count are actually reachable in play.

We also have some freedom in choosing which states are interesting enough to count. Because the game ends when we obtain a 2048 tile, we won’t care about where that tile is or what else is on the board, so we can condense all of the states with a 2048 tile into a special “win” state. Similarly, if we lose, we won’t care exactly how we lost; we can condense all of losing states into a special “lose” state.

Canonicalization and Symmetry

Many other non-interesting states that we’d like to avoid counting arise from the fact that some states are trivially related to each other by rotation or reflection. For example, in the game on a 2x2 board, the states

and

are just mirror images — they are reflections through the vertical axis. If we swiped left in the first state, it would be essentially equivalent to swiping right in the second. We can therefore reduce the number of states we have to worry about by treating these states as equivalent. In general, the number of such equivalent states is the number of elements in the dihedral group for the square, $D_4$, which is 8. For the first state above, these eight states are:

In this diagram, which is called a cycle graph, ⤴ denotes a rotation counterclockwise by 90° and ↔ denotes a reflection about the vertical axis. The three states at the top are obtained by one or more rotations; for example, ⤴² means two rotations of 90°, which add up to a rotation of 180°. The four states at the bottom are obtained by zero or more rotations followed by a reflection.

When the tiles in a state are arranged in a symmetrical pattern, the number of equivalent states may be less than 8, because the symmetry means that some of the states in the above diagram will be identical. For example, the state has only four, because it is symmetric along the diagonal. However, particularly on 3x3 and 4x4 boards, states with symmetric arrangements of tiles are not so common, so overall we can expect to reduce the number of states we need to count by roughly a factor of 8 by choosing only one of these equivalent states as the canonical state.

States as Numbers

Which of the equivalent states should we choose as the canonical state? To answer this question, it will be helpful to think about states as numbers. We’ll also see that this has significant computational benefits in the appendices.

To write a state as a number, we can start in the top left and read the cells by rows; for each cell, we write a $0$ digit if the cell is empty, and the digit $i$ if the cell contains the $2^i$ tile. For example, on a 2x2 board, the state would be written as the number $2310$, because $4 = 2^2$, $8 = 2^3$, $2 = 2^1$, and the last cell is empty.

Since we’re only interested in numbers up to 2048, which is $2^{11}$, we could in principle write any state as a number in base 12 (rather than base 10, as we are accustomed to). However, because computers run on binary numbers, it will be more convenient to think of each state as a number in base 16 — that is, hexadecimal. The state above would therefore be more properly written as 0x2310 for computer scientists or $2310_{16}$ for mathematicians.

To find the canonical state, we simply try all eight possible rotations and reflections, convert the resulting states to numbers, and then pick the smallest one. For our example state, , the candidates and their corresponding numbers are:

State	Number
	`0x2310`
	`0x3201`
	`0x1023`
	`0x2130`
	`0x0312`
	`0x1203`
	`0x0132`
	`0x3021`

The state has the smallest number, 0x0132, so that is the canonical state for . In general, this system for choosing canonical states tends to put tiles with larger values toward the bottom right corner.

Enumeration

Now we’re ready to start enumerating states — it’s all computation from now on. The idea is to generate all possible (canonical) start states, then for each one try each possible move, then for each move generate all possible (canonical) successor states. In code (ruby) form, the basic algorithm for enumerating states looks like this:

def enumerate(board_size, max_exponent)
  # Open all of the possible canonicalized start states.
  opened = find_canonicalized_start_states(board_size)
  closed = Set[]

  while opened.any?
    # Treat opened as a stack, so this is a depth-first search.
    state = opened.pop

    # If we've already processed the state, or if this is
    # a win or lose state, there's nothing more to do for it.
    next if closed.member?(state)
    next if state.win?(max_exponent) || state.lose?

    # Process the state: open all of its possible canonicalized successors.
    [:left, :right, :up, :down].each do |direction|
      state.move(direction).random_successors.each do |successor|
        opened.push(successor.canonicalize)
      end
    end

    closed.add(state)
  end

  closed
end

The code for generating start states is here, and the code for the rest of the State methods, such as random_successors and canonicalize, is here.

If we run this code for the game on the 2x2 board played up to the 32 tile, which we’ve previously established is the highest tile reachable on the 2x2 board, we get 57 states (click to enlarge):

In this diagram, each edge is a possible state-successor pair. For example, from the state , we could swipe left or right, which would result in a 4 tile, and then the game adds a 2 or 4 tile at random, which leads to , , or after canonicalization; or we could swipe up or down, which would leave the two 2 tiles unmerged and lead to or after canonicalization.

Not shown are the special ‘lose’ and ‘win’ states, so in total we have 59 states for the 2x2 game to the 32 tile. This compares favorably to the estimate of 529 states from the (simple) combinatorics arguments in the previous blog post. We’ve saved about one order of magnitude!

When we try to run this ruby code on the 3x3 or 4x4 boards, however, we quickly hit two problems: it’s very slow, and it runs out of memory for the closed set. I’ve included three appendices with the details of how to speed up the calculations and manage the large amounts of data involved, but first let’s see some results.

Results

The numbers of states for the various games we’ve looked at in this series of blog posts are:

Board Size	Maximum Tile	Combinatorics Bound	Actual
2x2	32	529	59
3x3	1024	786,513,819	25,179,014
4x4	64	2,816,814,934,817	40,652,843,435
4x4	2048	44,096,167,159,459,777	$\gg$ 1.3 trillion

Just like the 32 tile is the highest reachable tile on the 2x2 board, the 1024 tile is the highest on the 3x3 board. Exhaustive enumeration of states shows that the 3x3 game contains about 25 million states, which is a factor of 31 lower than the rough ‘Combinatorics Bound’ from the previous post. For the game on the 4x4 board to the 64 tile, which is the largest game on the 4x4 board that I was able to completely enumerate, the factor is even larger, at 69. As expected, the (very basic) combinatorics bounds were quite loose, because they count many states that can’t occur in the game or are trivially related to each other.

For the 3x3 game to the 1024 tile, there are too many states to draw a diagram like the one for the 2x2 game above. However, we can gain some insight into that 25 million figure by counting states in groups by (1) the sum of the tiles on the board and (2) the value of the maximum tile on the board. The sum of the tiles on the board increases by either 2 or 4 with each move ¹, so the game generally progresses from left to right on this graph:

Early in the game, when the sum of the tiles is small, the number of states grows fairly smoothly and linearly with the sum of tiles. However, later in the game when the board fills up, there are sharp drops around where the sum of tiles reaches a larger power of two, for example at around sums 128 and 256. These drops indicate that the 3x3 game is tightly constrained by the small size of the board — there are not many ways to survive past these drops without merging most of the tiles together into a larger one.

It’s also notable that the same structure seems to repeat each time a larger maximum tile is reached (that is, each time the shade of blue in the plot gets darker). The 64, 128, 256 and 512 max tile curves each have a similar slope at the start and a ‘step’ at about 26,000 states per tile sum. In terms of gameplay, this repetition reflects the fact that once you merge most of the tiles together to get the next largest one, the board is mostly empty again, except for the newly merged tile, so the game sort of ‘resets’ at that point.

We might hope that the game on the 4x4 board would also show some of these characteristics, but at least up to tile sum 380, this is apparently not the case. After running the enumeration for one month and counting over 1.3 trillion states, the results to date for the full game of 2048 look like:

We see smooth and uninterrupted growth in the total number of states. Whereas the game on the 3x3 board topped out at about 80 thousand states per tile sum, the game on the 4x4 board shows no sign of slowing down at 27 billion states per tile sum ².

We can still see the rise and fall in the number of states with each maximum tile value, and it becomes clearer if we unstack these counts and plot them on a logarithmic scale:

The top line in black shows the total number of states, summing over all the maximum tile values, which are again shown in shades of blue. Each blue arc shows the growth and later decay in the number of states with a given maximum tile value, as the game progresses. The bending down of the total (black line) shows that the growth in the number of states per tile sum tapers off as the game progresses, but the numbers are already quite large.

Conclusion

We’ve improved our estimates for the number of states in the game of 2048 on the 2x2 and 3x3 boards by one to two orders of magnitude, compared to the previous (basic) combinatorial estimates. The number of states for the game on the 4x4 board remains too large to enumerate in full, but we have at least managed to completely enumerate the states for the 4x4 game to the 64 tile, and we’ve made an attempt at enumerating the states for the full game.

The explicit enumeration of states counts only states that can be reached in actual game play, but even with that restriction there are still some surprising states included in the count. For example, in the last figure above, the arc that shows the number of states with at most a 32 tile does not stop until tile sum 348. There are four states with tile sum 348 and no 64 tile, one of which I chose for the cover image of this post: ³

It contains seven 32 tiles and seven 16 tiles arranged in a nice striped pattern that makes it very difficult to merge any of them. I find it quite surprising that it’s possible to play to such a state without losing, and indeed it seems quite improbable, especially if one is ‘playing well’.

In the next post, we’ll explore what it means to ‘play well’ in a rigorous way by modeling the game of 2048 as a Markov Decision Process and finding an optimal policy for the 2x2 and 3x3 games — that is, we will find a strategy for playing those game that we can show mathematically to be at least as good as any other possible strategy.

Appendix A: Bit Bashing for Efficiency

Profiling the ruby code above revealed that most of the time was being spent on state manipulation — for example, moving tiles, counting available tiles, and reflecting or rotating states to find canonical states. Let’s see how we can speed it up.

The hexadecimal numerical representation for states has a convenient property: for the 4x4 board, we need to store 16 numbers, where each number takes 4 bits, for a total of 64 bits. Now that we all use 64-bit computers, this is an auspicious number: the whole board state can fit into a single 64-bit (8-byte) machine word. For example, the 4x4 state (which is a winning state, because it has a 2048 tile)

would be written 0x022b013020100000 in hexadecimal, or more clearly with the addition of line breaks to make a 4x4 grid:

With the state represented as a 64-bit integer, we can also implement many common manipulations on states very efficiently using bit mask and shift operations, often without loops ⁴. For example, the C++ function to reflect a 4x4 board state horizontally looks like:

uint64_t reflect_horizontally(uint64_t state) {
  uint64_t c1, c2, c3, c4;
  c1 = state & 0xF000F000F000F000ULL;
  c2 = state & 0x0F000F000F000F00ULL;
  c3 = state & 0x00F000F000F000F0ULL;
  c4 = state & 0x000F000F000F000FULL;
  return (c1 >> 12) | (c2 >> 4) | (c3 << 4) | (c4 << 12);
}

While initially quite opaque, all this is doing is shuffling bits around. The role of first bit mask, 0xF000F000F000F000ULL, becomes clearer if we omit the 0x and ULL, which just tell the compiler that this is non-negative 64-bit integer in hexadecimal, and again add line breaks to make a 4x4 grid:

F000
F000
F000
F000

The binary representation of the hexadecimal digit F is 1111, so the effect of this bit mask is to make c1 contain only the values in the first column of the board and zero bits everywhere else. The bit shift c1 >> 12 at the end of the function moves the first column 12 bits, which is to say three 4-bit cells, to the right, which makes it the last column. Similarly, c2 selects the second column, and c2 >> 4 moves it one cell to the right, and so on with the third and fourth columns to reverse the order of the columns.

Some functions take a bit more work to decipher. If you’re in the mood for a puzzle, here’s a function that counts the number of cells available (value zero) in a 4x4 state (you can find my explanation in comments here):

int cells_available(uint64_t state) {
  state |= (state >> 2);
  state |= (state >> 1);
  state &= 0x1111111111111111ULL;
  uint64_t count = ((state * 0x1111111111111111ULL) >> 60);
  if (count == 0 && state != 0) return 0;
  return 16 - count;
}

Profiling (with perf) showed that such tricks made a big difference — compared to the obvious implementations with arrays and loops, these functions require very few CPU instructions and contain few or no branches, which allows the CPU to keep its instruction pipelines full.

Appendix B: Layers and MapReduce for Parallelism

The next challenge is that we can’t keep the whole closed set in memory. To break up the state space into manageably sized pieces, we can use the following property of the game: The sum of the tiles on the board increases by either 2 or 4 with each move. This property holds because merging two tiles does not change the sum of the tiles on the board, and the game then adds either a 2 or a 4 tile. ¹

This property is useful here because it means that we can organize the states into layers according to the sum of their tiles. We can therefore generate the whole state space by working through a single layer at a time, rather than having to deal with the whole state space at once.

To parallelize the work within each layer, we can use the MapReduce concept made famous by Google. The ‘map’ step here is to take one complete layer of states with sum $s$ and break it up into pieces; then, for each piece in parallel, generate all of the successor states, which will have either sum $s + 2$ or $s + 4$. The ‘reduce’ step is to merge all of the pieces for the layer with sum $s + 2$ together into a complete layer, removing any duplicates. The pieces with sum $s + 4$ are retained until the next layer, in which states have sum $s + 2$, is processed, at which point they will be included in the merge. This may be easier to see in an animation (made with d3):

In the map step for each piece, it is feasible to maintain the set of successor states in memory. To make the merge in the reduce step efficient, we want the map step to output a list of states for each piece in order by their state numbers so we can merge the pieces together in linear time. Here Google again comes to our aid: they have released a handy in-memory B-tree implementation that plays well with the C++ standard template library. B-trees are most commonly found in relational database systems, where they are often used to maintain indexes on columns. They keep data in order and provide logarithmic time lookup and also logarithmic time insertion — much better than logarithmic plus linear time insertion into a sorted list — with relatively little memory overhead.

To break the state space up into even smaller pieces, which reduces the amount of work we need to do in each merge step, we can exploit an another property of the game: the maximum tile value on the board must either stay the same or double with each move. This property holds because the maximum tile value never decreases, and when it does increase, it can only increase as the result of merging two tiles — that is, even if you have for example four 16 tiles in a row and you merge them, after one move the result is two 32 tiles, not one 64 tile.

Together with the property above, this means that from a list of states in which all states have tile sum $s$ and maximum tile value $k$, the generated successors will all fall into one of four pieces:

Piece	1	2	3	4
Tile Sum	$s+2$	$s+2$	$s+4$	$s+4$
Max Tile Value	$k$	$2k$	$k$	$2k$

There is a bit more bookkeeping to keep track of which pieces need to be merged together at each step, but it is basically the same idea. Here are the 57 states (excluding the ‘win’ and ‘lose’ states) for the 2x2 game to the 32 tile again, this time with the states grouped by tile sum and maximum tile value, here written $s / k$:

For example, in the leftmost piece, both states have tile sum 4 and maximum tile value 2. The transitions from that piece are to pieces with tile sum either 6 or 8 and maximum tile value 2 or 4. Compared to the original diagram of these states without grouping, it’s also easier to see that you always transition to a state with a tile sum that is either 2 or 4 larger, like in the first post in this series.

Appendix C: Encoding and Compression

When working with billions or trillions of states, and each state takes 8 bytes, even fitting them all on disk is not trivial (or at least not cheap). To reduce the storage space required, and also the amount of input/output required, we can exploit the fact that the states are stored as sorted lists of integers — rather than storing each integer in full, we can store the differences between successive integers. These differences will generally be smaller than the integers themselves, so it is usually possible to store the differences in a smaller number of bytes. Using a variable-width encoding scheme to store the differences ensures that we use only the number of bytes we need. ⁵

For example, the list of states for the 4x4 game to the 2048 tile with tile sum 380 and maximum tile value 128 contains 21,705,361,721 states ². At 8 bytes per state, that would be roughly 161GiB. However, with variable-width encoding, it takes only 35GiB — a compression factor of 4.6 — or about 1.7 bytes per state.

For longer term storage and transport over Internet, I also tried several compression programs on the variable-width encoded data. To evaluate the different programs, I used a smaller list of states with tile sum 260 and maximum tile value 32; it contained 30,954,422 states, which is about 1/700th the number in the 380/128 layer mentioned above, and it weighed in at 64MiB after variable-width encoding. For each of the programs, and for each of their supported compression levels, I measured the elapsed time for compression and the resulting compressed size using this smaller list of states.

Three programs emerged on the resulting Pareto frontier: Facebook’s Zstandard, Google’s Brotli and 7-zip. After scaling size and time up by a factor of 700 to estimate performance on the larger 380/128 layer, the frontier looks like:

Here closer to the origin is better; we see Zstandard performing best for relatively fast and light compression, and 7zip performing best for relatively slow and heavy compression. From the graph, we can see that Zstandard at compression level 11 is fairly close to the origin, and it turned out to minimize the particular linear cost function that I used. The results on the smaller layer suggested that Zstandard level 11 would reduce the 380/128 layer from 35GiB to about 10GiB. In fact, it did much better: it reduced it from 35GiB to 3.8GiB, for another factor of 9.2, or about 1.5 bits per state. I think that’s quite remarkable.

Thanks to Hope Thomas for reviewing drafts of this article.

If you’ve read this far, perhaps you should follow me on twitter, or even apply to work at Overleaf. :)

Footnotes

This property was called called Property 3 in the previous post, and we also used it anonymously in the first post. It is a very useful property. ↩ ↩²
These figures come from the one month build on the full 4x4 game, which used an additional technique for reducing the size of the state space: for each state, it looked one move ahead to see whether the next move was a definite win or a definite loss; if it was, the state was collapsed to a special ‘one move to win’ or ‘one move to lose’ state. This technique was not very effective, however, because states that are close to a loss tend to have few very successors anyway, and because win states only occur much later in the game, and the enumeration process never reached a win state. I therefore haven’t included it in the main body of the article, but the code is here. The state counts mentioned in this article for the partial enumeration of states with the 4x4 board to the 2048 tile are slightly reduced compared to what they would be without this technique, but only slightly. ↩ ↩²
In case you are wondering, I faked that screenshot; I didn’t actually play the game to that state! ↩
The bit bashing techniques used here are based mainly on 2048-ai by Robert Xiao and 2048-c by Kuang-che Wu, with some help from Bit Twiddling Hacks by Sean Eron Anderson. The original motivation for 2048-ai and 2048-c was to allow real time search for 2048 AI bots. ↩
Variable width encodings are most commonly encountered in the UTF-8 character encoding for unicode, which you are using as you read this webpage. They are also sometimes used for storing integer primary key indexes in some databases, which also face the problem of efficiently storing large lists of sorted integers. The implementation used here is libvbyte from Christoph Rupp, based on work by Daniel Lemire. ↩

Hour of Code 2017

Mike Kasberg's Blog — Sat, 09 Dec 2017 22:30:00 GMT

The Hour of Code is an annual event to encourage Computer Science education during Computer Science Education Week. I participated this year by volunteering at an elementary school, where I spent about an hour introducing the students to computer programming through Scratch. I think Scratch is a great way to introduce people (both young and old) to programming because it is simple enough that beginners can get started very quickly, but also powerful enough to create good-looking games (like this Flappy Bird Clone).

Asteroid Field (A Game)

I designed my own lesson for the Hour of Code 2017. I wanted my lesson to be centered around designing a simple game. The game needed to be fun to play, but still exceptionally simple - simple enough that 5th grade students could program a more-or-less functional copy in roughly an hour. I designed an Asteroid Field Game.

Now, although the game linked above is pretty simple (roughly 60 lines of Scratch code), I needed to simplify it even more to ensure that:

the program was not too complex for 5th-graders, and
we could program it in about an hour.

To that end, I created an even more simple version of the same game. This one is only 20 lines of Scratch code, and some of those (such as the score) are optional. I felt that programming a game like this was within the reach of 5th grade students, and I planned the lesson in chunks so that we would finish something that resembles the Asteroid Field game, even if we didn’t finish completely.

Overall, the lesson was a huge success. The class loved it, and I think many of them came away from the lesson with a better understanding of what programming is than they had going in. Perhaps more importantly, I think that for many kids this opened a doorway to a new world. I hope that for at least a few students I sparked an interest in computer science that they will pursue for many years to come.

If you’re interested in seeing the lesson, you can check out the lesson plan and the slides. The Scratch game and source code, along with some of my other Scratch projects, can be found on my Scratch profile.

NIPS 2017

Rachit Singh — Sat, 09 Dec 2017 10:47:37 GMT

I'm starting this blog to share research ideas that I have, and some solutions to problems I find along the way. I've been helped immensely by other people's blogs in the past, and want to do the same. Also it'll give me a chance to communicate the way I approach problems, and hopefully people will give me alternative perspectives either by email (rachitsingh@outlook.com) or in the comments, once I figure out how that works.

I'm going to steadily post my thoughts on NIPS 2017 (summary: lots of exposure to new ideas, very glad I went); but as a start I'm going to explain how we went: Jeffrey Ling and I submitted a short paper on Indian Buffet Processes to the NIPS workshop on Advances in Approximate Bayesian Inference (AABI), and we were accepted. See the repo for runnable code and some more information. I'll blog about the IBP as well soon, possibly after we have our ArXiv preprint up (soon!).

Recurse Center 21: nom and combinators

Pete's Points — Sat, 09 Dec 2017 03:37:35 GMT

Last night I asked RC about whether nom or untrusted seemed like better crate choices for parsing mixed text and binary file formats. This morning I woke up super early for no reason and was ready to go so I watched RC alumnus Stefanie Schirmer's talk on parser combinators.

Then I was excited to dig in so after I commuted in to RC early I searched on crates for "nom pem" since the PEM format was what I needed to parse next. I found an existing crate nom_pem that looked promising so I gave it a whirl. I was able to get it to parse a test file but I couldn't get at the parsed data. I was getting a "that field is private" error which was odd. How was I supposed to do anything with this parser if the parser output is private? So I looked again at the crate and saw "Last updated 2 hours ago". That was not the most recent update, that was the only update ever. Someone had published a nom PEM parsing crate to crates.io while I was watching the combinator video!

So I was download number 5 (probably a bunch of bots beat me to it) for that crate and the first github star. Within a few hours I had forked it and sent in 3 small PRs to fix trivial but showstopper issues, and by the afternoon they all had been merged. Go open source!

So I dove into learning nom which makes heavy use of macros which are hard. I spent basically all day starting with the most trivial example of like "can I tell that 'a' is the letter 'a'?" and eventually was at the point where I could parse the openssh public key format. Then I started to refactor that code into tealeaves but there's still more to do.

Today I also heard one of the RC mini-batchers talk about some cool clojure spec libraries and had a nice dinner out instead of take-out which was fun.

Today's unix command line tip is that there's a golang project by google called der-ascii that can translate both ways between the binary DER format used in some crypto stuff and an editable ascii version. Works like this:

grep -v - files/ssh-rsa-1024-a-private-key.pem| base64 -D | der2ascii
SEQUENCE {
  INTEGER { 0 }
  INTEGER { `00a3475d589021587893902869f524c6493856c3ad46f28eeee793a22da33dfd0c690f67a99b3acef3dcc877f4ad98fb91feb6c39d25a28871ad73a03bc7143a559de8eb22e3ca69a72ca6059a7ff5526878de96d29d82f2227be0308efb1e9f5fbf256c11f35e7ee8213961980b208666463f48c115a884aabc1ba7705f3293a7` }
  INTEGER { 65537 }
  INTEGER { `3bfa5915d14c0e7dac67061176159e2903630bda38f79cfdb15f8ff187c79b8ed580ed29667641d35ec4dd1baa314a282512e9e46e10b86259bee19b53d3e06140b6b2bfaaf9434b92569e5f67cd7d90deeeb0fb609fcdacb83518da6e7f39dc5c39bfb726362458fae49c03127338799e1104183cb4015a71b5f1cea91dd621` }
  INTEGER { `00d07dcd4433eebddf5ca318643c4b6066bc37ebd01e9b9468e8b97a0d9542b15ad236247a43f35b9a54acc7398ef9b0283658ca103e1b3e2eed4bf9f0b2abf4bf` }
  INTEGER { `00c87c1d8693dc1d624e30e6627fd79246da11aac0f95467193226a348cfb0ec2c605516cb73a472d558b54752666445046bb4e02057227260208f744832a29319` }
  INTEGER { `0b2b622ed0356f18a346dd8ca92a449cdc6286909cc32afd3fd287f66853ad5ab73e4d4ffb89f3135e8bef1467537b1b7c65df55656e62337365099bda8699d3` }
  INTEGER { `394f31697ceed8ff76f68784f4a27cd0001a38c37d02618e5445b33b67135e0fb961d9684320692b0f769272bd8e4024695d850bf99c8131755d4c922ed74651` }
  INTEGER { `54dfe6859a0ca3df5d85bbc031b6da4428af4328629157045495aedc9f3056b2538ebf513fbcc4bfe79d4a35a7cc3afedf79b7f856997cb3783dfad052a6ca06` }
}

Pretty handy! I couldn't find it in homebrew so I had to compile it by hand and manually install it, but it works.

Deep living

D13V — Sat, 09 Dec 2017 02:00:00 GMT

Imagine doing a lot of fun things, a lot of travel, and a lot of pleasure. Imagine laying down on the beach, sipping a Margarita. The salty rim of the glass. A feeling of being refreshed and cheered. Getting up from a beach bed and laying down on a massage table, relaxing and drifting off to sleep. Later on, a tasty dinner with the partner, dancing the calories away in the night club, and having sex.

We imagine ourselves in the above situation. And we think - “ah, this is good life”. Well, it’s good, but it is not deep. We are indulging our primal senses. Our neuron receptors. Our input layer.

That’s called vacation. That’s not living. Vacation is necessary for our low-level pleasure senses to recover from the numbness that surrounds them in the everyday grind. We work hard to make money to give ourselves pleasure, and as we work, we start to forget what pleasure is. That’s why we need vacation. That’s why we need weekends. To remind us. To strengthen and realign our understanding of pleasure. To develop our input layer.

And on top of the pleasure layer, we have our reason. Reason is the next layer in the deep neural network in our brain. Reason builds on top of pleasure and pain. We rely on it in everyday tasks, such as deciding what to eat, and how warm to dress. It lets us figure out how to maximize pleasure. It helps us deduce that we can increase the thermostat when we feel cold, or eat a pizza when we are hungry. Without reason, we’ll be worse off than most animals. Especially those damn raccoons. They’ll take our pizza.

But them raccoons get smarter and smarter, and we need to keep updating our reasoning if want pizza. That’s why we develop strategies as a layer on top of reason. We get a dog and teach it recognize racoon smell, so we’re alert next time they come to steal our pizza.

On top of strategies, we have principles and values. We value pizza, and we value not being cheated. Our principle is that we don’t let raccoons steal our pizza. We’re getting into emotion territory now. Our principles and values derive from our emotions. Our emotions dictate whether we feel sympathy or annoyance at those less strong and less smart, and whether we feel admiration or fear towards the stronger and smarter.

Emotions are at the opposite end of the chain of neural layers, and every swing of our emotions results in a swing in the layers after it - strategy, reasons, and pleasure. Emotions are another input layer. From top.

As we get exposed to physical and emotional pleasures and pain, the whole neural network shakes like a battle rope. But this rope of neurons is our identity. This is how we make decisions. We take the whole rope into account when deciding to eat a pizza, get a dog, or shoot a raccoon.

The more this rope shakes, the more confused we are, and less consistent in our decisions. When we change our strategy and principles, we get dumber, and eviler. It doesn’t matter that we may think of ourselves as good. Our actions define us, not our intentions.

When we act with our emotions and senses, we become losers. Because we let them attack and disrupt our decisions and actions and make them inconsistent with the principles and strategies we have. So these principles and strategies no longer work.

We are our decisions and actions, not our thoughts. Not our intentions. Our actions.

And here finally I can deliver my thesis. To live life fully, truly, madly, deeply, like winners, we need to pin down our reasoning, strategies and principles, we need to cut them out of granite so that emotions and senses can’t sway or bend them easily, and our decisions derive from a strong fixed point.

It’s not an easy thing to do. It’s not pleasure. Pleasure is input layer, and fear is the mind killer. We need to insulate our decision making from pleasure and emotions. Not to fully separate, because then we become robots, but to at least prevent minor swings in pleasure and emotion to shake us out of our path in life.

Living deeply, it’s a struggle. Struggle to stay focused and our paths, struggle to keep eating healthy, struggle to treat others kindly even when they don’t treat us kindly. Struggle to not be lazy and procrastinate. Struggle to not act stupid. To keep refining and improving our principles and strategies in a conscious way.

Living deeply doesn’t feel easy. Because we’re making hard decisions, putting effort in when we don’t feel like it, and abstaining from bad habits when they are tempting. It’s sacrificing our time in order to help others, or ignoring others in order to focus on the top priorities. The choice might go either way depending on our principles and the situation, but it will be hard, from emotional, and from comfort viewpoint.

Living deeply doesn’t feel exciting. It requires hard work which can often get boring, repetitive, and ungrateful. It involves hours and days of preparation and planning for a handful of interesting minutes of high impact.

Living deeply is mentally stressful. It requires us to ask the hard questions, to be humble and to examine our values. It requires us to have difficult conversations, and to do a lot of uncomfortable preventive actions.

And yet, living deeply feels like living. It feels like a breath of fresh cold air. Because we know we are on our own path.

Handling advanced product imports in WooCommerce

Daniel Immke — Sat, 09 Dec 2017 02:00:00 GMT

Learn how to handle advanced things like core fees when importing products into WooCommerce.

CLI for alerts via Slack

blog.harterrt.com — Fri, 08 Dec 2017 10:00:00 GMT

I finally got a chance to scratch an itch today.

Problem

When working with bigger ETL jobs, I frequently run into jobs that take hours to run. I usually either step away from the computer or work on something less important while the job runs. I don't have a good …

Recurse Center 20: Parsing is hard. Let's go shopping.

Pete's Points — Fri, 08 Dec 2017 05:13:00 GMT

So I've found that the pem crate I've been using doesn't parse headers, which are present in encrypted dsa private key files, and the mailparse crate doesn't parse properly with unix newlines, which seems to be what openssh generates even though in theory the RFCs involved require carriage return newline AFAIK.

So I spent most of the day writing my own PEM parser that handles headers and thinking about the things rust programmers think about:

Can this be done without allocating any extra memory?
Have I failed at life if I have to copy this 10-byte string before freeing it 4 nanoseconds later?
Can I represent all of my data as a giant enum so rust models it as a single u64 bitfield or something?

You know, that kind of stuff.

But after another shopping spree on crates dot io I'm leaning heavily toward learning to use the nom crate to parse this. There's even an open github issue asking for an example DER private key parser, which is what I've been working on. However, the untrusted crate also looks promising. Tomorrow I'll see how quickly I can get something easy done with nom.

I also gave a talk on Serverless today and pointed out my hexagonal-lambda sample project.

Here's the unix tip of the day. If you have some utf8 characters encoded as hexadecimal you can convert to regular utf8 like this:

echo 65636473612d736861322d6e69737470333834 | xxd -r -p

Which prints ecdsa-sha2-nistp384

Global Component Registration

Dylan Paulus' Blog — Fri, 08 Dec 2017 02:00:00 GMT

Introduction

Often, I find myself looking through open-source projects to see how they solve issues. Not bugs, but problems in code that cause repetition or excessive boilerplate. Modals have given me problems, specifically when having to create a specialized modal for different workflows. They are awkward. Modals, in the DOM, live outside of the scope of the current workflow. And, part of their state usually lives outside of the modal itself (hide/show). I came accross an interesting solution to the modal problem while looking through Insomnia's code. This is a different post than what I normally do. I'll just be posting code as an example, and filling in the explaination through comments. Hopefully this will work as a reference, and starting point to play around with the idea.

Code

// ModalHandler.js
const modals = {};

/*
 * This function takes in a component, and then adds a
 * reference to it through a friendly named key->value.
 */
export function registerModal(component) {
  if (!component) {
    return;
  }

  modals[component.name] = component;
}

/*
 * show() takes the component we want to show, and some additional
 * arguments. We'll then call the 'show' function on the component.
 * This allows us to 'activate' a component anywhere, but invert
 * control to the relative component.
 */
export function show(component, ...args) {
  const elem = modals[component.name];
  if (elem.show && typeof elem.show === "function") {
    return elem.show(...args);
  }

  console.warn(`show() function is missing from ${component.name}.`);
}

// App.jsx
import React, { Component } from "react";
import { registerModal } from "./ModalHandler";

class Base extends Component {
  render() {
    return (
      <div>
        <MyApp />
        {/*
          This component is pretty regular except for this next bit.
          We'll pass in registerModal() to the component's ref we'll
          show later on
        */}
        <Modal ref={registerModal} />
      </div>
    );
  }
}

import React, { Component } from "react";
import { show } from "./ModalHandler";
import Modal from "./Modal";

class MyApp extends Component {
  showModal = () => {
    // Will call the .show() function
    // on the Modal component
    show(Modal);
  };

  render() {
    return <button onClick={this.showModal}>Show Modal</button>;
  }
}

// Modal.jsx
import React, { Component } from "react";

class Modal extends Component {
  state = {
    isOpen: false,
  };

  // This is the show() function that'll be invoked
  // when this component is passed to ModalHandler.js' show() function
  show = () => {
    this.setState({ isOpen: true });
  };

  render() {
    if (!this.state.isOpen) {
      return null;
    }

    return (
      <div>
        Modal!
        <button onClick={() => this.setState({ isOpen: false })}>Close</button>
      </div>
    );
  }
}

Afrocubism

Blogs on Michail Rybakov — Thu, 07 Dec 2017 16:07:46 GMT

In the winter of 2013 I went to Guinea with a few friends. Guinea is one of my favourite destinations, since Silvia, the mother of my friend Christoph, is working there at a medical laboratory. And having nice people on the ground makes travelling to a foreign country much more enjoyable.

Guinea is an interesting country to visit. It is rich in natural resources, which leads to wealth being concentrated in the hands of a few powerful families. It also means that tourism is non-existent and the only white people present are professionals working at mining companies and NGOs. You can be the only white person in a densely populated area, and you get eyed on the streets - it feels like everything you do would be talked about later, so you end up being not only a guest but also an ambassador for your culture.

Experiments are releases

blog.harterrt.com — Thu, 07 Dec 2017 10:00:00 GMT

Mission Control was a major 2017 initiative for the Firefox Data team. The goal is to provide release managers with near-real-time release-health metrics minutes after going public. Will has a great write up here if you want to read more.

The key here is that the data has to be …

Recurse Center 19: ASN.1 Tagged Context

Pete's Points — Thu, 07 Dec 2017 01:22:37 GMT

Coding on tealeaves today was mostly pulling interesting bits out of ssh keys. Misc things discovered

openssh will only generate 1024 bit dsa keys, which are not secure, but longer key sizes are supported by other software
ecdsa keys the "bits" field represents either 256, 384, or 521 bit size curves and those are the only valid sizes
The specific curves are encoded in ASN.1 as object identifiers that appear to be assigned well-known OIDs which tell you which curve it is
At least one stackoverflow answerer thinks ecdsa is backdoored by the NSA

I also was struggling with tooling, which I guess is the name of the game in lower-level systems programming, and I ended up writing a little rust unix utility to hard wrap files at 32 characters after stripping whitespace so I can pipe hex dumps into it and get nice column widths. That was actually fairly fast and easy to do so for simple things it feels like once you're over the learning curve rust might actually deliver on its promise of scripting language ease with no-compromise performance at runtime.

Later I paired with one of the RC mini-batchers who taught me about fold -w 32 which is a standard unix tool that can do most of the same thing.

Desirable features of experimentation tools

blog.harterrt.com — Wed, 06 Dec 2017 10:00:00 GMT

Introduction

At Mozilla, we're quickly climbing up our Data Science Hierarchy of Needs ¹. I think the next big step for our data team is to make experimentation feel natural. There are a few components to this (e.g. training or culture) but improving the tooling is going to be …

5000 concepts for Europe? The first one is...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 06 Dec 2017 09:00:00 GMT

In the first 12 hours from its actual public announcement, my proposal of a “5000 concepts for Europe” book yeld some unexpected (for me, at least) results.

Reading Simple Haskell

λm.me — Wed, 06 Dec 2017 02:00:00 GMT

I wrote a presentation-style resource that tries to present a small part of Haskell and hopefully make it a little more accessible and a little less scary.

Simple JavaScript API wrapper

Muffin Man — Wed, 06 Dec 2017 02:00:00 GMT

For handling API calls I have a small snippet I'm copying from project to project. I decided to clean it up, make more generic and share it. It is intended to be a starting point, so you might want to customize it to your custom needs.

What it does? #

It is a simple wrapper around native fetch.If you need a polyfill isomorphic-fetch is a great one.

For successful requests it will parse the response and return it.
When HTTP error occursIt detects errors based on request's HTTP status. it will throw a custom error with status code, error message and response (parsed if it is a JSON).
If request never gets resolved, same custom error will be thrown but response will be set to null and status code to REQUEST_FAILED.

Please note that function will return a Promise so you need to handle how it resolves.

Code #

It is written as a native ES2015 module, which means you may need to transpile it depending on your browser support policy.

// ------------------------------------------------------ //
// Simple JavaScript API wrapper
// https://muffinman.io/blog/simple-javascript-api-wrapper
// ------------------------------------------------------ //

// For demo purposes I'm using this awesome Star Wars API
const API_URL = 'https://swapi.tech/api';

// Custom API error to throw
function ApiError(message, data, status) {
  let response = null;
  let isObject = false;

  // We are trying to parse response
  try {
    response = JSON.parse(data);
    isObject = true;
  } catch (e) {
    response = data;
  }

  this.response = response;
  this.message = message;
  this.status = status;
  this.toString = function () {
    return `${ this.message }\nResponse:\n${ isObject ? JSON.stringify(this.response, null, 2) : this.response }`;
  };
}

// API wrapper function
const fetchResource = (path, userOptions = {}) => {
  // Define default options
  const defaultOptions = {};
  // Define default headers
  const defaultHeaders = {};

  const options = {
    // Merge options
    ...defaultOptions,
    ...userOptions,
    // Merge headers
    headers: {
      ...defaultHeaders,
      ...userOptions.headers,
    },
  };

  // Build Url
  const url = `${ API_URL }/${ path }`;

  // Detect is we are uploading a file
  const isFile = options.body instanceof File;

  // Stringify JSON data
  // If body is not a file
  if (options.body && typeof options.body === 'object' && !isFile) {
    options.body = JSON.stringify(options.body);
  }

  // Variable which will be used for storing response
  let response = null;

  return fetch(url, options)
    .then(responseObject => {
      // Saving response for later use in lower scopes
      response = responseObject;

      // HTTP unauthorized
      if (response.status === 401) {
        // Handle unauthorized requests
        // Maybe redirect to login page?
      }

      // Check for error HTTP error codes
      if (response.status < 200 || response.status >= 300) {
        // Get response as text
        return response.text();
      }

      // Get response as json
      return response.json();
    })
    // "parsedResponse" will be either text or javascript object depending if
    // "response.text()" or "response.json()" got called in the upper scope
    .then(parsedResponse => {
      // Check for HTTP error codes
      if (response.status < 200 || response.status >= 300) {
        // Throw error
        throw parsedResponse;
      }

      // Request succeeded
      return parsedResponse;
    })
    .catch(error => {
      // Throw custom API error
      // If response exists it means HTTP error occured
      if (response) {
        throw new ApiError(`Request failed with status ${ response.status }.`, error, response.status);
      } else {
        throw new ApiError(error.toString(), null, 'REQUEST_FAILED');
      }
    });
};

export default fetchResource;

Usage #

fetchResource accepts two arguments, mandatory URL and optional options to be passed to the fetch request. If you wrap each API request in a function with it's specific options it will be easier to maintain.

Few examples:

// Simple get request
function getUsers() {
  return fetchResource('users');
}

// Post request with payload
function signIn(username, password) {
  return fetchResource('signin', {
    method: 'POST',
    body: {
      username,
      password,
    },
  });
}

// Put request, with file data and custom headers
function uploadAvatar(userId, file) {
  return fetchResource(`users/${ userId }/avatar/`, {
    method: 'PUT',
    body: file,
    headers: {
      'Content-Type': file.type,
    },
  });
}

Then you call newly created API functions, they will return a Promise, so you need to define then (and catch for errors).

getUsers()
  .then(userData => {
    // Do something with the "data"
  })
  .catch(error => {
    // Handle error
    // error.message (error text)
    // error.status (HTTTP status or 'REQUEST_FAILED')
    // error.response (text, object or null)
  })

Demo #

I've cooked a small demo to show it in practice, in which I used free Star Wars API.

Feel free to play with it on CodePen.

Recurse Center 18: SQL joins

Pete's Points — Wed, 06 Dec 2017 01:37:08 GMT

Today I mostly worked on some pgexercises. I learned some good things and did some fairly complex joins but I still feel like I need a lot more practice with joins before I'll actually get that clear light bulb of understanding. Our study group met and reviewed some stuff and made plans. I was hoping to also do some web security capture the flag exercises AND make progress on tealeaves but didn't even come close to getting to either of those today.

My general feeling is as per my focusaurus username, I do much better working on just 1 thing at a time and the past few days I've tried to sprinkle in some web security and SQL stuff and my main rust/tealeaves project basically came to a halt. I think mostly this is OK as I can work heads down alone on tealeaves when I get home but the other topics benefit a lot from working with my peers here. We'll see how I feel in another 2 days or so. I was casually hoping to have tealeaves be mostly working for modern ssh key formats by Thursday but it might not quite get there including CLI usability stuff.

How to use docker to run local databases

Pete's Points — Wed, 06 Dec 2017 01:25:41 GMT

Here's how I run use docker to run databases for local development. I think my approach is the simplest I've seen and if something is unnecessary I skip it. You don't need data containers or port mappings, so I don't use them. The goal here is to make it very easy to run databases while developing applications or doing DB development work. It's basically the same for macOS and linux but I'm currently on macOS so that's where my instructions come from.

Goals

Same process works mostly the same for many databases (postgresql, mysql, mongodb, redis, etc)
Data persists as expected

Can start/stop containers, reboot host without losing data

When we want to wipe data and start fresh, that's easy too
Easy to transfer files into the container and out of the container
Easy to get a shell inside the container to run tools there (psql etc)
Easy to purge when done

My Approach

Use a single docker-compose yaml file because 1 file is the easiest number of files to have
Each database gets 2 volume mounts
- 1 for ad hoc file transfer
  - This is always '~/docker-volumes/host:/host'
- 1 for the database data itself
  - The left/host path follows the pattern ~/docker-volumes/container-name
  - the right/container path needs to match the path the container stores data by default. You'll need to look at the container's docs to find the correct path to use. I've provided correct values for some popular databases below.
  - Look at hub.docker.com, search for your DB container by name, then search in the page for -v and you'll usually find it
Databases run on their default port bound to loopback
- Simplest thing. Least surprise.
- Also allows DB management GUIs to work easily.
If DB users are required, set their password to "password"
- This is local stuff that does not need much security

The compose file

# Run with:
# docker-compose -f ~/projects/dotfiles/docker-compose-local-dev.yml up -d
version: '2'
services:
  # https://hub.docker.com/_/postgres/
  postgres:
    container_name: 'postgres'
    ports: ['127.0.0.1:5432:5432']
    volumes: [
      # for ad-hoc transfers in/out of container
      '~/docker-volumes/host:/host',
      # for DB data itself
      '~/docker-volumes/postgres:/var/lib/postgresql/data'
      ]
    image: 'postgres'
    environment:
      POSTGRES_PASSWORD: 'password'
      PGDATA: '/var/lib/postgresql/data/pgdata'
  mysql:
    container_name: 'mysql'
    ports: ['127.0.0.1:3306:3306']
    volumes: [
      # for ad-hoc transfers in/out of container
      '~/docker-volumes/host:/host',
      # for DB data itself
      '~/docker-volumes/mysql:/var/lib/mysql'
      ]
    image: 'mysql'
    environment:
      MYSQL_ROOT_PASSWORD: 'password'
  mongo:
    container_name: 'mongo'
    ports: ['127.0.0.1:27017:27017']
    volumes: [
      # for ad-hoc transfers in/out of container
      '~/docker-volumes/host:/host',
      # for DB data itself
      '~/docker-volumes/mongo:/data/db'
      ]
    image: 'mongo'
  dynamodb:
   container_name: 'dynamodb'
   ports: ['127.0.0.1:8000:8000']
   volumes: [
        # for ad-hoc transfers in/out of container
        '~/docker-volumes/host:/host',
        # for DB data itself
        '~/docker-volumes/dynamodb:/var/dynamodb_local'
        ]
   image: 'peopleperhour/dynamodb'
  redis:
    container_name: 'redis'
    ports: ['127.0.0.1:6379:6379']
    volumes: [
      # for ad-hoc transfers in/out of container
      '~/docker-volumes/host:/host',
      # for DB data itself
      '~/docker-volumes/redis:/data'
      ]
    image: 'redis'

Check the docker compose file in my github dotfiles repo for latest version.

How to do basic operations

Commands are run from the host (macOS) terminal unless explicitly marked as run from within a container.

Start everything: docker-compose -f local-dev.yml up -d
Stop everything: docker-compose -f local-dev.yml down
Start some DBs but not all: docker-compose -f local-dev.yml up -d postgres mongo
Get a shell in the DB container: docker exec -i -t postgres bash
- From here you can run the DB repl in the container like psql, mongo, etc
Copy a file into the container: cp some-file.sql ~/docker-volumes/host
Read a file from within the DB container:
- Get a shell: docker exec -i -t postgres bash
- Read the file (container shell): more /host/some-file.sql
Copy a file from the container out to the host
- Get a shell: docker exec -i -t postgres bash
- Copy (container shell): cp /path/to/some/file /host
Fully wipe the data and start over for one database
- docker-compose -f local-dev.yml stop
- rm -rf ~/docker-volumes/postgres && mkdir ~/docker-volumes/postgres
- docker-compose -f local-dev.yml up -d postgres

Bonus docker shell aliases

These may come in handy:

alias dcf="docker-compose -f ~/projects/dotfiles/docker-compose-local-dev.yml"
alias dcup="docker-compose -f ~/projects/dotfiles/docker-compose-local-dev.yml up -d"
alias deit="docker exec --interactive --tty"
alias doco="docker-compose"
alias dpa="docker ps -a"
alias drit='docker run --rm --interactive --tty'

My First Program

Mike Kasberg's Blog — Tue, 05 Dec 2017 04:00:00 GMT

On the StackOverflow Podcast, they always ask their guests on the show how they got introduced to programming. Whenever I listen to that segment of the podcast, I always think back to my first experiences with programming, and how I built on those to get where I am today. I like the podcast segment a lot, and I think it’s fun to share these stories. So, in honor of Computer Science Education Week, I’m going to write a blog post about some of my first programming experiences.

Learn to Program BASIC

I always wanted to learn to program. From a pretty young age, I grasped the idea that there were people out there who could make computers do whatever they wanted them to. To me, this idea was enchanting. But I also realized that I had no idea how to do this, and didn’t really know anyone who did. I wanted to learn to program, but didn’t know how to get started.

I think I noticed Learn to Program BASIC in the back of a Scholastic book order. I convinced my mom to buy it for me. (After all, it was educational software.) Learn to Program BASIC was essentially a series of video lessons for the BASIC programming language. The software also came with its own BASIC compiler and execution environment, so it was relatively easy to write and run simple code. The first real program I ever wrote would have been some kind of “Hello World” program in BASIC, following one of the first lessons. (A video like this one. Wow, that’s so horrible. But still, nostalgia…) Incredibly, Learn to Program BASIC still has a product page on Amazon (although it appears to be sold out).

I worked through many of the lessons, but ultimately became frustrated. I could write a lot of simple text-based programs (add two numbers, ask a user’s name, etc), but never learned enough to create any kind of program on my own that wasn’t text-based. Perhaps I was still too young to completely grasp some of the more advanced concepts, but I don’t think I ever programmed anything really interesting in BASIC. I wanted to learn how to write computer games with real graphics, and my understanding of BASIC and the programming environment I was working in wasn’t enough to let me do anything exciting. Learn to Program BASIC got me off to a great start, but also left me thirsty for more knowledge.

Game Maker

As I grew a little older, one of my primary motivations for wanting to learn to program was to make computer games. Video games get expensive quickly, and my childhood brain thought that if I simply wrote the game myself, I could play whatever I wanted, essentially for free. My desire to write video games led me to an application called Game Maker.

Game Maker was great software (for my 13 year old self), and I used it to make a couple of games. Back in 2003, it was free. I remember that one of the games I wrote was a type of “laser tag” game, where multiple players could play on the same keyboard. One used WASD while the other used the arrow keys. Eventually, I think I added a 3rd and 4th player using other keyboard keys, to get even more friends in at once. Game maker produced a Windows executable (.exe) that I could put on a floppy disk and share with friends. I was really proud of the game I made, and learned a ton about programming in the process. It worked well, and the games I made were actually fun to play for a while.

Game Maker is still available, although it’s changed a lot. It’s now sold as GameMaker Studio by YoYo Games.

Scratch

Looking back on what I did with Game Maker, it’s really similar to what many kids are introduced to today with Scratch. Game Maker used it’s own programming language called GML (Game Maker Language). Using GML, you could essentially program sprites to respond to events. The language had simple commands (like move 10 left). Scratch is really the same thing, but it runs in a web browser and you program with blocks. I like to think that if I was a 10 year old today, I’d be learning to program by writing games in Scratch. So it’s great to see it gaining popularity in elementary and middle schools as an introduction to computer science.

CS Education Week

Computer Science Education Week is December 4-10, 2017. What are you doing this week to support computer science education? I’ll be doing an Hour of Code later this week at an elementary school, where I’ll be teaching one of my own lessons in Scratch. I’ve written several different games in Scratch over the last couple of years, primarily to use as teaching material. These days, when I write a game in Scratch, I try to keep the code (and the game) as simple as possible. My goal is to show the kids I’m teaching that you can make something fun with code that’s simple and easy to understand. I hope to write more about some of the Scratch lessons I’ve developed in the near future.

Recurse Center 17: ssh public keys

Pete's Points — Tue, 05 Dec 2017 03:48:47 GMT

This morning I paired on figuring out the structure of rsa private keys which was much easier since someone already wrote detailed documentation with fabulous color-coded binary structures. So mostly we just needed to figure out specifically what is meant when someone says "this is a 1024-bit rsa key". The answer is it's the size of the modulus field.

Then we needed the rust code to parse it out, which we used the yasna crate for. My pair had an excellent catch of the fact that we were getting an error due to "extra data at end of file" because we were only parsing the first few fields and ignore the rest which are uninteresting, but yasna expects you to fully parse the ASN.1 structure it seems.

After that we dug into the public key formats which are generally smaller and simpler. I think I have ed25519 and rsa figured out and can confirm you can match corresponding private/public keys by a common value in the payload. Also you can detect trivial tampering of the algorithm label field as that data is duplicated in the base64 payload (which you could of course also tamper with).

This afternoon there was a great web security presentation and a bunch of us will be doing a web security capture the flag exercise/contest this week. Looks fun but I think I'll have to time-box my effort as there are at least 48 vulnerabilities to be found and it's mostly unaided discovery of them.

searchcode plexus

Ben E. C. Boyter — Tue, 05 Dec 2017 03:12:02 GMT

Plexus “A combination of interlaced parts; a network.”

For a while I have been neglecting searchcode.com while working on searchcode server. This is of course not an ideal situation and as such I have started working on it again. The following is just a brief list of things I am considering, problems, issues etc…

So back when I first started working on searchcode I wrote it using PHP. For searchcode next (the current version) I rewrote it using Python and Django. I had intented to continue using this stack for the forseeable future when I chose it. I quite like writing Python code and at the time the performance was acceptable. When I started writing searchcode server however I wanted to make it as easy to install as possible. The reason being that I considered it a competitive advantage to have users installing searchcode without any pain and be off and running. At the time I considered writing it in Go or Rust but had little experience with either language. I decided to pull on my sensible pants and chose Java which is a language I had worked with before and had excellent library support. With the 8th version Java had become considerably less painful to work with.

Links - December 5th, 2017

Avy Faingezicht — Tue, 05 Dec 2017 02:00:00 GMT

This is another crypto heavy post with plenty of the links in some way discussing crypto assets, but bear with me: there's other good stuff to read, too.

An introduction to Ada's simple numeric types

Juan Cruz Viotti — Tue, 05 Dec 2017 02:00:00 GMT

This article introduces the numeric types available in Ada 2012

December 2017 Projects

Venam's Blog — Patrick Louis (Lebanon) — Tue, 05 Dec 2017 00:00:00 GMT

Five months have past since my last post about personal projects and endeavours but it feels like I've been stagnating for much longer.

The Empire of Oil

Blogs on Michail Rybakov — Mon, 04 Dec 2017 16:07:46 GMT

Costa Compagnie was preparing the Empire of Oil 360° movie to show at Ballhaus Ost in Berlin.

Two days before the premiere:

Submission Date vs Activity Date

blog.harterrt.com — Mon, 04 Dec 2017 10:00:00 GMT

My comments on Bug 1422892 started to get long, so I started untangling my thoughts here.

From the bug:

We experimented with using activity_date instead of submission_date when developing the clients_daily etl job. We should summarize our findings and decide on which of these measures we'd like to standardize against …

5000 concepts for Europe: a book proposal

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 04 Dec 2017 03:00:00 GMT

Thirty years ago (*) a Mr E. D. Hirsch Jr. wrote a book titled “Cultural Literacy: What Every American Needs to Know”

Frequently Answered Questions about: 5000 concepts for Europe

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 04 Dec 2017 03:00:00 GMT

These are the Frequently Answered Questions for the book project 5000 concepts for Europe.

Flutter for React Native Devs in 30 Seconds

swyx's site RSS Feed — Sun, 03 Dec 2017 18:06:18 GMT

You may have heard of Flutter, Google's answer to React Native. What should you know?

The Leftovers: Season 3

🦉 olshansky 🦁 — Sun, 03 Dec 2017 11:36:12 GMT

Olshansky's review of The Leftovers: Season 3

Aventador Renders Revisited

Code & Visuals — Sun, 03 Dec 2017 02:00:00 GMT

A long time ago, I made some posts that featured a cool Lamborghini Aventador model. Recently, I revisited that model and made some new renders using the current version of Takua, mostly just for fun. To me, one of the most important parts of writing a renderer has always been being able to actually use the renderer to make fun images. The last time I rendered this model was something like four years ago, and back then Takua was still in a very basic state; the renders in those old posts don’t even have any shading beyond 50% grey lambertian surfaces! The renders in this post utilize a lot of advanced features that I’ve added since then, such as a proper complex layered Bsdf and texturing system, advanced bidirectional light transport techniques, huge speed improvements to ray traversal, advanced motion blur and generalized time capabilities, and more. I’m way behind in writing up many of these features and capabilities, but in the meantime, I thought I’d post some for-fun rendering projects I’ve done with Takua.

All of the renders in this post are directly from Takua, with a basic white balance and conversion from HDR EXR to LDR PNG being the only post-processing steps. Each render took about half a day to render (except for the wireframe render, which was much faster) on a 12-core workstation at 2560x1440 resolution.

Shading the Aventador model was a fun, interesting exercise. I went for a orange-red paint scheme since, well, Lamborghinis are supposed to look outrageous and orange-red is a fairly exotic paint scheme (I suppose I could have picked green or yellow or something instead, but I like orange-red). I ended up making a triple-lobe shader with a metallic base, a dielectric lobe, and a clear-coat lobe on top of that. The base lobe uses a GGX microfacet metallic Brdf. Takua’s shading system implements a proper metallic Fresnel model for conductors, where the Fresnel model includes both a Nd component representing refractive index and a k component representing the extinction coefficient for when an electromagnetic wave propagates through a material. For conductors, the final Fresnel index of refraction for each wavelength of light is defined by a complex combination of Nd and k. For the base metallic lobe, most of the color wound up coming from the k component. The dielectric lobe is meant to simulate paint on top of a car’s metal body; the dielectric lobe is where most of the orange-red color comes from. The dielectric lobe is again a GGX microfacet Brdf, but with a dielectric Fresnel model, which has a much simpler index of refraction calculation than the metallic Fresnel model does. I should note that Takua’s current standard material implementation actually only supports a single primary specular lobe and an additional single clear-coat lobe, so for shaders authored with both a metallic and dielectric component, Takua takes a blend weight between the two components and for each shading evaluation stochastically selects between the two lobes according to the blend weight. The clear-coat layer on top has just a slightly amount of extinction to provide just a bit more of the final orange look, but is mostly just clear.

All of the window glass in the render is tinted slightly dark through extinction instead of through a fixed refraction color. Using proper extinction to tint glass is more realistic than using a fixed refraction color. Similarly, the red and yellow glass used in the head lights and tail lights are colored through extinction. The brake disks use an extremely high resolution bump map to get the brushed metal look. The branding and markings on the tire walls are done through a combination of bump mapping and adjusting the roughness of the microfacet Brdf; the tire treads are made using a high resolution normal map. There’s no displacement mapping at all, although in retrospect the tire treads probably should be displacement mapped if I want to put the camera closer to them. Also, I actually didn’t really shade the interior of the car much, since I knew I was going for exterior shots only.

Eventually I’ll get around to implementing a proper car paint Bsdf in Takua, but until then, the approach I took here seems to hold up reasonable well as long as the camera doesn’t get super close up to the car.

I lit the scene using two lights: an HDR skydome from HDRI-Skies, and a single long, thin rectangular area light above the car. The skydome provides the overall soft-ish lighting that illuminates the entire scene, and the rectangular area light provides the long, interesting highlights on the car body that help with bringing out the car’s shape.

For all of the renders in this post, I used my VCM integrator, since the scene contains a lot of subtle caustics and since the inside of the car is lit entirely through glass. I also wound up modifying my adaptive sampler; it’s still the same adaptive sampler that I’ve had for a few years now, but with an important extension. Instead of simply reducing the total number of paths per iteration as areas reach convergence, the adaptive sampler now keeps the number of paths the same and instead reallocates paths from completed pixels to high-variance pixels. The end result is that the adaptive sampler is now much more effective at eliminating fireflies and targeting caustics and other noisy areas. In the above render, some pixels wound up with as few as 512 samples, while a few particularly difficult pixels finished with as many as 20000 samples. Here is the adaptive sampling heatmap for Figure 1 above; brighter areas indicate more samples. Note how the adaptive sampler found a number of areas that we’d expect to be challenging, such as the interior through the car’s glass windows, and parts of the body with specular inter-reflections.

I recently implemented support for arbitrary camera shutter curves, so I thought doing a motion blurred render would be fun. After all, Lamborghinis are supposed to go fast! I animated the Lamborghini driving forward in Maya; the animation was very basic, with the main body just translating forward and the wheels both translating and rotating. Of course Takua has proper rotational motion blur. The motion blur here is effectively multi-segment motion blur; generating multi-segment motion blur from an animated sequence in Takua is very easy due to how Takua handles and understands time. I actually think that Takua’s concept of time is one of the most unique things in Takua; it’s very different from how every other renderer I’ve used and seen handles time. I intend to write more about this later. Instead of an instantaneous shutter, I used a custom cosine-based shutter curve that places many more time samples near the center of the shutter interval than towards the shutter open and close. Using a shutter shape like this wound up being important to getting the right look to the motion blur; even the car is moving extremely quickly, the overall form of the car is still clearly distinguishable and the front and back of the car appear more motion-blurred than the main body.

Since Takua has a procedural wireframe texture now, I also did a wireframe render. I mentioned my procedural wireframe texture in a previous post, but I didn’t write about how it actually works. For triangles and quads, the wireframe texture is simply based on the distance from the hitpoint to the nearest edge. If the distance to the nearest edge is smaller than some threshold, draw one color, otherwise, draw some other color. The nearest edge calculation can be done as follows (the variable names should be self-explanatory):

float calculateMinDistance(const Poly& p, const Intersection& hit) const {
    float md = std::numeric_limits<float>::infinity();
    const int verts = p.isQuad() ? 4 : 3;
    for (int i = 0; i < verts; i++) {
        const glm::vec3& cur = p[i].m_position;
        const glm::vec3& next = p[(i + 1) % verts].m_position;
        const glm::vec3 d1 = glm::normalize(next - cur);
        const glm::vec3 d2 = hit.m_point - cur;
        const float l = glm::length((cur + d1 * glm::dot(d1, d2) - hit.m_point));
        md = glm::min(md, l * l);
    }
    return md;
};

The topology of the meshes are pretty strange, since the car model came as a triangle mesh, which I then subdivided:

The material in the wireframe render only uses the lambertian diffuse lobe in Takua’s standard material; as such, the adaptive sampling heatmap for the wireframe render is interesting to compare to Figure 2. Overall the sample distribution is much more even, and areas where diffuse inter-reflections are present got more samples:

Takua’s shading model supports layering different materials through parameter blending, similar to how the Disney Brdf (and, at this point, most other shading systems) handles material layering. I wanted to make an even more outrageous looking version of the Aventador than the orange-red version, so I used the procedural wireframe texture as a layer mask to drive parameter blending between a black paint and a metallic gold paint:

Rocket - A Rust game running on WASM

Consulting on Adolfo Ochagavía — Sun, 03 Dec 2017 02:00:00 GMT

Two weeks ago, Alex Crichton’s PR adding a target for WebAssembly to the Rust compiler was merged. There are many differences between this target and the Emscripten one, but the important one for me is that it doesn’t depend on external stuff like the Emscripten SDK (which IIRC used to be a pain to get working on Windows, but seems to be better now). After seeing the examples on hellorust.com, I thought it would be interesting to try to adapt my game Rocket to work on the browser through the wasm32-unknown-unknown target.

Serverless Machine Learning at Google

swyx's site RSS Feed — Sun, 03 Dec 2017 00:25:33 GMT

---

Are we human? Or are we reCAPTCHA?

swyx's site RSS Feed — Sat, 02 Dec 2017 23:49:06 GMT

---

Firebase Analytics in 30 Seconds

swyx's site RSS Feed — Sat, 02 Dec 2017 22:57:12 GMT

---

Medical Machine Learning in 30 Seconds

swyx's site RSS Feed — Sat, 02 Dec 2017 18:41:14 GMT

---

React Window decorators

Muffin Man — Sat, 02 Dec 2017 02:00:00 GMT

Two days ago I released two npm packages, window-scroll-manager and react-window-decorators.

Scroll manager #

Scroll managerPlx uses the same scroll manager so I extracted it to the standalone package. is just a simple wrapper around scroll event, that broadcasts custom window-scroll event once per requestAnimationFrame. Idea comes from MDN.

React decorators #

or higher order components are the bees knees, as they bring easy way to track window scroll and resize events in React. They will track and inject props directly into React components. Check the demo.

Two decorators included are:

withScroll decorator #

tracks and injects window scroll position (by using scroll manager)

withWindow decorator #

tracks and injects window size, orientation and breakpointUser needs to pass breakpoints definition.

How do they work? #

Both scroll and resize decorators are listening to custom events, rather than the native ones. This is because both scroll and resize eventsEspecially scroll event. are triggered a lot and can cause performance issues.

That's why I introduced custom window-scroll and window-resize events. Plus, all updates are wrapped in requestAnimationFrame and resize event is debounced (configurable, 250ms by default).

So the heavy lifting is done by scroll and window managers, which are both singletons, while the React decorators are slim components that inject props to the wrapped component.

Please note that using withScroll on a lot of components can introduce performance issues. Then again ten-ish should work just fine. I used these on numerous projects, they are battle tested, but it took me some time to put in the effort to release them as a standalone package.

Check the code #

Of course, both packages are available on GitHub:

Recurse Center 16: tealeaves is useful

Pete's Points — Sat, 02 Dec 2017 00:53:15 GMT

Today I refactored all my various scratch iterations of code to dig into the details of an ed25519 openssh private key file and integrated that into the main tealeaves executable file. I added some unit tests for edge cases and I've so far been pleased with how unit testing is in rust. The cargo command line is weird for testing: I often have to do something like RUST_BACKTRACE=1 cargo test --jobs=1 long_field -- --nocapture to try to debug which seems silly.

But basically I can run tealeaves now on my ~/.ssh and it prints and accurate and useful description of what each file is.

There's also now a SQL study group happening so I'll be spending some time on that which was my minor project for RC which I have yet to actually spend any time on.

Tip: Array.from

Qubyte Codes — Fri, 01 Dec 2017 22:00:00 GMT

SPOILER ALERT: If you're doing the advent of code this year, you may not want to read onward. This post does not give any solutions away, but does contain information about how I approached a part of the first challenge.

Onward!

Today I started doing the 2017 advent of code challenges (don't worry, I won't give solutions away). The first challenge involves summing numbers in a sequence. The sequence itself is too long to express as an integer, and that wouldn't be particularly useful for the task anyway. The most convenient form is probably an array of integers. The sequence is given as raw text, and the quickest way to get it into JS is as a string.

There are a few ways to do this which spring to mind. One might argue that you can loop directly over a string anyway, but getting a list of integers avoids parsing each character in the string more than once for these tasks, so let's ignore that option.

The first thing to come to mind was a loop:

const numbers = [];

for (let i = 0; i < sequence.length; i++) {
  numbers.push(parseInt(sequence[i], 10));
}

There's a slightly nicer to read version of this introduced by ES2015:

const numbers = [];

for (const character of sequence) {
  numbers.push(parseInt(character, 10));
}

Both suffer from being a bit bulky though. ES2015 also introduced Array.from and the spread operator (...), which offer a similar way of creating an array when used in combination with Array.prototype.map:

const numbers = [...sequence].map(char => parseInt(char, 10));
// or
const numbers = Array.from(sequence).map(char => parseInt(char, 10));

The first spreads characters from the sequence into an array, and maps these characters to a fresh array of numbers. The second uses Array.from to get an array of characters from a string, and uses the same mapping to get an array of integers.

I prefer the latter of the two (it's a little easier for humans to parse). It would be nice to avoid the explicit mapping though, right? The solution I settled upon was to use the little-known second parameter to Array.from. It turns out you can include a function to map elements to be inserted into the returned array:

const numbers = Array.from(sequence, char => parseInt(char, 10));

Which is my tip! In essence, don't do this:

[...stuff].map(func);

Or this:

Array.from(stuff).map(func);

Do this:

Array.from(stuff, func);

Becoming Your Future Self

Coding VC — Fri, 01 Dec 2017 18:50:00 GMT

In my five years as a VC, I’ve worked closely with the founders of 50+ companies.

OpenSSH ed25519 private key file format

Pete's Points — Fri, 01 Dec 2017 18:46:17 GMT

Today I finished understanding the openssh private key format for ed25519 keys. Yesterday's analysis had a few remaining mysteries that a fellow RCer helped me solve plus a pair of mistakes that threw off some fields. So here for the record is a "complete" byte-by-byte analysis of what's inside an openssh ed25519 private key (both with and without a passphrase). This was done with OpenSSH_7.5p1, LibreSSL 2.5.4 on macOS 10.13. I say "complete" in quotes because you could go one layer deeper into the actual key payloads but for our purposes those details don't have specific meaning.

First, let's generate a test keypair (hit ENTER for no passphrase when prompted):

mkdir /tmp/test-keys
cd /tmp/test-keys
ssh-keygen -t ed25519 -f ssh-ed25519-private-key.pem
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ssh-ed25519-private-key.pem.
Your public key has been saved in ssh-ed25519-private-key.pem.pub.
The key fingerprint is:
SHA256:qXcj+K4QZHxJ712c/M1PPutkyxT/Hn+1RKBIHLlypY4 plyons@avi
The key's randomart image is:
+--[ED25519256]--+
|      . ..o      |
|   . . o + + o   |
|    + o o = * .  |
|   o . o B o . + |
|    .   S .   o.+|
|     . E .     +=|
|    . o o o   .**|
|     . o o .  =.O|
|      .oo     .*=|
+----[SHA256]-----+

Now let's get at those tasty bytes of payload as a hex dump:

grep -v - ssh-ed25519-private-key.pem | base64 -D | xxd -p                  
6f70656e7373682d6b65792d763100000000046e6f6e65000000046e6f6e
650000000000000001000000330000000b7373682d656432353531390000
00203cfe2afb025f46582e502b97f7dfa5a08dea09f87abfa8d5bfcaabf2
9fbb369500000090a2224bbaa2224bba0000000b7373682d656432353531
39000000203cfe2afb025f46582e502b97f7dfa5a08dea09f87abfa8d5bf
caabf29fbb36950000004002a1965d1a2684d50d29f2be0efd8e2fae3c5b
b013d06f7818416333955271a53cfe2afb025f46582e502b97f7dfa5a08d
ea09f87abfa8d5bfcaabf29fbb36950000000a706c796f6e734061766901
0203

OK here's how it breaks down.

# ASCII magic "openssh-key-v1" plus null byte
6f70656e7373682d6b65792d7631 00
00000004 int length = 4
6e6f6e65 string cipher = none
00000004 int length = 4
6e6f6e65 string kdfname = none
00000000 int length = 0
# zero-length kdfoptions placeholder here
00000001 int number of public keys = 1
00000033 int length first public key = 51 (4 + 11 + 4 + 32)
0000000b int length = 11
7373682d65643235353139 string key type = ssh-ed25519
00000020 int length = 32
# public key payload 32 bytes
# probably encoding a point on the ed25519 curve
3cfe2afb025f46582e502b97f7dfa5a0
8dea09f87abfa8d5bfcaabf29fbb3695

00000090 int length = 144 size of remaining payload
# 8 + 4 + 11 + 4 + 32 + 4 + 64 + 4 + 10 + 3
a2224bbaa2224bba iv/salt? (Not sure about these 8 bytes)

# Here's a repeat of the public key (part of the private key pair)
0000000b int length = 11
7373682d656432353531 39 string key type = ssh-ed25519
00000020  int length = 32
# public key payload 32 bytes
# probably encoding a point on the ed25519 curve
3cfe2afb025f46582e502b97f7dfa5a0
8dea09f87abfa8d5bfcaabf29fbb3695

00000040 int length = 64
# 32 bytes private key payload 1
02a1965d1a2684d50d29f2be0efd8e2f
ae3c5bb013d06f7818416333955271a5
# 32 bytes is the public/point again
3cfe2afb025f46582e502b97f7dfa5a0
8dea09f87abfa8d5bfcaabf29fbb3695

0000000a int length = 10
706c796f6e7340617669 private key payload 2
010203 padding 3 bytes incrementing integers

Now let's generate one and encrypt with a passphrase

ssh-keygen -t ed25519 -f ssh-ed25519-passphrase-private-key.pem
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ssh-ed25519-passphrase-private-key.pem.
Your public key has been saved in ssh-ed25519-passphrase-private-key.pem.pub.
The key fingerprint is:
SHA256:QfPM5uIVVM6yT5sXeCvSjETCcP5zmTH+Rt7wY/91Ik8 plyons@avi
The key's randomart image is:
+--[ED25519256]--+
|      . + ...    |
|       * * o     |
|        = O =    |
|         B = *   |
|        S B B =  |
|       . + O B * |
|        . o O Eo=|
|           . B..=|
|              . +|
+----[SHA256]-----+

Here's the dump

grep -v - ssh-ed25519-passphrase-private-key.pem | base64 -D | xxd -p
6f70656e7373682d6b65792d7631000000000a6165733235362d63626300
0000066263727970740000001800000010d08f6b8fd17593f246db4ac6c4
5a11930000001000000001000000330000000b7373682d65643235353139
0000002062837be86c63712896b8e0e7543e367c3abd0c0b5ad3e764ea0e
4f8ddd7d00ef000000901e60c56ef30d0ff02e07b57bf14645076c32c86c
88ecad545ca28424e4739aff5895bebd6778e70b6c54b309b9fdb0c94102
bf8cef5b97d3d75636967e67e4b9c1ee72ae81074b0ce0f7e540e051d569
05da263af3e383342cc75b3145242abb75257586a119c9d3673dfb7eabe4
696350904e7c7af3cd77f28bea10374e15bc6536c2e1029438fdd3930bee
bbc5ac30

Here's the annotated structure:

#ASCII magic "openssh-key-v1" plus null byte
6f70656e7373682d6b65792d7631 00
0000000a int length = 10
6165733235362d636263 string cipher = aes256-cbc
00000006 int length = 6
626372797074 string kdfname = bcrypt
00000018 int length = 24 (kdfoptions)
00000010 int length = 16
d08f6b8fd17593f246db4ac6c45a1193 salt/iv for bcrypt
00000010 int work factor = 16
00000001 int number of public keys = 1
00000033 int length = 51 public key 1 size (4 + 11 + 4 + 32)

# Public key 1
0000000b int length = 11
7373682d65643235353139 string key type = ssh-ed25519
00000020 int length = 32
# public key payload 32 bytes
# probably encoding a point on the ed25519 curve
62837be86c63712896b8e0e7543e367c
3abd0c0b5ad3e764ea0e4f8ddd7d00ef

00000090 int length = 144 encrypted aes256-cbc output (16x9)
1e60c56ef30d0ff02e07b57bf1464507
6c32c86c88ecad545ca28424e4739aff
5895bebd6778e70b6c54b309b9fdb0c9
4102bf8cef5b97d3d75636967e67e4b9
c1ee72ae81074b0ce0f7e540e051d569
05da263af3e383342cc75b3145242abb
75257586a119c9d3673dfb7eabe46963
50904e7c7af3cd77f28bea10374e15bc
6536c2e1029438fdd3930beebbc5ac30

So there you have it. You could dig even deeper into the keys themselves to extract the various specific numbers (usually they have 1-letter names from the math equations they belong to like e,n,p,q etc) but I didn't bother for this since for this key format all I want to know is that it's an ed25519 private key and whether or not it has a passphrase.

Other bits I learned while pairing are that ed25519 is the name of a specific curve in elliptic curve cryptography and the numbers represent ((2^255) - 19) as a Mersenne prime (out of my depth here, could be wrong). Part of it is the x coordinate and then you can compute the y bit if you know the sign or something like that.

Also in the openssh source code they explicitly error out if the number of keys field isn't 1 so only 1 key is supported.

For further reading check out How do Ed5519 keys work? by Brian Warner.

This post is technically my Recurse Center 15 post but it was suggested that this post would make a handy reference so I didn't want to muck up the clarity/SEO of the title.

Java 9 Strings

Studiofreya SSG Site — Thu, 30 Nov 2017 13:59:17 GMT

String in Java 9 is a data type just like integer and floating point, but it is used to represent text rather than numbers or a single character.

String variable is comprised of a set of characters that can also contain spaces, numbers and special characters.

For instance, the word “tennis” and the phrase “I play tennis” are an example of string variable. Even a number “5445” could be considered a string in Java, if specified correctly.

A Gentle Introduction to Erasure Codes

Fred Akalin — Thu, 30 Nov 2017 10:00:00 GMT

1. Overview

This article explains Reed-Solomon erasure codes and the problems they solve in gory detail, with the aim of providing enough background to understand how the PAR1 and PAR2 file formats work, the details of which will be covered in future articles.

I’m assuming that the reader is familiar with programming, but has not had much exposure to coding theory or linear algebra. Thus, I’ll review the basics and treat the results we need as a “black box”, stating them and moving on. However, I’ll give self-contained proofs of those results in a companion article.

So let’s start with the problem we’re trying to solve! Let’s say you have $n$ files of roughly the same size, and you want to guard against $m$ of them being lost or corrupted. To do so, you generate $m$ parity files ahead of time, and if in the future you lose up to $m$ of the data files, you can use an equal number of parity files to recover the lost data files.

cashcat0.jpg

cashcat1.jpg

cashcat2.jpg

$\xmapsto{\mathtt{GenerateParityFiles}}$

cashcats.p00

cashcats.p01

Figure 1 Using parity codes to protect against the loss or corruption of up to two images (out of three) of cashcats.

$\xmapsto{\mathtt{ReconstructDataFiles}}$

Figure 2 With a corrupted and a missing file, recovering the original cashcat images using the parity files from Figure 1.

Note that this works even if you lose some of the parity files also; as long as you have $n$ files, whether they be data or parity files, you’ll be able to recover the original $n$ data files. Compare making $n$ parity files with simply making a copy of the $n$ data files (for $n > 1$). In the latter case, if you lose both a data file and its copy, that data file becomes unrecoverable! So parity files take the same amount of space but provide superior recovery capabilities.

Now we can reduce the problem above to a byte-level problem as follows. Have ComputeParityFiles pad all the data files so they’re the same size, and then for each byte position i call a function ComputeParityBytes on the ith byte of each data file, and store the results into the ith byte of each parity file. Also take a checksum or hash of each data file and store those (along with the original data file sizes) with the parity files. Then, ReconstructDataFiles can detect corrupted files using the checksums/hashes and treat them as missing, and then for each byte position i it can call a function ReconstructDataBytes on the ith byte of each good data and parity file to recover the ith byte of the corrupted/missing data files.

A byte error where we know the position of the dropped/corrupted byte is called an erasure. Then, the pair of functions ComputeParityBytes and ReconstructDataBytes which behave as described above implements what is called an optimal erasure code; it’s an erasure code because it guards only against byte erasures, and not more general errors where we don’t know which data bytes have been corrupted, and it’s optimal because in general you need at least $n$ known bytes to recover the $n$ data bytes, and that bound is achieved.

In detail, an optimal erasure code is composed of some set of possible $(n, m)$ pairs, and for each possible pair, a function

ComputeParityBytes<n, m>(data: byte[n]) -> (parity: byte[m])

that computes $m$ parity bytes given $n$ data bytes, and a function

ReconstructDataBytes<n, m>(partialData: (byte?)[n], partialParity: (byte?)[m]) -> ((data: byte[n]) | Error)

that takes in a partial list of data and parity bytes from an earlier call to ComputeParity, and returns the full list of data bytes if there are at least $n$ known data or parity bytes (i.e., there are no more than $m$ omitted data or parity bytes), and an error otherwise.

(In the above pseudocode, I’m using T[n] to mean an array of n objects of type T, and byte? to mean byte | None. Also, I’ll omit the -Bytes<n, m> suffix from now on.)

By the end of this article, we’ll find out exactly how the following example works:

Example 1: `ComputeParity` and `ReconstructData`

`ComputeParity`

Let d = [ da, db, 0d ] be the input data bytes and let m = 2 be the desired parity byte count. Then the output parity bytes are p = [ 52, 0c ].

Let d_partial = [ ??, db, ?? ] be the input partial data bytes and p_partial = [ 52, 0c ] be the input partial parity bytes. Then the output data bytes are d = [ da, db, 0d ].

2. Erasure codes for $m = 1$

The simplest erasure codes are when $m = 1$. For example, define

ComputeParitySum(data: byte[n]) {
  return [data[0] + … + data[n-1]]
}

where we consider byte to be an unsigned type such that addition and subtraction wrap around, i.e. byte arithmetic is done modulo $256$. Then also define

ReconstructDataSum(partialData: (byte?)[n], partialParity: (byte?)[1]) {
  if there is more than one entry of partialData or partialParity set to None {
    return Error
  } else if partialData has no entry set to None {
    return partialData
  }

  i := partialData.firstIndexOf(None);
  partialSum = partialData[0] + … + partialData[i-1] + partialData[i+1] + … + partialData[n-1]
  return partialData[0:i] ++ [partialParity[0] - partialSum] ++ partialData[i+1:n]
}

where a[i:j] means the subarray of a starting at i and ending (without inclusion) at j, and ++ is array concatenation.

This simple erasure code uses the fact that if you have the sum of a list of numbers, then you can recover a missing number by subtracting the sum of the other numbers from the total sum, and also that this works even if you do the arithmetic modulo $256$.

Another erasure code for $m = 1$ uses bitwise exclusive or (denoted as xor, ^, or $\oplus$) instead of arithmetic modulo $256$. Define

ComputeParityXor(data: byte[n]) {
  return [data[0] ⊕ … ⊕ data[n-1]]
}

and

ReconstructDataXor(partialData: (byte?)[n], partialParity: (byte?)[1]) {
  if there is more than one entry of partialData or partialParity set to None {
    return Error
  } else if partialData has no entry set to None {
    return partialData
  }

  i := partialData.firstIndexOf(None);
  partialXor = partialData[0] ⊕ … ⊕ partialData[i-1] ⊕ partialData[i⊕1] ⊕ … ⊕ partialData[n-1]
  return partialData[0:i] ++ [partialParity[0] ⊕ partialXor] ++ partialData[i+1:n]
}

This relies on the fact that $a \oplus a = 0$, so given the xor of a list of bytes, you can recover a missing byte by xoring with all the known bytes.

3. Erasure codes for $m = 2$ (almost)

Now coming up with an erasure code for $m = 2$ is more involved, but we can get an inkling of how it could work by letting $n = 3$ for simplicity, and also letting the output of ComputeParity be non-negative integers, instead of just bytes (i.e., less than $256$). In that case, we can consider parity numbers that are weighted sums of the data bytes. For example, like in the $m = 1$ case, we can have the first parity number be \[ p_0 = d_0 + d_1 + d_2\text{,} \] (using $d_i$ for data bytes and $p_i$ for parity numbers) but for the second parity number, we can pick different weights, say \[ p_1 = 1 \cdot d_0 + 2 \cdot d_1 + 3 \cdot d_2\text{.} \] We want to make sure that the weights for the second parity number are “sufficiently different” from that of the first parity number, which we’ll clarify later, but for example note that setting \[ p_1 = 2 \cdot d_0 + 2 \cdot d_1 + 2 \cdot d_2 \] can’t add any new information, since then $p_1$ will always be equal to $2 \cdot p_0$.

So then our ComputeParity function looks like

ComputeParityWeighted(data: byte[3]) {
  return [
    int(data[0]) +     int(data[1]) +     int(data[2]),
    int(data[0]) + 2 * int(data[1]) + 3 * int(data[2]),
  ]
}

As for ReconstructData, if we have two missing data bytes, say $d_i$ and $d_j$ for $i < j$, and $p_0$ and $p_1$, we can rearrange the equations \[ \begin{aligned} p_0 &= d_0 + d_1 + d_2 \\ p_1 &= 1 \cdot d_0 + 2 \cdot d_1 + 3 \cdot d_2 \end{aligned} \] to get all the unknowns on the left side, letting $d_k$ be the known data byte: \[ \begin{aligned} d_i + d_j &= X = p_0 - d_k \\ (i+1) \cdot d_i + (j+1) \cdot d_j &= Y = p_1 - (k + 1) \cdot d_k\text{.} \end{aligned} \] We can then multiply the first equation by $i + 1$ and subtract it from the second to cancel the $d_i$ term and get \[ d_j = (Y - (i + 1) \cdot X) / (j - i)\text{,} \] and then we can use the first equation to solve for $d_i$: \[ d_i = X - d_j = ((j + 1) \cdot X - Y) / (j - i)\text{.} \] Thus with these equations, we can implement ReconstructData:

ReconstructDataWeighted(partialData: (byte?)[3], partialParity: (int?)[2]) {
  Handle all cases except when there are exactly two entries set to none in partialData.

  [i, j] := indices of the unknown data bytes
  k := index of the known data byte

  X := partialParity[0] - partialData[k]
  Y := partialParity[1] - (k + 1) * partialData[k];

  d_i := ((j + 1) * X - Y) / (j - i)
  d_j := (Y - (i + 1) * X) / (j - i)

  return an array with d_i, d_j, and d[k] in the right order
}

(Generalizing this to larger values of $n$ is straightforward; $p_0$ will still have a weight of $1$ for each data byte, and $p_1$ will have a weight of $i + 1$ for $d_i$. $X$ and $Y$ will then have terms for all known bytes, and everything else proceeds the same after that.)

Now what goes wrong if we just try to do everything modulo $256$? The most obvious difference from the $m = 1$ case is that solving for $d_i$ or $d_j$ involves division, which works fine for non-negative integers as long as there’s no remainder, but it is not immediately clear how division can make sense modulo $256$.

One possible way to define division modulo $256$ would be to first define the multiplicative inverse modulo $256$ of an integer $0 \le x \lt 256$ as the integer $0 \le y \lt 256$ such that $(x \cdot y) \bmod 256 = 1$, if it exists, and then define division by $x$ modulo $256$ to be multiplication by $y$ modulo $256$. But this immediately runs into problems; $2$ has no multiplicative inverse modulo $256$, and the same holds for any even number, so reconstruction will fail if, for example, we have the first and third data bytes missing, since then we’d be trying to divide by $j - i = 2$.

But for now, let’s leave aside the problem of generating parity bytes instead of parity numbers, and instead focus on how we can generalize the above for larger values of $m$. To do so, we need to first review some linear algebra.

4. Just enough linear algebra to get by^[1]

In our $n = 3, m = 2$ example in the previous section, the equations for the parity numbers have the form \[ p = a_0 \cdot d_0 + a_1 \cdot d_1 + a_2 \cdot d_2 \] for constants $a_0$, $a_1$, and $a_2$. We call such a weighted sum of the $d_i$s a linear combination of the $d_i$s, and we write this in a tabular form \[ p = \begin{pmatrix} a_0 & a_1 & a_2 \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ d_1 \\ d_2 \end{bmatrix}\text{,} \] where we define the multiplication of a row vector and a column vector by the equation above, generalized in the straightforward manner for any $n$.

Then since we have two parity numbers $p_0$ and $p_1$, each a linear combination of the $d_i$s, i.e. \[ \begin{aligned} p_0 &= a_{00} \cdot d_0 + a_{01} \cdot d_1 + a_{02} \cdot d_2 \\ p_1 &= a_{10} \cdot d_0 + a_{11} \cdot d_1 + a_{12} \cdot d_2\text{,} \end{aligned} \] we can write this in a single tabular form as \[ \begin{bmatrix} p_0 \\ p_1 \end{bmatrix} = \begin{pmatrix} a_{00} & a_{01} & a_{02} \\ a_{10} & a_{11} & a_{12} \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ d_1 \\ d_2 \end{bmatrix}\text{,} \] where we define the multiplication of a matrix and a column vector by the equations above.

Now if we restrict parity numbers to be linear combinations of the data bytes, then we can identify a function ComputeParity using some set of weights with the matrix formed from that set of weights as above. This holds in general: if a function is defined as a list of linear combinations of its inputs, then it can be represented using a matrix as above, and we call it a linear function. Then we have a correspondence between linear functions that take $n$ numbers to $m$ numbers and matrices with $m$ rows and $n$ columns, which are denoted as $m \times n$ matrices.

As the first example of this correspondence, note that we denote the elements of the matrix above as $a_{ij}$, where the first index is the row index and the second index is the column index. Looking back to the parity equations, we also see that the first index corresponds to the output arguments of ComputeParity, and the second index corresponds to the input arguments.^[2]

The usefulness of the correspondence between linear functions and matrices is that we can store and manipulate a linear function by storing and manipulating its corresponding matrix of weights, which you wouldn’t be able to easily do for functions in general. For example, as we’ll see below, we’ll be able to compute the inverse of a linear function by matrix operations, which will be useful for ReconstructData.

First, let’s examine some simple matrix operations and their effects on the corresponding linear function:

Deleting a row of a matrix corresponds to deleting an output of a linear function.
Swapping two rows of a matrix corresponds to swapping two outputs of a linear function.
Appending a row to a matrix corresponds to adding an output to a linear function.

In general, matrix row operations correspond to manipulations of a linear function’s outputs.

An important operation on functions is composition: if $f$ takes $k$ inputs to $m$ outputs, and $g$ takes $m$ inputs to $n$ outputs, then we can define $(g \circ f)(x_0, \dotsc, x_k) = g(f(x_0, \dotsc, x_k))$ which takes $k$ inputs to $n$ outputs. It turns out that the composition of two linear functions is again a linear function, and so there must be an operation which takes the corresponding $m \times k$ matrix $F$ and the $n \times m$ matrix $G$ and yields a $n \times k$ matrix. This important operation, the bane of high-schoolers everywhere, is called matrix multiplication, denoted by $F \cdot G$. If $H = F \cdot G$, then the explicit formula for its elements is \[ h_{ij} = \sum_{k=0}^{m-1} f_{ik} \cdot g_{kj}\text{,} \] which corresponds to the following code:

matrixMultiply(f: Matrix, g: Matrix) {
  if (f.columns != g.rows) {
    return Error
  }

  h := new Matrix(f.rows, g.columns)
  for i := 0 to f.rows - 1 {
    for j := 0 to g.columns - 1 {
      t := 0
      for k := 0 to f.columns - 1 {
        t += f[i, k] * g[k, j]
      }
      h[i, j] = t
    }
  }
  return h
}

You can convince yourself that the above formula and code is correct by trying to compose some small linear functions by hand.

A useful property of matrix multiplication is that it’s a generalization of the product of a row vector and a column vector, and the product of a matrix and a column vector as we defined above.

I would be remiss if I didn’t talk about the consequences of defining matrix multiplication as the matrix of the composition of the corresponding linear functions. First, this immediately implies that you can only multiply matrices if the left matrix has the same number of rows as the number of columns of the right matrix, which corresponds to the fact that you can only compose functions if the left function takes the same number of inputs as the number of outputs of the right function. Furthermore, even if you have two $n \times n$ matrices $F$ and $G$, unlike numbers, it is not true that $F \cdot G = G \cdot F$, which corresponds to the fact that in general, for two functions that take $n$ inputs to $n$ outputs, it is not true that $f \circ g = g \circ f$. If you learned matrix multiplication just from the formula above, then these facts are much less obvious!

Finally, an important function is the identity function $\mathrm{Id}_n$, which return its $n$ inputs as its outputs. It corresponds to the identity matrix \[ I_n = \begin{pmatrix} 1 & 0 & \cdots & 0 & 0 \\ 0 & 1 & \cdots & 0 & 0 \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & \cdots & 1 & 0 \\ 0 & 0 & \cdots & 0 & 1 \end{pmatrix}\text{.} \]

For a linear function $f$ that takes $n$ inputs to $n$ outputs, if there is a function $g$ such that $f \circ g = \mathrm{Id}_n$, then we call $g$ the inverse of $f$, and denote it as $f^{-1}$. (It is also true that $f^{-1} \circ f = \mathrm{Id}_n$, i.e. $(f^{-1})^{-1} = f$.) Not all linear functions taking $n$ inputs to $n$ outputs have inverses, but if the inverse exists, it is also linear (and unique, which is why we call it the inverse). Therefore, we can define the inverse of an $n \times n$ (or square) matrix $M$ as the unique matrix $M^{-1}$ such that $M \cdot M^{-1} = M^{-1} \cdot M = I_n$, if it exists; also, if $M$ has an inverse, we say that $M$ is invertible.

Example 2: The matrix/linear function correspondence

Let \[ M = \begin{pmatrix} 1 & 2 \\ 3 & 4\end{pmatrix}\text{.} \] This corresponds to the linear function

f(x: rational[2]) {
  return [
    1 * x[0] + 2 * x[1],
    3 * x[0] + 4 * x[1],
  ]
}

where rational is an arbitrary-precision rational number type.

$M$ is invertible with inverse \[ M^{-1} = \begin{pmatrix} -2 & 1 \\ 3/2 & -1/2\end{pmatrix}\text{.} \] This corresponds to the linear function

g(y: rational[2]) {
  return [
    -2 * x[0] + 1 * x[1],
    (3/2) * x[0] + (-1/2) * x[1],
  ]
}

so g is the inverse function of f. Indeed, f([5, 6]) is [17, 39] and g([17, 39]) is [5, 6].

So now we’ve reduced the problem of finding the inverse of a linear function taking $n$ inputs to $n$ outputs to finding the inverse of an $n \times n$ matrix. Before we tackle the question of computing those inverses, let’s first recast our problem in the language of linear algebra and see why we need to find the inverse of a linear function.

5. Erasure codes in general

So, revisiting our $n = 3, m = 2$ erasure code from above, we have the linear function

ComputeParityWeighted(data: byte[3]) {
  return [
    int(data[0]) +     int(data[1]) +     int(data[2]),
    int(data[0]) + 2 * int(data[1]) + 3 * int(data[2]),
  ]
}

which therefore corresponds to the parity matrix \[ P = \begin{pmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{pmatrix}\text{.} \] So in mathematical notation, ComputeParityWeighted looks like: \[ \begin{bmatrix} p_0 \\ p_1 \end{bmatrix} = \mathtt{ComputeParityWeighted}(d_0, d_1, d_2) = \begin{pmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ d_1 \\ d_2 \end{bmatrix}\text{.} \]

So let’s now reimplement ReconstructDataWeighted using linear algebra. First, append the rows of $P$ to the identity matrix $I_3$ to get the matrix equation \[ \begin{bmatrix} d_0 \\ d_1 \\ d_2 \\ p_0 \\ p_1 \end{bmatrix} = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 1 & 1 \\ 1 & 2 & 3 \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ d_1 \\ d_2 \end{bmatrix}\text{,} \] which corresponds to a linear function that returns the input data bytes in addition to computing the parity numbers. Now let’s say we lose the data bytes $d_0$ and $d_2$. Then let’s remove the rows corresponding to those bytes: \[ \begin{bmatrix} \xcancel{d_0} \\ d_1 \\ \xcancel{d_2} \\ p_0 \\ p_1 \end{bmatrix} = \begin{pmatrix} \xcancel{1} & \xcancel{0} & \xcancel{0} \\ 0 & 1 & 0 \\ \xcancel{0} & \xcancel{0} & \xcancel{1} \\ 1 & 1 & 1 \\ 1 & 2 & 3 \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ d_1 \\ d_2 \end{bmatrix}\text{,} \] which turns into \[ \begin{bmatrix} d_1 \\ p_0 \\ p_1 \end{bmatrix} = \begin{pmatrix} 0 & 1 & 0 \\ 1 & 1 & 1 \\ 1 & 2 & 3 \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ d_1 \\ d_2 \end{bmatrix}\text{,} \] which corresponds to a linear function that maps the input data bytes to the non-lost data bytes and the parity bytes. This is the inverse of the function we want, so we want to invert the $3 \times 3$ matrix above, which we’ll call $M$. That inverse is \[ M^{-1} = \begin{pmatrix} -1/2 & 3/2 & -1/2 \\ 1 & 0 & 0 \\ -1/2 & -1/2 & 1/2 \end{pmatrix}\text{.} \] Multiplying both sides above by $M^{-1}$, we get \[ \begin{bmatrix} d_0 \\ d_1 \\ d_2 \end{bmatrix} = \begin{pmatrix} -1/2 & 3/2 & -1/2 \\ 1 & 0 & 0 \\ -1/2 & -1/2 & 1/2 \end{pmatrix} \cdot \begin{bmatrix} d_1 \\ p_0 \\ p_1 \end{bmatrix}\text{,} \] which is exactly what we want: the original data bytes in terms of the known data bytes and the parity numbers!^[3]

Comparing this equation to the one we manually computed previously, they don’t look immediately similar, but some rearrangement will reveal that they indeed compute the same thing. As a sanity check, notice that the second row of $M^{-1}$ means that the first input argument is mapped unchanged to the second output argument, which is exactly what we want for the known byte $d_1$.

Now what does this look like in general, i.e. for arbitrary $n$ and $m$? First, we have to generate an $m \times n$ parity matrix $P$ whose rows have to be “sufficiently different” from each other, which we still have to clarify. Then ComputeParity just multiplies $P$ by $[d]$, the column matrix of input bytes, like so: \[ \begin{bmatrix} p_0 \\ \vdots \\ p_{m-1} \end{bmatrix} = \mathtt{ComputeParity}(d_0, \dotsc, d_{n-1}) = \begin{pmatrix} p_0 \\ \vdots \\ p_{m-1} \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ \vdots \\ d_{n-1} \end{bmatrix}\text{,} \] where the $p_i$ are the rows of $P$.

As for ReconstructData, we first append the rows of $P$ to $I_n$, whose rows we’ll denote as $e_i$: \[ \begin{bmatrix} d_0 \\ \vdots \\ d_{n-1} \\ p_0 \\ \vdots \\ p_{m-1} \end{bmatrix} = \begin{pmatrix} e_0 \\ \vdots \\ e_{n-1} \\ p_0 \\ \vdots \\ p_{m-1} \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ \vdots \\ d_{n-1} \end{bmatrix}\text{.} \] Now assume that the indices of the missing $k \le m$ data bytes are $i_0, \dotsc, i_{k-1}$. Then we remove the rows corresponding to the missing data bytes, and keep some $k$ parity rows, e.g. $p_0$ to $p_{k-1}$. This yields the equation \[ \begin{bmatrix} d_{j_0} \\ \vdots \\ d_{j_{n-k-1}} \\ p_0 \\ \vdots \\ p_{k-1} \end{bmatrix} = \begin{pmatrix} e_{j_0} \\ \vdots \\ e_{j_{n-k-1}} \\ p_0 \\ \vdots \\ p_{k-1} \end{pmatrix} \cdot \begin{bmatrix} d_0 \\ \vdots \\ d_{n-1} \end{bmatrix}\text{,} \] where $j_0, \dotsc, j_{m-k-1}$ are the indices of the present $n - k$ data bytes. Call that $n \times n$ matrix $M$, and compute its inverse $M^{-1}$. If $P$ was chosen correctly, $M^{-1}$ should always exist, so if the inverse computation fails, raise an error. Therefore, ReconstructData just multiplies $M^{-1}$ by the column matrix of present data bytes and chosen parity numbers: \[ \begin{bmatrix} d_0 \\ \vdots \\ d_{n-1} \end{bmatrix} = \mathtt{ReconstructData}(d_{j_0}, \dotsc, d_{j_{n-k-1}}, p_0, \dotsc, p_{k-1}) = M^{-1} \cdot \begin{bmatrix} d_{j_0} \\ \vdots \\ d_{j_{n-k-1}} \\ p_0 \\ \vdots \\ p_{k-1} \end{bmatrix}\text{.} \]

As an optimization, some rows of $M^{-1}$ correspond to just shuffling around the known data bytes $d_{j_*}$, so we can just remove those rows, compute the missing data bytes, and do the shuffling ourselves.

So we now have outlines of implementations of both ComputeParity and ReconstructData, but we still have missing pieces. In particular,

How do we compute matrix inverses?
How do we generate “optimal” parity matrices so that $M^{-1}$ always exists?
How do we compute parity bytes instead of parity numbers?

So first, let’s see how to compute matrix inverses using row reduction.

6. Finding matrix inverses using row reduction

We developed the theory of matrices by identifying them with linear functions of numbers. To show how to find matrix inverses, we have to look at them in a slightly different way by identifying matrix equations with systems of linear equations of numbers.

For example, consider the matrix equation \[ M \cdot x = y\text{,} \] where \[ M = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}\text{,} \quad x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \text{,} \quad \text{and } y = \begin{bmatrix} y_1 \\ y_2 \end{bmatrix}\text{.} \] This expands to \[ \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \cdot \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} y_1 \\ y_2 \end{bmatrix}\text{,} \] or \[ \begin{aligned} y_1 &= 1 \cdot x_1 + 2 \cdot x_2 \\ y_2 &= 3 \cdot x_1 + 4 \cdot x_2\text{,} \end{aligned} \] which is a linear system of equations of numbers. Letting $M$ be any matrix, and $x$ and $y$ be appropriately-sized column matrices of variables, we can see that the matrix equation $M \cdot x = y$ is shorthand for a system of linear equations of numbers.

If we could find $M^{-1}$, we could solve the matrix equation easily by multiplying both sides by it: \[ \begin{aligned} M^{-1} \cdot (M \cdot x) &= M^{-1} \cdot y \\ x &= M^{-1} \cdot y\text{,} \end{aligned} \] and therefore solve the linear system for $x$ in terms of $y$. Conversely, if we were able to solve the linear system for $x$, we’d then be able to read off $M^{-1}$ from the new linear system.

But how do we solve a linear system? From the theory of linear systems of equations, we have a few tools at our disposal:

swapping two equations,
multiplying an equation by a number,
adding one equation to another, possibly multiplying the equation by a number before adding.

All of these are valid transformations because they don’t change the solution set of the linear system.

For example, in the equation above, the first step would be to subtract $3$ times the first equation from the second equation to yield \[ \begin{aligned} y_1 &= x_1 + 2 \cdot x_2 \\ y_2 - 3 \cdot y_1 &= -2 \cdot x_2\text{.} \end{aligned} \] Then, add the second equation back to the first equation: \[ \begin{aligned} y_2 - 2 \cdot y_1 &= x_1 \\ y_2 - 3 \cdot y_1 &= -2 \cdot x_2\text{.} \end{aligned} \] Finally, divide the second equation by $-2$: \[ \begin{aligned} y_2 - 2 \cdot y_1 &= x_1 \\ (3/2) \cdot y_1 - (1/2) \cdot y_2 &= x_2\text{.} \end{aligned} \] This is equivalent to the matrix equation \[ \begin{pmatrix} -2 & 1 \\ 3/2 & -1/2 \end{pmatrix} \cdot \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\text{,} \] so \[ M^{-1} = \begin{pmatrix} -2 & 1 \\ 3/2 & -1/2 \end{pmatrix}\text{.} \]

So how do we translate the above process to an algorithm operating on matrices? First, express our matrix equation in a slightly different form: \[ M \cdot x = I \cdot y\text{.} \] Using the example above, this is \[ \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \cdot \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \cdot \begin{bmatrix} y_1 \\ y_2 \end{bmatrix}\text{.} \] Then, you can see that the first step above corresponds to subtracting $-3$ times the first row from the second row to yield: \[ \begin{pmatrix} 1 & 2 \\ 0 & -2 \end{pmatrix} \cdot \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{pmatrix} 1 & 0 \\ -3 & 1 \end{pmatrix} \cdot \begin{bmatrix} y_1 \\ y_2 \end{bmatrix}\text{.} \] We don’t even need to keep writing the $x$ and $y$ column matrices; we can just write the “augmented” matrix. \[ A = \left( \hskip -5pt \begin{array}{cc|cc} 1 & 2 & 1 & 0 \\ 0 & -2 & -3 & 1 \end{array} \hskip -5pt \right) \] and operate on it.

Thus, the operations listed above on linear systems have corresponding operations on augmented matrices:

swapping two equations corresponds to swapping two rows;
multiplying an equation by a number corresponds to multiplying a row by a number; and
adding an equation to another, possibly multiplying the equation by a number before adding, corresponds to adding a row to another row, possibly multiplying the row by a number before adding.

Then, the goal is to use these row operations to transform the initial augmented matrix, where the right side looks like the identity matrix, into one where the left side looks like the identity matrix. Then, translating the augmented matrix back into a matrix equation, that would give $M^{-1}$ on the right side.^[4]

When doing this by hand, one usually works with the linear system itself, trying to see which variables can be easily eliminated so as to minimize arithmetic. However, to translate this to an algorithm, we’re more interested in a systematic way of doing this. Fortunately, there’s an easy two-step process:

Turn the left side of $A$ into a unit upper triangular matrix, which means that all the elements on the main diagonal are $1$, and all elements below the main diagonal are $0$, i.e. that $a_{ii} = 1$ for all $i$, and $a_{ij} = 0$ for all $j > i$.
Then turn the left side of $A$ into the identity matrix.

This algorithm is called row reduction. The first step can be further broken down:

For each column $i$ of the left side in ascending order:
1. If $a_{ii}$ is zero, look at the rows below the $i$th row for a row $j > i$ such that $a_{ji} \ne 0$, and swap rows $i$ and $j$. If no such row exists, return an error, as that means that $A$ is non-invertible.
2. Divide the $i$th row by $a_{ii}$, so that $a_{ii} = 1$.
3. For each row $j > i$, subtract $a_{ji}$ times the $i$th row from it, which will set $a_{ji}$ to $0$.

The second step can be similarly broken down:

For each column $i$ of the left side, in order:
1. For each row $j < i$, subtract $a_{ji}$ times the $i$th row from it, which will set $a_{ji}$ to $0$.

Note that we’re assuming that all arithmetic is exact, i.e. we use a arbitrary-precision rational number type. If we use floating point numbers, we’d have to worry a lot more about the order in which we do operations and numerical stability.

Example 3: Matrix inversion via row reduction

Let

    / 0 2 2 \
M = | 3 4 5 |
    \ 6 6 7 /.

The initial augmented matrix A is

/ 0 2 2 | 1 0 0 \
| 3 4 5 | 0 1 0 |
\ 6 6 7 | 0 0 1 /.

We need A₀₀ to be non-zero, so swap rows 0 and 1:

/ 0 2 2 | 1 0 0 \     / 3 4 5 | 0 1 0 \
| 3 4 5 | 0 1 0 | --> | 0 2 2 | 1 0 0 |
\ 6 6 7 | 0 0 1 /     \ 6 6 7 | 0 0 1 /.

We need A₀₀ to be 1, so divide row 0 by 3:

/ 3 4 5 | 0 1 0 \     / 1 4/3 5/3 | 0 1/3 0 \
| 0 2 2 | 1 0 0 | --> | 0  2   2  | 1  0  0 |
\ 6 6 7 | 0 0 1 /     \ 6  6   7  | 0  0  1 /.

We need A₂₀ to be 0, so subtract row 0 scaled by 6 from row 2:

/ 1 4/3 5/3 | 0 1/3 0 \     / 1 4/3 5/3 | 0 1/3 0 \
| 0  2   2  | 1  0  0 | --> | 0  2   2  | 1  0  0 |
\ 6  6   7  | 0  0  1 /     \ 0 -2  -3  | 0 -2  1 /.

We need A₁₁ to be 1, so divide row 1 by 2:

/ 1 4/3 5/3 |  0  1/3 0 \     / 1 4/3 5/3 |  0  1/3 0 \
| 0  2   2  |  1   0  0 | --> | 0  1   1  | 1/2  0  0 |
\ 0 -2  -3  |  0  -2  1 /     \ 0 -2  -3  |  0  -2  1 /.

We need A₂₁ to be 0, so subtract row 1 scaled by −2 from row 2:

/ 1 4/3 5/3 |  0  1/3 0 \     / 1 4/3 5/3 |  0  1/3 0 \
| 0  1   1  | 1/2  0  0 | --> | 0  1   1  | 1/2  0  0 |
\ 0 -2  -3  |  0  -2  1 /     \ 0  0  -1  |  1  -2  1 /.

We need A₂₂ to be 1, so divide row 2 by −1, which makes the left side of A a unit upper triangular matrix:

/ 1 4/3 5/3 |  0  1/3 0 \     / 1 4/3 5/3 |  0  1/3 0 \
| 0  1   1  | 1/2  0  0 | --> | 0  1   1  | 1/2  0  0 |
\ 0  0  -1  |  1  -2  1 /     \ 0  0   1  | -1   2 -1 /.

We need A₁₂ to be 0, so subtract row 2 from row 1:

/ 1 4/3 5/3 |  0  1/3 0 \     / 1 4/3 5/3 |  0  1/3 0 \
| 0  1   1  | 1/2  0  0 | --> | 0  1   0  | 3/2 -2  1 |
\ 0  0   1  | -1   2 -1 /     \ 0  0   1  | -1   2 -1 /.

We need A₀₂ to be 0, so subtract row 2 scaled by 5/3 from row 0:

/ 1 4/3 5/3 |  0  1/3 0 \     / 1 4/3  0  | 5/3 -3 5/3 \
| 0  1   0  | 3/2 -2  1 | --> | 0  1   0  | 3/2 -2  1  |
\ 0  0   1  | -1   2 -1 /     \ 0  0   1  | -1   2 -1  /.

We need A₀₁ to be 0, so subtract row 1 scaled by 4/3 from row 0, which makes the left side of A the identity matrix:

/ 1 4/3  0  | 5/3  -3 5/3 \     / 1  0   0  | -1/3 -1/3 1/3 \
| 0  1   0  | 3/2 -2   1  | --> | 0  1   0  |  3/2  -2   1  |
\ 0  0   1  | -1   2  -1  /     \ 0  0   1  |  -1    2  -1  /.

Since the left side of A is the identity matrix, the right side of A is M^-1. Therefore,

         / -1/3 -1/3 1/3 \
M^{-1} = |  3/2  -2   1  |
         \  -1    2  -1  /.

Now notice one thing: if $M$ has a row that is proportional to another row, then row reduction would eventually zero out one of the rows, causing the algorithm to fail, and signaling that $M$ is non-invertible. In fact, a stronger statement is true: $M$ has a row that can be expressed as a linear combination of other rows of $M$ exactly when $M$ is non-invertible. Informally, this means that the linear function corresponding to $M$ has one of its outputs redundant with the other outputs, so it is essentially a a linear function taking $n$ inputs to fewer than $n$ outputs, and such functions aren’t invertible.

This gets us a partial explanation for what “sufficiently different” means for our parity functions. If one parity function is a linear combination of other parity functions, then it is redundant, and therefore not “sufficiently different”. Therefore, we want our parity matrix $P$ to be such that no row can be expressed as a linear combination of other rows.

However, this criterion for $P$ isn’t quite enough to guarantee that all possible matrices $M$ computed as part of ReconstructData are invertible. For example, this criterion holds for the identity matrix $I_n$, but if $n > 1$ and you pick $I_n$ as the parity matrix for $n = m$, you can certainly end up with a constructed matrix $M$ with repeated rows, since you’re starting by appending another copy of $I_n$ on top of $P = I_n$! This explains in a different way why simply making a copy of the original data files makes for a poor erasure code, unless of course you only have one data file. We’re led to our next topic: what makes a parity matrix “optimal”?

7. Optimal parity matrices

Recall from above that we form the square matrix \[ M = \begin{pmatrix} e_{j_0} \\ \vdots \\ e_{j_{n-k-1}} \\ p_0 \\ \vdots \\ p_{k-1} \end{pmatrix} \] by prepending some rows of the identity matrix to the first $k$ rows of the parity matrix. We can generalize this a bit more, since we don’t have to take the first $k$ rows, but instead can take any $k$ rows of the parity matrix, whose indices we denote here as $l_0, \dotsc, l_{k-1}$: \[ M = \begin{pmatrix} e_{j_0} \\ \vdots \\ e_{j_{n-k-1}} \\ p_{l_0} \\ \vdots \\ p_{l_{k-1}} \end{pmatrix}\text{.} \] So we want to construct $P$ so that any such square matrix $M$ formed from the rows of $P$ is invertible. Therefore, we call a parity matrix $P$ optimal if it satisfies this criterion.

Fortunately, there is a simpler criterion for optimal parity matrices. First, define a submatrix of a matrix $P$ to be a matrix that you get by deleting any number of rows or columns, and call a matrix non-empty if it has at least one row and one column. Then:

(Theorem 1.) A parity matrix $P$ is optimal exactly when any non-empty square submatrix of $P$ is invertible.^[5]

Note that this criterion is stronger than the one in the previous section, where we want a parity matrix $P$ to have no row that can be expressed as a linear combination of other rows. That is, if any non-empty square submatrix of $P$ is invertible, that means that no row can be expressed as a linear combination of other rows.^[6] However, it is possible to have a matrix $P$ where no row can be expressed as a linear combination of other rows, but which is not optimal. We’ve already seen an example above: $I_n$ for $n \gt 1$, and indeed, \[ I_2 = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}\text{,} \] has the $1 \times 1$ non-invertible submatrix $\begin{pmatrix} 0 \end{pmatrix}$.

Example 4: A optimal parity matrix for $m = 2$

Recall the parity matrix \[ P = \begin{pmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{pmatrix} \] that we were using for our $n = 3, m = 2$ example. For any $n$, this matrix looks like \[ P = \begin{pmatrix} 1 & 1 & \cdots & 1 \\ 1 & 2 & \cdots & n-1 \end{pmatrix}\text{.} \] A $1 \times 1$ matrix is invertible exactly when its single element is non-zero, so any $1 \times 1$ submatrix of $P$ is invertible. Any $2 \times 2$ submatrix of $P$ looks like \[ A = \begin{pmatrix} 1 & 1 \\ a & b \end{pmatrix} \] for $a \ne b$, which, using the formula for inverses of $2 \times 2$ matrices, has inverse \[ A^{-1} = \begin{pmatrix} b/(b-a) & -1/(b-a) \\ -a/(b-a) & 1/(b-a) \end{pmatrix}\text{.} \] These are all the possible square submatrices of $P$, so therefore this $P$ is a optimal parity matrix for $m = 2$.

Then, finally, we now have a complete definition of what makes a list of parity functions “sufficiently different”; it is exactly when the corresponding parity matrix is optimal as we’ve defined it above.

Now this leads us to the question: how do we find such optimal matrices? Fortunately, there’s a whole class of matrices that are optimal: the Cauchy matrices.

Let $a_0, \dotsc, a_{m+n-1}$ be a sequence of distinct integers, meaning that no two $a_i$ are equal, and let $x_0, \dotsc, x_{m-1}$ be the first $m$ integers of $a_i$ with $y_0, \dotsc, y_{n-1}$ the remaining integers. Then form the $m \times n$ matrix $A$ by setting its elements according to: \[ a_{ij} = \frac{1}{x_i - y_j}\text{,} \] which is always defined since the denominator is never zero, by the distinctness of the $a_i$. Then $A$ is a Cauchy matrix.

What makes Cauchy matrices useful is the following theorem:

(Theorem 2.) Any non-empty square Cauchy matrix is invertible.

Combining this with the simple fact that any submatrix of a Cauchy matrix is also a Cauchy matrix, we get:

(Corollary 1.) Any non-empty square submatrix of a Cauchy matrix is invertible, and thus any Cauchy parity matrix is optimal.

Example 5: Cauchy matrices

Let x = [ 1, 2, 3 ] and y = [ -1, 4, 0 ]. Then, the Cauchy matrix constructed from x and y is

/ 1/2 -1/3  1  \
| 1/3 -1/2 1/2 |
\ 1/4  -1  1/3 /,

which has inverse

/ -36/5 96/5 -36/5 \
| -3/10  9/5  -9/5 |
\  9/2  -9     3   /.

Therefore, to generate a optimal parity matrix for any $(n, m)$, all we need to do is to generate an $m \times n$ Cauchy matrix. We can pick any sequence of distinct $m + n$ integers, so for simplicity let’s just use \[ x_i = n + i \quad \text{and} \quad y_i = i\text{.} \]

Example 6: Cauchy parity matrices for $m = 2$

For $n = 3, m = 2$, we have the sequences \[ x_0 = 3, x_1 = 4 \quad \text{and} \quad y_0 = 0, y_1 = 1, y_2 = 2\text{,} \] so the corresponding Cauchy parity matrix is \[ P = \begin{pmatrix} 1/3 & 1/2 & 1 \\ 1/4 & 1/3 & 1/2 \end{pmatrix}\text{.} \] Similarly, for any $n$, \[ P = \begin{pmatrix} 1/n & \cdots & 1/2 & 1 \\ 1/{n + 1} & \cdots & 1/3 & 1/2 \end{pmatrix}\text{.} \] All entries of $P$ are non-zero, so any $1 \times 1$ submatrix of $P$ is invertible. Any $2 \times 2$ submatrix of $P$ looks like \[ A = \begin{pmatrix} 1/a & 1/b \\ 1/(a+1) & 1/(b+1) \end{pmatrix} \] for $a \ne b$, which, using the formula for inverses of $2 \times 2$ matrices, has inverse \[ A^{-1} = \begin{pmatrix} \frac{ab(a+1)}{b-a} & -\frac{a(a+1)(b+1)}{b-a} \\ -\frac{ab(b+1)}{b-a} & \frac{b(a+1)(b+1)}{b-a} \end{pmatrix}\text{.} \] These are all the possible square submatrices of $P$, so therefore this $P$ is a optimal parity matrix for $m = 2$.

Note that our first parity matrix for $n = 3, m = 2$ isn’t a Cauchy matrix, since no Cauchy matrix can have repeating elements in a single row. That means that there are other possible optimal parity matrices that aren’t Cauchy matrices.^[7]

Also, our previous parity matrices had integers, and Cauchy matrices have rational numbers (i.e., fractions). This means that our parity numbers are now fractions. This isn’t a serious difference, though, since we’d have to deal with fractions when computing matrix inverses anyway. You could also change a parity matrix with fractions into one without by simply multiplying the entire matrix by some non-zero number that gets rid of all the fractions, which doesn’t change the optimality of the matrix. For example, we can multiply \[ \begin{pmatrix} 1/3 & 1/2 & 1 \\ 1/4 & 1/3 & 1/2 \end{pmatrix} \] by $12$ to get the equivalent parity matrix \[ \begin{pmatrix} 4 & 6 & 12 \\ 3 & 4 & 6 \end{pmatrix}\text{.} \]

Now our only remaining missing piece is this: how do we compute parity bytes instead of parity numbers? Answering this would render the above discussion moot. However, to do so, we first have to take another look at how we’re doing linear algebra.

8. Linear algebra over fields

We ultimately want our parity numbers to be parity bytes, which means that we want to work with matrices of bytes instead of matrices of rational numbers. In order to do that, we need to define an interface for matrix elements that preserves the operations and properties we care about, and then we have to figure out how to implement that interface using bytes.

Looking at the rule for matrix multiplication, we need to be able to add and multiply matrix elements. Looking at how we do matrix inversion, we also need to be able to subtract and divide matrix elements. Finally, there are certain properties that hold for rational numbers that we implicitly assume when doing matrix operations, but that we have to make explicit for matrix elements.

This leads us to the concept of a field, which essentially defines the interface that matrix elements should implement. Here it is:

interface Field<T> {
  static Zero: T, One: T

  plus(b: T): T
  negate(): T

  times(b: T): T
  reciprocate(): T

  equals(b: T): bool

  minus(b: T) = this.plus(b.negate())
  dividedBy(b: T) = this.times(b.reciprocate())
}

We need to be able to add and multiply field elements, which we’ll denote generically by $\oplus$ and $\otimes$. We also need to be able to take the negation (additive inverse) of an element $x$, which we’ll denote by $-x$, and the reciprocal (multiplicative inverse) of a non-zero element $x$, which we’ll denote by $x^{-1}$. Then we can define subtraction of field elements to be \[ a \ominus b = a \oplus -b \] and division of field elements to be \[ a \cldiv b = a \otimes b^{-1}\text{,} \] when $b \ne 0$.

Also, an implementation of Field must satisfy further properties, which are copied from the number laws you learn in school:

Identities: $a \oplus 0 = a \otimes 1 = a$.
Inverses: $a \oplus -a = 0$, and for $a \ne 0$, $a \otimes a^{-1} = 1$.
Associativity: $(a \oplus b) \oplus c = a \oplus (b \oplus c)$, and $(a \otimes b) \otimes c = a \otimes (b \otimes c)$.
Commutativity: $a \oplus b = b \oplus a$, and $a \otimes b = b \otimes a$.
Distributivity: $a \otimes (b \oplus c) = (a \otimes b) \oplus (a \otimes c)$.

Of the above, guaranteeing the existence of reciprocals of non-zero elements is usually the non-trivial part. Now the rational numbers satisfy all of the above, since \[ (p/q)^{-1} = q/p\text{,} \] so we say that they form a field. However, the integers do not form a field, since for example $2$ has no integer reciprocal; only $1$ and $-1$ have integer reciprocals. Furthermore, as we saw above, the integers modulo $256$, i.e. the numbers from $0$ to $255$ with standard arithmetic operations modulo $256$, do not form a field, as we saw earlier, since $(2 \cdot b) \bmod 256 \ne 1$ for any $b$.

However, we can construct a field with $257$ elements, using the fact that $257$ is a prime number, and the following theorem:

(Theorem 3.) Given a prime number $p$, for every integer $0 \lt a \lt p$, there is exactly one $0 \lt b \lt p$ such that $(a \cdot b) \bmod p = 1$.

There are clever ways to find multiplcative inverses mod $p$, but since $257$ is so small, we can just brute-force it. So an implementation would look like:

class Field257Element : implements Field<Field257Element> {
  plus(b) { return (this + b) % 257 }
  negate() { return (257 - this) }
  times(b) { return (this * b) % 257 }
  reciprocate() {
    if (this == 0) { return Error }
    for i := 0 to 256 {
      if (this.times(b) == 1) { return i; }
    }
    return Error
  }
  ...
}

Example 7: Field with 257 elements

Denote operations on the field with 257 elements by a ₂₅₇ subscript, and let a = 23 and b = 54. Then

a +₂₅₇ b = (23 + 54) mod 257 = 77;
−₂₅₇b = (257 − 54) mod 257 = 203;
a −₂₅₇ b = a +₂₅₇ −₂₅₇b = (23 + 203) mod 257 = 226;
a ×₂₅₇ b = (23 × 54) mod 257 = 214;
54 ×₂₅₇ 119 = 1, so b^-1₂₅₇ = 119;
a ÷₂₅₇ b = a ×₂₅₇ b^-1₂₅₇ = (23 × 119) mod 257 = 167, and indeed b ×₂₅₇ (a ÷₂₅₇ b) = (54 × 167) mod 257 = 23 = a.

So this gets us closer, since we can use Field257Element instead of a rational number type when implementing ComputeParity and ReconstructData, and if we’ve abstracted our Matrix type correctly, almost everything should just work. However, there is one thing we need to check: Are Cauchy parity matrices still optimal if we use fields other than the rational numbers? Fortunately, the answer is yes:

(Theorem 1, general version.) A parity matrix $P$ over any field is optimal exactly when any non-empty square submatrix of $P$ is invertible.

(Theorem 2, general version.) Any non-empty square Cauchy matrix over any field is invertible.

(Corollary 1, general version.) Any square submatrix of a Cauchy matrix over any field is invertible, and thus any Cauchy parity matrix over any field is optimal.

However, note that to construct an $m \times n$ Cauchy matrix, we need $m + n$ distinct elements. So if we’re working with a field with $257$ elements, then this imposes the condition that $m + n \le 257$, i.e. using a finite field limits the number of data bytes and parity numbers you can have.

Now the question remains: can we construct a field with $256$ elements? As we saw above, we can’t do so the same way as we constructed the field with $257$ elements. In fact, we need to start with defining different arithmetic operations on the integers. This brings us to the topic of binary carry-less arithmetic.

9. Binary carry-less arithmetic

The basic idea with binary carry-less (which I’ll henceforth shorten to “carry-less”) arithmetic is to express all integers in binary, then perform all arithmetic operations using binary arithmetic, except ignoring all the carries.^[8]

How does this work with addition? Let’s denote binary carry-less add as $\clplus$,^[9] and let’s see how it behaves on single binary digits: \[ \begin{aligned} 0 \clplus 0 &= 0 \\ 0 \clplus 1 &= 1 \\ 1 \clplus 0 &= 1 \\ 1 \clplus 1 &= 0\text{.} \end{aligned} \] This is just the exclusive or operation on bits, so if we do carry-less addition on any two integers, it turns out to be nothing but xor! Since xor can also be denoted by $\clplus$, we can conveniently think of $\clplus$ as meaning both carry-less addition and xor.

Example 8: Carry-less addition

Let a = 23 and b = 54. Then, with carry-less arithmetic,

  a = 23 =  10111b
^ b = 54 = 110110b
           -------
           100001b

so a ⊕ b = 100001_b = 33.

What about subtraction? Recall that $(a \clplus b) \clplus b = a$ for any $a$ and $b$. Therefore, every element $b$ is its own (carry-less binary) additive inverse, which means that $a \clminus b = a \clplus b$, i.e. carry-less subtraction is also just xor.

Carry-less multiplication isn’t as simple, but recall that binary multiplication is just adding shifted copies of $a$ based on which bits are set in $b$ (or vice versa). To do carry-less multiplication, just ignore carries when adding the shifted copies again, i.e. xor shifted copies instead of adding them.

Example 9: Carry-less multiplication

Let a = 23 and b = 54. Then, with carry-less arithmetic,

   a = 23 =       10111b
^* b = 54 =      110110b
            ------------
                 10111
          ^     10111
          ^   10111
          ^  10111
            ------------
             1111100010b

so a ⊗ b = 1111100010_b = 994.

Finally, we can define carry-less division with remainder. Binary division with remainder is subtracting shifted copies of $b$ from $a$ until you get a remainder less than the divisor; then carry-less binary division with remainder is xor-ing shifted copies of $b$ with $a$ until you get a remainder. However, there’s a subtlety; with carry-less arithmetic, it’s not enough to stop when the remainder (for that step) is less than the divisor, because if the highest set bit of the remainder is the same as the highest set bit of the divisor, you can still xor with the divisor one more time to yield a smaller number (which then becomes the final remainder).

Consider the example below, where we’re dividing $55$ by $19$. The first remainder is $17$, which is less than $19$, but still shares the same highest set bit, so we can xor one more time with $19$ to get the remainder $2$.

Example 10: Carry-less division

Let a = 55 and b = 19. Then, with carry-less arithmetic,

                     11b
                --------
b = 19 = 10011b )110111b = 55 = a
               ^ 10011
                 -----
                  10001
                ^ 10011
                  -----
                     10b

so a &odiv b = 11_b = 3 with remainder 10_b = 2.

This leads to an interesting difference between the carry-less modulo operation and the standard modulo operation. If you mod by a number $n$, you get $n$ possible remainders, from $0$ to $n - 1$. However, if you clmod (carry-less mod) by a number $2^k \le n \lt 2^{k+1}$, you get $2^k$ possible remainders, from $0$ to $2^k-1$, since those are the numbers whose highest set bit is lower than the highest set bit of $n$.

In particular, if you clmod by a number $256 \le n < 512$, you always get $256$ possible remainders. This is very close to what we want—now the hope is to find some $256 \le n < 512$ so that doing binary carry-less arithmetic clmod $n$ yields a field, which will then be a field with $256$ elements!

10. The finite field with $256$ elements

Since there are only a few numbers between $256$ and $512$, we can just try each one of them to see if clmod-ing by one of them yields a field. However, with a bit of math, we can gain more insight into which numbers will work.

Recall the situation with the standard arithmetic operations: arithmetic mod $p$ yields a field exactly when $p$ is prime.^[10] But recall the definition of a prime number: it is an integer greater than $1$ whose positive divisors are only itself and $1$. Stated another way, a prime number is an integer $p \gt 1$ that cannot be expressed as $p = a \cdot b$, for $a, b \gt 1$.

Thus, the concept of a prime number is determined by the multiplication operation, and therefore we can define a “carry-less” prime number to be an integer $ p \gt 1$ that cannot be expressed as $p = a \clmul b$, for $a, b \gt 1$.^[11]

The only question remaining is whether there is an equivalent of Theorem 3 for carry-less arithmetic. And indeed there is:

(Theorem 4.) Given a carry-less prime number $2^k \lt p \le 2^{k+1}$, for every integer $0 \lt a \lt 2^k$, there is a exactly one $0 \lt b \lt 2^k$ such that $(a \clmul b) \bclmod p = 1$.

Now we just need to find a carry-less prime number $256 \le p < 512$. However, the set of prime numbers and the set of carry-less prime numbers are not necessarily related, so for example, even though $257$ is a prime number, it is not a carry-less prime number.

It is easy enough to test each number $256 \le n < 512$ for carry-less primality though; doing so, we find the lowest one, $283$.^[12]

So finally, we have a field with $256$ elements: the integers with binary carry-less arithmetic clmod $283$. An implementation would look like:

class Field256Element : implements Field<Field256Element> {
  plus(b) { return this ^ b }
  negate() { return b }
  times(b) { return clmod(clmul(this, b), 283) }
  reciprocate() {
    if (this == 0) { return Error }
    for i := 0 to 255 {
      if (this.times(b) == 1) { return i; }
    }
    return Error
  }
  ...
}

Similarly to how we find reciprocals mod $257$, we brute-force finding reciprocals clmod $283$ also.

Example 11: Field with 256 elements

Denote operations on the field with 256 elements by a ₂₅₆ subscript, and let a = 23 and b = 54. Then

a ⊕₂₅₆ b = 23 ⊕ 54 = 33;
&ominus₂₅₆b = b = 54;
a &ominus₂₅₆ b = a ⊕₂₅₆ &ominus₂₅₆b = a ⊕₂₅₆ b = 33;
a ⊗₂₅₆ b = (23 ⊗ 54) clmod 283 = 207;
54 ⊗₂₅₆ 102 = 1, so b^-1₂₅₆ = 102;
a ø₂₅₆ b = a ⊗₂₅₆ b^-1₂₅₆ = (23 ⊗ 102) clmod 283 = 19, and indeed b ⊗₂₅₆ (a ø₂₅₆ b) = (54 × 19) clmod 283 = 23 = a.

11. The full algorithm

Now we have all the pieces we need to construct erasure codes for any $(n, m)$ such that $m + n \le 256$. First, we can compute an $m \times n$ Cauchy parity matrix over the field with $256$ elements. (Recall that this needs $m + n$ distinct field elements, which is what imposes the condition $m + n \le 256$.)

Example 12: Cauchy matrices in general

Working over the field with 256 elements, let x = [ 1, 2, 3 ] and y = [ 4, 5, 6 ]. Then, the Cauchy matrix constructed from x and y is

/  82 203 209 \
| 123 209 203 |
\ 209 123  82 /,

which has inverse

/ 130  31 176 \
| 252 219  31 |
\ 108 252 130 /.

Then we can implement matrix multiplication over arbitrary fields, and thus we can implement ComputeParity.

Example 13: `ComputeParity` in detail

Let d = [ da, db, 0d ] be the input data bytes and let m = 2 be the desired parity byte count. Then, with the input byte count n = 3, the m × n Cauchy parity matrix computed using x_i = n + i and y_i = i is

/ f6 8d 01 \
\ cb 52 7b /.

Therefore, the parity bytes are computed as

                _    _     _    _
/ f6 8d 01 \   |  da  |   |  52  |
\ cb 52 7b / * |  db  | = |_ 0c _|,
               |_ 0d _|

and thus the output parity bytes are p = [ 52, 0c ].

Then we can implement matrix inversion using row reduction over arbitrary fields.

Example 14: Matrix inversion via row reduction in general

Working over the field with 256 elements, let

    / 0 2 2 \
M = | 3 4 5 |
    \ 6 6 7 /.

The initial augmented matrix A is

/ 0 2 2 | 1 0 0 \
| 3 4 5 | 0 1 0 |
\ 6 6 7 | 0 0 1 /.

We need A₀₀ to be non-zero, so swap rows 0 and 1:

/ 0 2 2 | 1 0 0 \     / 3 4 5 | 0 1 0 \
| 3 4 5 | 0 1 0 | --> | 0 2 2 | 1 0 0 |
\ 6 6 7 | 0 0 1 /     \ 6 6 7 | 0 0 1 /.

We need A₀₀ to be 1, so divide row 0 by 3:

/ 3 4 5 | 0 1 0 \     / 1 245 3 | 0 246 0 \
| 0 2 2 | 1 0 0 | --> | 0  2  2 | 1  0  0 |
\ 6 6 7 | 0 0 1 /     \ 6  6  7 | 0  0  1 /.

We need A₂₀ to be 0, so subtract row 0 scaled by 6 from row 2:

/ 1 245 3 | 0 246 0 \     / 1 245  3 | 0 246 0 \
| 0  2  2 | 1  0  0 | --> | 0  2   2 | 1  0  0 |
\ 6  6  7 | 0  0  1 /     \ 0  14 13 | 0  2  1 /.

We need A₁₁ to be 1, so divide row 1 by 2:

/ 1 245  3 | 0 246 0 \     / 1 245  3 |  0  246 0 \
| 0  2   2 | 1  0  0 | --> | 0  1   1 | 141  0  0 |
\ 0 14  13 | 0  2  1 /     \ 0 14  13 |  0   2  1 /.

We need A₂₁ to be 0, so subtract row 1 scaled by 14 from row 2:

/ 1 245  3 |  0  246 0 \     / 1 245  3 |  0  246 0 \
| 0  1   1 | 141  0  0 | --> | 0  1   1 | 141  0  0 |
\ 0 14  13 |  0   2  1 /     \ 0  0   3 |  7   2  1 /.

We need A₂₂ to be 1, so divide row 2 by 3, which makes the left side of A a unit upper triangular matrix:

/ 1 245  3 |  0  246 0 \     / 1 245  3 |  0  246  0  \
| 0  1   1 | 141  0  0 | --> | 0  1   1 | 141  0   0  |
\ 0  0   3 |  7   2  1 /     \ 0  0   1 | 244 247 246 /.

We need A₁₂ to be 0, so subtract row 2 from row 1:

/ 1 245  3 |  0  246  0  \     / 1 245  3 |  0  246  0  \
| 0  1   1 | 141  0   0  | --> | 0  1   0 | 121 247 246 |
\ 0  0   1 | 244 247 246 /     \ 0  0   1 | 244 247 246 /.

We need A₀₂ to be 0, so subtract row 2 scaled by 3 from row 0:

/ 1 245  3 |  0  246  0  \     / 1 245  0 |  7  244  1  \
| 0  1   0 | 121 247 246 | --> | 0  1   0 | 121 247 246 |
\ 0  0   1 | 244 247 246 /     \ 0  0   1 | 244 247 246 /.

We need A₀₁ to be 0, so subtract row 1 scaled by 245 from row 0, which makes the left side of A the identity matrix:

/ 1 245  0 |  7  244  1  \     / 1 0 0 |  82  82  82 \
| 0  1   0 | 121 247 246 | --> | 0 1 0 | 121 247 246 |
\ 0  0   1 | 244 247 246 /     \ 0 0 1 | 244 247 246 /.

Since the left side of A is the identity matrix, the right side of A is M^-1. Therefore,

         /  82  82  82 \
M^{-1} = | 121 247 246 |
         \ 244 247 246 /.

Finally, we can use that to implement ReconstructData.

Example 15: `ReconstructData` in detail

Let d_partial = [ ??, db, ?? ] be the input partial data bytes and p_partial = [ 52, 0c ] be the input partial parity bytes. Then, with the data byte count n = 3 and the parity byte count m = 2, and appending the rows of the m × n Cauchy parity matrix to the n × n identity matrix, we get

/ X01X X00X X00X \
|  00   01   00  |
| X00X X00X X01X |
|  f6   8d   01  |
\  cb   52   7b  /,

where the rows corresponding to the unknown data and parity bytes are crossed out. Taking the first n rows that aren’t crossed out, we get the square matrix

/ 00 01 00 \
| f6 8d 01 |
\ cb 52 7b /

which has inverse

/ 01 d0 d6 \
| 01 00 00 |
\ 7b b8 bb /.

Therefore, the data bytes are reconstructed from the first n known data and parity bytes as

                _    _     _    _
/ 01 d0 d6 \   |  db  |   |  da  |
| 01 00 00 | * |  52  | = |  db  |
\ 7b b8 bb /   |_ 0c _|   |_ 0d _|,

and thus the output data bytes are d = [ da, db, 0d ].

And we’re done!

12. Further reading

Next time we’ll talk about the PAR1 file format, which is a practical implementation of an erasure code very similar to the one described above, and the various challenges to make it perform well on sets of large files.

Also, for those of you interested in the mathematical details, I’ll also write a companion article. (This article is already quite long!)

I gave a 15-minute presentation for WaffleJS covering the same topics as this article but at a higher-level and more informally.

I got the idea for explaining the finite field with $256$ elements in terms of binary carry-less arithmetic from A Painless Guide to CRC Error Detection Algorithms, which is an excellent document in its own right.

Most sources below use Vandermonde matrices, which I plan to cover in the next article on PAR1, instead of Cauchy matrices. Cauchy matrices are more foolproof, which is why I started with them. templexxx, whose Go implementation I cite below, feels the same way. (His blog post is in Chinese, but using Google Translate or a similar service translates it well enough to English.)

I started learning about erasure codes from James Plank’s papers. See A Tutorial on Reed-Solomon Coding for Fault-Tolerance in RAID-like systems, but also make sure to read the very important correction to it! Optimizing Cauchy Reed-Solomon Codes for Fault-Tolerant Storage Applications covers Cauchy matrices, although in a slightly different context. The first part of Plank’s All About Erasure Codes slides also contains a good overview of the encoding/decoding process, including a nifty color-coded matrix diagram.

As for implementations, klauspost and templexxx have good ones written in Go. They were in turn inspired by Backblaze’s Java implementation. Backblaze’s accompanying blog post is also a good overview of the topic. The toy JS implementation powering the demos on this page are also available on my GitHub.

An Introduction to Galois Fields and Reed-Solomon Coding^[13] covers much of the same material as I do, albeit assuming slightly more mathematical background.

Going further afield, Russ Cox, Jeremy Kun, and Nayuki also wrote about finite fields and Reed-Solomon codes.

Thanks to Ying-zong Huang, Ryan Hitchman, Charles Ellis, and Josh Gao for comments/corrections/discussion.

Like this post? Subscribe to my feed or follow me on Twitter .

Footnotes

[1] This discussion of linear algebra is necessarily abbreviated for our purposes. For a more general but still basic treatment, see Khan Academy. ↩

[2] Here and throughout this document, I index vectors and matrices starting with $0$, to better match array indices in code. Most math texts index vectors and matrices starting at $1$. ↩

[3] Now would be a good time to talk about the conventions I and other texts use. Following Plank, I use $n$ for the data byte count and $m$ for the parity byte count, and I represent arrays and vectors as column vectors, where multiplication with a matrix is done with the column vector on the right, which is the standard in most of math. However, in coding theory, $k$ is used for the data byte count, which they call the message length, and $n$ is used for the sum of the data and parity byte counts, which they call the codeword length. Furthermore, contrary to the rest of math, coding theory treats arrays and vectors as row vectors, where multiplication with a matrix is done with the row vector on the left, and the matrix used would be the transpose of the matrix that would be used with a column vector. ↩

[4] Khan Academy has a video stepping through an example for a $3 \times 3$ matrix. ↩

[5] People with experience in coding theory might recognize that a parity matrix $P$ being optimal is equivalent to the corresponding erasure code being MDS. ↩

[6] An equivalent statement which is easier to see is that if a row could be expressed as a linear combination of other rows, then one would be able to construct a non-empty square submatrix of $P$ with those rows, which would then be non-invertible. ↩

[7] It is instead a (transposed) Vandermonde matrix, which we’ll cover when we talk about the PAR1 file format in a follow-up article. ↩

[8] People with experience in abstract algebra might recognize this as arithmetic over $\mathbb{F}_2[x]$, the polynomials with coefficients in the finite field with $2$ elements. ↩

[9] Our use of $\clplus$, $\clminus$, $\clmul$, and $\cldiv$ to denote carry-less arithmetic clashes with our use of the same symbols to denote generic field operations. However, we’ll never need to talk about both at the same time, so whichever one we mean should be obvious in context. ↩

[10] This is a slightly stronger statement than Theorem 3. ↩

[11] People with experience in abstract algebra might recognize carry-less primes as irreducible elements of $\mathbb{F}_2[x]$. ↩

[12] Coincidentally, $283$ is also a regular prime number. Using another carry-less prime number $256 \le p \lt 512$ would also yield a field with $256$ elements, but is important to consistently use the same carry-less modulus; different carry-less moduli lead to fields with $256$ elements that are isomorphic, but not identical.

Borrowing notation from CRCs, the carry-less modulus is sometimes represented as a hexadecimal number with the leading digit (which is always $1$) omitted. For example, $283$ would be represented as $\mathtt{0x1b}$, and we can say that we’re using the field with $256$ elements defined by $\mathtt{0x1b}$. ↩

[13] Galois field is just another name for finite field. ↩

OKRs and 4DX

blog.harterrt.com — Thu, 30 Nov 2017 10:00:00 GMT

I feel like I'm swimming in acronyms these days.

Earlier this year, my team started using Objectives and Key Results (OKRs) for our planning. It's been a learning process. I had some prior experience with OKRs at Google, but I've never felt like I was fully taking advantage of the …

Recurse Center 14: OpenSSH key files

Pete's Points — Thu, 30 Nov 2017 03:05:34 GMT

So today (my 15th day of RC sabbatical) was where the rubber was supposed to hit the road. I was working on actually digging into the meaty bits of the OpenSSH private key PEM files and starting to peek into the base64 encoded data.

I did some research this morning and futzed around with both bash pipelines and rust code to start to understand how these files work.

First fun thing I found was looking at the raw data inside a PEM-encoded private key with this pipeline:

# strip out the header/footer lines (they have dashes)
grep -v - ./local/test_files/test-ssh-ed25519-2-passphrase.privatekey |
  # Decode the base64, this handles newlines no problem
  base64 -D |
  # dump that as hex
  hexdump
0000000 6f 70 65 6e 73 73 68 2d 6b 65 79 2d 76 31 00 00
0000010 00 00 0a 61 65 73 32 35 36 2d 63 62 63 00 00 00
0000020 06 62 63 72 79 70 74 00 00 00 18 00 00 00 10 d9
0000030 3b fe b4 54 93 c8 05 a7 77 21 7d 1c a5 69 1d 00
0000040 00 00 10 00 00 00 01 00 00 00 33 00 00 00 0b 73
0000050 73 68 2d 65 64 32 35 35 31 39 00 00 00 20 2e 40
0000060 78 7a 53 e2 89 d8 8f 54 9d 4b a3 3e 56 58 85 c7
0000070 1d 52 2e 7d 78 4e 9b e3 a5 89 1b 61 79 2a 00 00
0000080 00 90 68 c8 c3 da e6 6b cd d2 2f 4a 74 ad e9 7b
0000090 75 6a a6 8f d1 6d 24 86 fa 1e b1 77 25 0c e3 fa
00000a0 30 6d da 7c 72 11 be ac 98 0e b0 7b 7d 85 87 aa
00000b0 d1 49 c7 97 0a b4 5f 6e 61 5d 10 7c cd 55 c5 38
00000c0 45 e8 4c b8 2d 23 67 7b 23 18 97 06 7a 9d 0d bc
00000d0 69 d1 88 73 ba 34 8e 39 4e 50 01 cc 36 a7 c8 8b
00000e0 58 57 e9 c5 f4 b0 eb 79 ab 1b e6 64 a0 78 59 a6
00000f0 2b 05 03 00 57 bb 7b d4 19 ed 62 98 b1 db 4e 51
0000100 27 8e b7 7e f8 e2 42 d0 18 01 99 b6 c2 30 ad 9c
0000110 33 9e                                          
0000112

After some futzing I was sure I had the right data and extracted some telltale ASCII strings indicating I was definitely on the right track.

# remove the PEM header/footer dash lines
grep -v - ./local/test_files/test-ssh-ed25519-2-passphrase.privatekey |
  # Decode the base64
  base64 -D |
  # find stuff that looks like ASCII words
  strings

openssh-key-v1
aes256-cbc
bcrypt
ssh-ed25519
(some gobbledygook omitted)

Bingo! Now that I know some key words to look for, let me make a handy reference from ASCII to hex for these words.

for word in openssh-key-v1 aes256-cbc bcrypt ssh-ed25519 none
do
  echo -n "${word}: "
  echo -n "${word}" |
  hexdump |
  head -1 |
  cut -d " " -f 2-
done

openssh-key-v1: 6f 70 65 6e 73 73 68 2d 6b 65 79 2d 76 31
aes256-cbc: 61 65 73 32 35 36 2d 63 62 63
bcrypt: 62 63 72 79 70 74
ssh-ed25519: 73 73 68 2d 65 64 32 35 35 31 39
none: 6e 6f 6e 65

Nice! OK so I've been studying that and got some RC help with the openssh C source that generates it. Here's what I discovered about the byte-by-byte details of this openssh private key file.

14 bytes magic ASCII "openssh-key-v1"
- We can see our hex: 6f 70 65 6e 73 73 68 2d 6b 65 79 2d 76 31
1 NULL (zero) byte terminating that string
- This is a bit weird because there's a mix of null-terminated C strings and length-prefixed string structs, but that's what it is
4 byte length for next string
N bytes string cipher name
- "none" for no passphrase (cleartext), otherwise "aes256-cbc"
- We can see in our hex aes256-cbc 61 65 73 32 35 36 2d 63 62 63
4 byte length for next string
N bytes string kdfname (kdf means Key Derivation Function)
- We can see in our hex bcrypt 62 63 72 79 70 74
4 byte length for next string
N bytes kdfoptions
4 byte integer number of public keys in this file
one string for each public key
then an encrypted and padded blob for the private keys

Part of the struggle researching this is AFAICT there's no clear, googlable name for this file format. It's just the bespoke openssh-key-v1 format but nobody calls it that and there's no file extension.

So mostly I think I'm actually coding a command line tool in rust that will be useful to me now. I'm excited to keep pushing forward tomorrow.

Beginner interaction design with Principle

Daniel Immke — Thu, 30 Nov 2017 02:00:00 GMT

I was able to pick up this powerful motion design tool in about 10 minutes and you can too!

Advanced Oracle Troubleshooting seminar in 2018!

Tanel Poder Blog — Thu, 30 Nov 2017 00:24:06 GMT

A lot of people have asked me to do another run of my Oracle Troubleshooting training or at least get access to previous recordings – so I decided to geek out over the holiday period, update the material with latest stuff and run one more AOT class in 2018!

The online training will take place on 29 January – 2 February 2018 (Part 1) & 26 February – 2 March 2018 (Part 2).

Observations from Costa Rica, 2017

Avy Faingezicht — Wed, 29 Nov 2017 02:00:00 GMT

Every time I come back home to visit my family I notice how differently things run here in Costa Rica than in the US. There are some blaringly obvious contrasts, while others are details so subtle that it took me 7 years abroad to even consider. On this trip, I hung out with a lot of non-Costa Ricans (from the US and elsewhere) who came for my brother's wedding, so they kept asking me about peculiarities as they noticed them on their own. Seeing other people point out things that seemed odd to them made me become more aware myself. Here are a few of these observations: * Unlike in the US, retail in Costa Rica is thriving. The first shopping malls here started popping up in the early 90s, and the gold rush has not stopped. With rising standards of living, and a growing economy came purchasing power and the desire to buy into big name brands. People are eager to let go of their cash for generic status symbols. Compare this to the US and the rise of artisanal/bespoke/craft goods over homogeneous retail (see also Venkatesh Rao's Premium Mediocre). Interestingly, the Costa Rican retail boom is fully physical - ecommerce is nearly non-existent and people love going to the malls to do their shopping. Some possible explanations: while broadband penetration is high relative to our neighbors it is still pretty low, there is no robust logistics infrastructure (think UPS), and no company has tried to kickstart online shopping here. This is a large business opportunity that's up for grabs. * The ways that Costa Ricans use the apps on their smartphones are very different from the US. Everyone here uses Waze. Everyone. It is well known that our streets have no names and that people get around by reference points, but this time around I saw a lot of billboards and signs with variations of Waze's smiling car logo, and URLs to the Waze "pin" for a given store. Similarly, businesses use WhatsApp to communicate directly with their clients. We heard at least two ads on the radio prompting listeners to get in touch via WhatsApp. For example, when we rented a car to go to the beach the clerk insisted that we WhatsApp him copies of our documents, instead of sending them via email. And yet, everyone uses Uber to get around (hi, Alan!). * The press in Costa Rica is quite yellow, and there is little interest in long-form, in-depth local content, so the market doesn't supply it. We have no equivalent of the New Yorker or the Atlantic. There are opportunities for investigative journalism, with lots of cool stories waiting for someone to write. The stories are amazing, and the kind that the public in the US would be interested in. Hannah and I spent a few days in Puerto Viejo, and we talked a lot about the Afro-Caribbean culture and how different it feels from San José. For historic reasons, there is a much higher concentration of black people there than elsewhere in the country. People in the US would read about that. Everyone rides bikes there, which is unusual in the rest of the country, and we think may be due to an entrepreneur deciding to heavily market it to tourists. People in the US would read about that. We noticed thousands of dead puffer fish laying on the beach, and when we asked around, no one knew the real reason why (I could only find one article on it, and it was fairly weak). People in the US would read about that. We went to a restaurant owned by a guy who was a community activist, became a representative in the Costa Rica legislature, and is deeply tied to the town's development. People in the US would read about that. Notice that all these ideas come from 3 days on the Caribbean coast, a place I had never been to before. It sometimes takes an outsider to see there is a story to be told. I can imagine a study abroad program where journalism schools send people to live in Costa Rica for a summer and churn these out, arbitraging journalism. * A while back I had told Hannah that I find the number of American flags in cars, stores, and streets in the US kind of crazy, and that this kind of noisy nationalism bothers me. Turns out there are also flags all over Costa Rica, I just didn't really notice them until my girlfriend decided to start pointing out every one we passed. This was an odd realization of the power of indoctrination: I see the American flags plastered all over NYC, Chicago, or San Francisco as loud and obnoxious, but fail to notice the equivalent in Costa Rica. The longer I am away, the more I start to understand these subtle differences and catch the details that I totally ignored before. Coming back home makes me see my birthplace with new eyes. It is its own version of having a beginner's mind. What are some other oddities that you've noticed in your trips abroad? Let me know on Twitter.

Photo: by me, also posted on Puerto Viejo.

Puerto Viejo

Avy Faingezicht — Wed, 29 Nov 2017 02:00:00 GMT

I was just in Costa Rica for my brother's wedding. The trip was amazing, but somehow I only ended up taking photos when we went to the beach. It was my first time in the Caribbean side of the country, and we had a lot of fun. You can read some more in-depth commentary on the cultural aspect of my trip and some of the realizations I had during it here.

Recurse Center 13: ssh private key PEM files

Pete's Points — Wed, 29 Nov 2017 00:48:55 GMT

Today I used the rust pem crate to check whether a given file was or was not parseable as PEM encoded. PEM stands for Privacy Enhanced Mail. I learned that most ssh private keys are stored as PEM files and there are ED25519, ECDSA, RSA, and DSA types (maybe more but those are the ones I'm familiar with and read about a bit in the openssh source code). Still not clear on whether the public keys are in SSLEAY format or not. Also did some pairing and showed another RCer the rust ? operator.

rust custom Display and sorting

Pete's Points — Tue, 28 Nov 2017 23:16:34 GMT

So as an exercise I wanted to make a rust struct with custom order and blog the process. I did this in my tealeaves project to get some data to group/sort by severity (errors then warnings then infos) and it worked nicely but it was mostly the derived implementations and I wanted to review. So let's make a basic Person struct (very similar to the example in the official rust docs, but we'll get fancier).

struct Person {
    name: String,
}

fn main() {
    println!("{}", Person { name: "Sheena".to_string() });
}

"No can do!" says the compiler:

error[E0277]: the trait bound `Person: std::fmt::Display` is not satisfied
 --> src/main.rs:6:20
  |
6 |     println!("{}", Person { name: "Sheena".to_string() });
  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `Person` cannot be formatted with the default formatter; try using `:?` instead if you are using a format string
  |
  = help: the trait `std::fmt::Display` is not implemented for `Person`
  = note: required by `std::fmt::Display::fmt`

error: aborting due to previous error

OK, let's make Person displayable.

use std::fmt;

struct Person {
    name: String,
}

impl fmt::Display for Person {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "{}", self.name)
    }
}

fn main() {
    println!("{}", Person { name: "Sheena".to_string() });
}

We run that and it prints "Sheena". Pretty neat!

OK but now I wanted to make my Person struct sortable by age. As is often the case in rust, one part of this was way harder than in most languages and another part was also way harder than in other languages :-p.

First, we need a way to compute a person's age based on their birthday, so I extended my Person struct to also store a born property, but in order to model that as a calendar date, we need the chrono crate and a while reading the docs to figure out what type we should use. After my research, I decided chrono::NaiveDate would be viable.

extern crate chrono;
use chrono::NaiveDate;
use std::fmt;

struct Person {
    name: String,
    born: NaiveDate,

}

impl fmt::Display for Person {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "{}", self.name)
    }
}

fn main() {
    println!("{}", Person { name: "Sheena".to_string(), born: NaiveDate::from_ymd(1970, 1, 17) });
}

OK now I wanted a more concise way to define instances of Person, so I added a constructor function.

extern crate chrono;
use chrono::NaiveDate;
use std::fmt;

struct Person {
    name: String,
    born: NaiveDate,
}

impl Person {
    pub fn new(name: String, year: i32, month: u32, day: u32) -> Self {
        Person {
            name,
            born: NaiveDate::from_ymd(year, month, day),
        }
    }
}

impl fmt::Display for Person {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "{}", self.name)
    }
}

fn main() {
    println!("{}", Person::new("Sheena".to_string(), 1970, 1, 17));
}

Next let's write an age function so we can see how old a person is at this moment. Again I had to read the docs for a long time, and there may be a better way to do this, but my basic approach ended up being get a Duration between now and their birthday, convert that to a number of days, and divide by 365 to get something approximately yearish. While we're at it we'll update our Display trait to print their age.

extern crate chrono;
use chrono::NaiveDate;
use chrono::Local;
use std::fmt;

struct Person {
    name: String,
    born: NaiveDate,
}

impl Person {
    pub fn new(name: String, year: i32, month: u32, day: u32) -> Self {
        Person {
            name,
            born: NaiveDate::from_ymd(year, month, day),
        }
    }

    pub fn age(&self) -> i64 {
        NaiveDate::signed_duration_since(Local::today().naive_local(), self.born).num_days() / 365
    }
}

impl fmt::Display for Person {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "{} (age: {})", self.name, self.age())
    }
}

fn main() {
    println!("{}", Person::new("Sheena".to_string(), 1970, 1, 17));
}

OK now to make them sortable by age we need 4 total traits: Eq, PartialEq, Ord, and PartialOrd. Eq we can derive satisfactorily. The other three are mostly boilerplate we copy from the rust documentation and adjust to sort people based on their born attribute with youngest first.

extern crate chrono;
use chrono::{NaiveDate, Local};
use std::{cmp, fmt};

#[derive(Eq)]
struct Person {
    name: String,
    born: NaiveDate,
}

impl Person {
    pub fn new(name: String, year: i32, month: u32, day: u32) -> Self {
        Person {
            name,
            born: NaiveDate::from_ymd(year, month, day),
        }
    }

    pub fn age(&self) -> i64 {
        NaiveDate::signed_duration_since(Local::today().naive_local(), self.born).num_days() / 365
    }
}

impl PartialOrd for Person {
    fn partial_cmp(&self, other: &Person) -> Option<cmp::Ordering> {
        Some(other.cmp(self))
    }
}

impl Ord for Person {
    fn cmp(&self, other: &Person) -> cmp::Ordering {
        self.born.cmp(&other.born)
    }
}

impl PartialEq for Person {
    fn eq(&self, other: &Person) -> bool {
        self.born == other.born
    }
}

impl fmt::Display for Person {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "{} (age: {})", self.name, self.age())
    }
}

fn main() {
    let mut people = vec![];
    people.push(Person::new("Imogen Heap".to_string(), 1977, 12, 9));
    people.push(Person::new("Fatboy Slim".to_string(), 1963, 7, 31));
    people.push(Person::new("Weird Al".to_string(), 1959, 10, 23));
    people.push(Person::new("Zoë Keating".to_string(), 1972, 2, 2));
    people.sort();
    for person in people.iter() {
        println!("{}", person);
    }
}

So now we run this and we see a nice display format for people and they are sorted youngest first:

Imogen Heap (age: 39)
Zoë Keating (age: 45)
Fatboy Slim (age: 54)
Weird Al (age: 58)

The Poor Man's 3D Camera

Evan Todd — Tue, 28 Nov 2017 10:00:00 GMT

Each of us have our own giants to face. This is a story about one of my giants. Something I never imagined could make a grown man cry, until it did.

A 3D camera.

No one can face your giants for you. This is a story, not a walkthrough. Expect no useful information. For that I recommend 50 Game Camera Mistakes by thatgamecompany's John Nesky. His job title is literally "camera designer".

Firefox 57 CPU usage is insane

Muffin Man — Tue, 28 Nov 2017 02:00:00 GMT

This is a sad rant :(

I love Firefox (and Mozilla too) and have never switched to ChromeAlthough I use Chrome for development. But few days ago I switched to Opera, because new Firefox's (aka Quantum) CPU usage is insane.

Disabling extensions didn't help, it just hogs all available resources. MacOS is giving me "Service Battery" warning when Firefox is running. And battery last about four times(!) shorter.

So I checked my options, and decided to go with Opera. One of my favorite features in Firefox, tabs on the side, is not supported in Chrome and Safari, so only decent browser left was Opera.

Opera has Tree Tabs extension, which is not as good as Tree Style Tab but it is good enough. I'm getting used to it, and so far so good. Although I do hope Firefox will resolve CPU issues.

If they haven't hyped it so much, I probably wouldn't be writing this. But they raised my expectations, how much faster and better it is, how they finally caught up... and it is bad, like really bad. I've tried both beta and developer editions some time ago, and fans were going crazy, CPU usage was going through the roof, battery drainage... But I hoped that it will get resolved before they release a stable version.

I'm really sad, but if they don't fix this I won't be switching back to Firefox. Still, I'm going to continue supporting and donating to Mozilla. On the plus side, MDN is one of the best sites for web development resources.

Update 02. Decemeber 2017

It seems that guys from Mozilla are aware of it and that they are trying to fix it. Hooray!

Update 04. December 2018

Unfortunately, even today with version 63.0.3 issue doesn't seem to be resolved. I'm sticking with Opera, but I switched to Simple Vertical Tabs extension.

Update 26. September 2019

It is fixed!

Recurse Center 12: AsRef

Pete's Points — Tue, 28 Nov 2017 01:10:25 GMT

I spent a long time today trying to understand how the AsRef trait works. I failed.

I was able to split my tealeaves project code out into a handful of files and more strongly model the filesystem checks as Enums in the type system. If you are printing filesystem permission octals, don't forget the {:o} format string.

I find if I battle the compiler too long or just in general struggle to understand something, it drains me really quickly. No dopamine hits from little victories. This is one of the reasons I like unit testing - lots of little frequent victories and most of the time no time debugging weird production issues whilst praying for the sweet kiss of death.

Software Interview Questions

TOGoS's Project Log — Mon, 27 Nov 2017 22:58:11 GMT

Some kid was assigned to ask people who worked in the software industry questions about it and I ended up being the one to answer them.

Running .NET on the Raspberry Pi

Daniel Little Dev — Mon, 27 Nov 2017 06:59:57 GMT

I recently got my hands on an older Raspberry Pi, which was a great excuse to see if I could get .NET running on it! If you're interested in…

Blog redesign

Muffin Man — Mon, 27 Nov 2017 02:00:00 GMT

Muffin Man is back! This time with new visual identity.

This was long overdue, but new design took me more than I wanted to.Mostly because of the trillion things I'm working on. Now I can get back to writing.

Alogn with the new design, some other things got updated too, hopefully making content easier to browse. To name a few:

Archive is now one page with category accordion (every category used to have a dedicated page)
You can now find related posts under each post (instead of the latest ones)
Search results are now navigable using arrows

I ditched custom Jekyll plugins to be able to use GitHub to automatically build the whole thing. Some of them got rewritten to Liquid.

Hope you like it!

Adding Blog Here

Ben Overmyer's Site — Mon, 27 Nov 2017 02:00:00 GMT

I'm starting to use Hugo for blogging now. I might move all of the content currently on Pathfinder over to here, if it works as well as I hope it does.

Also, I'm now hosting this site on Netlify, instead of deploying it myself to a VPS. This makes things much easier from a content management perspective.

Google Code-in 2017

kyrofa's blog — Mon, 27 Nov 2017 02:00:00 GMT

Tomorrow, it begins. The culmination of weeks of research, documentation, and preparation by multiple open-source projects, just waiting to start hacking away with students. Tomorrow, students aged 13-17 can begin participating in Google Code-in 2017. Ubuntu is one of the mentoring organizations, with several of my colleagues and me as mentors. Just in case you haven’t heard of it, Ubuntu is an open-source operating system for servers, desktops, and embedded devices.

Red-letter theology

Honza Pokorný — Sun, 26 Nov 2017 19:00:00 GMT

Several weeks ago, I had a few friends over to our house to watch Les Lanphere’s excellent Calvinist documentary. I assumed this would be their first exposure to Reformed theology, so I expected some lively discussion. About half way through the movie, we had to pause to answer a simple clarification question. Once that was done, one of the guys volunteered an interesting observation. He said: “I hope they show some quotes from Jesus, too. So far it’s just been Paul in Romans and Ephesians. I wonder what Jesus has to say about that.”

This little comment has been on my mind ever since. The traditional typography of Bibles on the North American usually includes words of Christ in red. Having grown up in Europe, I didn’t encounter this until I came to Canada about ten years ago. The motivation behind this typographic design is to give reverence to the words of Christ because when he speaks, God himself speaks. Unfortunately, what this leads to is that people treat other parts of the Bible as less important. If it’s not red, it’s as if you don’t really have to believe it. This is a really bad way of reading the Bible and doing theology in general.

A small typographic detail can make a huge difference in how we think about what we’re reading. It might be helpful for us to stop and think about the features of our Bibles to see how all the references, footnotes, verse numbers, section headings, verse-by-verse layouts affect our reading and understanding of the text of Scripture.

Related: Bible Typography Manifesto

Twitter Plays Zork: @DemocraticZork

Thomas Hunter II — Sun, 26 Nov 2017 02:00:00 GMT

Ready Player One

Krishna's blog — Sun, 26 Nov 2017 02:00:00 GMT

Ready Player One is a fast-paced book filled with 80s references, but falls short with one-dimensional characters and a lack of new ideas.

THE big question about the Internet of Things is...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 25 Nov 2017 06:00:00 GMT

There has recently been a MIT workshop on “Low-Latency Wireless Random-Access” which, from a report I’ve just seen, may have been focused on the wrong questions about the Internet of Things.

Comparing editions of the Institutes

Honza Pokorný — Fri, 24 Nov 2017 19:00:00 GMT

The way I see it, you have three options when selection an edition of John Calvin’s Institutes of the Christian Religion.

The final edition of 1559 translated by Henry Beveridge, published in 1867.
The final edition of 1559 translated by Ford Lewis Battles, published in 1960.
The French edition of 1541 translated by Robert White, published in 2014.

Beveridge

With Beveridge, you get the full text of the Institutes. The language is somewhat archaic and a bit convoluted sometimes. It’s a lot easier to read than the Puritans. This translation is in the public domain, so you can read it for free online, and printed editions are relatively cheap.

Book 1, Chapter 1, Section 2

On the other hand, it is evident that man never attains to a true self-knowledge until he have previously contemplated the face of God, and come down after such contemplation to look into himself. For (such is our innate pride) we always seem to ourselves just, and upright, and wise, and holy, until we are convinced, by clear evidence, of our injustice, vileness, folly, and impurity. Convinced, however, we are not, if we look to ourselves only, and not to the Lord also—he being the only standard by the application of which this conviction can be produced. For, since we are all naturally prone to hypocrisy, any empty semblance of righteousness is quite enough to satisfy us instead of righteousness itself. And since nothing appears within us or around us that is not tainted with very great impurity, so long as we keep our mind within the confines of human pollution, anything which is in some small degree less defiled delights us as if it were most pure: just as an eye, to which nothing but black had been previously presented, deems an object of a whitish, or even of a brownish hue, to be perfectly white. No, the bodily sense may furnish a still stronger illustration of the extent to which we are deluded in estimating the powers of the mind.

If, at mid-day, we either look down to the ground, or on the surrounding objects which lie open to our view, we think ourselves endued with a very strong and piercing eyesight; but when we look up to the sun, and gaze at it unveiled, the sight which did excellently well for the earth is instantly so dazzled and confounded by the refulgence, as to oblige us to confess that our acuteness in discerning terrestrial objects is mere dimness when applied to the sun. Thus, too, it happens in estimating our spiritual qualities. So long as we do not look beyond the earth, we are quite pleased with our own righteousness, wisdom, and virtue; we address ourselves in the most flattering terms, and seem only less than demigods.

Battles

With Battles, you get the full text in its final form. This is the academic reference edition. If you are going to write a paper or a book, and you want to quote Calvin, use this. The language is much more modern than Beveridge but still a bit technical at times. The biggest downside of this edition is that it comes in two volumes and it’s very expensive.

Book 1, Chapter 1, Section 2

Again, it is certain that man never achieves a clear knowledge of himself unless he has first looked upon God’s face, and then descends from contemplating him to scrutinize himself. For we always seem to ourselves righteous and upright and wise and holy—this pride is innate in all of us—unless by clear proofs we stand convinced of our own unrighteousness, foulness, folly, and impurity. Moreover, we are not thus convinced if we look merely to ourselves and not also to the Lord, who is the sole standard by which this judgment must be measured. For, because all of us are inclined by nature to hypocrisy, a kind of empty image of righteousness in place of righteousness itself abundantly satisfies us. And because nothing appears within or around us that has not been contaminated by great immorality, what is a little less vile pleases us as a thing most pure—so long as we confine our minds within the limits of human corruption. Just so, an eye to which nothing is shown but black objects judges something dirty white or even rather darkly mottle to be whiteness itself. Indeed, we can discern still more clearly from the bodily senses how much we are deluded in estimating the powers of the soul.

For if in broad daylight we either look down upon the ground or survey whatever meets our view round about, we seem to ourselves endowed with the strongest and keenest sight; yet when we look up to the sun and gaze straight at it, that power of sight which was particularly strong on earth is at once blunted and confused by a great brilliance, and thus we are compelled to admin that our keenness in looking upon things earthly is sheer dullness when it comes to the sun. So it happens in estimating our spiritual goods. As long as we do not look beyond the earth, being quite content with our own righteousness, wisdom and virtue, we flatter ourselves most sweetly, and fancy ourselves all bug demigods.

White

With White, you get an earlier version of the Institutes. It’s missing some sections from the final, but not much. This is translated from the French edition which is known for its lively prose. The biggest draw for this edition is its recency, only 2014. The language is very modern and flows nicely. The printed edition costs a bit more than Beverage but still a lot less than Battles. This edition will serve hobbyists well. The newness of the language will provide you with higher likelihood of finishing.

Book 1, Chapter 1, Section 2

Conversely, we observe that no one ever attains clear knowledge of self unless he has first gazed upon the face of the Lord, and then turns back to look upon himself. Deeply rooted in all of us is an arrogance which persuades us that we are righteous, truthful, wise and holy. Only clear evidence that we are unrighteous, deceitful, foolish and vile will convince us of the contrary. We feel no such conviction if all we do is look upon ourselves and not also upon the Lord. He is the one and only standard with which our judgment must accord. But because hypocrisy is something to which we are all naturally prone, we are quite content with an empty show of righteousness rather than with its reality. And because there is nothing around us which is not greatly defiled, whatever is a little less grubby appears to us as purity itself, as long as we confine our attention to the limits of our own—debased—humanity. It is like the eye which, used to seeing only objects that are dark, judges things which are vaguely white or even semi-grey to be the whitest there is. An analogy based on physical sight will help us better understand how badly we misjudge our soul’s powers.

If in broad daylight we look down at the ground or attend to things which are round about us, we have no trouble believing our sight is extremely sharp and keen. When, however, we look straight up at the sun, the power that served us so well on earth is dazed and dazzled by so intense a light, forcing us to admit that our ability clearly to see earthly objects is weak and feeble when it comes to gazing at the sun. This is how it is when we try to estimate our spiritual strengths. As long as we do not look beyond earth’s horizons, we are perfectly content with our own righteousness, wisdom and power. We flatter and congratulate ourselves, and are not far from thinking we are demigods!

Conclusion

In conclusion, let me quote J.I. Packer on the subject of English translations of Calvin’s Institutes:

No English translation fully matches Calvin’s Latin; that of the Elizabethan, Thomas Norton, perhaps gets closest; Beveridge gives us Calvin’s feistiness but not always his precision; Battles gives us the precision but not always the punchiness, and fleetness of foot; Allen is smooth and clear, but low-key.

David W. Hall and Peter A. Lillback, eds. A Theological Guide to Calvin’s Institutes: Essays and Analysis. The Calvin 500 Series. (Phillipsburg, N.J: P&R Pub, 2008), x

Bio

John Jacobsen — Fri, 24 Nov 2017 02:00:00 GMT

I grew up¹[some old pictures of me]]] in Madison, Wisconsin and studied art and physics at the University of Wisconsin — Madison, receiving my doctorate in physics there in 1996 under Prof. Francis Halzen. I also worked for a year at CERN in Geneva, Switzerland, under Prof. Sau Lan Wu on the ALEPH experiment.

My doctoral work and postdoc focused on the AMANDA neutrino detector constructed at the Geographic South Pole, and the subsequent, much larger IceCube Neutrino Observatory. My physics career is described in more depth in my Curriculum Vitae; the IceCube project is described quite engagingly and at length by Mark Bowen in his book, The Telescope in the Ice: Inventing a New Astronomy at the South Pole.

While in graduate school I also studied painting, drawing and printmaking in the UW-Madison graduate art program, mostly with Jack Damer, Richard Long and David H. Becker.

In the 21st century I have been a software engineer and painter. I travelled to the Geographic South Pole ten times for AMANDA and IceCube, going at first as a physicist, and then as a software consultant (blog posts here). Some of my early paintings were described in William Fox's book, Terra Antarctica: Looking Into the Emptiest Continent.

Since 2015 I have maintained a regular studio practice focusing on figurative art based on observation and imagination. This effort has involved some re-training, mostly in the form of workshops at the Palette and Chisel Academy of Fine Art with Andrew S. Conklin, Stephen Assael, and Larry Paulsen. My exhibition history is here, while recent work and work in progress are shown on my Instagram and Tumblr feeds.

Just for fun, here are [[./more-pix-of-me.html

Links - November 24, 2017

Avy Faingezicht — Fri, 24 Nov 2017 02:00:00 GMT

This was in drafts for a while, so posting without comments.

Quick fix: MongoDB fails when saving Decimal fields

Nicola Iarocci — Thu, 23 Nov 2017 12:05:28 GMT

So today I was trying to store some decimal fields on a MongoDB 3.4 instance and I kept failing, miserably.

In Compass I was getting a somewhat obscure “connection lost” message on every save attempt. Upgrading to Mongo 3.4.10 (I was on 3.4.4) improved things, as the error message was now hinting at the setFeatureCompatibilityVersion setting which, apparently, was preventing some features to be made available.

Now that made sense, as this specific database had been upgraded all the way from up Mongo 2.6. A quick lookup revealed that, when upgrading to 3.4, setFeatureCompatibilityVersion is automatically set to "3.2". As we all know, support for Decimal type only comes with Mongo 3.4.

Blog CD Pipeline with AWS CodePipeline

Brian's Waste of Time — Thu, 23 Nov 2017 03:09:31 GMT

Jumped out of order from my earlier checklist and set up some automagic build and deploy. I'd wanted an excuse to try out CodePipeline, so this was it!

So, how does this blog work? It is deployed to an S3 bucket (skife.org) with CloudFront in front of it. CloudFront is set up to use the free SNI certs to provide TLS. Previously, I pushed manually via s3cmd, which worked well with some incantation fiddling.

I won't write a full CodeBuild and CodeDeploy tutorial, Amazon has that well covered, but a couple bits were funny to work out, so will talk about those.

First, CodePipeline needs to trigger things. This is important as CodeBuild has no mechanism (which I could find) to only care about particular branches. CodeBuild really just does builds (kind of). Conceptually, do everything through CodePipeline and other stuff is just steps which react to the pipeline.

Given this is a static site, the build step just builds a tarball:

version: 0.2

phases:
  install:
    commands:
      - wget https://github.com/gohugoio/hugo/releases/download/v0.31/hugo_0.31_Linux-64bit.deb
      - dpkg -i ./hugo_0.31_Linux-64bit.deb
  build:
    commands:
      - hugo
      - tar -C public -cvzf skife.org.tgz .
artifacts:
  files:
    - skife.org.tgz

Nothing fancy here, but having the artifact for CodePipeline to pass around is important.

My first pass just had the deploy at the end of the build, but I want to be able to insert some basic tests before I deploy new versions. Just things like link verification, HTML5 validation, and maybe running stuff like Lighthouse against a test instance before letting it out. The test part :-) Because of this, I wanted to seperate build from deploy. It turns out "deploy by copying into an S3 bucket" is not a thing CodeDeploy has any concept for.

So my "deploy" is just another CodeBuild build:

version: 0.2
phases:
  build:
    commands:
      - tar -xf skife.org.tgz
      - rm skife.org.tgz
      - aws s3 sync . s3://skife.org --acl public-read --cache-control public,max-age=600

You can feed one build to another, it doesn't mind. When I configured the second build I had to set it up with an output artifact or CodePipeline wouldn't let me add it. After I saved the CodePipeline changes, I could go back and remove that output. The other decent path is probably to set up a Lambda function that takes apart the tarball and copies things over... but this build approach seems simpler.

I tried to put a cloudfront invalidation into the last step as well, but the version of the aws cli on the build image is old and it is not supported. I'll sort that out later. Once I do, will change max-age to max-age=31536000 or so and add something like:

- aws cloudfront create-invalidation --distribution-id E1DTTO3T6ZPN9M --paths / /index.html /404.html /archive.html /index.xml

to the build commands, and voila!

Six Easy Pieces

Krishna's blog — Thu, 23 Nov 2017 02:00:00 GMT

Six Easy Pieces by Richard Feynman is a fun read that provides an intuitive grasp of physics concepts, but falls short of teaching techniques.

Notes from Priceonomics' 2017 Content Marketing Conference

Coding VC — Wed, 22 Nov 2017 07:06:00 GMT

I’ve been a fan of Priceonomics for a long time.

QEFs, Eigenvalues, and Normals

Matt Keeter — Wed, 22 Nov 2017 02:00:00 GMT

Recurse Center 11: rsfs rust associated types

Pete's Points — Wed, 22 Nov 2017 01:01:17 GMT

So I'm working on a tool that will interact heavily with the filesystem. This has always presented a unit testing problem both from a test speed perspective but as well as the perspective of simulating unusual errors.

Yesterday I shopped on crates.io and found the filesystem crate which looked to be built exactly to what I needed. However, I quickly discovered it was missing some key things like exposing a file's size without having to read it and getting detailed permissions.

Today I shopped again and found rsfs which had the detailed unix stuff I needed. In theory I want to be able to use the real filesystem when my tool is run for real and a throw-away in-memory filesystem for unit tests (including being able to simulate rare error cases). In my mind this meant passing in some abstract filesystem interface to the guts of my module as a function argument. So I set out to learn the basics of the rsfs crate.

Here's the first thing that prints out the permissions mode in octal of a real file that exists on my filesystem.

extern crate rsfs;
use rsfs::GenFS; // trait gives us .metadata()
use rsfs::Metadata; // trait gives us .permissions()
use rsfs::unix_ext::PermissionsExt; // trait gives us .mode()

fn main() {
    let fs  = rsfs::disk::FS;
    let meta = fs.metadata("/tmp/foo.txt").unwrap();
    let perms = meta.permissions();
    let mode = perms.mode();
    println!("{:o}", mode);
}

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/fstool`
644

$ chmod 755 /tmp/foo.txt

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/fstool`
755

OK, so far so good. Now, can we simulate that with an in-memory filesystem?

extern crate rsfs;
use rsfs::GenFS; // trait gives us .metadata()
use rsfs::Metadata; // trait gives us .permissions()
use rsfs::unix_ext::PermissionsExt; // trait gives us .mode()

fn main() {
    let fs  = rsfs::mem::unix::FS::new();
    fs.create_dir_all("/tmp").unwrap();
    fs.create_file("/tmp/foo.txt").unwrap();
    let meta = fs.metadata("/tmp/foo.txt").unwrap();
    let perms = meta.permissions();
    let mode = perms.mode();
    println!("{:o}", mode);
}

// fn main_disk() {
//     let fs  = rsfs::disk::FS;
//     let meta = fs.metadata("/tmp/foo.txt").unwrap();
//     let perms = meta.permissions();
//     let mode = perms.mode();
//     println!("{:o}", mode);
// }

$ cargo run
   Compiling fstool v0.1.0 (file:///private/tmp/x)
    Finished dev [unoptimized + debuginfo] target(s) in 1.11 secs
     Running `target/debug/fstool`
666

Yup, looks acceptable.

OK let's try to extract a helper function that works on either. My first thought would be the filesystem argument would be of type rsfs::GenFS which is the "generic filesystem" trait whose whole purpose is to be agnostic about which specific filesystem implementation is used.

extern crate rsfs;
use rsfs::GenFS; // trait gives us .metadata()
use rsfs::Metadata; // trait gives us .permissions()
use rsfs::unix_ext::PermissionsExt; // trait gives us .mode()

fn get_mode(fs: &GenFS, path: &str) -> u32 {
    let meta = fs.metadata(path).unwrap();
    let perms = meta.permissions();
    let mode = perms.mode();
    mode
}

fn main() {
    let mem = rsfs::mem::unix::FS::new();
    mem.create_dir_all("/tmp").unwrap();
    mem.create_file("/tmp/foo.txt").unwrap();

    let disk = rsfs::disk::FS;

    println!("{:o}", get_mode(mem, "/tmp/foo.txt"));
    println!("{:o}", get_mode(disk, "/tmp/foo.txt"));
}

$ cargo run
   Compiling fstool v0.1.0 (file:///private/tmp/x)
error[E0191]: the value of the associated type `DirBuilder` (from the trait `rsfs::GenFS`) must be specified
 --> src/main.rs:6:18
  |
6 | fn get_mode(fs: &GenFS, path: &str) -> u32 {
  |                  ^^^^^ missing associated type `DirBuilder` value

error[E0191]: the value of the associated type `Permissions` (from the trait `rsfs::GenFS`) must be specified
 --> src/main.rs:6:18
  |
6 | fn get_mode(fs: &GenFS, path: &str) -> u32 {
  |                  ^^^^^ missing associated type `Permissions` value

error[E0191]: the value of the associated type `ReadDir` (from the trait `rsfs::GenFS`) must be specified
 --> src/main.rs:6:18
  |
6 | fn get_mode(fs: &GenFS, path: &str) -> u32 {
  |                  ^^^^^ missing associated type `ReadDir` value

error[E0191]: the value of the associated type `Metadata` (from the trait `rsfs::GenFS`) must be specified
 --> src/main.rs:6:18
  |
6 | fn get_mode(fs: &GenFS, path: &str) -> u32 {
  |                  ^^^^^ missing associated type `Metadata` value

error[E0191]: the value of the associated type `OpenOptions` (from the trait `rsfs::GenFS`) must be specified
 --> src/main.rs:6:18
  |
6 | fn get_mode(fs: &GenFS, path: &str) -> u32 {
  |                  ^^^^^ missing associated type `OpenOptions` value

error[E0191]: the value of the associated type `DirEntry` (from the trait `rsfs::GenFS`) must be specified
 --> src/main.rs:6:18
  |
6 | fn get_mode(fs: &GenFS, path: &str) -> u32 {
  |                  ^^^^^ missing associated type `DirEntry` value

error[E0191]: the value of the associated type `File` (from the trait `rsfs::GenFS`) must be specified
 --> src/main.rs:6:18
  |
6 | fn get_mode(fs: &GenFS, path: &str) -> u32 {
  |                  ^^^^^ missing associated type `File` value

error: aborting due to 7 previous errors

error: Could not compile `fstool`.

To learn more, run the command again with --verbose.

Damn, the compiler sure is mad about a bunch of associated types. OK let's try to specify the associated types, at least for the bits we think we really need.

We tried many variants and sadly I can't really walk you through the whole sequence of compiler error, code change, rinse repeat, but it was probably 19 iterations. We knew that basically when we called .metadata().permissions() we needed to tell the type system "Hey, use PermissionsExt there because that has the .mode() function we need and don't just give me back Permissions because there's no .mode() function there". But because that's nested underneath the Metadata associated type, we had to first say, OK for Metadata, we want a Permissions associated type that has all the combined functionality of both Permissions and PermissionsExt, which looks like Permissions + PermissionExt in trait sublanguage.

However, the syntax here doesn't let us write it exactly that way. We get an error the trait rsfs::Permissions cannot be made into an object. At this point I'm basically 2 levels below my depth here but a fellower RCer was pairing with me and knew that we basically needed to define that combination of associated types as P and then explain to the typesystem to use it both for our Metadata associated type and our GenFS type. That ultimately ends up looking like the next snippet. Even though the compiler initially complained about needing ALL the associated types specified, I think because we are only actually using APIs related to permissions, we got away with only specifying the associated types necessary for the API calls we actually use in our code.

extern crate rsfs;
use rsfs::GenFS; // trait gives us .metadata()
use rsfs::Metadata; // trait gives us .permissions()
use rsfs::unix_ext::PermissionsExt; // trait gives us .mode()
use rsfs::Permissions;

fn get_mode<P: Permissions + PermissionsExt,
            M: Metadata<Permissions = P>,
            F: GenFS<Permissions = P, Metadata = M>>
    (fs: &F,
     path: &str)
     -> u32 {
    let meta = fs.metadata(path).unwrap();
    let perms = meta.permissions();
    let mode = perms.mode();
    mode
}

fn main() {
    let mem = rsfs::mem::unix::FS::new();
    mem.create_dir_all("/tmp").unwrap();
    mem.create_file("/tmp/foo.txt").unwrap();

    let disk = rsfs::disk::FS;

    println!("{:o}", get_mode(&mem, "/tmp/foo.txt"));
    println!("{:o}", get_mode(&disk, "/tmp/foo.txt"));
}

$ cargo run
   Compiling fstool v0.1.0 (file:///private/tmp/x)
    Finished dev [unoptimized + debuginfo] target(s) in 1.5 secs
     Running `target/debug/fstool`
666
755

Success! We now have a business logic function that works identically on both an in-memory filesystem and the real filesystem. This is pretty exciting for me especially since the rsfs crate docs indicate you can synthesize error cases, which lights up my "100% code coverage" light bulb.

Avoiding The Infosec Extinction Part 1.

Privacy, Power, & Protection In The Cyber Century — Tue, 21 Nov 2017 15:00:00 GMT

800 words, 3 1/2 minutes. Making Your 1st Decision This is the first of a short series of posts about the Cyber Security market. This market is interesting now because I believe it’s at a juncture where we can choose one of two possible futures. We being the product builders, investors, and customers. The choice being whether to align ourselves with reality or fantasy. Plenty of markets perpetuate a degree of fantasy.

Recurse Center 10: filesystem palindrome

Pete's Points — Tue, 21 Nov 2017 02:05:33 GMT

This morning I worked on my "tealeaves" crypto diagnostic library. Just the basics of statting files but some interesting type system modeling and grokking what #[derive(PartialOrd)] does for enums and structs. I basically got rust to handle printing errors then warnings then OK messages via the sorting mechanisms.

I found a could-be-awesome crate called filesystem which gives a consistent interface for both an in-memory filesystem for fast unit tests as well as the real filesystem for production code. It's something I've always wanted. However, at the moment the API is not quite rich enough for what I want to do so I'm not sure it will work unless I enhance it.

Late in the day a group of us did a polyglot pairing exercise where we paired up to do some basic exercises in a programming language neither of us knew. I paired with JB in clojure and we implemented a program to find the longest palindrome in a sentence. I learned a bunch of stuff about clojure and it was a fun exercise.

How to Make a Churn Model in R

Luke Singham — Tue, 21 Nov 2017 02:00:00 GMT

The following post details how to make a churn model in R. It was part of an interview process for which a take home assignment was one of the stages. The company stated this should take 2hrs, which is entirely unrealistic. To minimise the time cost, my analysis is very succinct and short on the exploratory analysis and amount of models compared. The churn model got me to the final stage, however little in the way of feedback was offered. There is considerable debate in the tech industry as to whether take home exams are a fair assessment or even a reasonable ask on an individual. My assessment is that this is a lazy way to interview and a very high cost on the interviewee. If you really want to work for a particular company, then sure you might be prepared to do what it takes. I doubt I will do another take home assignment.

The analysis was done in Rmarkdown and the output copied into this blog post following these helpful pointers. If you would like to play with the data and full unabridged code please visit this github repo.

Aim #

Please create a model that predicts which businesses are likely to churn at the start of 2015 based on the vertical and incorporation_date.
We have an inkling that a dropoff in the number of mandates added might be an advance indicator of someone churning. Please can you assess whether this might be true, and if you think it is useful, incorporate it into your model from part (1).

# Load packages
suppressPackageStartupMessages({
    library(data.table)   # Fast I/O
    library(dplyr)        # Data munging
    library(tidyr)        # Data munging
    library(lubridate)    # Makes dates easy
    library(plotly)       # Interactive charts
    library(magrittr)     # pipe operators
    library(caret)        # Handy ML functions
    library(rpart)        # Decision Trees
    library(rpart.plot)   # Pretty tree plots
    library(ROCR)         # ML evaluation
    library(e1071)        # Misc stat fns
    library(randomForest) # rf
})

set.seed(42)

# Read data and drop row number column
df <- fread("monthly_data_(2)_(2).csv", drop = 1)

# Have a glimpse of the data
glimpse(df)

## Observations: 902
## Variables: 27
## $ company_id          <int> 4, 5, 6, 7, 8, 10, 11, 12, 13, 14...
## $ 2014-01-01_payments <dbl> 8, 0, 2, 3, 0, 0, 0, 0, 0, 0, 4, ...
## $ 2014-02-01_payments <dbl> 4, 0, 2, 3, 0, 0, 0, 2, 0, 11, 1,...
## $ 2014-03-01_payments <dbl> 7, 39, 1, 1, 6, 0, 0, 0, 8, 0, 1,...
## $ 2014-04-01_payments <dbl> 7, 0, 3, 7, 50, 1, 0, 0, 2, 0, 0,...
## $ 2014-05-01_payments <dbl> 1, 54, 1, 4, 119, 0, 1, 0, 0, 0, ...
## $ 2014-06-01_payments <dbl> 2, 0, 2, 1, 151, 3, 0, 0, 0, 0, 2...
## $ 2014-07-01_payments <dbl> 2, 0, 2, 7, 182, 0, 0, 0, 3, 0, 0...
## $ 2014-08-01_payments <dbl> 4, 22, 1, 2, 167, 0, 0, 0, 2, 0, ...
## $ 2014-09-01_payments <dbl> 3, 0, 1, 5, 180, 0, 0, 0, 0, 9, 5...
## $ 2014-10-01_payments <dbl> 5, 0, 2, 8, 157, 1, 0, 0, 0, 2, 1...
## $ 2014-11-01_payments <dbl> 5, 0, 1, 2, 105, 0, 0, 0, 0, 0, 0...
## $ 2014-12-01_payments <dbl> 9, 0, 3, 8, 57, 0, 0, 0, 0, 0, 0,...
## $ 2014-01-01_mandates <dbl> 0, 4, 0, 0, 0, 4, 4, 0, 0, 22, 0,...
## $ 2014-02-01_mandates <dbl> 0, 31, 1, 0, 2, 3, 8, 0, 0, 20, 1...
## $ 2014-03-01_mandates <dbl> 0, 24, 0, 0, 0, 5, 19, 0, 0, 11, ...
## $ 2014-04-01_mandates <dbl> 53, 18, 0, 1, 0, 0, 0, 0, 1, 11, ...
## $ 2014-05-01_mandates <dbl> 0, 8, 0, 0, 0, 0, 0, 0, 1, 15, 0,...
## $ 2014-06-01_mandates <dbl> 0, 7, 0, 0, 0, 0, 0, 0, 0, 13, 5,...
## $ 2014-07-01_mandates <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 13, 16...
## $ 2014-08-01_mandates <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 14, 8,...
## $ 2014-09-01_mandates <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 16, 2,...
## $ 2014-10-01_mandates <dbl> 0, 0, 0, 0, 1, 0, 0, 0, 0, 16, 1,...
## $ 2014-11-01_mandates <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 0, ...
## $ 2014-12-01_mandates <dbl> 0, 0, 0, 3, 0, 0, 0, 0, 0, 11, 0,...
## $ vertical            <chr> "gym/fitness", "freelance d...
## $ incorporation_date  <chr> "2003-09-25", "2008-10-22", ...

Data Munging #

Some data munging needs to occur for our binary classifiers to make use of the data within. This includes handling dates.

# Reshape data and create new columns
df %<>%
    gather(key = date, value = quantity, starts_with("20")) %>%
    separate(date, c("date","paymentMandate"), "_") %>%
    spread(paymentMandate, quantity) %>%
    mutate(
      incorporation_date = as.Date(incorporation_date),
      date = as.Date(date),
      incorporation_time = round(
        as.numeric(difftime(as.Date("2014-12-01"),
        as.Date(incorporation_date),
        unit="weeks")) / 52.25,
        digits = 1)
    ) %>%
    arrange(date)

# What does the new data look like?
glimpse(df)

## Observations: 10,824
## Variables: 7
## $ company_id         <int> 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14,...
## $ vertical           <chr> "gym/fitness", "gym/fitness", "freelance de...
## $ incorporation_date <date> 2013-05-30, 2003-09-25, 2008-10-22, 2005-0...
## $ date               <date> 2014-01-01, 2014-01-01, 2014-01-01, 2014-0...
## $ mandates           <dbl> 1, 0, 0, 0, 4, 0, 0, 0, 4, 4, 0, 0, 22, 0, ...
## $ payments           <dbl> 0, 1, 6, 8, 0, 2, 3, 0, 0, 0, 0, 0, 0, 4, 4...
## $ incorporation_time <dbl> 1.5, 11.2, 6.1, 9.4, 1.2, 9.1, 4.0, 2.9, 1....

What is Churn #

For the purposes of this project I have defined ‘churn’ as zero mandates and payments for the first instance of that after the last mandate/payment made.

# Create binary 'churn' column
df$churn <- 0

# For use in for loop - upper bound of data
max.date <- max(df$date)

# Mark all companies as churned in the month immediately their last activity
for (company in unique(df$company_id)) {

    # Subset data to company
    df.sub <- subset(df, df$company_id == company)

    # Index of last positive mandate OR payment
    last.pos.idx <- tail(which(df.sub$mandates != 0 | df.sub$payments != 0), 1)

    # Get date of last activity
    last.activity.date <- df.sub$date[last.pos.idx]

    # If less than max.date of dataset mark churn ELSE do nothing i.e. positive at end of period
    if (last.activity.date < max.date) {

        # Get churn date (last positive month plus 1mth)
        churn.date <- last.activity.date %m+% months(1)

        # Mark month of churn as 1
        df[df$date == churn.date & df$company_id == company, ]$churn <- 1
    }
}

# Multiple rows per company, filter for last month or churn month values...
# Get churners
df %>% filter(churn == 1) -> churners

# Get max date row of remainers (non-churners)
df %>%
    filter(churn == 0 & !(company_id %in% churners$company_id) & date == max(date)) -> remainers

# Combine and variables coded ready for modelling
churners %>%
    rbind(remainers) %>%
    mutate(vertical = as.factor(vertical),
           churn    = as.factor(churn)) -> model.df

Churns over the year #

# Plot churners
model.df %>%
    filter(churn == 1) %>%
    group_by(date) %>%
    summarise(n = n()) %>%
    plot_ly( x = ~date, y = ~n, type = 'scatter', mode = 'lines')

Could do more on exploratory but this is the easiest to cut back for this report.

Balance of the data #

The proportion of churn is 32.04%. Representing an imbalanced dataset. Accuracy is an inappropriate measure (I could get 67.96% accuracy predicting no businesses leave), so I will focus on recall and precision.

# Loyal vs Churn
table(model.df$churn)
##
##   0   1
## 613 289

Model #

Survival models and binary classifiers are common approaches to ‘Churn’ models. I will approach the model using the latter, though if I had more time I would investigate other classifiers and a survival model. I will limit the range of models to a logistic, a decision tree and an ensemble.

Split Data #

# 80/20 train test split
index    <- createDataPartition(model.df$churn, p = 0.8, list = FALSE)
train.df <- model.df[index, ]
test.df  <- model.df[-index, ]

# Check balance of the training split
table(train.df$churn)
##
##   0   1
## 491 232

# Check balance of the test split
table(test.df$churn)
##
##   0   1
## 122  57

Logistic #

# Run model
Logistic.model <- glm(churn ~ incorporation_time + vertical,
                      data   = train.df,
                      family = binomial(link = 'logit'))

# Predict
log.pred <- predict(Logistic.model, newdata = test.df, type = 'response')

# Convert probs to binary
log.pred <- as.factor(ifelse(log.pred > 0.5, 1, 0))

# Evaluation Metrics
log.result    <- confusionMatrix(data = log.pred, test.df$churn)
log.precision <- log.result$byClass['Pos Pred Value']
log.recall    <- log.result$byClass['Sensitivity']
log.F1        <- log.result$byClass['F1']

Decision Tree #

# Train model
tree.model <- rpart(churn ~ incorporation_time + vertical,
                    data = train.df,
                    method = "class",
                    control = rpart.control(xval = 10))

# Plot
rpart.plot(tree.model)

# Evaluation metrics
tree.pred      <- predict(tree.model, newdata = test.df, type = "class")
tree.result    <- confusionMatrix(data = tree.pred, test.df$churn)
tree.precision <- tree.result$byClass['Pos Pred Value']
tree.recall    <- tree.result$byClass['Sensitivity']
tree.F1        <- tree.result$byClass['F1']

Random Forest (Ensemble) #

# Train model
forest.model <- randomForest(churn ~ incorporation_time + vertical,
                       data = train.df,
                       ntree=200,
                       type="classification")

# See error reduction with number of trees ( not much gained beyond ~25 trees)
plot(forest.model)

# Look at the variable Importance from the random forest
varImpPlot(forest.model, sort = T, main="Variable Importance")

# Evaluation metrics
forest.pred      <- predict(forest.model, newdata = test.df, type = "class")
forest.result    <- confusionMatrix(data = forest.pred, test.df$churn)
forest.precision <- forest.result$byClass['Pos Pred Value']
forest.recall    <- forest.result$byClass['Sensitivity']
forest.F1        <- forest.result$byClass['F1']

Evaluation Metrics #

log.precision;
## Pos Pred Value
##      0.8333333
tree.precision;
## Pos Pred Value
##      0.8088235
forest.precision;
## Pos Pred Value
##      0.8134328

log.recall;
## Sensitivity
##   0.9016393
tree.recall;
## Sensitivity
##   0.9016393
forest.recall;
## Sensitivity
##   0.8934426

log.F1;
##        F1
## 0.8661417
tree.F1;
##        F1
## 0.8527132
forest.F1;
##        F1
## 0.8515625

Surprisingly, the logistic regression model performs the best, with the top precision score and equal recall score with that of the decision tree. With more time, I’d see if tweaks to the decision tree and random forest models could change this. It's also possible over/undersampling could help. From here on I will use the logistic regression model.

Examine the incorporation of time information #

Per the second aim, examine the inclusion of time information. This requires more data munging. Taking code used from above, I will extend to include the derivation of a very basic time period variable.

# Create binary 'leading_indicator' column
model.df$leading_indicator <- 0

# Min date for which a leading_indicator can be calculated (lower limit of data)
min.date <- min(df$date)

# If month before 'churn' (churn-1), is below the level of mandates of the month 2 months prior (churn-2) then
# make leading_indicator == 1
for (company in churners$company_id) {

    # Subset data to company
    df.sub <- subset(df, df$company_id == company)

    # Get month prior to churn
    month.prior <- df.sub$date[df.sub$churn == 1] %m-% months(1)

    # Get two months prior to churn
    two.month.prior <- df.sub$date[df.sub$churn == 1] %m-% months(2)

    # If two months prior is within dataset date range and level of mandates is greater than 0
    if ((two.month.prior > min.date) && (df.sub$mandates[df.sub$date == two.month.prior] > 0)) {

        # Compare number of mandates 1 month prior to 2 months prior, if less, mark 'leading_indicator' as '1'
        if (df.sub$mandates[df.sub$date == month.prior] < df.sub$mandates[df.sub$date == two.month.prior]) {
            model.df[model.df$company_id == company, ]$leading_indicator <- 1
        }
    }
}

Of the churners, how many have a leading indicator? #

model.df %>%
    filter(churn == 1) %>%
    group_by(leading_indicator) %>%
    summarise(n=n()) %>%
    mutate(percent = round(n / sum(n) * 100, 1))

## # A tibble: 2 x 3
##   leading_indicator     n percent
##               <dbl> <int>   <dbl>
## 1                 0   264    91.3
## 2                 1    25     8.7

Re-train model and evaluate #

# Re-split data so 'leading_indicator' is in columns (the index remains the same)
train.df <- model.df[index, ]
test.df  <- model.df[-index, ]

Logistic #

# Run model
lead.logistic.model <- glm(churn ~ incorporation_time + vertical + leading_indicator,
                      data   = train.df,
                      family = binomial(link = 'logit'))

# Predict
log.pred <- predict(lead.logistic.model, newdata = test.df, type = 'response')

# Convert probs to binary
log.pred <- as.factor(ifelse(log.pred > 0.5, 1, 0))

# Evaluation Metrics
lead.log.result    <- confusionMatrix(data = log.pred, test.df$churn)
lead.log.precision <- log.result$byClass['Pos Pred Value']
lead.log.recall    <- log.result$byClass['Sensitivity']
lead.log.F1        <- log.result$byClass['F1']

Evaluation Metrics #

Compare the precision and recall of the logistic model with and without the lead_indicator.

log.precision
## Pos Pred Value
##      0.8333333

lead.log.precision
## Pos Pred Value
##      0.8333333

log.recall
## Sensitivity
##   0.9016393

lead.log.recall
## Sensitivity
##   0.9016393

# Have a look at the model coefficients and p-values
summary(lead.logistic.model)

## Call:
## glm(formula = churn ~ incorporation_time + vertical + leading_indicator,
##     family = binomial(link = "logit"), data = train.df)
##
## Deviance Residuals:
##     Min       1Q   Median       3Q      Max  
## -1.4429  -0.8252  -0.4957   0.9938   2.5564  
##
## Coefficients:
##                              Estimate Std. Error z value Pr(>|z|)
## (Intercept)                   0.74066    0.20007   3.702 0.000214 ***
## incorporation_time           -0.26140    0.03045  -8.584  < 2e-16 ***
## verticalfreelance developer   0.12614    0.21737   0.580 0.561714
## verticalgym/fitness          -0.91099    0.22342  -4.077 4.55e-05 ***
## leading_indicator            18.06219  482.97671   0.037 0.970168
##
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
##     Null deviance: 907.42  on 722  degrees of freedom
## Residual deviance: 736.18  on 718  degrees of freedom
## AIC: 746.18
##
## Number of Fisher Scoring iterations: 15

Generally, it is best to minimise the AIC. The Logistic.model model AIC is 800.5 compared with the lead.logistic.model incorporating the lead_indicator is 746.18. The lead_indicator has not changed the predictive power on this dataset, but since the AIC favours a better model fit whilst penalising for additional predictors, the model to choose is the lead.logistic.model.

Re-train model and save #

# Re-train on all data
final.model <- glm(churn ~ incorporation_time + vertical + leading_indicator,
                      data   = model.df,
                      family = binomial(link = 'logit'))

Recommendations for further investigation/Comments #

The lead_indicator was a quick and dirty test, I’d spend more time looking at a better construction (e.g. moving avg)
If the cost of retention activities vs losing a customer was known then the optimal trade-off in terms of business cost could be found.
Incorporating additional data, e.g. CRM data showing interactions. Potentially assessing sales/account staff.
If it were a production model prompting staff to do retention calls, evaluate the impact of such calls through modelling e.g. a/b testing
Test model performance with a change to balancing the classes e.g. under/oversampling, bootstrap samples. This may explain the relative underperformance of tree based models in this exercise.
Try other binary classifier models.

Cryptocurrencies

rob.sh — Tue, 21 Nov 2017 00:30:00 GMT

The concept of having no central entity control currency seems attractive to a lot of people. However, trust in some decentralised entity means trusting the way that they operate their business - how they’re able to audit their code, and write bug-free software. The only thing I know about software is that it isn’t ever bug-free. To reduce MTBF, you implement things multiple times, in disjoint environments.

That said, the latest hack on a cryptocurrency/smart wallet I saw, made me wonder about their testing. How many bugs are they looking for, in terms of both their standard operating environment, as well as for security.

PyEnv is the new Conda

bastibe.de — Mon, 20 Nov 2017 02:00:00 GMT

How to install Python? If your platform has a package manager, you might be tempted to use that to install Python. But I don't like that. That version is often outdated, and you risk messing with an integral part of your operating system. Instead, I like to install a separate Python in my home directory. I used to use Anaconda (or WinPython, or EPD) to do this. But now there is a better way: PyEnv

The thing is: PyEnv installs (any version of) Python. That's all it does.

So why would I choose PyEnv over the more popular Anaconda? Because Anaconda is a Python distribution, a package manager, an environment manager, and a platform for paid packages. In the past, they once did break pip because they wanted to promote conda instead. Some features of conda require a login, some require a paid subscription. When you install packages through conda, you get binaries and source code from anaconda's servers, not the official packages from PyPi, which might or might not be up-to-date and feature-complete. For every package you install, you have to make a choice of using pip or conda, and the same goes for specifying your dependencies.

As an aside, many of these complaints are just as true for package-manager-provided Python packages (which often break pip, too!). Just like Anaconda, package managers want to be the true and only source of packages, and don't like to interact with Python's own package manager.

In contrast, with PyEnv, you install a Python. This can be a version of CPython, PyPy, IronPython, Jython, Pyston, stackless, miniconda, or even Anaconda. It downloads the sources from the official repos, and compiles them on your machine ¹. Plus, it provides an easy and transparent way of switching between installed versions (including any system-installed versions). After that, you use Python's own venv and pip.

I find this process much simpler, and easier to manage, because it relies on small, orthogonal tools (pyenv, venv, pip) instead of one integrated conda that kind of does everything. I also like to use these official tools and packages instead of conda's parallel universe of mostly-open, mostly-free, mostly-standard replacements.

Mind you, conda solved real problems back in the day (binary package distributions, Python version management, and environment management), and arguably still does (MKL et al, paid packages). But ever since wheels became ubiquitous and painless, and virtualenv was integrated into Python, and the development of PyEnv, these issues now have better solutions, and conda is no longer needed for my applications.

the downside of compilation is: no Windows support.↩

Server security scan #manpage

Yasha Solutions — Mon, 20 Nov 2017 02:00:00 GMT

chkrootkit – Linux rootkit scanner What it does : Find rootkits Where to find it : in your distribution – or on chkrootkit.org Malware protection Option 1: ISPProtect – 90 euro / year but worth it. Where to find it : ispprotect.com Step #1: Get a licence Step #2: Download and install and run Then indicate your key number or trial if you want a trial version. Indicate /var/www to start the scan on the web installs.

Nextcloud snap to be updated to v12

kyrofa's blog — Mon, 20 Nov 2017 02:00:00 GMT

If you’re a user of the Nextcloud snap, you may have noticed that upstream Nextcloud released v12 a while back, but you’re still on v11. That’s because v12 introduced several regressions that made us maintainers decide to hold off on the upgrade. Thankfully, things are finally looking up. As of today, it seems that all the blocking regressions have been fixed. Which means that, barring any additional regressions being introduced between now and then, v12.

Knock, knock. This thing still on?

Brian's Waste of Time — Sun, 19 Nov 2017 03:09:31 GMT

I've attempted to wake this blog up a couple times, but between Jekyll changing, Pygments changing, and whatnot, it has been more pain than any given post seemed worth. I've recently had three folks independently chastise me for no longer writing, however, and three is a magic number.

So, this is basically just a test post as I try converting over to Hugo. Jekyll resisted hard enough that I declared bankruptcy. We'll try this one. If I get annoyed at it, Gutenberg is next on the list. Torsten says nice things about it :-)

All my old posts should be exactly where I left them, no sense breaking URLs.

For a test post I need to include some code, that being most of the point of this thig, so some Rust from wsf.

impl From<hyper::error::Error> for CliError {
    fn from(err: hyper::error::Error) -> CliError {
        CliError::Http(err)
    }
}

So far, so good.

Build with Hugo
Code is pretty
Ensure Atom/RSS is sane
Add some kind of commenting (staticman?)
Better favicon

Off to deploy and see if this thing works!

Installing Emacs on Windows

bastibe.de — Sun, 19 Nov 2017 02:00:00 GMT

The official website states that you need to download Emacs from a nearby GNU mirror. However, this does not install gnutls, which is required for installing packages from melpa or marmalade. The documentation says that this can be obtained from ezwinports.

However, I have found that this does not work any more: As of Emacs 25, Emacs is available in 64 bit, but ezwinport only supplies 32 bit binaries. I had to search a bit, but the (in retrospect, obvious) solution is to download the required binaries from GnuTLS's website, directly. Then unpack all *.dll from the bin directory to your Emacs's bin directory, and you are good to go.

This situation is really a bit sad. Installing Emacs should not be this hard. But there is talk about providing a Windows installer for Emacs in one of the next versions, which will hopefully fix these issues.

Voice & IOT

Yasha Solutions — Sun, 19 Nov 2017 02:00:00 GMT

Voice and IOT Amazon Echo, Alexa, Google Home, Apple HomePod and Siri (and Microsoft Cortana) are all fighting to control the voice market. While they get full respect for doing so, I am not gonna say I am particularly trusting any of these companies to provide us with the best level of privacy, service and control over what we do with our devices and how is the data used. Therefore I am compiling here some resources which I’ll probably update as we go on interesting DIY tools to build similar devices.

The Brothers Karamazov

Krishna's blog — Sun, 19 Nov 2017 02:00:00 GMT

The Brothers Karamazov by Fyodor Dostoevsky is a challenging but rewarding read, with richly detailed characters and authentic relationships.

Game Review: WildSnake - SNES

KernelCurry — Sat, 18 Nov 2017 02:00:00 GMT

Wildsnake Cartridge

If you thought Tetris was fun, get ready for Tetris with snakes! Yeah, actual Snakes; Wild Snakes. Alexey Pajitnov, the creator of Tetris, thought it would be a good idea to add snakes to the classic game. To be honest, when I started playing this game I thought it was just a crappy rip off of Tetris, but NOPE the creator of Tetris himself thought this was a good idea… sigh.

An overview of arrays in Ada 2012

Juan Cruz Viotti — Sat, 18 Nov 2017 02:00:00 GMT

This article is an overview of array types in Ada 2012

Drafting

Lambda Land — Sat, 18 Nov 2017 02:00:00 GMT

I once asked my dad over email how to improve my the potency of my words and thoughts. His reply came back as one word:

Revision.

Good writing does not emerge spontaneously; it comes as one practices writing. A “draft” is a pass of writing a particular work. It’s like sketching in art: when an artist starts a painting, they usually start with a rough sketch outlining where the ﬁgures will be, what the landscape will be like, etc. These sketches are composed of many lines: each coming closer to their ﬁnal position.

When I draft, I start out with a few different freewrites. I usually start with a stream of consciousness to get my ideas out on paper and to sort of flesh out what I actually want to talk about. Then I will beat out a very rough draft. After this initial draft, I usually will have a good idea of how I want to treat the subject matter. I will do things like read my paper aloud to hear how it flows, check for grammatical inconsistencies, look for more concise phrases that convey my ideas more clearly. Occasionally I will decide that I need to change how I structure my argument entirely. In these situations I will often do a complete rewrite.

I usually use something akin to semantic versioning—the first version number of my draft indicates the major write of the paper, (e.g. 0.0.1 for the rough draft, 1.0.0 for the first full draft after the rough draft, 2.0.0 for the next full rewrite, etc.) the second number might indicate significant changes in the text, but not complete structural changes, and the third number indicates minor changes, such as spelling and grammar errors.

I don’t follow this system too rigorously; it’s just a kind of fun thing I use to keep track of my drafts. My brain is messy some times, and I’m okay with that.

Recurse Center 9: rayon par iter

Pete's Points — Sat, 18 Nov 2017 00:05:44 GMT

Today I refactored my little rust blockchain miner to divide the mining work across multiple CPUs. The rayon crate's par_iter() provides the magic for this. Before coding that, I was imagining how I'd handle that given I'm searching a 32-bit integer space where each number is equally likely to be a "golden nonce" (meaning a successful block mine). The most straightforward approach I could think of would be to break the single range into equal-sized partial ranges and ask the OS to distribute the load across cores. So instead of trying 0 then 1 then 2 in serial, you'd have 4 workers starting at (using 100 for obviousness) 0, 25, 50, 75 for the first round then each would advance so you'd have 1, 26, 51, 76 and so on.

I coded a simpler scratch version and printed debug output and was delighted to see at least part of that holding true with rayon. I could definitely see several sequences of incrementing nonces being tested in ascending order, they just weren't obviously mapped to the range in a way I could grok.

I also found out the testing I did yesterday with the shaman crate had a bug and you can indeed both give it read-only access to an existing vector of the payload data plus you can call .insert() more than once, so I refactored to just call .insert() with a reference to the payload then again with a reference to the 4-byte nonce slice and that avoids any memory copies and mutation, which I believe is nearly as good as you can do in rust.

So now my macbook can mine 8-16x faster than yesterday. However, the concurrent version no longer has deterministic behavior, so finding the same nonce on the same payload doesn't always take the same number of hashes, and the lower difficulty setting the larger the number of possible golden nonces so on separate runs you can mine different nonces. But however with multiple runs you do see it is just a small set of nonces that repeat.

How to print progress in a unit-test-friendly way is still unclear, at least in terms of what would be the idiom in rust. Maybe since output is captured by default you just don't worry about it? Not sure but I tried to use a channel such that each worker could send a message to the parent thread whenever it wanted to report a batch of a million non-golden nonces tested, allowing the parent thread to handling printing that as a dot to stdout, but I couldn't get that to compile.

One helpful development pattern I'm using a lot is focusing on the core function signatures and building a mock-up in a separate "scratch" project I have where the guts of most functions are stubbed out to just return a hard coded value or some dummy data but all the type signatures are exactly what I need for the real project. Then I iterate on that and do my compiler battling, and when that's over only then do I start adding implementation logic to the function bodies. That has been helpful this week for sure.

In other news I made Firefox Quantum my default browser today, partially due to rust fanboyism. So far so good. The "Vertical Tabs Reloaded" add-on is nice.

Oh random shell tip when debugging checksums. I wasn't sure if I was using the rust shaman crate correctly so I wrote a scratch rust program to feed it data from the first command line argument, then simulate a nonce as "aaaa" and print the hex of that hash. Then I verified with an independent implementation via the command line: echo -n abcdefghijkaaaa| shasum -a 256 and when both hashes were identical I knew I had the right code. (Don't forget the -n or you'll accidentally include the newline as your hash payload).

Never Graduate

Pete's Points — Fri, 17 Nov 2017 23:11:59 GMT

Neo: Whoa. Déjà vu. Déjà vu.

Trinity: What did you see, Neo?

Neo: It felt like I just did three Code and Coffees back to back.

Morpheus: Code and Coffee is a system. A system with rules. Rules like "Code and Coffee is once a week". Rules like "Code and Coffee is nine to noon". Some of these rules can be bent. Others can be broken. What if I told you that you could do sixty Code and Coffees in a row?

Neo: That's impossible.

Morpheus: Is that what you believe? What if I told you some people do one hundred and eighty Code and Coffees back to back? Impossible? Free your mind. Never graduate.

(Morpheus slow-mo backflips over a desk, reaches overhead to start typing on a laptop while still upside down before landing into a swivel chair)

Use the telephone for outgoing requests, email for incoming requests

Databases & Life — Fri, 17 Nov 2017 16:35:38 GMT

I have been thinking about emails vs phone, or more generally asynchronous vs synchronous communication, and have realized:

If you receive a call, it breaks your flow, which is bad.
If you send an email, it breaks your flow, which is bad. (As you have to wait for an answer, and do something else in the meantime.)

So:

If you have a question, the optimal strategy (for you) is to make a call.

Lord of the Flies

Apurva Shukla's RSS Feed — Fri, 17 Nov 2017 13:53:02 GMT

⭐ ⭐ ⭐ ⭐ Thought provoking, revealing the hidden precipice of our human nature; animalistic and savage. Golding cleverly guides the audience…

Recurse Center 8: binary literals

Pete's Points — Fri, 17 Nov 2017 04:18:51 GMT

Not that much to report today mostly some tweaks to my rust blockchain miner exercise. I remembered rust has binary literals and bitwise operations are one place they actually are useful. I've watched all the Into Rust tutorials/exercises and they were great but there's only a handful. I did an exercism on isograms and demoed my rustblock program during Thursday evening demos. As tends to be the case, I learned a lot from the RCers while talking about the miner and apparently the bitcoin difficulty factor is so high that you need datacenter scale computing power to even hope to mine one block in a small number of minutes. My rust miner starts to take more than a few seconds searching for a golden nonce with about 27 or 28 leading zeros so presumably the bitcoin work factor is quite a bit larger than that (with a maximum of 256 since that's the sha size but practically has to be quite a bit lower than that to allow space for enough blocks to get unique hashes for all of bitcoin's lifetime I guess).

How to root Nexus 7 (2012) and flash it to LineageOS, from Mac OSX

Subclassed — Fri, 17 Nov 2017 02:56:59 GMT

I've finally got fed up enough with the old Nexus 7 (2012) that we originally bought to let the kids play - it's never really worked properly, getting laggy every 5 minutes and running out of battery after an hour. The kids moved on to an iPad, so I "installed" this crappy tablet to the kitchen wall, hoping to use it for calendaring and podcasts, but it was still horrendously laggy. I've decided to give a chance to LineageOS (previously CyanogenMod) to see if it helps. I'd be interested to know if anybody else uses a tablet wall-mounted in their kitchen, and what apps they installed - at the moment I have Paprika, the BBC players and weather, and PodcastAddict.

Here are the full steps required for flashing - I write them here because some of these are at risk to disappear from the internet for good after the demise of CyanogenMod wiki.

Note: THIS WILL WIPE YOUR DEVICE, YOU WILL LOSE EVERYTHING THAT IS ON IT so first make backups etc etc.

First you need to install the Android tools. I recommend doing this with homebrew, install it if you don't have it yet (it's extremely useful in many many cases).
Then open a terminal on your mac and type:

brew tap caskroom/cask
brew tap caskroom/versions
brew cask install java8
brew cask install android-sdk

Enable Developer Mode on the tablet by tapping 7 times on the build number under Settings -> About tablet.
Back to Settings, tap on Developer Options and enable USB Debug mode and Stay Awake.
Go back to your mac terminal and type:

sudo adb start-server

This starts the android debugger server.
Connect the tablet with a real USB cable (beware charging-only cables! Those won't work.)
On the tablet screen, you should be prompted to authorize the device; do it.
Back to terminal:

adb reboot bootloader

once the device comes back in bootloader mode:

fastboot oem unlock

Accept the disclaimer on the tablet by clicking the power button when Yes is highlighted (you can move with the volume buttons).
If it doesn't reboot on its own, select Start (again with the power button). At this point you should see an unlocked pad at the bottom of the screen as Android loads.
Note: if you plan to flash another OS, there is no point in going through the whole setup at this point, skip as much as you can.
Re-enable Developer mode and USB Debugging.

At this point the machine is rooted. The following steps are necessary only if you want to install LineageOS or other hacks; at the very minimum, though, you should install Trimmer (fstrim) from the Play store and use it liberally. Anyway, on with the flashing...

Get the latest image for "grouper" from https://twrp.me/, and unzip it.
Back to the terminal:

cd folder/where/you/have/your-downloaded-image.img
fastboot flash recovery your-downloaded-image.img
adb reboot bootloader

At this point you have a custom recovery image with a bunch of nifty features that make it very easy to hack the device. Tap Wipe and select system, cache, and dalvik.
Get the image you want to install. I'm currently trying this but in general anything after CM10 / Android 4.1 may be on the slow side for the original Nexus. You probably want the Google Apps from opengapps.org - choose ARM / 7.1 and the Micro option. Save both the apps zip and the image zip in the same folder.
Back to Terminal, let's copy these files to the device:

cd folder/where/you/have/your/zips
adb push your-downloaded-image.zip /
adb push your-downloaded-gapps.zip /

On the tablet, in the recovery screen, select Install, tap on Install ZIP, then select the image file and install. Repeat for the gapps file. If gapps refuse to go in because of space constraints, download this file, rename it gapps-config.txt, then copy it to the device in the same location as the gapps zip:

adb push gapps-config.txt /

and try again to install the gapps image.

Reboot the tablet. You might have to re-enable developer mode and usb debugging.
Install the Trimmer (fstrim) app, which is absolutely necessary. Launch it and click Trim Now. Click on the settings icon and enable autotrim as frequently as possible. In fact, every time you install an app or write a bunch of files, before you launch the new app, open Trimmer and do a trim, or it will all get laggy again.
Another good app is Android SSH Server, although it's a bit old it still works fine and it's easy to configure. You just have to use the options -t -oHostKeyAlgorithms=+ssh-dss -p XXXXX (where XXXXX is the custom port configured in the app, Android won't allow the classic one) when connecting.

And that's it.

My first custom element

Qubyte Codes — Thu, 16 Nov 2017 03:45:00 GMT

After some years of browser vendors working out what web components should look like, they're almost ready for the prime time. The part which I find most intriguing (custom elements) has finally stabilised. With custom elements, you can make new HTML elements which have custom behaviour which you define using JavaScript. In this post I'll demonstrate a custom element for fuzzy counting.

Custom elements are created in two parts. Firstly we need to extend an element with a JavaScript class.

class FuzzyCount extends HTMLElement {

}

So far this describes no custom behaviour. All this is is the extended class, with identical behaviour to an HTMLElement. We can customize the constructor to add the behaviour we want. In order to give the element data to use, it must be passed in as an attribute. We'll use an attribute called count.

class FuzzyCount extends HTMLElement {
  constructor() {
    // The parent constructor must be called
    // before using `this`.
    super();

    // rawCount is a string.
    const rawCount = this.getAttribute('count');
    const count = parseInt(rawCount, 10);

    // Set text content based on the count.
    if (count === 0) {
      this.textContent = 'none';
    } else if (count === 1) {
      this.textContent = 'one';
    } else if (count === 2) {
      this.textContent = 'a couple';
    } else if (count < 5) {
      this.textContent = 'a few';
    } else if (count < 10) {
      this.textContent = 'several';
    } else {
      this.textContent = 'lots';
    }
  }
}

The element reads and parses the count attribute, giving itself a text content accordingly.

To make the element available to a page, it must be registered. One interesting restriction placed upon custom elements is that they must contain a hyphen so that they can be distinguished from built-in elements. We're going to register our element as fuzzy-count, but in the real world you should prefix it with a namespace. For example, if we're working at a place called Funky Corp, we could name the element funkycorp-fuzzy-count.

customElements.define('fuzzy-count', FuzzyCount);

Now we can use this to make elements on the page!

<fuzzy-count count="3"></fuzzy-count>

Any matching custom elements which exist before the element is defined will be upgraded. This means a page can be sent by the server with custom elements included, and everything will be rendered properly once the custom element is registered.

But what if we want to create elements in JS? We can try:

const fuzzyCount = document.createElement('fuzzy-count');

but an error will be thrown. It turns out that newly constructed elements cannot contain other stuff (the text content). Even if no error was thrown, we'd have an element with no count attribute, and so we've missed our chance since all the logic is in the constructor. It turns out that the constructor is the wrong place for this stuff.

Thankfully we can make a few changes to defer the setting of text content.

class FuzzyCount extends HTMLElement {
  // Called when the element is inserted into
  // the document or upgraded.
  connectedCallback() {
    this.setTextContent();
  }

  setTextContent() {
    const rawCount = this.getAttribute('count');

    if (!rawCount) {
      this.textContent = '';
      return;
    }

    const count = parseInt(rawCount, 10);

    if (count === 0) {
      this.textContent = 'none';
    } else if (count === 1) {
      this.textContent = 'one';
    } else if (count === 2) {
      this.textContent = 'a couple';
    } else if (count < 5) {
      this.textContent = 'a few';
    } else if (count < 10) {
      this.textContent = 'several';
    } else {
      this.textContent = 'lots';
    }
  }
}

The logic which sets the textContent has moved to its own method setTextContent, and connectedCallback calls it when the element is "connected" to avoid the error when constructing the element in JS. Connected is called when the element is inserted into the document, or the element is upgraded by custom element registration.

This is enough to get the behaviour we want, so long as the count attribute of a new fuzzy-count element is set before it is appended to the document.

const fuzzy = document.createElement('fuzzy-count');
fuzzy.setAttribute('count', 10);
document.body.appendChild(fuzzy);

We can take one more step to make the element react to changes to the count attribute...

class FuzzyCount extends HTMLElement {
  // Called for each watched attribute when
  // the element is added to the document or
  // upgraded. Also fired for a watched
  // attribute when it is added, updated, or
  // removed.
  attributeChangedCallback() {
    this.setTextContent();
  }

  // A static getter because we can't add a
  // static value within a class declaration
  // directly (yet).
  static get observedAttributes() {
    return ['count'];
  }

  setTextContent() {
    // Same as before.
  }
}

connectedCallback has been replaced with attributeChangedCallback, and a static getter defines the list of attributes to watch. For each watched attribute, when an element is inserted, upgraded, or the attribute is added, updated, or removed, this attributeChangedCallback is called. These changes allow us to update elements before appending as before, and also after appending.

const fuzzy = document.createElement('fuzzy-count');
document.body.appendChild(fuzzy);
fuzzy.setAttribute('count', 10);

Overall I'm impressed. I've only covered some of the API available here. It is a little fiddly, but upon consideration it all seems to make sense so far. It strikes me that a good way to use it would be for small components, or as a low level primitive for a front end framework.

Finally, a note on compatibility. At the time of writing Chrome and Safari support custom elements with no additional effort. Other browsers must be polyfilled for support.

Recurse Center 7: blockchain mining in rust

Pete's Points — Thu, 16 Nov 2017 02:01:27 GMT

This morning I tweaked my hexagonal lambda tooling and tests. No big deal. I also started watching the "Into Rust" screencasts. The first one is great for answering the "Why Rust?" question which I got asked twice today already.

After lunch I paired on the inner guts of a blockchain miner in rust. Just enough to combine a nonce with an array of block payload bytes and hunt for a sha256 "golden nonce" with a given number of leading zero bits.

We initially arbitrarily picked a suboptimal crate for the sha256 algorithm that needed a mutable reference to the block+nonce data so we had to clone it every time we had to hash a different nonce. I eventually did a second shopping trip to crates.io and found shaman which just needed an immutable reference to our data, which was perfect. There's a little bit of bitwise logic and bit shifting required which is normally "scary low level" land from which I run away but with a pair to check me on things like "If I want the second bit of a u8 to be a 1, that's 64, right?" it was OK.

We increased the difficulty factor up to 24 leading zero bits and were able to mine blocks quickly on my 2014 mackbook, but at 25 bits it started to take a minute or so. I haven't tried 26 but maybe I'll try tonight and leave it running with a timer.

Some of the RCers played "Coding Bee" this evening which was a very amusing spectacle. It's done in 2-coder teams. You are read a small programming exercise of self-selected difficulty (easy/medium/hard). You then code a solution by each taking turns speaking a single character to a dedicated typist who types it up onto the projector, but the players are not allowed to look at the screen. There's also no backspace, just "clear line" and you have to speak all indentation and punctuation. Amazingly, some teams actually wrote correct programs.

org-babel Objective-C support

xenodium.com @alvaro — Thu, 16 Nov 2017 02:00:00 GMT

Wanted to quickly execute an Objective-C snippet. org-babel didn't support it out of the box, but adding it was straightforward (looked at ob-C.el and ob-java.el):

(require 'ob)

(defcustom org-babel-objc-compile-command "clang -x objective-c -framework Foundation"
  "For example: \"clang -x objective-c -framework Foundation\"."
  :group 'org-babel
  :version "24.3"
  :type 'string)

(defun org-babel-execute:objc (body params)
  "Compile Objective-C BODY with org PARAMS and execute binary."
  (let* ((src-file (org-babel-temp-file "org-babel-objc-block-" ".m"))
         (cmpflag (or (cdr (assq :cmpflag params)) ""))
         (full-body (org-babel-expand-body:generic body params))
         (bin-file
          (org-babel-process-file-name
           (org-babel-temp-file "org-babel-objc-block" org-babel-exeext))))
    (with-temp-file src-file (insert full-body))
    (org-babel-eval
     (concat org-babel-objc-compile-command " " cmpflag " " src-file " " "-o" " " bin-file) "")

    ;; Using 2>&1 since org babel does not include stderr in output from NSLog.
    (let ((results (org-babel-eval (concat (org-babel-process-file-name bin-file) " 2>&1")  "")))
      (org-babel-reassemble-table
       (org-babel-result-cond (cdr (assq :result-params params))
         (org-babel-read results)
         (let ((tmp-file (org-babel-temp-file "c-")))
           (with-temp-file tmp-file (insert results))
           (org-babel-import-elisp-from-file tmp-file)))
       (org-babel-pick-name
        (cdr (assq :colname-names params)) (cdr (assq :colnames params)))
       (org-babel-pick-name
        (cdr (assq :rowname-names params)) (cdr (assq :rownames params)))))))

(provide 'ob-objc)

Add objc to org-babel-load-languages, and you can subsequently compile and run Objective-C blocks like:

#import <Foundation/Foundation.h>

int main() {
  NSLog(@"Hello World");
  return 0;
}

Olaf's Frozen Adventure

Code & Visuals — Thu, 16 Nov 2017 02:00:00 GMT

After an amazing 2016, Walt Disney Animation Studios is having a bit of a break year this year. Disney Animation doesn’t have a feature film this year; instead, we made a half-hour featurette called Olaf’s Frozen Adventure, which will be released in front of Pixar’s Coco during Thanksgiving. I think this is the first time a Disney Animation short/featurette has accompanied a Pixar film. Olaf’s Frozen Adventure is a fun little holiday story set in the world of Frozen, and I had the privilege of getting to play a small role in making Olaf’s Frozen Adventure! I got an official credit as part of a handful of engineers that did some specific, interesting technology development for Olaf’s Frozen Adventure.

Olaf’s Frozen Adventure is really really funny; because Olaf is the main character, the entire story takes on much more of a self-aware, at times somewhat absurdist tone. The featurette also has a bunch of new songs- there are six new songs in total, which is somehow pretty close to the original film’s count of eight songs, but in a third of the runtime. Olaf’s Frozen Adventure was originally announced as a TV special, but the wider Walt Disney Company was so happy with the result that they decided to give Olaf’s Frozen Adventure a theatrical release instead!

Something I personally find fascinating about Olaf’s Frozen Adventure is comparing it visually with the original Frozen. Olaf’s Frozen Adventure is rendered entirely with Disney’s Hyperion Renderer, compared with Frozen, which was rendered using pre-RIS Renderman. While both films used our Disney BRDF [Burley 2012] and Ptex [Burley and Lacewell 2008], Olaf’s Frozen Adventure benefits from all of the improvements and advancements that have been made during Big Hero 6, Zootopia, and Moana. The original Frozen used dipole subsurface scattering, radiosity caching, and generally had fairly low geometric complexity relative to Hyperion-era films. In comparison, Olaf’s Frozen Adventure uses brute force subsurface scattering, uses path-traced global illumination, uses the full Disney BSDF (which is significantly extended from the Disney BRDF) [Burley 2015], uses our advanced fur/hair shader developed during Zootopia [Chiang et al. 2016], and has much greater geometric complexity. A great example of the greater geometric complexity is the knitted scarf sequence [Staub et al. 2018], where 2D animation was brought into Hyperion as a texture map to drive the colors on a knitted scarf that was modeled and rendered down to the fiber level. Some shots even utilize an extended version of the photon mapped caustics we developed during Moana; the photon mapped caustics system on Moana only supported distant lights as a photon source, but for Olaf’s Frozen Adventure, the photon mapping system was extended to support all of Hyperion’s existing light types as photon sources. These extensions to our photon mapping system is one of the things I worked on for Olaf’s Frozen Adventure, and was used for lighting the ice crystal tree that Elsa creates at the end of the film. Even the water in Arendelle Harbor looks way better than in Frozen, since the FX artists were able to make use of the incredible water systems developed for Moana [Palmer et al. 2017]. Many of these advancements are discussed in our SIGGRAPH 2017 Course Notes [Burley et al. 2017].

One of the huge advantages to working on an in-house production rendering team in a vertically integrated studio is being able to collaborate and partner closely with productions on executing long-term technical visions. Because of the show leadership’s confidence in our long-term development efforts targeted at later shows, the artists on Olaf’s Frozen Adventure were willing to take on and try out early versions of a number of new features in Hyperion that were originally targeted at later shows. Some of these “preview” features wound up making a big difference on Olaf’s Frozen Adventure, and lessons learned on Olaf’s Frozen Adventure were instrumental in making these features much more robust and complete on Ralph Breaks the Internet.

One major feature was brute force path-traced subsurface scattering; Peter Kutz, Matt Chiang, and Brent Burley had originally started development during Moana’s production on brute force path-traced subsurface scattering [Chiang 2016] as a replacement for Hyperion’s existing normalized diffusion based subsurface scattering [Burley 2015]. This feature wasn’t completed in time for use on Moana (although some initial testing was done using Moana assets), but was far enough along by Olaf’s Frozen Adventure was in production that artists started to experiment with it. If I remember correctly, the characters in Olaf’s Frozen Adventure are still using normalized diffusion, but path-traced subsurface wound up finding extensive use in rendering all of the snow in the show, since the additional detail that path-traced subsurface brings out helped highlight the small granular details in the snow. A lot of lessons learned from using path-traced subsurface scattering on the snow were then applied to making path-traced subsurface scattering more robust and easier to use and control. These experiences gave us the confidence to go ahead with full-scale deployment on Ralph Breaks the Internet, which uses path-traced subsurface scattering for everything including characters.

Another major development effort that found experimental use on Olaf’s Frozen Adventure were some large overhauls to Hyperion’s ray traversal system. During the production of Moana, we started running into problems with how large instance groups are structured in Hyperion. Moana’s island environments featured vast quantities of instanced vegetation geometry, and because of how the instancing was authored, Hyperion’s old strategy for grouping instances in the top-level BVH wound up producing heavily overlapping BVH leaves, which in extreme cases could severely degrade traversal performance. On Moana, the solution to this problem was to change how instances were authored upstream in the pipeline, but the way that the renderer wanted instances organized was fairly different from how artists and our pipeline like to think about instances, which made authoring more difficult. This problem motivated Peter Kutz and I to develop a new traversal system that would be less sensitive to how instance groups were authored; the system we came up with allows Hyperion to internally break up top-level BVH nodes with large overlapping bounds into smaller, tighter subbounds based on the topology of the lower-level BVHs. It turns out this system is conceptually essentially identical to BVH rebraiding [Benthin et al. 2017], but we developed and deployed this system independently before Benthin 2017 was published. As part of this effort, we also wound up revisiting Hyperion’s original cone-based packet traversal strategy [Eisenacher et al. 2013] and, motivated by extensive testing and statistical performance analysis, developed a new, simpler, higher performance multithreading strategy for handling Hyperion’s ultra-wide batched ray traversal. Olaf’s Frozen Adventure has a sequence where Olaf and Sven are being pulled down a mountainside through a forest by a burning sled; the enormous scale of the groundplane and large quantities of instanced trees proved to be challenging for Hyperion’s old traversal system. We were able to partner with the artists to deploy a mid-development prototype of our new traversal system on this sequence, and were able to cut traversal times by up to close to an order of magnitude in some cases. As a result, the artists were able to render this sequence with reasonable render times, and we were able to field-test the new traversal system prior to studio-wide deployment and iron out various kinks that were found along the way.

The last major mid-development Hyperion feature that saw early experimental use on Olaf’s Frozen Adventure was our new, next-generation spectral and decomposition tracking [Kutz et al. 2017] based null-collision volume rendering system, which was written with the intention of eventually completely replacing Hyperion’s existing residual ratio tracking [Novák 2014] based volume rendering system [Fong 2017]. Artists on Olaf’s Frozen Adventure ran into some difficulties with rendering loose, fluffy white snow, where the bright white appearance is the result of high-order scattering requiring large numbers of bounces. We realized that this problem is essentially identical to the problem of rendering white puffy clouds, which also have an appearance dominated by energy from high-order scattering. Since null-collision volume integration is specifically very efficient at handling high-order scattering, we gave the artists an early prototype version of Hyperion’s new volume rendering system to experiment with rendering loose fluffy snow as a volume. The initial results looked great; I’m not sure if this approach wound up being used in the final film, but this experiment gave both us and the artists a lot of confidence in the new volume rendering system and provided valuable feedback.

As usual with Disney Animation projects I get to work on, here are some stills in no particular order, from the film. Even though Olaf’s Frozen Adventure was originally meant for TV, the whole studio still put the same level of effort into it that goes into full theatrical features, and I think it shows!

Here is a credits frame with my name! I wasn’t actually expecting to get a credit on Olaf’s Frozen Adventure, but because I had spent a lot of time supporting the show and working with artists on deploying experimental Hyperion features to solve particularly difficult shots, the show decided to give me a credit! I was very pleasantly surprised by that; my teammate Matt Chiang got a credit as well for similar reasons.

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Carsten Benthin, Sven Woop, Ingo Wald, and Attila T. Áfra. 2017. Improved Two-Level BVHs using Partial Re-Braiding. In Proc. of High Performance Graphics (HPG 2017). Article 7.

Brent Burley. 2012. Physically Based Shading at Disney. In ACM SIGGRAPH 2012 Course Notes: Practical Physically-Based Shading in Film and Game Production.

Brent Burley. 2015. Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering. In ACM SIGGRAPH 2015 Course Notes: Physically Based Shading in Theory and Practice.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2017. Recent Advances in Disney’s Hyperion Renderer. In ACM SIGGRAPH 2017 Course Notes: Path Tracing in Production Part 1

Brent Burley and Dylan Lacewell. 2008. Ptex: Per-face Texture Mapping for Production Rendering. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering) 27, 4 (Jun. 2008), 1155-1164.

Matt Jen-Yuan Chiang, Benedikt Bitterli, Chuck Tappan, and Brent Burley. 2016. A Practical and Controllable Hair and Fur Model for Production Path Tracing. Computer Graphics Forum (Proc. of Eurographics) 35, 2 (May 2016), 275-283.

Matt Jen-Yuan Chiang, Peter Kutz, and Brent Burley. 2016. Practical and Controllable Subsurface Scattering for Production Path Tracing. In ACM SIGGRAPH 2016 Talks. Article 49.

Christian Eisenacher, Gregory Nichols, Andrew Selle, and Brent Burley. 2013. Sorted Deferred Shading for Production Path Tracing. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering) 32, 4 (Jul. 2013), 125-132.

Julian Fong, Magnus Wrenninge, Christopher Kulla, and Ralf Habel. 2017. Production Volume Rendering. In ACM SIGGRAPH 2017 Courses. Article 2.

Peter Kutz, Ralf Habel, Yining Karl Li, and Jan Novák. 2017. Spectral and Decomposition Tracking for Rendering Heterogeneous Volumes. ACM Transactions on Graphics (Proc. of SIGGRAPH) 36, 4 (Aug. 2017), Article 111.

Jan Novák, Andrew Selle and Wojciech Jarosz. 2014. Residual Ratio Tracking for Estimating Attenuation in Participating Media. ACM Transactions on Graphics (Proc. of SIGGRAPH Asia) 33, 6 (Nov. 2014), Article 179.

Sean Palmer, Jonathan Garcia, Sara Drakeley, Patrick Kelly, and Ralf Habel. 2017. The Ocean and Water Pipeline of Disney’s Moana. In ACM SIGGRAPH 2017 Talks. Article 29.

Josh Staub, Alessandro Jacomini, Dan Lund. 2018. The Handiwork Behind “Olaf’s Frozen Adventure”. In ACM SIGGRAPH 2018 Talks. Article 26.

Isso Comments

Tyler Hallada - Blog — Wed, 15 Nov 2017 02:00:00 GMT

I’ve been meaning to add a commenting system to this blog for a while, but I couldn’t think of a good way to do it. I implemented my own commenting system on my old Django personal site. While I enjoyed working on it at the time, it was a lot of work, especially to fight the spam. Now that my blog is hosted statically on Github’s servers, I have no way to host something dynamic like comments.

Disqus seems to be the popular solution to this problem for other people that host static blogs. The way it works is that you serve a javascript client script on the static site you own. The script will make AJAX requests to a separate server that Disqus owns to retrieve comments and post new ones.

The price you pay for using Disqus, however, is that they get to sell all of the data that you and your commenters give them. That reason, plus the fact that I wanted something more DIY, meant this blog has gone without comments for a few years.

Then I discovered Isso. Isso calls itself a lightweight alternative to Disqus. Isso allows you to install the server code on your own server so that the comment data never goes to a third party. Also, it does not require logging into some social media account just to comment. Today, I installed it on my personal AWS EC2 instance and added the Isso javascript client script on this blog. So far, my experience with it has been great and it performs exactly the way I expect.

I hit a few snags while installing it, however.

Debian Package

I don’t recommend using the Debian package anymore as it frequently goes out of date and breaks on distribution upgrades. See bottom edit.

There is a very handy Debian package that someone has made for Isso. Since my server runs Ubuntu 16.04, and Ubuntu is based off of Debian, this is a package I can install with my normal ubuntu package manager utilities. There is no PPA to install since the package is in the main Ubuntu package archive. Just run sudo apt-get install isso.

I got a bit confused after that point, though. There seems to be no documentation I could find about how to actually configure and start the server once you have installed it. This is what I did:

sudo cp /etc/default/isso /etc/isso.d/available/isso.cfg
sudo ln -s /etc/isso.d/available/isso.cfg /etc/isso.d/enabled/isso.cfg

Then you can edit /etc/isso.d/available/isso.cfg with your editor of choice to configure the Isso server for your needs. Make sure to set the host variable to the URL for your static site.

Once you’re done, you can run sudo service isso restart to reload the server with the new configuration. sudo service isso status should report Active (running).

Right now, there should be a gunicorn process running the isso server. You can check that with top or running ps aux | grep gunicorn, which should return something about “isso”.

Nginx Reverse Proxy

In order to map the URL “comments.hallada.net” to this new gunicorn server, I need an nginx reverse proxy.

To do that, I made a new server block: sudo vim /etc/nginx/sites-available/isso which I added:

server {
    listen 80;
    listen [::]:80;
    server_name comments.hallada.net;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Script-Name /isso;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://localhost:8000;
    }
}

Then I enabled this new server block with:

sudo ln -s /etc/nginx/sites-available/isso /etc/nginx/sites-enabled/isso
sudo systemctl restart nginx

DNS Configuration

I added a new A record for “comments.hallada.net” that pointed to my server’s IP address to the DNS configuration for my domain (which I recently switched to Amazon Route 53).

After the DNS caches had time to refresh, visiting http://comments.hallada.net would hit the new isso nginx server block, which would then pass the request on to the gunicorn process.

You can verify if nginx is getting the request by looking at /var/log/nginx/access.log.

Adding the Isso Script to my Jekyll Site

I created a file called _includes/comments.html with the contents that the Isso documentation provides. Then, in my post template, I simply included that on the page where I wanted the comments to go:

<div class="card">
  <div class="row clearfix">
    <div class="column full">
      <script data-isso="https://comments.hallada.net/"
              src="https://comments.hallada.net/js/embed.min.js"></script>

      <section id="isso-thread">
        <noscript>Javascript needs to be activated to view comments.</noscript>
      </section>
    </div>
  </div>
</div>

Another thing that was not immediately obvious to me is that the value of the name variable in the Isso server configuration is the URL path that you will need to point the Isso JavaScript client to. For example, I chose name = blog, so the data-isso attribute on the script tag needed to be http://comments.hallada.net/blog/.

The Uncaught ReferenceError

You won’t need to fix this if you install Isso from PIP! See bottom edit.

There’s an issue with that Debian package that causes a JavaScript error in the console when trying to load the Isso script in the browser. I solved this by uploading the latest version of the Isso embeded.min.js file to my server, which I put at /var/www/html/isso/embeded.min.js. Then I modified the nginx server block to serve that file when the path matches /isso:

server {
    listen 80;
    listen [::]:80;
    server_name comments.hallada.net;
    root /var/www/html;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Script-Name /isso;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://localhost:8000;
    }

    location /isso {
        try_files $uri $uri/ $uri.php?$args =404;
    }
}

Now requesting http://comments.hallada.net/isso/embeded.min.js would return the newer script without the bug.

Sending Emails Through Amazon Simple Email Service

I already set up Amazon’s SES in my last blog post. To get Isso to use SES to send notifications about new comments, create a new credential in the SES UI, and then set the user and password fields in the isso.cfg to what get’s generated for the IAM user. The SES page also has information for what host and port to use. I used security = starttls and port = 587. Make sure whatever email you use for from is a verified email in SES. Also, don’t forget to add your email as the to value.

Enabling HTTPS with Let’s Encrypt

Let’s Encrypt allows you to get SSL certificates for free! I had already installed the certbot/letsencrypt client before, so I just ran this to generate a new certificate for my new sub-domain “comments.hallada.net”:

sudo letsencrypt certonly --nginx -d comments.hallada.net

Once that successfully completed, I added a new nginx server block for the https version at /etc/nginx/sites-available/isso-https:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name comments.hallada.net;
    root /var/www/html;

    ssl_certificate /etc/letsencrypt/live/comments.hallada.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/comments.hallada.net/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/comments.hallada.net/fullchain.pem;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Script-Name /isso;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://localhost:8000;
    }

    location /isso {
        try_files $uri $uri/ $uri.php?$args =404;
    }
}

And, I changed the old http server block so that it just permanently redirects to the https version:

server {
    listen 80;
    listen [::]:80;
    server_name comments.hallada.net;
    root /var/www/html;

    location / {
        return 301 https://comments.hallada.net$request_uri;
    }
}

Then I enabled the https version:

sudo ln -s /etc/nginx/sites-available/isso-https /etc/nginx/sites-enabled/isso-https
sudo systemctl restart nginx

I checked that I didn’t get any errors visiting https://comments.hallada.net/, and then changed my Jekyll include snippet so that it pointed at the https site instead of http.

Now you can securely leave a comment if you want to yell at me for writing the wrong thing!

EDIT 5/28/2019:

I don’t recommend using the Debian package anymore since it frequently goes out of date and breaks when upgrading your Linux distribution.

Instead, follow the Isso docs by creating a virtualenv and then run pip install isso and pip install gunicorn from within the virtualenv. Then, when creating a systemd service, make sure to point to the gunicorn executable in that virtualenv (e.g.
/opt/isso/bin/gunicorn). It should load and run Isso from the same virtualenv.

Recurse Center 6: nth prime

Pete's Points — Wed, 15 Nov 2017 00:33:52 GMT

This morning I wrote a script to manage AWS session tokens and asked for a code review. Not optimistic that I'll ultimately use it and I'll probably go back to aws-vault, but at least it works now and I can test drive it.

I did the 3 next exercism rust exercises. There are 70+ of them and I'm worried they are too much about basic programming problems and not enough about how to write idiomatic rust, so I think starting tomorrow I'm going to just cherry pick a few that look the most interesting.

Heading to Etsy to hear Suz Hinton talk about Code as Craft this evening.

C++ Smart Pointers: weak_ptr

Studiofreya SSG Site — Tue, 14 Nov 2017 22:16:06 GMT

C++ smart pointers are pointers that wrap raw C++ pointers. There are several types of smart pointers in C++.

This article will be about the weak pointer (std::weak_ptr) in the C++ standard that is a type of a smart pointer. It’s closely related to the shared pointer (std::shared_ptr) that is also a type of C++ smart pointers.

Other types are unique_ptr and scoped_ptr. There was also an auto_ptr, but it was removed in C++17.

cURL to Postman

Daniel Little Dev — Tue, 14 Nov 2017 02:18:32 GMT

Postman is a fantastic tool for testing any HTTP endpoint. But if you're using your browser to look around and you find something of…

How to Deploy Plotly's Dash using Shinyproxy

Luke Singham — Tue, 14 Nov 2017 02:00:00 GMT

In a previous post I established that I could easily deploy a 'Hello World' flask.py web application using Shinyproxy. Therefore, I thought it would be straightforward to deploy a Dash app which is built on top of flask.py. However, it proved to be a little more difficult than that. This blog post runs through the errors and eventual solution to deploying a Dash app on Shinyproxy.

The Issue #

Dash apps, like most web apps, load additional static resources, but upon deploying on Shinyproxy these aren't being delivered to the client (browser) as the URL path is wrong. This results in the all too familiar 404 Not Found. Disclaimer: I haven't programmed in Java or SpringBoot (yet!). However, as far as I can see from looking at Chrome dev tools there are two ways Shinyproxy delivers static files to the client. I'm taking a look first at how it works with a Shiny app.

Webjars - I can see that the main template of shinyproxy loads the bootstrap CSS which also gets utilised by each Shiny app.
Load from the container - Additional static content for shiny apps is requested by using the container name, in this case peaceful_jepsen e.g.:

Request URL:http://<my-ip-address>/peaceful_jepsen/dt-core-1.10.12/css/jquery.dataTables.min.css

Now let's move on to it not working with Dash...

Deploying a Dash App on Shinyproxy #

Using react.min.js as an example, I'm going to contrast how the app fetches static content on my local machine in a container vs Shinyproxy on a remote server. I'll do this by inspecting the path for the GET requests using Chrome dev tools. Firstly, what is the default behaviour of Dash.

By default, dash serves the JavaScript and CSS resources from the online CDNs.
Source: Dash Docs

Dash Container on my Local Machine

All resources loaded and the application displayed correctly.

Dash Container on Shinyproxy

On Shinyproxy react.min.js gets loaded however _dash-layout and _dash-dependencies fail to GET the necessary resources from the Request URL: http://<my-ip-address>/_dash-layout. As we know from the Shiny app, the container name should be in there e.g. Request URL: http://<my-ip-address>/<container_name>/_dash-layout. This is how Dash is supposed to behave, looking at the source code we can see a configurable routes_pathname_prefix (I'll come back to this later in the post). So a possible solution is to get the Dash app to obtain the container name from which it is running.

The Dash Docs have the following lines of code to serve content locally.

app.css.config.serve_locally = True
app.scripts.config.serve_locally = True

Let's see how that changes the behaviour.

Dash Container on my Local Machine

Since all static content is now requested from Dash this is probably going to prevent all content from being loaded on Shinyproxy.

Dash Container on Shinyproxy

Yup! A whole lot of red 404s.

Essentially it doesn't matter if you configure Dash to serve static files locally, the underlying issue is the relative URL route prefix. We know from the Dash source code how we could prefix the URL route. However, Shinyproxy is already doing this for Shiny apps, what's different for the requests from Dash?

I started scouring the Shinyproxy Java code base. I found the construction of the containerPath in Shinyproxy's AppController.java. But since I am not a Java developer I decided that this would probably not be the best allocation of my efforts. However, what the containerPath does show is the construction of the path begins with '/'.

Another approach I thought of was obtaining the container name from within the Dash application and appending that to the GET URL requests initiated by Dash. Alas, Docker doesn't allow that. You can access the ContainerID but not the ContainerName. There is a 3 year old open issue regarding this. I then started looking for an easy way to pass into the container the containerName but I couldn't find a non-hacky way to do that either.

Sidenote - I was wondering what the logic was on the container name generation e.g. adorin_raman. If you look at the docker source code you will find some go-lang code called names-generator.go. This contains two lists, a list of adjectives and a list of "notable scientists and hackers". The function that randomly combines these into a container name formatted as "adjective_surname" has one funny exception:

if name == "boring_wozniak" /* Steve Wozniak is not boring */ {
 goto begin
}

Two other approaches that weren't very appealing would have been managing the resources as webjars or an Nginx redirect.

The Final Solution #

I posted a question to the shinyproxy forum, Frederick one of the main authors responded with:

Then that means the href being used was /_dash-layout and not _dash-layout.
Do you control the hrefs? If so, removing the slash may solve your issue. If not, it becomes trickier… you could use javascript and extract the correct URL from window.location.

If we dig into the dash.py code We can see the construction of the URL paths for _dash-layout _dash-dependencies occurs on lines 73-80:

add_url(
    '{}_dash-layout'.format(self.config['routes_pathname_prefix']),
    self.serve_layout)

add_url(
 '{}_dash-dependencies'.format(self.config['routes_pathname_prefix']), self.dependencies)

The next question is, where is routes_pathname_prefix defined? On lines 46-51 routes_pathname_prefix is mapped to url_base_pathname:

self.url_base_pathname = url_base_pathname
self.config = _AttributeDict({
    'suppress_callback_exceptions': False,
    'routes_pathname_prefix': url_base_pathname,
    'requests_pathname_prefix': url_base_pathname
})

Which begs the question, where is url url_base_pathname defined... on lines 20-28:

class Dash(object):
    def __init__(
        self,
        name=None,
        server=None,
        static_folder=None,
        url_base_pathname='/',
        **kwargs
    ):

As Frederick said, the request should not have the '/' prefix in the GET request. Unsurprisingly, having '//' in your request paths is not good practice (see this SO answer and this one) as it can cause problems depending on how requests are handled. The software layers of Shinyproxy are reasonably complex with requests being handled by multiple technologies. The best solution after this trouble-shooting exercise is to use the following code in your Dash app.

# In order to work on shinyproxy (and perhaps other middleware)
app.config.supress_callback_exceptions = True
app.config.update({
    # remove the default of '/'
    'routes_pathname_prefix': '',

    # remove the default of '/'
    'requests_pathname_prefix': ''
})

Talk - Clustered, Distributed File and Volume Storage with GlusterFS

smcleod.net — Tue, 14 Nov 2017 02:00:00 GMT

Using GlusterFS to provide volume storage to Kubernetes as a replacement for our existing file and static content hosting.

This talk was given at Infracoders on Tuesday 14th November 2017.

Click below to view slides (PDF version):

Direct download link

Come To The Edge by Christopher Logue

ho.dges.online — Tue, 14 Nov 2017 02:00:00 GMT

Come to the edge.
We might fall.
Come to the edge.
It’s too high!
COME TO THE EDGE!
And they came,
And he pushed,

Recurse Center 5: command line arg parsing

Pete's Points — Tue, 14 Nov 2017 00:52:32 GMT

This morning I integrated the rust clap crate into some of my little CLI exercise utilities and learned a bit about dealing with a mix of Option and Result types, which is annoying. clap is really nice. I found a problem with the docs and filed a github issue and sent them a PR. It's actually ultimately an issue with the crates.io website code so I'm hoping one of those maintainers makes a fix that will handle the situation for all crates.

I've been basically trying to make a customized aws-vault and hitting some frustrations. Not sure if this approach makes sense but we'll see.

Hello World!

The Technical Blog of James on purpleidea.com — Mon, 13 Nov 2017 16:18:15 GMT

Hello! This is a new version of my website and blog. If you experience any issues, please let me know!

Happy Hacking, James

You can hire James and his team at m9rx corporation.
You can follow James on Mastodon for more frequent updates and other random thoughts.
You can follow James on Twitter for more frequent updates and other random thoughts.
You can support James on GitHub if you'd like to help sustain this kind of content.
You can support James on Patreon if you'd like to help sustain this kind of content.

Practical Coprocesses

Anardil — Mon, 13 Nov 2017 10:00:00 GMT

A practical example using Bash coprocesses

San Francisco, Downtown

Avy Faingezicht — Mon, 13 Nov 2017 02:00:00 GMT

>Light can be instantaneous. You must be ready, and you must have the damn camera with you. > > -Jay Maisel, Light Gesture And Color I have been trying to take my camera with me more often when I go about my daily life. I usually bike to work, but when I take public transit instead, this is what happens:

Be interesting to somebody

Yasha Solutions — Mon, 13 Nov 2017 02:00:00 GMT

Not to anybody. You cannot be interesting to anybody. Whatever medium you are using to express yourself – it won’t work with every human being. So instead of trying to please the world – go make sure the people your information is intended to reach – actually catch their attention. If you don’t know – just pick one and go with it. You will have an other opportunity tomorrow to write for the other crowd you have ignored today.

Quick Thoughts: Machine Learning

Pete's Points — Sun, 12 Nov 2017 16:33:55 GMT

Things machine learning plus UX teams seemingly can't get right yet:

Is this non-friend facebook message important or not?
Do I care if this person liked some random thing on facebook?
Do I need notifications or directions to an event I'm already at?
Am I in a vehicle right now?
What zoom level would be useful for this map?

Cleaning up locally installed Rubies & Gems

Caius Theory — Sun, 12 Nov 2017 13:30:24 GMT

Given the following is broadly true:

Rubies installed to ~/.rubies (via ruby-install most likely)
Gems for each ruby version installed under ~/.gem (via chruby most likely)

What to do when you want to reclaim some disk space? Delete unused ruby versions of course! Pretty straight forward, look in ~/.rubies for ones you want to remove, then delete them.

$ ls ~/.rubies
ruby-2.0.0
ruby-2.1.7
ruby-2.3.1
ruby-2.4.1
$ rm -r ~/.rubies/ruby-{2.0.0,2.1.7}

Then the problem is we’re left with artifacts hanging around, namely any gems we installed for ruby 2.0.0 or 2.1.7 are still present under ~/.gem using up disk space. We could go through and find them by hand, or we could get the computer to delete anything under ~/.gem that doesn’t have a corresponding runtime under ~/.rubies.

diff --old-line-format= --unchanged-line-format= --new-line-format=$HOME/.gem/ruby/%L \
 <(ls ~/.rubies | sed -Ee 's/ruby-|-p[0-9]+//g') <(ls ~/.gem/ruby) \
 | xargs -pL1 rm -r

(xargs -pL1 will prompt with each command it wants to run before running it - answer y to proceed, anything else to prevent it running that command. Lets you see what ruby versions it is removing before it does so.)

$ ls ~/.gem/ruby
2.0.0
2.1.7
2.3.1
2.4.1
$ diff --old-line-format= --unchanged-line-format= --new-line-format=$HOME/.gem/ruby/%L \
<(ls ~/.rubies | sed -Ee 's/ruby-|-p[0-9]+//g') <(ls ~/.gem/ruby) \
| xargs -pL1 rm -r
rm -r /Users/caius/.gem/ruby/2.0.0?...y
rm -r /Users/caius/.gem/ruby/2.1.7?...y
$ ls ~/.gem/ruby
2.3.1
2.4.1

And now revel in your reclaimed disk space. (Hunting other large folders/items on your disk? ncdu is a great tool for that.)

iOS dev command-line goodies

xenodium.com @alvaro — Sun, 12 Nov 2017 02:00:00 GMT

Install ipa on device

Get utility with:

npm install -g ipa-deploy
npm install -g ios-deploy

Install ipa on connected iPhone:

ipa-deploy path/to/your/App.ipa

Install app on booted simulator

Install ipa on connected iPhone:

xcrun simctl install booted path/to/your/App.app

Install ipa on booted simulator

#!/bin/bash

# Unzip ipa, install app, and run on booted simulator.

set -o nounset
set -o errexit

readonly IPA_PATH=$1
readonly TEMP_DIR_PATH=$(mktemp -d)
readonly BASENAME=$(basename ${IPA_PATH})
readonly NAME=${BASENAME%.*}
readonly APP_DIR_PATH="${TEMP_DIR_PATH}/Payload/${NAME}.app"
readonly PLIST_FILE_PATH="${APP_DIR_PATH}/Info.plist"

trap "rm -rf ${TEMP_DIR_PATH}" EXIT

unzip -o "${IPA_PATH=}" -d "${TEMP_DIR_PATH}"

readonly BUNDLE_ID=$(/usr/libexec/PlistBuddy -c "Print CFBundleIdentifier" ${PLIST_FILE_PATH})

xcrun simctl install booted "${APP_DIR_PATH}"
xcrun simctl launch booted "${BUNDLE_ID}"

Halloween 2017

Avy Faingezicht — Sun, 12 Nov 2017 02:00:00 GMT

This year for Halloween we went out in the Castro. It was a fun night, but to be honest the costumes were not as interesting or outrageous as I expected them to be. I had planned to go to the Dia De Los Muertos event in the Mission a couple of days later, and post photos from both days in one set, but I messed up, and went to MoMa the day of instead, so you only get the Halloween photos. Sorry.

Recurse Center 4: Reservoir Sampling

Pete's Points — Sat, 11 Nov 2017 01:08:36 GMT

This morning I battled the borrow checker for a while trying to properly pass a variable through a closure to a child thread. I eventually figured out calling .to_owned() in the right place was needed, but it took a long time to get to that point. My normal urge is to try to understand what's happening by studying closely the compiler error message and the standard library docs, but for pragmatic reasons I need to let go of that and not wait to just google "how to X in rust" sooner rather than later.

In the afternoon I paired with Casey on refining my rust basics for my little toy CLI apps and learned some good idioms there with the question operator. Then we refactored my tirefox program to use the reservoir sampling algorithm from The Art of Computer Programming so we could sample our words file in a single pass instead of needing a separate pass to count the words.

I managed to get in 2 sets of physical exercise which some of the RCers do periodically and two halfway decent naps which made a huge difference (it's after 6PM and I'm still functioning). Another takeaway is that I learn a lot while pairing so I intend to request to pair very frequently and accept all offers of pairing.

Recurse Center 3: Box

Pete's Points — Fri, 10 Nov 2017 06:58:32 GMT

Today I wrote a little unix utility to ensure a given line is present in a file. It's not that much longer than I would expect in a scripting language (60 lines vs probably ~10 for python), but the number of machinations I had to go through was vastly larger. To get basename of argv[0] I had to make a Path struct which brings lots of Option/Result handling into things. I'm getting less terrible at reading the standard library reference documentation. I continue to be frustrated that the expandable sections for each method are all expanded by default. Maybe I'll work on a PR for that later assuming it's not baked into the tool they use to generate the docs.

The timing in NYC is indeed proving to be tricky as I predicted. I work best first thing in the morning, but that's when the subways are most crowded. So RC mostly starts late around 10:30ish and goes late. I also start to fade fast into late afternoon exhaustion around 4PM. Today I just came in early. The commute wasn't that bad, but I had to let 2 trains pass because they were totally full. I think I'll end up working at home or from a coffee shop in Park Slope or from Brooklyn Boulders in the mornings for about 2 hours, then commute into RC, and either head home on the early side or find exhaustion-compatible things to fill the last 2 or so hours before commuting back home.

Oh and I learned 3 new ways to make coffee so far: chemex, aeropress, and keurig (not as easy as you might assume). Only tried 2 myself but tomorrow I'll try making a mug via aeropress.

I also paired with Casey at the end of the day and got a good verbal explanation of the differences between Refs and Boxes in rust.

Something Goes Here

Caius Theory — Fri, 10 Nov 2017 02:27:07 GMT

Something goes here
and something else
over there slowly
wrapping lines a
little each time
oh no, you fucked
up one little comma
and now everything is running away drastically from you
End it.

Easy Recipe For Making Chili at Home with Matt Crampton

MattCrampton.com — Thu, 09 Nov 2017 08:00:00 GMT

Easy Recipe For Making Chili at Home with Matt Crampton Cooking chili doesn't have to be a pain, use this easy recipie to make it at home. Easy Recipe For Making Chili at Home Cooking chili is easy if you make it my way. Recipe as follows... Ingredients: 1 pound of ground beef 1 onion 1 can of Ro-Tel (chopped tomatoes and green chilis) 1 can of creamed corn 1 can of tomato paste 1 packet of chili seasoning 2 cubes of frozen garlic (or just chop up a bunch of garlic) Instructions: Throw your ground beef in a pot. (I use a cast iron dutch oven.) Brown the meat over medium heat until mostly brown. Chop up the onion and add it to the pot. Pour the contents of the Ro-Tel, creamed corn and tomato paste into the pot. Drop in the garlic. Simmer at low heat. Stir every 5 to 10 minutes. Cook for as long as you like, at least 45 min. (Also remember to get your bread ahead of time).

Links - November 09, 2017

Avy Faingezicht — Thu, 09 Nov 2017 02:00:00 GMT

Go v1.9.1 vs Node v8.9.0 Microbenchmark

Thomas Hunter II — Thu, 09 Nov 2017 02:00:00 GMT

Four Fun Phobias

huphtur — Thu, 09 Nov 2017 02:00:00 GMT

Running terraform and docker on my home server

Nemo's Home — Thu, 09 Nov 2017 02:00:00 GMT

The last time I’d posted about my Home Server build in September, I’d just gotten it working. Since then, I’ve made a lot of progress. It is now running almost 10 services, up from just Kodi back then. Now it has a working copy of:

Kodi

I was running kodi-standalone-service, set to run on boot, as per the ArchLinux Wiki, but switched in favor of openbox to a simple autorun.

Steam

The current setup uses Steam as the application launcher. This lets me ensure that the Steam Controller works across all applications.

Openbox

Instead of running Kodi on xinit, I’m now running openbox with autologin against a non-privileged user.

PulseAudio

I tried fighting it, but it was slightly easier to configure compared to dmix. Might move to dmix if I get time.

btrfs

I now have the following disks:

128GB root volume. (Samsung EVO-850)
1TB volume for data backups
3TB RAID0 configuration across 2 disks. There are some btrfs subvolumes in the 3TB raid setup, including one specifically for docker volumes. The docker guide recommends running btrfs subvolumes on the block device, which I didn’t like, so I’m running docker volumes in normal mode on a btrfs disk. I don’t have enough writes to care much yet, but might explore this further.

Docker

This has been an interesting experiment. Kodi is still installed natively, but I’ve been trying to run almost everything else as a docker container. I’ve managed to do the configuration entirely via terraform, which has been a great learning experience. I’ve found terraform much more saner as a configuration system compared to something like ansible, which gets quite crazy. (We have a much more crazy terraform config at work, though).

Terraform

I have a private repository on GitLab called nebula which I use as the source of truth for the configuration. It doesn’t hold everything yet, just the following:

Docker Configuration (not the docker service, just the container/volumes)
CloudFlare - I’m using bb8.fun as the root domain, which is entirely managed using the CloudFlare terraform provider.
MySQL - Running a MariaDB container, which has been configured by-hand till this PR gets merged.

Gitea

Running as a docker container, provisioned using terraform. Plan to proxy this using git.captnemo.in.

Emby

Docker Container. Nothing special. Plan to set this up as the Kodi backend.

Couchpotato

Experimental setup for now. Inside a docker container.

Flexget

I wish I knew how to configure this. Also inside docker.

traefik

Running as a simple reverse proxy for most of the above services

elibsrv

A simple OPDS server, which I use against my Kindle. If you don’t know what OPDS is, you should [check this out][]. Running on a simple apache setup on the archlinux box for now. WIP for dockerization.

ubooquity

Simple ebook server. Proxied over the internet. Has a online ebook reader, which is pretty cool.

MariaDB

I set this up planning to shift Kodi’s data to this, but now that I have emby setup - I’m not so sure. Still, keeping this running for now.

Transmission

Hooked up to couchpotato,flexget, and sickrage so it can do things.

Sickrage

Liking this more than flexget so far, much more easier to configure and use.

AirSonic

This is the latest fork of libresonic, which was itself forked off subsonic. My attempt at getting off Google Play Music.

Learnings

Moved these to a separate blog post

TODO

A few things off my TODO list:

Create a Docker image for elibsrv that comes with both ebook-convert and kindlegen pre-installed
~~Do the same for ubooquity as well~~ (Using the linuxserver/ubooquity docker image)

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Rapid Website Development with Mojolicious and Polymer

Lambda Land — Thu, 09 Nov 2017 02:00:00 GMT

My girlfriend works for BYU SA—it’s the division of BYU that’s responsible for planning and running events. As part of her job, she has to review song lyrics and make sure that the song is okay to play at BYU functions.

This can get rather irksome. Imagine reading text looking for vulgar words or phrases. Yuck. I took some time this evening to write a little website that checks MetroLyrics for any vulgar words or phrases. I have an extensible blacklist which gets initialized at server start by some phrases from FrontGate.

I coded this up with Mojolicious and Polymer Web Components. Polymer might have been a little overkill, but it saved me from writing lots of boring jQuery to set variable.

Here’s what the site looks like:

You can look at the code for the repository here.

This all took about 3 hours, the last of which was spent mostly getting some of the finer points of the user interface to work.

I’d be happy to get some contributions to my project, if anyone is feeling a little bored. ;-) Some more robust song searching would be nice, as well as better heuristics for bad phrases and whatnot.

Recurse Center 2: Generics

Pete's Points — Thu, 09 Nov 2017 00:27:21 GMT

Worked through 3 of the really tricky chapters of Rust By Example today. Still not a ton of lightbulbs illuminating but it will come with more coding. I feel hesitant to code more exercises until I get through the next 3 chapters on error handling etc and then I'll have a bit more knowledge whereas this morning hacking through stuff it was clear I didn't have the basics down when I had working code and just wanted to extract it unchanged into a function, which should in theory be straightforward, but I was unable to express the type signatures correctly.

Also paired a little bit on passport.js and nginx with my batchmates.

Already had a bit of panic about what if I need 2 weeks just to get basic fluency in rust then I have only 4 weeks to code on a project.

C++ Hello World with Classes

Studiofreya SSG Site — Wed, 08 Nov 2017 22:13:37 GMT

In this article we will show you how to write C++ Hello World program using classes with constructors and destructors. We will create different beginner C++ programs that will output “Hello, World!” as an example.

Hello World

The most trivial example is to create a simple cpp HelloWorld file inside your C++ project and a class with the same name. Now we are going to call a method in this class that prints “Hello World!”.

Showing how poor performance affects user behaviour

Phil Booth's Blog — Wed, 08 Nov 2017 09:18:22 GMT

“Bad performance hurts user engagement” is a sentiment that feels intuitively true but can be hard to sell in product conversations. The best way to persuade a non-believer is to point them at hard evidence and for that you need to do a few things with your performance data.

Firstly, don’t store it in a silo. That might sound obvious but silos occur incidentally for all kinds of organisational reasons that can be hard to resolve. Conway’s law is real and causes inertia. If you want to draw meaningful inferences from your data, it must be possible to link it to other events that track user behaviour.

Consider the following graph (numbers redacted):

Here, the x-axis is time and the y-axis is the count of users that successfully signed in to Firefox Accounts. The different coloured lines represent evenly-sized cohorts of users, grouped according to how quickly the initial page loaded. If web performance had no effect on user behaviour, those lines would roughly overlay each other. Instead, they show that the 10% of users who experienced the fastest initial page load were about twice as likely to sign in to their account, as the 10% of users who experienced the slowest. If you’re having trouble convincing others to invest engineering effort in improving performance, having graphs like this can help. And they’re only possible if you can directly link performance data to user events.

Secondly, be careful if you’re using a time-series database like Graphite. These are often configured to gradually expire data, which is good for storage costs but bad for precision. The expiry is achieved by storing averages of your metrics in progressively coarser-grained buckets as they get older, which reduces precision and can produce strange jumps at the bucket boundaries when rendered in a chart. People looking at those might mistake them for genuine performance changes.

Instead of a time-series database, consider storing your data somewhere that can handle it en masse. We use Redshift for this, maintaining parallel datasets that trade history length for sample rate. We have an unsampled set that contains 3 months of history, a 50%-sampled set that goes back 6 months and a 10%-sampled set that includes 2 years. After the initial sampling is performed for each set, we keep all of the data until it expires. Data is automatically popped from the end of each set as we push it at the start. Limiting the size of the sets keeps query times reasonable. In practice, we rarely need to consult the sampled histories but it’s good to have them available for occasional longer-term analysis and maintaining them in parallel means we can always compare like with like. Medians and percentiles are computed on the fly inside queries, which can be scheduled to run at regular intervals in the background so we don’t have to wait for them every time we want to view a chart.

Finally, produce multiple visualisations of your data. For instance, the chart I showed earlier has a partner that maps the percentile boundaries to concrete timings:

This is useful, but it’s a bit annoying to look at a separate chart when you want to know the timing value. So we have another chart that tries to combine the work of both into a single view:

Here, each band is between concrete timing values and the y-axis is the percent of users in each band that successfully signed in to their account. It tells the same story as before, with some extra spikiness due to the interaction between weekend behaviour and performance. Backing up an argument from one chart with evidence from another makes our case more compelling.

It also helps make our dashboards useful for other purposes, like keeping an eye on performance regressions and tracking how we’re doing over time. Here are the same 1-second timing bands rendered to show how many people are in each:

Ideally, we want to see users moving from the red bands at the top of the chart towards the green bands at the bottom. If the opposite occurs, we’re doing something wrong and we know how that affects user behaviour.

Recurse Center 1: Shuffle Iterator

Pete's Points — Wed, 08 Nov 2017 01:38:20 GMT

This post is pretty stream of consciousness.

Thoughts on rust language and syntax so far (about ~45 hours spent studying, trickled in over the last 6 months or so).

The Hard

rust stuff that are hard for me, unfamiliar, or I haven't studied yet

ref/stack/heap
move/borrow
lifetimes
traits
generics
ampersand ref syntax, tick lifetime syntax
keeping track of intermediate type conversions when chaining methods like .iter().map().collect()
super/self unclear when/how these are OK to use, especially self

The Easy

rust stuff that is easy for me or familiar

pub, access
most syntax
- function
- let assignment
- match
- loop
- if
- most keywords
Cargo.toml
most cargo subcommands are clear enough for the basics
mod keyword (mostly I think)
Option/Some/None, Result/OK/Err (similar to elm)

The Bad

rust stuff I dislike

double colon namespacing, one should suffice
C-style abbreviations
turbofish type hinter seems weird both syntax and position
- unclear when I need an annotation before the assignment, a turbofish, or both/neither
println macro is so verbose
something feels weird about the use statement when bringing a trait into scope but not getting a name for that. Why is this necessary?
I mostly don't like aliasing you can do with use. I acknowledge it's necessary but linters should check it's only used when necessary. If you alias something from the std lib just because you don't like the name, you discard a bunch of easy understanding and readability by Un-Ronsealing (https://en.wikipedia.org/wiki/Does_exactly_what_it_says_on_the_tin) stuff
attribute syntax is overly verbose
Needing doc comments on every line. I'd so much rather just have a start and end delimiter so hard wrapping via editor tooling was easy
- maybe make an atom plugin for this?
attribute config language. Why is there a not(fool = "bar") syntax instead of foo != bar like actual rust

The Good

rust stuff I like

lower_snake_case for most things
both separating external API tests but keeping internal unit tests in the same file as the code feel good to me
benchmarking support baked in, not that I've ever benchmarked anything, but it's cool

Notes and Questions from Today

don't skim the rust by example snippets, read them line by line or you'll miss key stuff
are super and self always referring to module scopes?
I'm losing/lost interest in the web because I feel like it's actively and rapidly and irrecoverably getting worse (mostly due to advertising and corporate interests) vs unix/command line which is at least a stable level of brokenness and not getting worse.
Need a totally new-to-me coding process to deal with rust
- First just get the type signatures right and return a hard coded basic value
- one line at a time try to get something working, anything
- separate src/scratch.rs file I can periodically wipe clean to have a low-context area to attempt things in isolation
social screw ups: I referred to people as "guys" once. I'm sorry.

Recurse Center 0: Rust Anagrams

Pete's Points — Tue, 07 Nov 2017 05:05:19 GMT

Today was my first day at Recurse Center! I was excited. Like Christmas excited. And in fact, I woke up at 5AM due to said excitement. We weren't supposed to show up earlier than 10AM so I had a while to wait around in the morning.

All day was orientation type stuff about the logistics of the space, time to meet people and chat over food, panels about how best to use our time here, and some lightning talks.

We did a pair programming exercise which I did in rust. We managed to get a no-op version compiling and then spent the remaining 45 minutes with various flavors of non-compiling code and battling lifetimes. After dinner I took another crack at it and had something passing all the tests after another 2 hours most of which was reading online trying to find examples that did the type conversions I needed to appease the compiler.

https://github.com/focusaurus/rust-basics/blob/master/anagram/src/lib.rs

I was pleased that RC encourages naps and plan to nap at least once a day. There are also 2 people from the F2 batch doing some rust and a bunch of folks knowledgeable about rust on the RC chat server which I think will be great because getting in-person help can save tons of time with hard-to-google rust type conversion errors such as I had tonight which amounts to "I have Vec<&&str> but I need Vec<&str>" which as you might predict is tough as a web search term.

I found these two good blog posts in my search:

I also learned what a Functor is in one of the lightning talks and no it's not as hard as it sounds but oh man it is nothing like I what I thought it might be. In fact, after the talk I asked a question that amounts to "Wait so which of your examples is actually the functor itself: A or B" and the answer was "neither, it's C" and that was very surprising.

Today was I believe the one and only structured day at RC. Starting tomorrow it's entirely self-directed.

2017-11-07

ho.dges.online — Tue, 07 Nov 2017 02:00:00 GMT

Sense8: Season 2

🦉 olshansky 🦁 — Sun, 05 Nov 2017 02:18:10 GMT

Olshansky's review of Sense8: Season 2

Data Storage on Unix

Venam's Blog — Patrick Louis (Lebanon) — Sun, 05 Nov 2017 00:00:00 GMT

Libraries and banks, amongst other institutions, used to have a filing system, some still have them. They had drawers, holders, and many tools to store the paperwork and organise it so that they could easily retrieve, through some documented process, at a later stage whatever they needed. That's where the name filesystem in the computer world emerges from and this is one of the subject of this episode. We're going to discuss data storage on Unix with some discussion about filesystem and an emphasis on storage device.

Four Simple Steps To Post To Twitter Using Python

MattCrampton.com — Fri, 03 Nov 2017 07:00:00 GMT

Four Simple Steps To Post To Twitter Using Python NOTE: Don't miss the part 2 of this blog post: Posting Photos to Twitter with Python Automating Twitter can be an easy coding exercise once you have everything set up, but figuring out all the steps that Twitter puts you through to generate your access keys and tokens can be a challenge. I put these four steps together to make it easier to walk to the process, hopefully it's useful for someone else out there. Create An Application On app.twitter.com Install Python Dependencies Write The Python Code Post Your First Tweet TLDR; If you just want the code, you can grab the code repo for this tutorial here: https://github.com/mattccrampton/post_to_twitter_using_python Step #1 - Create An Application on app.twitter.com First head over to https://apps.twitter.com/, sign in to your twitter account and then click on the "Create New App" button. Enter your application's information. For the purposes of this demo I'm creating an application to ...

Digital Security Coach

Danny Guo — Fri, 03 Nov 2017 02:00:00 GMT

My journey to find the best Bible

Honza Pokorný — Thu, 02 Nov 2017 18:00:00 GMT

… or, ESV Pitt Minion in black goatskin review

I have been reading Mark Bertrand’s Bible Design Blog for a few years now. Through his long-form writing, I have come to appreciate beautiful typography, fine craftsmanship, single-column layouts, and attention to detail in modern Bible-making. I even helped him with a website update several years ago.

For the longest time, though, my reading was just that—reading. I couldn’t imagine spending so much money on a single Bible. I already had a few Bibles in the house, and there wasn’t any pressing need to buy another one. This spring, after saving up some money, I decided to invest in a premium Bible. I made a list of requirements, and I read and watched just about every review I could find on the world wide web. I was searching for the one Bible, the elusive unicorn of typographic design. I read articles about how that Bible doesn’t exist, and that I should just come to terms with the fact that I shall have to acquire a few different Bibles, each one for its own intended purpose.

I became addicted to the idea of single-column design. I read all about it. I could absolutely sign Mark Ward’s Bible Typography Manifesto. Two years before the search began, my wife bought me the single-volume ESV Reader’s Bible.

This Bible is relatively cheap, and gorgeous. It looks a lot more like a regular book than a dictionary or a reference book. It features a large font, single-column layout, … but its significance is in what it lacks: verse numbers, references, section headings, book introductions, etc. It’s very clean and a joy to read. I highly recommend it to anyone, and often show it off to visitors to our house. My friends are usually dismissive of the idea of a single column Bible because it doesn’t look like a real Bible. The inertia of Bible design is so tremendous that people experience a mild culture shock when looking at modern Bibles.

Unfortunately, the Reader doesn’t work as the one Bible.

From the start, I knew I didn’t want to become a Bible collector. I have no desire to own thousands of dollars worth of Bibles. I wanted to acquire a single Bible that I could use in many different scenarios. The primary motivation for this is that over time, you get to know your Bible, and you will learn where things are, and you can find them quickly. Professor Grant Horner is an advocate of this argument in his 10 chapters per day reading plan which I enjoy.

So, let’s see if we can put together a list of priorities. The Bible must work during private reading, study, public worship, family worship, and adult Bible study with others. It must be easy to read, and yet be small enough that it’s practical to carry everywhere. It must be available in the English Standard Version. It must be pleasing to the eye and to the touch. It should be rendered in single-column layout. Since I want it to be as practical as possible, it should come with verse numbers, and section headings. References are unimportant.

I was immediately captivated by Crossway’s ESV Heirloom Single Column Legacy, and Cambridge’s Clarion.

Here is the Heirloom:

(Photos via evangelicalbible.com)

Judging by the photos in Mark Bertrand’s review of the new ESV Heirloom Legacy, the Bible looks huge. This means two things for me: it features a large, readable page, and it’s likely too big to carry everywhere. This would make a great sit-at-home-and-read-for-hours Bible, but I wouldn’t drag it to church with me. Still, the page looks amazing, single column; I love the headings on the side instead of in the text. I like that it has verse numbers and no references. It makes the text look clean. This Bible will set you back 275 USD.

And here is the Clarion:

(Photos via evangelicalbible.com)

The Clarion is short and stout. I like the shape of it better than the Heirloom Legacy. Given its supple cover, it can be difficult to hold. It features references and a single-column text block. For some strange reason, the references are on the outside. This causes the text of the Bible to dip into the fold near the spine. Crossway’s Personal Reference Bible is also rendered in a single column setting but its references are in the middle. This is a much better solution in my opinion. Unfortunately, the Clarion seems to have problems with ghosting (also known as show-through), page curling, and while it’s smaller than the Heirloom, it’s still rather bulky. It’s about the same size as my ESV Reader’s Bible which definitely isn’t very portable. The Clarion costs about 255 USD.

These two Bibles are absolutely excellent, and very expensive. If I were a collector, I think both of these should be in my collection. However, that’s not why we’re here.

After dismissing the Heirloom and the Clarion, I was browsing publishers' websites, and came across another Bible from Cambridge: the Pitt Minion. The size of this Bible is absolutely spot on: not too big (portable), and not too small (readable), not too short and not fat. I absolutely love the black goatskin edition with the red accent of a ribbon. Only problem? It’s double column.

Initially, I dismissed it—almost immediately. Who would buy a double column Bible in 2017? But I kept thinking about the Pitt Minion, and kept coming back to it. I read all of Mark Bertrand’s reviews of it over and over. I think there are at least four articles on his website dedicated to the Pitt Minion. Mark’s argument for the Pitt Minion is that its columns are magic. There is something about the ratio between the column’s width, the font size, and the leading that makes it just work. This leads to high readability in spite of the tiny 6.75 font. Also, the size of the Bible is somewhere in between a pocket Bible and a medium sized Bible. This makes it unique in the kind of situations you’d want to use it in.

I also greatly benefited from Matthew Everhard’s videos. Matthew is a pastor who uses a Pitt Minion has his take-it-with-you-everywhere Bible, and does a video review once a year to show how the Bible is doing with daily use. I love how the leather ages, and how the board covers soften as you use it.

Once I was willing to seriously consider the double-column layout, I went back to the drawing board, as it were, and had another look at the available options. I have yet to see a Bible from R. L. Allan that I like (sorry folks). But Schuyler is a different story! Their Quentel is very intriguing, especially the Personal size version.

(Photos via evangelicalbible.com)

The only problem is the price. I live in Canada, and shipping is around 50 USD, and I’m probably going to get hit with import tax fees, etc. The Bible itself is about 175 USD. Evangelical Bible really needs to figure a better distribution model for their awesome Bibles. A lot more people would buy them if the shipping were cheaper (e.g. partner with Amazon).

After much thought, I decided to order the Pitt Minion. Another huge reason was its price. It costs about 150 USD and I even found a great deal on Amazon, so I ended up paying about 100 USD (shipped). Compare that to the 225+ USD for the Quentel. The Pitt Minion is a steal! I opted for the black goatskin which to my eyes and heart looks the best. Just as a side note, you can get the Clarion for about that price but with a much inferior leather cover.

Just look at that delicious grain!

The book opens flat after a few days of handling it.

Notice the red under gold art gilding on the sides of the pages. Just gorgeous.

I love the red accent that the ribbons provide.

(Photos by me, you can use them for any purpose)

While I was waiting for it to arrive in the mail, I was rather nervous about the whole thing. But as soon as I opened the box, I was relieved and happy with the new Bible. It far exceeded by expectations. It’s even more beautiful than in the pictures.

Right out of the box, the Pitt Minion doesn’t open flat. You have to handle it and read it for about a week, and then it will happily oblige. The quality of the print is exceptional. Many reviewers have said that the font is small. That might be true in the technical sense, but given the high quality of the printing, the font doesn’t feel small. Think of it like this: your normal Bible is like an old TV, it has very low resolution and unnatural colors. The Pitt Minion is your latest 4K smart TV with life-like colors. Premium Bibles stand out not just wit their binding and covers, but also with the quality of the printing. I find the font size of the Pitt Minion luxurious.

Then there is the question of the double column. I remain convinced on a theoretical level that the single column layout is superior in readability. However, it does in a sense waste space in poetry sections. This leads to more bulk in the final form factor of the Bible. In practice, I don’t feel any hindrance in reading the Pitt Minion. My eye can track the words easily, and I don’t find I lose my place often when reading for extended periods of time. Moreover, there is something to be said about the Bible looking like a Bible, and double column is certainly the more traditional look. I realize this is a pretty weak argument. With Cambridge, you are purchasing a piece of history or Bible publishing history. Cambridge University Press is the world’s oldest Bible printer and publisher. They have been in continuous operation since 1591 when they published the Geneva Bible.

Once you handle a premium Bible, you won’t settle for inferior quality. I have been reading from the Pitt Minion for several months now. I have carried it many places. I travelled to Europe with it. I certainly don’t baby this Bible. I fold back one side of the cover when reading, I roll it up when it’s closed, and I take it with me when going out. I find it delightful, and it’s truly a great all-around Bible.

Java Hello World – How to Run

Studiofreya SSG Site — Thu, 02 Nov 2017 12:55:37 GMT

In this article we will show you how to start programming in Java from scratch. We will use Java 9 for this beginner tutorial and write and run a “Hello, World!” example program on your computer.

Installing Java

In order to start programming in Java you will need to install Java on your computer.

Step 1. Install JDK
Go to the Oracle website and download and install the latest JDK pack. We used Java 9 JDK. JDK stands for Java Development Kit and is a library that you need to compile and run Java applications.

How I Structure GameObjects - Components and Mixin-based Inheritance

Varun Ramesh's Blog — Thu, 02 Nov 2017 11:04:17 GMT

My recent game projects have used a GameObject architecture that unifies Component-based GameObjects and Mixin-based inheritance.

Newsletter email confirmation – NextCloud UX copy

Yasha Solutions — Thu, 02 Nov 2017 02:00:00 GMT

Nextcloud is an awesome solution to create and manage your own cloud you should really check them out if you have never heard of them. Here we have a quick look at the newsletter confirmation – an always challenging issue for open rate and engagement with your users. I found this one pretty good – so I have added it here. Why is it good? It gets personal It gets personnal real quick – which force you to care about that guy who is writing to you.

The Mega List Of Free Image Sites For Blogging

MattCrampton.com — Wed, 01 Nov 2017 07:00:00 GMT

The Mega List Of Free Image Sites For Blogging Blogging is hard work, and finding good images to make your content compelling can be even harder. Luckily a number of talented photographers and engineers have taken it upon themselves to make a huge number of free images available for commercial use. Take a look at the list below for a huge set of sites you can use for your next project. This list is a work in progress. If you have a site you think should be on the list, or if any of these links are broken, please tweet it at me. Artvee Discover Classical Art, Browse and download high-resolution, public domain artworks. https://artvee.com/ Albumarium Author Vilem Ries, developed by Filip Skokan & Lukas Prvy http://albumarium.com/ Bucketlistly A free creative common collection of over 5000+ travel photos from all over the world anyone can use. http://photos.bucketlistly.com/ Burst Free stock photos for everyone https://burst.shopify.com/ Cupcake Free (do what ever you want) photos. htt...

Eshell pcomplete company completion

xenodium.com @alvaro — Wed, 01 Nov 2017 02:00:00 GMT

Howard Abrams's Introduction to eshell video prompted me to poke at eshell some more. This time, I got eshell context aware completion by glueing the excellent company and pcomplete packages.

(require 'cl-lib)
(require 'company)
(require 'dash)
(require 'pcomplete)
(require 's)

(defun company-pcomplete--overlap-tail (a b)
  "When A is \"SomeDev\" and B is \"Developer\", return \"eloper\"."
  (let ((prefix a)
        (remaining nil))
    (while (and (not remaining) (> (length prefix) 0))
      (when (s-starts-with? prefix b)
        (setq remaining (substring b (length prefix))))
      (setq prefix (substring prefix 1)))
    remaining))

(defun company-pcomplete--candidates (prefix)
  "Get candidates for PREFIX company completion using `pcomplete'."
  ;; When prefix is: "~/Down" and completion is "Downloads", need
  ;; to find common string and join into "~/Downloads/".
  (-map (lambda (item)
          (if (s-starts-with? prefix item)
              item
            (concat prefix (company-pcomplete--overlap-tail prefix item))))
        (all-completions prefix (pcomplete-completions))))

(defun company-pcomplete (command &optional arg &rest ignored)
  "Complete using pcomplete. See `company''s COMMAND ARG and IGNORED for details."
  (interactive (list 'interactive))
  (case command
    (interactive (company-begin-backend 'company-pcomplete))
    (prefix (company-grab-symbol))
    (candidates
     (company-pcomplete--candidates arg))))

Don't forget to add company-pcomplete to company-backends, and if you want an explicit binding, use something like:

(bind-key "<backtab>" #'company-complete eshell-mode-map)

Challenger Bank

Craig Pardey — Wed, 01 Nov 2017 02:00:00 GMT

When I first heard about the so-called UK Challenger Banks like Monzo and Atom, I was amazed that there wasn’t a similar offering in Canada. The Canadian retail banking sector desperately needs new players bring compelling products to the market but nobody was doing it. There are some options like PC Financial and Tangerine but they’re both backed by one of the Big Banks (CIBC and Scotiabank, respectively). Nobody was attempting a ground-up rebuild of a bank using modern technology and practices.

Postgres - the non-code bits

CRAIG KERSTIENS — Tue, 31 Oct 2017 22:55:56 GMT

Postgres is an interesting open source project. It’s truly one of a kind, it has its own license to prove it as opposed to falling under something like Apache or GPL. The Postgres community structure is something that is pretty well defined if you’re involved in the community, but to those outside it’s likely a little less clear. In case you’re curious to learn more about the community here’s a rundown of a few various aspects of it:

PostgreSQL License

Let’s start with the legal part first. First IANAL. PostgreSQL is under its own license. For those who don’t regularly follow software licensing it’s extremely liberal and flexible. You can take Postgres, fork it, change it, package it up, and resell it. This is actually one of the reasons you see Postgres at the core of so many other databases like Par Accel, Asterdata, etc. That and that it’s such a solid code base that is capable of being extended. I once heard someone describe how they don’t really like writing C, but they enjoy writing Postgres C ;)

A thing you can’t do is profit off the PostgreSQL logo without any approval from the core team.

The people

Within the Postgres community there are 2 major sets of people.

The core team

The core team is a smaller team within the Postgres community. The core team is effectively a steering committee for Postgres. They’re responsible for coordinating releases, handling confidential issues (read: security issues), managing permissions around PostgreSQL code and infrastructure, defining policy.

The core team is a very small list of people, at the moment 5 individuals.

Contributors

Yes, anyone can contribute to Postgres, and with each release there are a laundry of people that write some code that goes into Postgres. If fact Postgres 10 had 325 people that contributed in some form. That said there is a hierarchy that exists. The two biggest ones are committers and major contributors.

Committers gain access after years of contributing to Postgres showing sustained commitment to the project. New committers are voted on each year at PgCon which happens in March/April in Ottawa. If you’re ever curious for a conference of what’s coming and being deep in the internals of Postgres it’s one to check out. Once you do gain your commit bit you’re expected to contribute every couple of years to the project. And of course there are a number of qualifications such as contributing high quality code and perhaps most key is helping review others contributions.

Major contributors are another notable group. Major contributors don’t have full sole commit access, but are held in a higher regard from consistently contributing major features as well as providing review for others.

Contributors in general are another area worth calling out. While they may not have a flagship feature to their name like the major contributors, Postgres is what it is because of the contributions of everyone.

PostgreSQL the company

Well, it turns out there isn’t a company behind Postgres, it’s one thing that makes it unique–no one can ever “own it”. There are some official PostgreSQL non-profits though in particular the US non-profit and the EU non-profit. These non-profits ensure that the core guidelines are enforced and also give coverage for the community to help put on official community conferences. A few of these happen each year which include:

PGCon - The hackers conference
PostgresOpen SV - A consolidation of PostgresOpen and PGConf Silicon Valley
PGConf EU - The largest PG European Conference which moves around each year

If you’re looking for a way to support the PostgreSQL non-profit organization I’d encourage you to consider joining PostgreSQL.us.

Engaging

So you want to jump into the community, where do you even start? The first place I’d encourage is to subscribe to the mailing list or check out the slack channel. The users mailing is a great one to just jump in and help answer questions and see what people need help with. The hackers list is where you go to get a peek at all the fun debates/discussions/development.

If you’re thinking about contributing it’s a good idea to lurk on the hackers list for a bit first. Then when the commitfest comes chip in and help review some patches and do some testing. Oh and of course, you can always blog about what you’re doing with Postgres and will aim to get it included into Postgres Weekly.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

The Internet worm of 1988: A Tour of the Worm

Blog by Jade Michael Thornton — Tue, 31 Oct 2017 17:00:00 GMT

On the evening of November 2, 1988, a self-replicating program was released upon the Internet. Within the space of hours this program had spread across the US.

PDO vs MySQLi speed comparison

Jim Westergren — Tue, 31 Oct 2017 10:34:00 GMT

My conclusion Based on my testing and the advantages of PDO (info) I recommend the following: Use PDO with PDO::ATTR_EMULATE_PREPARES set to TRUE and use the charset=utf8mb4 in the dsn (to be equally safe as if emulation was turned off). The speed increase is around 80% and this is not the default with for example [...]

reMarkable review

Michael Mior — Tue, 31 Oct 2017 02:00:00 GMT

As an academic, I spend a lot of time reading papers. I generally hate the idea of printing out a paper for a one-time read since it feels wasteful, but I also don’t enjoy the reading experience of viewing papers on a desktop screen. I’ve always liked the look of the Sony Digital Paper that my advisor uses but it’s rather expensive and not easy to find. I used a Samsung Galaxy Tab 3 for several years which freed me from my desk and made note-taking easier, but still wasn’t as nice as paper.

This is why I was excited to hear about the reMarkable. The stated goal of reMarkable is to be a tablet for paper lovers. The cost is still fairly significant for a grad student, and I actually cancelled my first preorder before changing my mind and committing. While I did experience shipping delays that seem inevitable with crowd funded projects, I finally got my device this past Thursday.

So far, I’m pretty impressed by the hardware and the writing experience. There’s definitely no perceivable latency when writing and most other actions happen quite quickly as well. The software still needs a lot of work (especially the Android app) and I’ve had some issues with syncing not working correctly. My biggest disappointment so far, is that the Wi-Fi on the device doesn’t currently support networks which use usernames and passwords for authentication, which means I can’t use wireless connectivity in the office. This is a pretty big drawback, but there’s already a promise to fix this in a future software update. Overall, I’m looking forward to seeing what happens.

The Gym

Martin Rue — Tue, 31 Oct 2017 02:00:00 GMT

Anyone who knows me will know that the gym is a fairly big part of my life. While my goal certainly isn't to become the next Eddie Hall or Phil Heath, during the last 3 years the gym has been a valuable source of discipline and improvement for me.

Born to Run

Krishna's blog — Tue, 31 Oct 2017 02:00:00 GMT

Born to Run is a low-quality pseudo-science book that pushes the agenda of barefoot running without disclosing the lack of conclusive evidence.

A VIM User's Experiments With GUI Code Editors

MattCrampton.com — Mon, 30 Oct 2017 07:00:00 GMT

When You Pry It From My Cold, Dead Hands... I've been a die hard VIM user since the days of co-founding the Kalamazoo Linux Users Group back in 1998. It's a clean command line editor with a simple user interface and once you've mastered the admittedly steep learning curve of its robust keyboard shortcuts, I haven't found any substitutes that measure up for raw horsepower coding. But... Relatively recently there's been a lot of advancement around pairing code editors with debugging utilities which has started to leave editors like VIM behind. The ability to instantly set break points in React code and have that matched in real time with your browser is starting to make me want to lift my head out of my Ubuntu VM and see what all the cool kids are using these days. Visual Studio Code Chrome debugging If you're like me and hail back to the old school days of awk, grep and sed, you might be cringing and the very thought of a mouse click as an integral part of your coding flow, but let's...

Repubblica: crappy journalism from The Usual Suspects

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 30 Oct 2017 04:50:00 GMT

Do you remember “Who is Keyser Soze?”? That was the recurring line of a great movie, which is now related to another question: who, in the staff of italian newspaper Repubblica (and many other newspapers too, of course) still makes, or allows, crap like this?

Light mallet

june.kim — Mon, 30 Oct 2017 02:00:00 GMT

How to Become Invincible

Dan Stroot — Mon, 30 Oct 2017 02:00:00 GMT

Do you want to become invincible? I mean “rock solid”; impenetrable; bullet-proof? The answer is simple: strive to **be the truth**.

This post How to Become Invincible first appeared on Dan Stroot's Blog

Builders vs Maintainers

Byte Size Theories — Sun, 29 Oct 2017 23:44:17 GMT

I have found that software engineers tend to divide themselves into 2 broad categories: Builders (selfish) Maintainers (selfless) This is of course an oversimplification. Most of the time we do a bit of both, but in my experience engineers tend to lean towards one or the other. The beef I have with this is that maintainers rarely do this consciously. They also enjoy building, it’s just that they typically have some trait (perfectionism, humility, shyness, …) that prevents them from pushing to get the opportunity to be a builder.

Quick Dive into React

Josh Thompson — Sun, 29 Oct 2017 15:00:00 GMT

As usual, this is a work in progress. At a high level, I’m familiarizing myself with Phoenix/Elixir, and need to sharpen my React knowledge along the way.

After working through part 1 of a slack clone in Elixir/Phoenix tutorial, I ran into some errors getting the React app up and running.

I expect errors, but don’t have enough familiarity with React to effectively debug, so I’m going to take a quick spin through the tool, see what I can quickly learn, and then debug the errors I saw.

Robots displacing humans... or humans becoming robots?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 29 Oct 2017 09:00:00 GMT

There is lot of fear around about robots, or more exactly software-powered automation, destroying jobs by the millions. But maybe there is still not enough awareness, and concern, about software creating jobs that only a robot would be happy, and fit, to do.

Tips for a successful WooCommerce project

Daniel Immke — Sun, 29 Oct 2017 03:00:00 GMT

Building your first WooCommerce site? Here are 4 things to consider.

Elixir/Phoenix part deux

Josh Thompson — Sat, 28 Oct 2017 10:18:52 GMT

I planned on working through this tutorial for building a slack clone, but half-way through the set-up instructions, after I installed Elixir and Phoenix, I took a long detour through the basic set-up guide. Built some custom routes, along with controllers/views/templates, etc.

I’m now a bit more comfortable with the basics of the framework and language, so I’ll now be working through the tutorial, and Rubber Duck Debugging by writing about it as I go.

This will be a bit of a work-in-progress. Onward.

OK. Basic setup done. Scaffolded the Phoenix and React app, made the initial commit, and we’re cooking with gas.

The setup is pretty straight forward - a quick note on the React/Redux Boilerplate section: I’ve had only passing experience with React/Redux, so I might take a detour through some resources for those tools sometime during this tutorial.

I don’t find much value in blindly following guides without having a good idea of what’s going on. This desire is tempered by me knowing that I cannot absorb everything my first pass, and things will become clear on my third and fourth exposure to a topic that I simply cannot pick up my first or second.

OK, down to the bottom of the first tutorial section, and the React App is not working as expected. Here’s where I’m at right now, and while it’s compiling successfully, I’ve got lots of errors:

Broadcast.js:104 Uncaught TypeError: Cannot read property 'object' of undefined
    at Object../node_modules/react-broadcast/Broadcast.js (Broadcast.js:104)
    at __webpack_require__ (bootstrap 4fb1954d0e8840de1bbc:669)
    at fn (bootstrap 4fb1954d0e8840de1bbc:87)
    at Object../node_modules/react-broadcast/index.js (index.js:6)
    at __webpack_require__ (bootstrap 4fb1954d0e8840de1bbc:669)
    at fn (bootstrap 4fb1954d0e8840de1bbc:87)
    at Object../node_modules/react-router/Broadcasts.js (Broadcasts.js:12)
    at __webpack_require__ (bootstrap 4fb1954d0e8840de1bbc:669)
    at fn (bootstrap 4fb1954d0e8840de1bbc:87)
    at Object../node_modules/react-router/Link.js (Link.js:13)
    at __webpack_require__ (bootstrap 4fb1954d0e8840de1bbc:669)
    at fn (bootstrap 4fb1954d0e8840de1bbc:87)
    at Object../node_modules/react-router/index.js (index.js:6)
    at __webpack_require__ (bootstrap 4fb1954d0e8840de1bbc:669)
    at fn (bootstrap 4fb1954d0e8840de1bbc:87)
    at Object../src/containers/App/index.js (index.js:11)
    at __webpack_require__ (bootstrap 4fb1954d0e8840de1bbc:669)
    at fn (bootstrap 4fb1954d0e8840de1bbc:87)
    at Object../src/index.js (index.js:7)
    at __webpack_require__ (bootstrap 4fb1954d0e8840de1bbc:669)
    at fn (bootstrap 4fb1954d0e8840de1bbc:87)
    at Object.0 (index.js:11)
    at __webpack_require__ (bootstrap 4fb1954d0e8840de1bbc:669)
    at bootstrap 4fb1954d0e8840de1bbc:715
    at bundle.js:719

As I mentioned earlier, I don’t have tons of React experience, so I’m going to go take a quick dive into the world of React. I expect, on the other end, to have a better framework for debugging and understanding the design decisions the tutorial author is making.

I think part of the challenge is this guide was written in January 2017, or approaching a year ago. React changes quickly, and in Ruby/Rails, I’ve always been able to fill in the info gaps when debugging problems, but React (and Elixir) I’m not yet there.

Quick pass through React

Once more: online or e-voting? No. Just no

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 28 Oct 2017 10:00:00 GMT

OK, let’s summarize this again.

The Long Road to Async/Await in JavaScript at NodeConfAr17

Thomas Hunter II — Sat, 28 Oct 2017 03:00:00 GMT

Hidden Hope For Control

Venam's Blog — Patrick Louis (Lebanon) — Sat, 28 Oct 2017 00:00:00 GMT

This article is about a rarely discussed component that resides in the background of our lives. No word clearly describes this phenomenon as it's a fusion of different ideas. We blueprint ourselves according to media that display extravagant versions of winners and losers. The profiles on social media are facade created around individuals — Profile being a well-chosen word as it misses the other angles. All across those there's an implicit demand of mechanized efficiency which slipped from our workplace to our interaction with ourselves. We are our own employee and are expecting results.

First pass with Elixir/Phoenix

Josh Thompson — Fri, 27 Oct 2017 10:18:52 GMT

I’m digging into Elixir and Phoenix. I’m working through this tutorial to cloning Slack.

The tutorial author says

At the time of writing, I have ~1 week experience with Phoenix. Similar to Rubber Ducky Debugging, I am writing this blog post to force myself to think differently about the problems I am solving and therefore gain a better understanding of the language and framework.

Couldn’t agree more myself. This post serves as my Rubber Ducky Debugging process for getting started with Elixir/Phoenix.

Similar to the author, I come from a Ruby/Rails background, but undoubtedly less experienced than the author. So, this article will capture tidbits and pieces useful to me as I work through the tutorial.

I’m intentional in how I learn, as well as in how I ask questions, so this is just me, doing what I do, to learn things.

Onward!

Installing Elixir and Phoenix

Installation is super easy.

The recommended tutorials + some clicking around was all I needed. Brew handled most of it, and elixir-lang’s installation guide pairs well with the same for phoenix

I just cd‘ed into workspace/elixir (where my otherwise homeless bits of code lives) and test-drove Phoenix’s equivalent of rails new. I’m taking a quick detour through this guide at the moment.

OK, it was more than a quick detour. The rest of this post is me familiarizing myself with the above guide. Read Elixir/Phoenix part deux for more

$ mix phx.new hellow gives us a bunch of output, and eventually a new project.

(I tend to intentionally misspell variables and such when following tutorials - helps me detangle the language/framework’s nomenclature from my own, and tunes me in a bit closer to ways to make errors. So, this project is named “hellow”, and any further code snippets will reflect that. Be wary, this strategy has led me into more than a little debugging as I walk through tutorials. But I come out stronger on the other end…)

The directory structure looks familiar:

.
├── README.md
├── _build
├── assets
├── config
├── deps
├── lib
├── mix.exs
├── mix.lock
├── priv
└── test

deps is foreign to me, though. Inside of that directory is a bunch of other directories, all looking greek to me.

Poking around a bit, I’m seeing files with .ex and .exs extensions. StackOverflow says .ex is for “compiled” Elixir code, while .exs is “scripting” Elixir code. Elixir is my first brush with a functional programming language, which is also my first exposure to compiled languages, so I’m putting a flag in this as Significant.

Ah, and .eex is “embedded Elixir”, or in my Rails-centric background “elixir’s version of embedded ruby/.erb”. Good, because no one wants to write raw HTML.

If Elixir doesn’t have some version of pry, I’m going to be sad.

Per the instructions, $ mix ecto.create, a few missing dependencies (say Y), and

=> The database for Hellow.Repo has been created

Wahoo.

$ mix phx.server seems to be rails server. So handy having a mental framework to hang new info on.

A quick check of localhost:4000 and we’re in business:

On to part 2 of the intro guide:

The writeup mentions that most of our work will live in the /lib directory

Lots of interesting stuff that seems quite familiar after working with a MVC model, but we’ll dig into it all later.

Routes

Ah, routes. Makes me nostalgic for config/routes.rb

Elixir’s equivalent lives in hellow/lib/hellow_web/router.ex, which we know is a compiled file (I think?).

By default, it looks pretty manageable:

defmodule HellowWeb.Router do
  use HellowWeb, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", HellowWeb do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
  end

end

As soon as seeing the reference to PageController, I had to see if there was an associated test.

Ah, software. It even reads just like RSpec. :)

# test/hellow_web/controllers/page_controller_test.exs

defmodule HellowWeb.PageControllerTest do
  use HellowWeb.ConnCase

  test "GET /", %{conn: conn} do
    conn = get conn, "/"
    assert html_response(conn, 200) =~ "Welcome to Phoenix!"
  end
end

Testing

We interrupt current programming to figure out how to run tests in Elixir

Looks like Mix is an Elixir build tool. We’ll be using it a lot.

ExUnit seems to be Elixir’s test-unit framework. AKA RSpec/Minitest. (source)

And tests

Looks like $ mix test does it.

Holy cow… There’s a ton of deprecation warnings and the test took ages. (Er, it passed by the way. )

Looks like this is a new and fixed issue, so the recommended fix is to use a more recent version of poison.

Jumping into mix.exs (aka Rail’s Gemfile), adding the newest version of poison:

defp deps do
    [
      {:poison, "~> 3.1"},
      {:phoenix, "~> 1.3.0"},
      .
      ]
end

and updating dependencies:

$ mix deps.get

and lets run the tests again:

$ mix test

Ah, lovely. No more dependency problems.

Two other notable events, though - I think the first time I ran the tests, Mix had a ton of compiling to do.

This is the output from the first test run:

# all of the many deprecation warnings removed
==> connection
Compiling 1 file (.ex)
Generated connection app
==> gettext
Compiling 1 file (.erl)
Compiling 20 files (.ex)
Generated gettext app
===> Compiling ranch
===> Compiling poolboy
==> decimal
Compiling 1 file (.ex)
Generated decimal app
==> poison
Generated poison app
==> db_connection
Compiling 23 files (.ex)
Generated db_connection app
==> phoenix_pubsub
Compiling 12 files (.ex)
Generated phoenix_pubsub app
===> Compiling cowlib
===> Compiling cowboy
==> mime
Compiling 1 file (.ex)
Generated mime app
==> plug
Compiling 1 file (.erl)
Compiling 44 files (.ex)
Generated plug app
==> phoenix_html
Compiling 8 files (.ex)
Generated phoenix_html app
==> phoenix
Compiling 74 files (.ex)
Generated phoenix app
==> postgrex
Compiling 62 files (.ex)
Generated postgrex app
==> ecto
Compiling 70 files (.ex)
Generated ecto app
==> phoenix_ecto
Compiling 6 files (.ex)
Generated phoenix_ecto app
==> hellow
Compiling 16 files (.ex)
Generated hellow app
....

Finished in 0.1 seconds
4 tests, 0 failures

Randomized with seed 769269

The second time I ran it, not only were there no deprecation warnings, there was basically no set-up:

mix test
Compiling 16 files (.ex)
Generated hellow app
....

Finished in 0.06 seconds
4 tests, 0 failures

Randomized with seed 608206

OK, again, I feel like all the setup steps are Significant, but don’t yet know why.

Onward!

Back to Routes/Routing

Lets add:

scope "/", HellowWeb do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
    get "/hello", HellowController, :index # Adding this bad boy
  end

When visiting localhost:4000/hello, we get a very Rails-esque error code, saying “I cannot find the controller you’re asking for”. Which is great, because HellowController doesn’t exist.

A new Controller

Jumping in to make a new controller file moves us along in error-driven development (EDD):

# lib/hellow_web/controllers/hellow_controller.ex
defmodule HellowWeb.HellowController do
  use HellowWeb, :controller

  def index(conn, _params) do
    render conn, "index.html"
  end
end

And we get:

UndefinedFunctionError at GET /hello
function HellowWeb.HellowView.render/2 is undefined (module HellowWeb.HellowView is not available)

Perfect. We are (I suspect) calling a non-existent file to render stuff to the browser. Phoenix complains.

A new View

I put together a quick view, templated off an existing view, and for the first time my MVC background failed me. I assumed views would be for presentation, but it seems like Views in Phoenix are more analagous to Models in Rails.

There’s a /templates directory that contains presentation-related data, so I’ll just run with “view == models” for now.

Phoenix views have several important jobs. They render templates. They also act as a presentation layer for raw data from the controller, preparing it for use in a template. Functions which perform this transformation should go in a view. (hexdocks)

Templates

Ah, interesting. As soon as I created lib/hellow_web/templates/hellow/index.html.eex, the errors in Mix went away, and I’m now loading an empty page.

(Even though the file is empty)

So, Phenix has live code-reloading (when I made the file, I saw the logs from mix phx.server show compiling and rendering data. The red went away.)

Also, the view is obviously embedded inside of an application layout, because the header/footer/things-that-make-HTML-work-on-the-internet kicked in and rendered my three lines of HTML.

A quick peek at lib/hellow_web/templates/layout/app.html.eex, and it’s almost indistinguishable from app/views/layouts/application.html.erb in Rails.

A second new page

The universally sound advice for education and training applies here quite well:

Do it again, faster

So, I did a quick trip through adding another page, to do a standard “handle a dynamic URL and present the value from the URL in the browser”. This is like the “hello world” of dynamic routing.

That’s the end of the tiny “getting started” tutorial. I’m ending this post here, and will continue with the slack clone in part 2

Everything And More, a short review

Avy Faingezicht — Fri, 27 Oct 2017 03:00:00 GMT

The first time I heard about David Foster Wallace was during my sophomore year of college. My girlfriend was powering through _Infinite Jest_, and was surprised I had never heard about the book or its author. I told her it sounded unbearable. A few years later, a friend picked The Weasel, Twelve Monkeys and the Shrub as reading material for our article club[^1], and I had no choice - I read it and hated it. Then this year, I was talking about GEB with my boss's boss's boss and he insisted that even if I hated DFW's writing, I had to read his book on infinity. I followed his advice. _Everything and More_ walks a thin line between being too technical for the untrained reader and too dumbed down for the expert. For better or for worse, I fall somewhere in between. I enjoyed reading about Mr. Chicken, the rooster who _inductively_ figured out that _t_(man + sack)=food, only to end up dead in the sack himself a few weeks later. I also enjoyed figuring out the inner workings of $$\aleph _{0}$$ and drudging through proofs of how certain kinds of infinities are greater than others. As someone without a pure math background, it seems like DFW did a good job of exposing the reader to a wide set of ideas. How thoroughly he explains them is a different issue. From some quick research it seems like the book got a lot of well-deserved flak for it's accuracy (or lack thereof) upon release. Regardless, it is worth reading not so much for its comprehensive proofs, but for the way it illuminates the development of modern mathematics. Early on, DFW explains that, in math, paradoxes arise from expressing logical systems as natural language. For example, halve the distance between two sidewalks enough times, and with a bit of rhetoric and pomp even the smartest people start to believe that crossing a street is impossible. Thanks, infinite regress! The whole book is peppered with the histories of these paradoxes, and of mathematicians trying to overcome them. From the Greeks to Galileo, over time there is a growing gap between the theorists (dealing in abstraction) and the engineers and physicists (dealing with reality). The two can't be easily reconciled, but that only concerns one of the two groups. The issue is especially exacerbated with the discovery of calculus in the 17th century, and the explotion of its applications in construction, warfare, navigation, astronomy, etc: > The situation of mathematics after 1700 is intensely weird [...] mathematical discoveries enabled scientific advances, which themselves motivated further math discoveries. [This] created for math a situation that resembled a tree with great lush proliferant systems of branches but no real roots. There were still no grounded, rigorous definitions of the differential, derivative, integral, limit, or convergent/divergent series. Not even of the function. There was constant controversy, and yet at the same time nobody seemed to care. > > David Foster Wallace - Everything and More, §5a For roughly a century and a half people solved a panoply of life-changing human problems via calculus, without understanding its underpinnings. Later on, the book compares this disparity to a market bubble. The appearance of analysis, set theory, and modern mathematics, as explained by DFW, are the response to this chaos, set in motion by the discovery of calculus. The rigor that these new branches of math brought to the field eventually became the foundation that powered the technological advances of the 19th and early 20th century. What is most striking is how much of a parallel we can draw between the development of calculus and today's state of the art in computer science, with machine learning being particularly pertinent - the whole field is based on empiricism and induction. The eighteenth and nineteenth century engineers treated the methods of calculus as black boxes, and trusted them enough to use their answers to construct buildings, to predict the movement of stars, and to sail across the Atlantic. _Whole complex human systems based on the output of black boxes._ Read that again and tell me that there isn't a parallel to software engineering and machine learning. It makes one wonder where our technology today might be headed. Notice that this is not a dis on the Bengios and Hintons of the world, nor on my peers who are practitioners in the field. To the contrary. If anything its a comparison to Newton and Leibniz kicking off this transformation, and to the Boyles and the Watts who put the knowledge in motion. What DFW's book in a sense revealed to me is that while today anyone can download Tensorflow and feed the black box with some data to solve real problems, there's a lot still to uncover. The frontier of knowledge is ever expanding, and it has not disappointed us so far. Perhaps all we need is our own Karl Weierstrass to come and wake us up from the stupor, and a George Cantor to lead the way forward. Computer science is as alive as ever, and for the last fifty years has driven the greatest technological revolution in history. Just imagine where we'll be when we move past the black box. [^1]: Article club? That's a story for another time.

Photo: by me, previously posted on Napa.

The shifting sweet spot for media

D13V — Fri, 27 Oct 2017 03:00:00 GMT

How informative is media really. The incentives on any media organization are like the incentives on any other company - to make profits. Media also aims to make people informed on important facts. That’s how people perceive the value of media. That, and entertainment. All media wants people to consume it, in order to get revenue from subscriptions, or from advertisement.

Let’s temporary make a distinction between media as a business, and media as an informer. Every media outlet has a bipolar disorder between these two. The decision makers in each media sometimes decide with their business hat, and sometimes they do it with their informant hat.

It makes sense to balance the two. If the media spends too much resources on developing the content, the final product will be of higher quality, but more expensive. But the slight increase in quality, would not justify a steep increase in price, for the customers. Most of them would stop using it, and that would make it not viable financially. Instead, if the media spends too little resources, then the quality of the content will go down, and customers will flee for the opposite reason.

Each media business needs to find a sweet spot in terms of price and quality. And that sweet spot derives from the current market conditions and the ability of the management and employees.

Internet affects the sweet spots in a profound way. Some might say “disruptive”. On the Internet, the distribution cost of media is a lot lower than before. They can move over to low-precision low cost land for their facts supply. I think it is great to have such choice. It improves market efficiency, but by adding a new lowest sweet spot, it triggers a gradual reconfiguration of all existing sweets spots before.

As some people flock to the low precision land for their supply of “facts”, this changes the demographics of those who remain willing to pay for high precision media. With fewer and fewer customers, there are more biases in their overall views. And each media, consciously or subconsciously will start tending more and more to the tastes of this population. And thus, each media becomes more and more biased.

But it’s not purely the media’s fault. It’s ours as well. We all tend to believe more the facts which agree with our opinions, and ignore the facts which disagree. And media feeds us only what we like to digest. The tasty facts. By forcing media to adapt to our tastes, we increasingly fake-ify it. Media had objectivity and coverage issues before, but now these are exacerbated, and much more fundamental to media’s existence.

It’s not clear how the media landscape will develop, and whether there will be a high precision, high recall media in the future. This depends on market conditions, and technology improvements. What we can do meanwhile is to apply pressure on media to be more objective, but not merely by complaining to the media about facts we don’t like.

No, each of us needs to be a little bit more critical of facts that seem too good or too bad to be true, and to more often look at facts we disagree with, with a little bit of benefit of the doubt. Just like we all know that a single unrecycld cardboard box trash won’t damage the environment, but if everyone recycles then we are much better off, we will all be better off if we are more critical about our media’s objectivity.

Are you a robot?

Yasha Solutions — Fri, 27 Oct 2017 03:00:00 GMT

Automation is coming. It is affecting nearly all industries. And it would be stupid to ignore it. But aren’t we taking the frenzy a tad too far? (Disclaimer: This article is a bit messy – I have tried my best to give it a decent shape – but if we are to be honnest, it needs some serious re-writing – which I will probably do – but I wanted to get this out now anyway – so take it as a work in progress with maybe a few unfinished thoughts / sentence here and there…)

Two design articles

Muffin Man — Fri, 27 Oct 2017 03:00:00 GMT

Just wanted to share two articles that helped me understand visual design better. I was already aware of the most of these things, but having someone explain it how and why was really helpful.

Learn more about design #

When asked how to learn more about design as a developer, design partner in my company recommended studying work of these two designers. I haven't taken my time to do it, but it is on my todo list.

Massimo Vignelli - author of NYC subway map
Paul Rand - author of IBM logo

Frontend developers should have at least basic understanding of visual design. Because great products are symbiosis of both great design and great tech. Hopefully, these links will help you in that regard.

Misleading advice in Europol's cybercrime page

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 26 Oct 2017 11:10:34 GMT

Europol is the EU’s law enforcement agency, whose main goal is “to achieve a safer Europe for the benefit of all the EU citizens”. The Europol page about public awareness and prevention mixes good advice with a few… misleading bits of information.

Evaluating New Tools

blog.harterrt.com — Thu, 26 Oct 2017 10:00:00 GMT

At Mozilla, we're still relatively early in our data science journey. As such, we're always evaluating new tools to improve our analysis workflow (jupyter vs. Rmd), or make our infrastructure more usable (our home-rolled ATMO vs. databricks), or scale our knowledge (knoledge-repo. vs. gitbook)

Most of these tools look like …

Wall Street Journal Live Interview with Matt Crampton