Hacker News Personal Blogs 2017 | Top 100 Blogs

Migrating from Google (and more)

Nemo's Home — Sun, 31 Dec 2017 02:00:00 GMT

Email
Google Play Music
Google Keep
Phone
LastPass
GitHub

As part of working on my home-server setup, I wanted to move off few online services to ones that I manage. This is a list of what all services I used and what I’ve migrated to .

Why: I got frustrated with Google Play Music a few times. Synced songs would not show up across all clients immediately (I had to refresh,uninstall,reinstall), and I hated the client limits it would impose. Decided to try microG on my phone at the same time, and it slowly came together.

Email

I’ve been using email on my own domain for quite some time (captnemo.in), but it was managed by a Outlook+Google combination that I didn’t like very much.

I switched to Migadu sometime last year, and have been quite happy with the service. Their Privacy Policy, and pro/cons section on the website is a pleasure to read.

Why: Email is the central-point of your online digital identity. Use your own-domain, at the very least. That way, you’re atleast protected if Google decides to suspend your account. Self-hosting email is a big responsibility that requires critical uptime, and I didn’t want to manage that, so went with migadu.

Why Migadu: You should read their HN thread.

Caviats: They don’t yet offer 2FA, but hopefully that should be fixed soon. Their spam filters aren’t the best either. Migadu even has a Drawbacks section on their website that you must read before signing up.

Alternatives: RiseUp, FastMail.

Google Play Music

I quite liked Google Play Music. While their subscription offering is horrible in India, I was a happy user of their “bring-your-own-music” plan. In fact, the most used Google service on my phone happened to be Google Play Music! I switched to a nice subsonic fork called [AirSonic][airsonic], which gives me the ability to:

Listen on as many devices as I want (Google has some limits)
Listen using multiple clients at the same time
Stream at whatever bandwidth I pick (I stream at 64kbps over 2G networks!)

I’m currently using Clementine on the Desktop (which unfortunately, doesn’t cache music), and UltraSonic on the phone. Airsonic even supports bookmarks, so listening to audiobooks becomes much more simpler.

Why: I didn’t like Google Play Music limits, plus I wanted to try the “phone-without-google” experiment.

Why AirSonic: Subsonic is now closed source, and the Libresonic developers forked off to AirSonic, which is under active development. It is supports across all devices that I use, while Ampache has spotty Android support.

Google Keep

I switched across to WorkFlowy, which has both a Android and a Linux app (both based on webviews). I’ve used it for years, and it is a great tool. Moreover, I’m also using DAVDroid sync for Open Tasks app on my phone. Both of these work well enough offline.

Why: I didn’t use Keep much, and WorkFlowy is a far better tool anyway.

Why WorkFlowy: It is the best note-taking/list tool I’ve used.

Phone

I switched over to the microG fork of lineageOS which offers a reverse-engineered implementation of the Google Play Services modules. It includes:

microG Core

Which talks to Google for Sync, Account purposes.

Why: Saves me a lot of battery. I can uninstall this, unlike Google Play Services.

Cons: Not all google services are supported very well. Push notifications have some issues on my phone. See the Wiki for Implementation Status.

UnifiedLP

Instead of the Google Location Provider. I use the Mozilla Location Services, along with Mozilla Stumbler to help improve their data.

Why: Google doesn’t need to know where I am.

Caviats: GALP (Google Assisted Location Provider) does GPS locks much faster in comparision. However, I’ve found the Mozilla Location Services coverage in Bangalore to be pretty good.

Maps

Stil looking for decent alternatives.

Uber

microG comes with a Google Maps shim that talks to Open Street Maps. The maps feature on Uber worked fine with that shim, however booking cabs was not always possible. I switched over to m.uber.com which worked quite well for some time.

Uber doesn’t really spend resources on their mobile site though, and it would ocassionaly stop working. Especially with regards to payments. I’ve switched over to the Ola mobile website, which works pretty well. I keep the OlaMoney app for recharging the OlaMoney wallet alongside.

Uber->Ola switch was also partially motivated by how-badly-run Uber is.

Calendar/Contacts

Most implementations support caldav/carddav for calendar/contacts sync. I’m using DAVDroid for syncing to a self-hosted Radicale Server.

Why: I’ve always had contacts synced to Google, so it was always my-single-source-of-truth for contacts. But since I’m on a different email provider now, it makes sense to move off those contacts as well. Radicale also lets me manage multiple addressbooks very easily.

Why Radicale: I looked around at alternatives, and 2 looked promising: Sabre.io, and Radicale. Sabre is no longer under development, so I picked Radicale, which also happened to have a regularly updated docker image.

Google Play Store

Switch to FDroid - It has some apps that Google doesn’t like, and some more. Moreover, you can use YALP Store to download any applications from the Play Store. You can even run a FDroid repository for the apps you use from Play Store, as an alternative. See this excellent guide on the various options.

Why: Play Store is tightly linked to Google Play Services, and doesn’t play nice with microG.

Why FDroid: FDroid has publicly verifiable builds, and tons of open-source applications.

Why Yalp: Was easy enough to setup.

If you’re looking to migrate to MicroG, I’d recommend going through the entire NO Gapps Setup Guide by shadow53 before proceeding.

LastPass

I’ve switched to pass along with a sync to keybase.

Why: LastPass has had multiple breaches, and a plethora of security issues (including 2 RCE vulnerabilities). Their fingerprint authentication on Android could be bypassed til recently. I just can’t trust them any more

Why pass: It is built on strong crypto primitives, is open-source, and has good integration with both i3 and firefox. There is also a LastPass migration script that I used.

Caviats: Website names are used as filenames in pass, so even though passwords are encrypted, you don’t want to push it to a public Git server (since that would expose the list of services you are using). I’m using my own git server, along with keybase git(which keeps it end-to-end encrypted, even branch names). You also need to be careful about your GPG keys, instead of a single master password.

GitHub

For bonus, I setup a Gitea server hosted at git.captnemo.in. Gitea is a fork of gogs, and is a single-binary go application that you can run easiy.

Just running it for fun, since I’m pretty happy with my GitHub setup. However, I might move some of my sensitive repos (such as this) to my own host.

Why Gitea: The other alternatives were gogs, and GitLab. There have been concerns about gogs development model, and GitLab was just too overpowered/heavy for my use case. (I’m using the home server for gaming as well, so it matters)

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Sourcing developer marketing content

CRAIG KERSTIENS — Thu, 28 Dec 2017 22:55:56 GMT

I spend a lot of time with dev tool and data companies. I think I’ve more or less banished myself to a life of working in the space, no consumer products for me. In that world a common topic that comes up amongst marketing teams is how do I get my team to contribute to content? Sometimes the person already has an idea of how they want the team to jump onto the bandwagon of their plan, sometimes they’re entirely open minded. I won’t get into pros and cons of various approaches here, rather after sharing some of my approaches in one on one settings I thought it could be useful to share more broadly here.

Turn emails into blogs

I’m a big fan of high quality content when it comes to developer focused products. Yes, you can publish x vs. y with FUD and get some traction with it. You can publish high level customer stories that don’t get down into the details. But publishing high quality deep technical content that other engineers appreciate gives you a huge head start in getting buy in from your customers. The best thing about much of this type of content is you likely already have it sitting within your organization.

Engineers spend a lot of time being thorough and articulate in their emails to their peers. If you see a well written email from one engineer to the engineers@ list and then others chime in either with questions, follow-up, or praise then it’s likely a great candidate for a blog post. We recently had one of these at Citus where another engineer replied with “Nice blog post :)”, a few weeks later we had a post for the rest of the world to read.

The thing about emails though is the engineers won’t usually take and turn them into a blog post. Get a technically minded person that likes to write and put some framing around it, pull out relevant details, and collaborate with the engineer. I’ve often found going from one of these emails to a fully published blog post can be anywhere from 2-8 hrs (including reviews and edits).

Beginner mindset with tickets

As your product people and engineers spend more and more time with the product they become numb to the cool feature from 2 years ago. That initial awe is just expected. Yes there are new advanced features and tech that is being built… but most of your users aren’t the advanced power users. You still need to speak to the people just now onboarding and discovering you.

The best way to maintain this beginner mindset is to capture the interactions with those that are newer to your product. Over the years I’ve made the practice of cataloging support tickets and the type of issue encountered. Any time the same issue is seen several times it becomes a candidate for a blog post or at the very least being clearly documented.

Make sales engineering obsolete

Your sales engineers spend weeks with customers helping them implement complex solutions with your product. In a lot of cases they get really good at helping re-implement a similar solution over and over. Similar to your process with support tickets they’re a great source to take what they’ve done a few times over and turn it into a blog post.

Usually after the first time implementing they have some ideas but there are rough edges. The second or third time they’ve helped a customer implement a particular approach it becomes a candidate for a post.

Training and Documentation don’t have to be siloed

I’m a big fan of not doing anything for one reason. You should absolutely document your product, and if you need to do training to support customers go for it. But that doesn’t mean those bits of content can’t be re-used. Documentation is a great place to take the factual how something works and then give it some narration of the end to end experience in a blog post.

The process

If you’re asking the question how do I get more good content out there, then you’re not leveraging the content you’re already sitting on top of. Though perhaps equal is getting a process in place. A few tips for that:

Don’t blanket ask to a list for volunteers, ask individuals and about specific posts. If you see a great email, respond and ask if you can work with them to get it turned into a blog post
You’ll need to navigate the process, and make sure to have a process
Separate messaging/flow from grammar
My typical process is: outline -> draft -> feedback from 2-3 on draft -> finalize draft -> feedback from 4-5 (often some external parties) -> publish

What’d I miss?

Have other tips that are key to the process? Know of other places where content may already exist but isn’t being leveraged? Would love to hear your thoughts.

Top album picks for 2017

smcleod.net — Thu, 28 Dec 2017 02:00:00 GMT

Here are my top album pics for 2017 (in no particular order)

Note: This is by no means an exhaustive list, it’s just the top albums that really stood out to me and in all fairness, I’ve thought of several others since so there may be a follow up post (or two).

Father John Misty - Pure Comedy

As a bonus, a short film was released with the album:

From Wikipedia:

HP 4951C Protocol Analyser

smcleod.net — Wed, 27 Dec 2017 02:00:00 GMT

My good friend Joel Shea received a most unlikely gift this Christmas - A vintage HP 4951 Protocol Analyser.

According to the HP Computer Museum:

Original Price: $3595 The 4951B was replaced by the 4951C and 4952A in 1986. Both new models handled Async, BSC, SDLC, HDLC, X.25 and SNA protocols. The 4951C also handled DDCMP, while the 4952A did not. The 4952A handled X.21 while the 4951C did not. Both new analysers used a floppy dive (618 KB) for removable media.

Guidance on performing retrospectives

CRAIG KERSTIENS — Tue, 26 Dec 2017 22:55:56 GMT

In my career I’ve had to conduct a number of retrospectives. Ahead of them it already sucked, there was an outage at some point, customers were impacted, and it was our fault. Never was it solely on our underlying infrastructure provider (AWS or Heroku), nope the blame was on us and we’d failed in some way. And as soon as the incident was resolved, it wasn’t time to go home and decompress with a beer, it was time start the process of a retrospective.

Finding the motivation to get right back to work is tough, but not losing time is important. There is probably a lot out there on retrospectives, and in general I was well rehearsed at them. But since I’d not performed a large scale one in a few years I found myself rusty and thought it’d be good to share some of our process.

Capture details immediately

It may not be clear if you’ve not been involved in many, but a retrospective is more than just a meeting to discuss what happened and how to fix it. It’s an overall process, it begins with capturing thorough details of what happened. The start is a timeline. The best thing to do is capture the details while they’re fresh. Start with a google doc and simply document the timeline of everything. Capture chat logs that are relevant while they’re fresh in history and easy to find.

The start of an outage likely wasn’t the start of the timeline, there may have been something that happened days, weeks, or even years ago. Don’t just start from the time things went offline, go back to the causes as much as possible. If code was committed a year ago that was the offender make sure to note that.

Running the retrospective (the meeting part)

There are a number of various good practices for running the retrospective itself. There are also a lot of different formats, all valid each with their own pros and cons. You can do with a basic timeline, what went well/didn’t, do a five whys analysis. I tend to prefer a clean and dry analysis of timeline, what went well and what didn’t and what we’re doing about it.

Some key tips to help a retrospective meeting be productive:

Explicitly set time bounds for each activity ahead of time, more so than maybe any other meeting it’s important to get through all your agenda. Hard time limits on the planned items is how you accomplish this.
Spend at least some time on what went well, retrospectives aren’t fun. Spending some time on the good parts of your process and response isn’t wasted, just don’t be overly self-back-patting (that’s not a word but you get it).
Don’t discuss people, or rather don’t point fingers. Yes, people will come up, but it’s about the technical and process errors not the person that performed them.

The important part

Every step in the retrospective is important, but the goal of them all is to get to how you can improve. With any retrospective there are likely two categories of improvements that will surface. The first is bugs that caused the issue or engineering that could go in place to help with the specific issue. The second are process improvements. If you don’t have improvements in both areas then spend more time thinking on the one you’re missing.

Improvements shouldn’t be isolated to the exact issue you saw. Yes you may see the exact same issue again, but there is also a lot more you can draw out that helps improve overall quality. It’s inevitable you’ll see different issues in the future, thinking of how you can improve your systems and processes to catch those future issues is time well spent.

You’ve done the hard work in the above, but it’s still good to share publicly and transparently. Within your public retrospective I tend to follow:

Apologize, and mean it
Show a firm understanding of your systems, and communicate the problem. Don’t try to be fancy technically, but don’t be too highlevel. Think goldilocks.
Share what you’re doing to improve

Credit to Mark for being the first person I’m aware of to lay it out like the above

Thanks to Mark Imbriaco, Blake Gentry, Daniel Farina, Lukas Fittl, Will Leinweber for input and feedback along the way.

Looking for more resources on the topic? Make sure to check out these two talks:

Books I read in 2017

Krishna's blog — Mon, 25 Dec 2017 02:00:00 GMT

I share my top book recommendations for 2017, including Trevor Noah's Born a Crime and The Dictator's Handbook by Bruce Bueno de Mesquita and Alastair Smith.

Man's Search for Meaning

Krishna's blog — Sun, 24 Dec 2017 02:00:00 GMT

Man's Search for Meaning offers a unique perspective on the Holocaust and introduces the theory of logotherapy.

And Then There Were None

Krishna's blog — Wed, 20 Dec 2017 02:00:00 GMT

A review of Agatha Christie's 'And Then There Were None' highlighting the absence of her typical detectives and the prevalence of casual racism in the 1930s.

Simply encrypt or decrypt a string using Boto3 Python and AWS KMS

Ben E. C. Boyter — Mon, 18 Dec 2017 04:54:59 GMT

Another one of those things I need to look up every now and then. Below is a snippet of how to encrypt and decrypt a string using Python and KMS in AWS. The interesting thing is that you don’t need to supply the KMS key alias in the decryption portion. So long as whatever role or key you are using can access the key it should work. For the encryption you can either supply the full ARN of the key or the alias so long as you prefix it with alias/

Learnings from building my own home server

Nemo's Home — Mon, 18 Dec 2017 02:00:00 GMT

Learnings

I forgot to do this on the last blog post, so here is the list:

archlinux has official packages for intel-microcode-updates.
wireguard is almost there. I’m running openvpn for now, waiting for the stable release.
While traefik is great, I’m concerned about the security model it has for connecting to Docker (uses the docker unix socket over a docker mounted volume, which gives it root access on the host). Scary stuff.
Docker Labels are a great signalling mechanism. Update: After seeing multiple bugs with how traefik uses docker labels, they have limited use-cases but work great in those. Don’t try to over-architect them for all your metadata.
Terraform still needs a lot of work on their docker provider. A lot of updates destroy containers, which should be applied without needing a destroy.
I can’t proxy gitea’s SSH authentication easily, since traefik doesn’t support TCP proxying yet.
The docker_volume resource in terraform is useless, since it doesn’t give you any control over the volume location on the host. (This might be a docker limitation.)
The upload block inside a docker_container resource is a great idea. Lets you push configuration straight inside a container. This is how I push configuration straight inside the traefik container for eg:
```
upload {
  content = "${file("${path.module}/conf/traefik.toml")}"
  file    = "/etc/traefik/traefik.toml"
}
```

Advice

This section is if you’re venturing into a docker-heavy terraform setup:

Use traefik. Will save you a lot of pain with proxying requests.
Repeat the ports section for every IP you want to listen on. CIDRs don’t work.

If you want to run the container on boot, you want the following:

 restart = "unless-stopped"
 destroy_grace_seconds = 10
 must_run = true

If you have a single docker_registry_image resource in your state, you can’t run terraform without internet access.
Breaking your docker module into images.tf, volumes.tf, and data.tf (for registry_images) works quite well.
Memory limits on docker containers can be too contrained. Keep an eye on logs to see if anything is getting killed.
Setup Docker TLS auth first. I tried proxying Docker over apache with basic auth, but it didn’t work out well.

MongoDB with forceful server restarts

Since my server gets a forceful restart every few days due to power-cuts (I’m still working on a backup power supply), I faced some issues with MongoDB being unable to recover cleanly. The lockfile would indicate a ungraceful shutdown, and it would require manual repairs, which sometimes failed.

As a weird-hacky-fix, since most of the errors were from the MongoDB WiredTiger engine itself, I hypothesized that switching to a more robust engine might save me from these manual repairs. I switched to MongoRocks, and while it has stopped the issue with repairs, the wiki stil doesn’t like it, and I’m facing this issue: https://github.com/Requarks/wiki/issues/313

However, I don’t have to repair the DB manually, which is a win.

SSHD on specific Interface

My proxy server has the following

eth0 139.59.22.234

And an associated Anchor IP for static IP usecases via Digital Ocean. (10.47.0.5, doesn’t show up in ifconfig).

I wanted to run the following setup:

eth0:22 -> sshd
Anchor-IP:22 -> simpleproxy -> gitea:ssh

where gitea is the git server hosting git.captnemo.in. This way:

I could SSH to the proxy server over 22
And directly SSH to the Gitea server over 22 using a different IP address.

Unfortunately, sshd doesn’t allow you to listen on a specific interface, and since the eth0 IP is non-static I can’t rely on it.

As a result, I’ve resorted to just using 2 separate ports:

22 -> simpleproxy -> gitea:ssh 222 -> sshd

There are some hacky ways around this by creating a new service that boots SSHD after network connectivity, but I thought this was much more stable.

Wiki.js public pages

I’m using wiki.js setup at https://wiki.bb8.fun. A specific requirement I had was public pages, so that I could give links to people for specific resources that could be browser without a login.

However, I wanted the default to be authenticated, and only certain pages to be public. The config for this was surprisingly simple:

YAML config

You need to ensure that defaultReadAccess is false:

auth:
  defaultReadAccess: false
  local:
    enabled: true

Guest Access

The following configuration is set for the guest user:

Now any pages created under the /public directory are now browseable by anyone.

Here is a sample page: https://wiki.bb8.fun/public/nebula

Docker CA Cert Authentication

I wrote a script that goes with the docker TLS guide to help you setup TLS authentication

OpenVPN default gateway client side configuration

I’m running a OpenVPN configuration on my proxy server. Howver, I don’t always want to use my VPN as the default route, only when I’m in an untrusted network. I still however, want to be able to connect to the VPN and use it to connect to other clients.

The solution is two-fold:

Server Side

Make sure you do not have the following in your OpenVPN server.conf:

push "redirect-gateway def1 bypass-dhcp"

Client Side

I created 2 copies of the VPN configuration files. Both of the them have identical config, except for this one line:

redirect-gateway def1

If I connect to the VPN config using this configuration, all my traffic is forwarded over the VPN. If you’re using Arch Linux, this is as simple as creating 2 config files:

/etc/openvpn/client/one.conf
/etc/openvpn/client/two.conf

And running systemctl start openvpn-client@one. I’ve enabled my non-defaut-route VPN service, so it automatically connects to on boot.

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Should I buy Bitcoin?

Krishna's blog — Mon, 11 Dec 2017 02:00:00 GMT

This article explores the question of whether or not to invest in Bitcoin, examining its value as a currency and asset, as well as its potential for future growth and stability.

searchcode plexus

Ben E. C. Boyter — Tue, 05 Dec 2017 03:12:02 GMT

Plexus “A combination of interlaced parts; a network.”

For a while I have been neglecting searchcode.com while working on searchcode server. This is of course not an ideal situation and as such I have started working on it again. The following is just a brief list of things I am considering, problems, issues etc…

So back when I first started working on searchcode I wrote it using PHP. For searchcode next (the current version) I rewrote it using Python and Django. I had intented to continue using this stack for the forseeable future when I chose it. I quite like writing Python code and at the time the performance was acceptable. When I started writing searchcode server however I wanted to make it as easy to install as possible. The reason being that I considered it a competitive advantage to have users installing searchcode without any pain and be off and running. At the time I considered writing it in Go or Rust but had little experience with either language. I decided to pull on my sensible pants and chose Java which is a language I had worked with before and had excellent library support. With the 8th version Java had become considerably less painful to work with.

Flutter for React Native Devs in 30 Seconds

swyx's site RSS Feed — Sun, 03 Dec 2017 18:06:18 GMT

You may have heard of Flutter, Google's answer to React Native. What should you know?

Serverless Machine Learning at Google

swyx's site RSS Feed — Sun, 03 Dec 2017 00:25:33 GMT

---

Are we human? Or are we reCAPTCHA?

swyx's site RSS Feed — Sat, 02 Dec 2017 23:49:06 GMT

---

Firebase Analytics in 30 Seconds

swyx's site RSS Feed — Sat, 02 Dec 2017 22:57:12 GMT

---

Medical Machine Learning in 30 Seconds

swyx's site RSS Feed — Sat, 02 Dec 2017 18:41:14 GMT

---

Adding Blog Here

Ben Overmyer's Site — Mon, 27 Nov 2017 02:00:00 GMT

I'm starting to use Hugo for blogging now. I might move all of the content currently on Pathfinder over to here, if it works as well as I hope it does.

Also, I'm now hosting this site on Netlify, instead of deploying it myself to a VPS. This makes things much easier from a content management perspective.

Ready Player One

Krishna's blog — Sun, 26 Nov 2017 02:00:00 GMT

Ready Player One is a fast-paced book filled with 80s references, but falls short with one-dimensional characters and a lack of new ideas.

Six Easy Pieces

Krishna's blog — Thu, 23 Nov 2017 02:00:00 GMT

Six Easy Pieces by Richard Feynman is a fun read that provides an intuitive grasp of physics concepts, but falls short of teaching techniques.

The Brothers Karamazov

Krishna's blog — Sun, 19 Nov 2017 02:00:00 GMT

The Brothers Karamazov by Fyodor Dostoevsky is a challenging but rewarding read, with richly detailed characters and authentic relationships.

How to root Nexus 7 (2012) and flash it to LineageOS, from Mac OSX

Subclassed — Fri, 17 Nov 2017 02:56:59 GMT

I've finally got fed up enough with the old Nexus 7 (2012) that we originally bought to let the kids play - it's never really worked properly, getting laggy every 5 minutes and running out of battery after an hour. The kids moved on to an iPad, so I "installed" this crappy tablet to the kitchen wall, hoping to use it for calendaring and podcasts, but it was still horrendously laggy. I've decided to give a chance to LineageOS (previously CyanogenMod) to see if it helps. I'd be interested to know if anybody else uses a tablet wall-mounted in their kitchen, and what apps they installed - at the moment I have Paprika, the BBC players and weather, and PodcastAddict.

Here are the full steps required for flashing - I write them here because some of these are at risk to disappear from the internet for good after the demise of CyanogenMod wiki.

Note: THIS WILL WIPE YOUR DEVICE, YOU WILL LOSE EVERYTHING THAT IS ON IT so first make backups etc etc.

First you need to install the Android tools. I recommend doing this with homebrew, install it if you don't have it yet (it's extremely useful in many many cases).
Then open a terminal on your mac and type:

brew tap caskroom/cask
brew tap caskroom/versions
brew cask install java8
brew cask install android-sdk

Enable Developer Mode on the tablet by tapping 7 times on the build number under Settings -> About tablet.
Back to Settings, tap on Developer Options and enable USB Debug mode and Stay Awake.
Go back to your mac terminal and type:

sudo adb start-server

This starts the android debugger server.
Connect the tablet with a real USB cable (beware charging-only cables! Those won't work.)
On the tablet screen, you should be prompted to authorize the device; do it.
Back to terminal:

adb reboot bootloader

once the device comes back in bootloader mode:

fastboot oem unlock

Accept the disclaimer on the tablet by clicking the power button when Yes is highlighted (you can move with the volume buttons).
If it doesn't reboot on its own, select Start (again with the power button). At this point you should see an unlocked pad at the bottom of the screen as Android loads.
Note: if you plan to flash another OS, there is no point in going through the whole setup at this point, skip as much as you can.
Re-enable Developer mode and USB Debugging.

At this point the machine is rooted. The following steps are necessary only if you want to install LineageOS or other hacks; at the very minimum, though, you should install Trimmer (fstrim) from the Play store and use it liberally. Anyway, on with the flashing...

Get the latest image for "grouper" from https://twrp.me/, and unzip it.
Back to the terminal:

cd folder/where/you/have/your-downloaded-image.img
fastboot flash recovery your-downloaded-image.img
adb reboot bootloader

At this point you have a custom recovery image with a bunch of nifty features that make it very easy to hack the device. Tap Wipe and select system, cache, and dalvik.
Get the image you want to install. I'm currently trying this but in general anything after CM10 / Android 4.1 may be on the slow side for the original Nexus. You probably want the Google Apps from opengapps.org - choose ARM / 7.1 and the Micro option. Save both the apps zip and the image zip in the same folder.
Back to Terminal, let's copy these files to the device:

cd folder/where/you/have/your/zips
adb push your-downloaded-image.zip /
adb push your-downloaded-gapps.zip /

On the tablet, in the recovery screen, select Install, tap on Install ZIP, then select the image file and install. Repeat for the gapps file. If gapps refuse to go in because of space constraints, download this file, rename it gapps-config.txt, then copy it to the device in the same location as the gapps zip:

adb push gapps-config.txt /

and try again to install the gapps image.

Reboot the tablet. You might have to re-enable developer mode and usb debugging.
Install the Trimmer (fstrim) app, which is absolutely necessary. Launch it and click Trim Now. Click on the settings icon and enable autotrim as frequently as possible. In fact, every time you install an app or write a bunch of files, before you launch the new app, open Trimmer and do a trim, or it will all get laggy again.
Another good app is Android SSH Server, although it's a bit old it still works fine and it's easy to configure. You just have to use the options -t -oHostKeyAlgorithms=+ssh-dss -p XXXXX (where XXXXX is the custom port configured in the app, Android won't allow the classic one) when connecting.

And that's it.

Talk - Clustered, Distributed File and Volume Storage with GlusterFS

smcleod.net — Tue, 14 Nov 2017 02:00:00 GMT

Using GlusterFS to provide volume storage to Kubernetes as a replacement for our existing file and static content hosting.

This talk was given at Infracoders on Tuesday 14th November 2017.

Click below to view slides (PDF version):

Direct download link

Running terraform and docker on my home server

Nemo's Home — Thu, 09 Nov 2017 02:00:00 GMT

The last time I’d posted about my Home Server build in September, I’d just gotten it working. Since then, I’ve made a lot of progress. It is now running almost 10 services, up from just Kodi back then. Now it has a working copy of:

Kodi

I was running kodi-standalone-service, set to run on boot, as per the ArchLinux Wiki, but switched in favor of openbox to a simple autorun.

Steam

The current setup uses Steam as the application launcher. This lets me ensure that the Steam Controller works across all applications.

Openbox

Instead of running Kodi on xinit, I’m now running openbox with autologin against a non-privileged user.

PulseAudio

I tried fighting it, but it was slightly easier to configure compared to dmix. Might move to dmix if I get time.

btrfs

I now have the following disks:

128GB root volume. (Samsung EVO-850)
1TB volume for data backups
3TB RAID0 configuration across 2 disks. There are some btrfs subvolumes in the 3TB raid setup, including one specifically for docker volumes. The docker guide recommends running btrfs subvolumes on the block device, which I didn’t like, so I’m running docker volumes in normal mode on a btrfs disk. I don’t have enough writes to care much yet, but might explore this further.

Docker

This has been an interesting experiment. Kodi is still installed natively, but I’ve been trying to run almost everything else as a docker container. I’ve managed to do the configuration entirely via terraform, which has been a great learning experience. I’ve found terraform much more saner as a configuration system compared to something like ansible, which gets quite crazy. (We have a much more crazy terraform config at work, though).

Terraform

I have a private repository on GitLab called nebula which I use as the source of truth for the configuration. It doesn’t hold everything yet, just the following:

Docker Configuration (not the docker service, just the container/volumes)
CloudFlare - I’m using bb8.fun as the root domain, which is entirely managed using the CloudFlare terraform provider.
MySQL - Running a MariaDB container, which has been configured by-hand till this PR gets merged.

Gitea

Running as a docker container, provisioned using terraform. Plan to proxy this using git.captnemo.in.

Emby

Docker Container. Nothing special. Plan to set this up as the Kodi backend.

Couchpotato

Experimental setup for now. Inside a docker container.

Flexget

I wish I knew how to configure this. Also inside docker.

traefik

Running as a simple reverse proxy for most of the above services

elibsrv

A simple OPDS server, which I use against my Kindle. If you don’t know what OPDS is, you should [check this out][]. Running on a simple apache setup on the archlinux box for now. WIP for dockerization.

ubooquity

Simple ebook server. Proxied over the internet. Has a online ebook reader, which is pretty cool.

MariaDB

I set this up planning to shift Kodi’s data to this, but now that I have emby setup - I’m not so sure. Still, keeping this running for now.

Transmission

Hooked up to couchpotato,flexget, and sickrage so it can do things.

Sickrage

Liking this more than flexget so far, much more easier to configure and use.

AirSonic

This is the latest fork of libresonic, which was itself forked off subsonic. My attempt at getting off Google Play Music.

Learnings

Moved these to a separate blog post

TODO

A few things off my TODO list:

Create a Docker image for elibsrv that comes with both ebook-convert and kindlegen pre-installed
~~Do the same for ubooquity as well~~ (Using the linuxserver/ubooquity docker image)

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Postgres - the non-code bits

CRAIG KERSTIENS — Tue, 31 Oct 2017 22:55:56 GMT

Postgres is an interesting open source project. It’s truly one of a kind, it has its own license to prove it as opposed to falling under something like Apache or GPL. The Postgres community structure is something that is pretty well defined if you’re involved in the community, but to those outside it’s likely a little less clear. In case you’re curious to learn more about the community here’s a rundown of a few various aspects of it:

PostgreSQL License

Let’s start with the legal part first. First IANAL. PostgreSQL is under its own license. For those who don’t regularly follow software licensing it’s extremely liberal and flexible. You can take Postgres, fork it, change it, package it up, and resell it. This is actually one of the reasons you see Postgres at the core of so many other databases like Par Accel, Asterdata, etc. That and that it’s such a solid code base that is capable of being extended. I once heard someone describe how they don’t really like writing C, but they enjoy writing Postgres C ;)

A thing you can’t do is profit off the PostgreSQL logo without any approval from the core team.

The people

Within the Postgres community there are 2 major sets of people.

The core team

The core team is a smaller team within the Postgres community. The core team is effectively a steering committee for Postgres. They’re responsible for coordinating releases, handling confidential issues (read: security issues), managing permissions around PostgreSQL code and infrastructure, defining policy.

The core team is a very small list of people, at the moment 5 individuals.

Contributors

Yes, anyone can contribute to Postgres, and with each release there are a laundry of people that write some code that goes into Postgres. If fact Postgres 10 had 325 people that contributed in some form. That said there is a hierarchy that exists. The two biggest ones are committers and major contributors.

Committers gain access after years of contributing to Postgres showing sustained commitment to the project. New committers are voted on each year at PgCon which happens in March/April in Ottawa. If you’re ever curious for a conference of what’s coming and being deep in the internals of Postgres it’s one to check out. Once you do gain your commit bit you’re expected to contribute every couple of years to the project. And of course there are a number of qualifications such as contributing high quality code and perhaps most key is helping review others contributions.

Major contributors are another notable group. Major contributors don’t have full sole commit access, but are held in a higher regard from consistently contributing major features as well as providing review for others.

Contributors in general are another area worth calling out. While they may not have a flagship feature to their name like the major contributors, Postgres is what it is because of the contributions of everyone.

PostgreSQL the company

Well, it turns out there isn’t a company behind Postgres, it’s one thing that makes it unique–no one can ever “own it”. There are some official PostgreSQL non-profits though in particular the US non-profit and the EU non-profit. These non-profits ensure that the core guidelines are enforced and also give coverage for the community to help put on official community conferences. A few of these happen each year which include:

PGCon - The hackers conference
PostgresOpen SV - A consolidation of PostgresOpen and PGConf Silicon Valley
PGConf EU - The largest PG European Conference which moves around each year

If you’re looking for a way to support the PostgreSQL non-profit organization I’d encourage you to consider joining PostgreSQL.us.

Engaging

So you want to jump into the community, where do you even start? The first place I’d encourage is to subscribe to the mailing list or check out the slack channel. The users mailing is a great one to just jump in and help answer questions and see what people need help with. The hackers list is where you go to get a peek at all the fun debates/discussions/development.

If you’re thinking about contributing it’s a good idea to lurk on the hackers list for a bit first. Then when the commitfest comes chip in and help review some patches and do some testing. Oh and of course, you can always blog about what you’re doing with Postgres and will aim to get it included into Postgres Weekly.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Born to Run

Krishna's blog — Tue, 31 Oct 2017 02:00:00 GMT

Born to Run is a low-quality pseudo-science book that pushes the agenda of barefoot running without disclosing the lack of conclusive evidence.

Applying syntax in Sublime based on the first file line

smcleod.net — Wed, 25 Oct 2017 03:00:00 GMT

In vim, you can add a comment at the top of files to set the syntax, e.g.:

# vim: syntax=ruby

In SublimeText there are many ways to detect syntax, one interesting approach I’ve recently found useful is to match on the top line in the file. For example, with Puppet there is a file called Puppetfile, it has no extension but it’s really Ruby syntax, so it’s useful to add linting incase you miss something simple like a , and break deployments.

How to build waifu2x command-line version on osx

Subclassed — Mon, 23 Oct 2017 05:49:54 GMT

UPDATE 23/10/2017: recent changes seem to be incompatible with clang. I have updated the steps below to check out the last commit that is known as working.

Waifu2x is a popular image converter backed by neural-network models, typically used to upscale images from anime. The various web versions are easy to use but typically don't allow for batch-processing, because it's a relatively intensive operation. However, there is a command-line version that is fairly easy to compile on Windows and Linux. OSX support was notably absent... until now.

These are the steps required to get it working on Mac:

brew tap science && brew install opencv3
(if you don't have Homebrew, go install it now - it's wonderful)
git clone https://github.com/DeadSix27/waifu2x-converter-cpp.git
cd waifu2x-converter-cpp
git checkout d69313040b0784662465fb1d2eca81a2b1ebccb2
cmake -DOVERRIDE_OPENCV=1 -DOPENCV_PREFIX=/usr/local/Cellar/opencv3/<your version here> . (remember the dot at the end!)
make -j4

At this point, you have the executable waifu2x-converter-cpp in the directory; you can make install if you really want to (I prefer to keep custom stuff in my home dir). To test it's working, the following command should return info on your system:

./waifu2x-converter-cpp --list-processor

If you need some images to test, here you can get quite a few stills from the gorgeous 5 Centimeters Per Second. Note: you might have to rename the folder "models_rgb" into "models" before converting.

BONUS: Qt GUI

If you hate the command-line, there is a simple QT wrapper. To compile it you will need Qt-Creator and Qt installed and configured. This used to be very complicated, but both items have now been packaged for Homebrew so it's all awesomely simple (well, compared to what it was, at least). Note that you still have to do the steps above - the wrapper needs the waifu2x executable to be already compiled.

Install and configure Qt and Qt-Creator

brew install qt && brew cask install qt-creator
Launch Qt Creator from Launchpad, then go to Qt Creator -> Preferences (Command + ,)
Select Build & Run, then the Qt Versions tab
Click on Add... and select “Macintosh HD”
Press Command + Shift + . to show hidden directories
select /usr/local/Cellar/qt5/<your version>/bin/qmake then OK and OK again to close the Preferences window
Reopen Preferences -> Build & Run, select the Kits tab and click Add
Set the following options:
- Name to something like "Qt CLang Desktop"
- Both Compiler options to Clang X86 64bit
- Qt Version to the version you just created
- Click on Make Default then OK to close Preferences.

Compile waifu2x-converter-qt

git clone https://github.com/toyg/waifu2x-converter-qt.git (the original repo looks abandoned, so I forked it and tweaked it a bit)
cd waifu-converter-qt && open waifu2x-converter-qt.pro (this should open Qt Creator; if it doesn't, launch it manually and File -> Open Project -> select the .pro file).
Select Build -> Run (or Command + R). The resulting .app bundle should now be in a folder called build-waifu2x-converter-qt-<something something>, just beside your main project folder.

When you first launch the GUI, you should probably go to Preferences and set the path to your waifu2x-converter-cpp executable.

Static Analysis: It's Simply Critical

Donat Studios — Sat, 21 Oct 2017 10:59:24 GMT

I am someone who spends large portions of their time working in weakly typed languages. Namely, PHP and JavaScript. I argue that static analysis is a must-have when working with any weakly typed language. It is nearly impossible to make reliable code without it.

Static analysis makes up for the missing type system and allows you to scale.

I recently got in an argument with another developer over static analysis. It's been an ongoing point of contention. He was angry that code that works fine but didn't document its undetectable return types was failing CI. He argued that if the code works, it should pass. He argued that if the code works, it is "correct".

I put forward the exact opposite position. Code that works but doesn't statically analyze is incorrect. Code that is correct has no potential of runtime error.

My intentions are not only to sell you on the idea that you should be statically analyzing your code, but to convince you that it is fundamental to responsible reliable development.

Humans are fallible; code written by humans doubly so. By it's very nature that code is broken until proven otherwise. If it cannot be statically analyzed, it cannot be proven correct.

If code runs as expected, this only proves it correct in the limited scope it's been ran it within. Be that ones, tens or even hundreds of thousands of parameter combinations, it's always limited. There therefore must be zero expectation of it continuing to work.

To the contrary, if code statically analyzes unerring, it will run without runtime error in all cases, save a failure of the analyzer. This code is proven. This code is correct.

Code which "works" does nothing to prove that it's used as you expect. In particular elsewhere within the project. It does nothing to prove it will remain correctly used in all cases in the future.

I know what you are thinking "But… but… my code runs correctly within my expected domain", and that's fine. That is what's expected of it. But throw exceptions if it happen it doesn't know how to handle, things that creep outside the expected domain. It lets the future developers, the analyzer, and the program itself know the intended domain and when it steps outside it.

No amount of fuzz testing can account for another developer doing something you never expected.

To give a trivial but near real life example, consider the following:

function getUserById( userId ) {
    if(userId > 0) {
        return new User($userId);
    }

    return null;
}

var user = getUserById( input.value );
console.log( "Hello userId: " + user.getId() );

The above code may work throughout all your testing. Every place you use it, every time you try it, everything your application throws at it - success.

This code works, and yet this code is broken. This code does not handle the potential null, and has the potential to trigger a runtime error. This code has the capability of a runtime error, and is therefore incorrect.

Making this code correct would be as simple as:

var user = getUserById( input.value );

if(!user instanceof User) {
    throw new Error("User not found");
}

console.log( "Hello userId: " + user.getId() );

An Exception, as opposed to a runtime error, is something the code and coder anticipates. It's an easy way to show you've thought about the domain of your code. Even unhandled, it's clear the developer considered the possibility. It's easily recoverable. Throwing an exception here is correct.

Obviously you'd probably want to handle that Error in the example somewhere, but that's outside the scope of this writing.

The trivial case above is reasonably obvious and easy to spot. Let's instead imagine the function to get a user is deep in your codebase. While we are at it, let's also say that the logical User|null union we've created gets passed around. Passed several levels deep, rather than using it immediately where it's fetched. It rapidly becomes much less obvious, nigh impossible, to notice that there is an issue. A call 8 methods deep has no idea it might be receiving a null. This is exactly the kind of thing a static analysis is for.

A static analyzer knows the full scope and range of possible values. It is able to pinpoint problems a human fails to notice. Noting all function calls within the scope of a project and knowing all possible.

All this withstanding, static analysis cannot prove the logic is correct. That's not what static analysis is for. Unit tests and other testing prove that. Static analysis proves that the code itself executes free of unhandled runtime errors. It is the type checker your language forgot.

Arguable the better solution is to switch to a type safe language; one with compile time type checks and ideally union types. That's not always an option for most people though, consider the recent meteoric growth of JavaScript.

TypeScript is without a doubt the best option for JavaScript. I highly recommend it — consider it static analysis on steroids. It's JavaScript with the addition of strong typing. There's no reason to be writing straight JavaScript anymore. TypeScript is really a wonderful all in one solution.

Elm is a neat alternative but a completely different paradigm. While it compiles to JavaScript, it's completely foreign and more of an all-in situation. It borrows much of its syntax from Haskell, and offers the aforementioned union types.

For PHP on the other hand I recommend a handful of tools. Scrutinizer is in my experience the most feature complete of the hosted CI services. I make extensive use of PHP CodeSniffer which is an amazing tool based around a PHP tokenizer. It's a lot of fun writing your own sniffs in my experience. I have begun experimenting with Phan - it's a little unforgiving, in a good way… And of course PHPStorm has a pretty decent analyzer built in.

Born a Crime

Krishna's blog — Thu, 19 Oct 2017 03:00:00 GMT

'Born a Crime' by Trevor Noah is a hilarious and thought-provoking memoir about his childhood in South Africa.

BBQ with a Dutch ICT/start up Delegation

Ben E. C. Boyter — Tue, 17 Oct 2017 03:22:18 GMT

So a few weeks back I received an interesting email that at first thought looked like a scam.

In short I was being invited to a BBQ at the Consul General’s residence in Sydney to meet an ICT/start up delegation from the Netherlands. Let me straight up say I have no idea why I would receive something like this. I am not dutch, have no dutch ancestry (that I am aware of) and while searchcode.com somehow is listed on the top startups in Sydney somehow its not as though it has lots of press coverage or is a real start-up (its more a hobby side business at the moment).

The Crusades: The Authoritative History of the War for the Holy Land

Krishna's blog — Tue, 17 Oct 2017 03:00:00 GMT

Discover the truth about the Crusades and why invoking them today is misguided, in this well-researched and engaging book.

Broadcom, Or How I Learned To Start Worrying And Drop The Packet

smcleod.net — Fri, 13 Oct 2017 03:00:00 GMT

Earlier this week we started the process to upgrade one of our hypervisor compute clusters when we encountered a rather painful bug with HP’s Broadcom NIC chipsets.

We were part way through a routine rolling pool upgrade of our hypervisor (XenServer) cluster when we observed unexpected and intermittent loss of connectivity between several VMs, then entire XenServer hosts.

The problems appeared to impact hosts that hadn’t yet upgraded to XenServer 7.2. We now attribute this to a symptom of extreme packet loss between the hosts in the pool and thanks to buggy firmware from Broadcom and HP.

Dear Postgres

CRAIG KERSTIENS — Thu, 12 Oct 2017 23:55:56 GMT

Dear Postgres,

I’ve always felt an affinity for you in my 9 years of working with you. I know others have known you longer, but that doesn’t mean they love you more. Years ago when others complained about your rigidness or that you weren’t as accommodating as others I found solace in your steadfast values:

Don’t lose data
Adhere to standards
Move forward with a balancing act between new fads of the day while still continuously improving

You’ve been there and seen it all. Years ago you were being disrupted by XML databases. As companies made heavy investment into what such a document database would do for their organization you proceeded to “simply” add a datatype that accomplished the same and brought your years of progress along with it.

In the early years you had the standard format of index b-tree that most database engines leveraged. Then quietly but confidently you started adding more. Then came K-nearest neighbor, generalized inverted indexes (GIN), and generalized search-tree (GiST), only to be followed by space partitioned GiST and block range indexes (BRIN). Now the only question is which do I use?

All the while there was this other camp using for something that felt cool but outside my world: GIS. GIS, geographical information systems, I thought was something only civil engineers used. Then GPS came along, then the iPhone and location based devices came along and suddenly I wanted to find out the nearest path to my Peets, or manage geographical region for my grocery delivery service. PostGIS had been there all along building up this powerful feature set, sadly to this day I still mostly marvel from the sideline at this whole other feature set I long to take advantage of… one day… one day.

A little over 5 years ago I fell in love with your fastly improving analytical capabilities. No you weren’t an MPP system yet, but here came window functions and CTEs, then I almost understood recursive CTEs (still working on that one). I can iterate over data in a recursive fashion without PL/PgSQL? Yes please! I only want to use it more.

And then five years ago, document stores start taking over the world. I feel like I’ve seen this story before, wasn’t XML going to change the internet? Enter JSON, the JSON datatype, and JSONB. Wow, this is really nice to mix relational, document storage, join against things. I suddenly don’t get why more don’t take this flexible approach to building on a good foundation and layering on the refinements.

Extensions! Where have you been all my life? There’s Citus, and HyperLogLog, and ZomboDB, with each I can add functionality to Postgres without it being limited to the standard release, they can be in C or not. Wait, all along so much has been built on this foundation? PostGIS, full-text search, hstore? I like all those things, why didn’t you tell me all along about this foundation? Postgres, I like what I’m seeing how you’re allowing others to do more without having it be in the core of Postgres. This extension stuff is really kinda cool that it’s Postgres and then some, kinda like C and then ++, wait nevermind scratch that analogy.

Sorry, I’ve rambled a bit. You’re a little over twenty years old now. I’ve known you for nearly ten of those years so I know there’s so much about your background I don’t know, I hope we get to spend the time together to share it all. This 10 release is really an exciting one to me. We’ve spent all this time together and I feel like each passing year the bond grows fonder.

Now you’ve brought me better parallelism so I can further utilize my system resources. I now have partitioning. Thank you! I don’t have to roll my own hacks to help age out old data for my time series database. Logical replication will make so many other things possible, such as more online upgrades and integration with other systems.

Postgres, I just want to say thank you for the past ten years together. Thank you for all you’ve done and for all you’ll continue to do in the future.

Fighting Cliqz in Firefox

Subclassed — Tue, 10 Oct 2017 16:18:31 GMT

Mozilla recently announced that some of their German users downloading Firefox will receive a version that tracks some of their web activity, reporting it to Cliqz.com. In the wake of this development, which is pretty awful from a privacy perspective, I went spelunking into my version of FF to see if anything had been enabled already in my build.

To my horror, cliqz.com is already mentioned in a few places, despite me never installing anything related to it. If you enter about:config in the address bar and search for cliqz, a few entries will pop up:

Straight out of the gate, it looks like something from cliqz.com has been whitelisted. What is it?

The social.* preferences are related to SocialAPI Services, a sort of framework to integrate social networks into Firefox. It was introduced several years ago but very few people actually use it or know about it. If you look up the related preferences, a few more entries are present:

If you are a fan of this sort of thing, you can go to that address https://activations.cdn.mozilla.net and install some of the available providers. I have no idea whether they still work or not; among others, delicious.com has recently been sold and it's in read-only mode, so that's unlikely to be useful.

I personally disabled it all (that social.remote-install.enabled freaked me out) by double-clicking all boolean properties (turning them to false) and double-clicking then clearing out social.whitelist.

The other mentions of cliqz seem to be special-casing whitelists aimed at making an extension work around some of the recent changes in the Firefox extensibility framework. If I understand correctly, dom.ipc.cpows.allow-cpows-in-compat-addons allows Cross-Process Object Wrappers (an internal communication mechanism that should slowly be removed) to be used even if the extension is marked as compatible with the new multiprocess architecture; and extensions.legacy.exceptions exempts the listed extensions from being marked as Legacy.

In those whitelists, there is one called testpilot@cliqz.com. Test Pilot is an official Firefox add-on from Mozilla that will periodically publish some proposed additions to the browser, allowing users to enable them, test them out, give feedback and so on. I personally like it, some of the proposed features are actually pretty good (although it doesn't look like any of them ever made it to the main build); considering its complexity (an extension-installing extension) it makes sense that it might require some special privileges, and after all it's an official Mozilla feature so why not?

Well, it looks like Mozilla is using cliqz.com to gather remote info about TestPilot usage, which is very disappointing. TestPilot was marketed as an official Mozilla project, there was no mention of third parties involved. I won't disable TestPilot, but I am again very disappointed in their cavalier attitude with my usage data.

To conclude, it looks like the "synergy" between Mozilla and Cliqz was already underway before the latest announcement. It's likely that more defensive hacks will be required in the near future to keep user-tracking at bay in Firefox. As a long-time fan of Mozilla since the '90s, this development is disappointing.

The Remains of the Day

Krishna's blog — Sat, 07 Oct 2017 03:00:00 GMT

Follow the story of a reserved butler as he reflects on his life and the meaning of his work in The Remains of the Day.

Computers Have Had Emotions for Quite Some Time

Svedic.org — Fri, 29 Sep 2017 23:31:52 GMT

A common assumption is that computers can’t have emotions. But there is a strong philosophical argument that AI systems have had emotions for many decades now. Before making an argument, we need to define “emotion”. That definition shouldn’t require consciousness … Continue reading →

Do Androids Dream of Electric Sheep

Krishna's blog — Fri, 29 Sep 2017 03:00:00 GMT

Do Androids Dream of Electric Sheep has potential, but is sidetracked by a confusing premise and missed opportunities.

Kernel Recipes 2017 day 3 notes

Linux Engineer's random thoughts — Fri, 29 Sep 2017 01:00:00 GMT

This is continuation of day 1 and day 2 of Kernel Recipes 2017.

Using Linux `perf` at Netflix

by Brendan Gregg

Brendan started with a ZFS on Linux case study, where it was eating 30% of the CPU resources, which it should never be doing. He started by generating a …

Kernel Recipes 2017 day 2 notes

Linux Engineer's random thoughts — Thu, 28 Sep 2017 01:00:00 GMT

This is continuation of yesterday's live blog of Kernel Recipes 2017.

Linux Kernel Self Protection Project

by Kees Cook

Presentation slides

The aim of the project is more than protecting the kernel.

Background

Kees' motivation for working on Linux, is the two billion Android devices running a Linux. The majority …

Kernel Recipes 2017 day 1 live-blog

Linux Engineer's random thoughts — Wed, 27 Sep 2017 01:00:00 GMT

Following last year attempt, I'm doing a live blog of Kernel Recipes 6th edition. There's also a live stream at Air Mozilla

What's new in the world of storage for Linux

by Jens Axboe

Jens started with the status of blk-mq conversions: most drivers are now converted: stec, nbd, MMC …

Embedded Recipes 2017 notes

Linux Engineer's random thoughts — Tue, 26 Sep 2017 01:00:00 GMT

Following last year attempt, I'm doing a live blog of Embedded Recipes 1st edition.

Understanding SCHED_DEADLINE

by Steven Rostedt

Every task starts as SCHED_OTHER, where each task gets a fair share of the CPU bandwidth.

Then comes SCHED_FIFO, where it's first in, first out, a task will run until it …

GlusterFS

smcleod.net — Mon, 25 Sep 2017 03:00:00 GMT

We’re in the process of shifting from using our custom ‘glue’ for orchestrating Docker deployments to Kubernetes, When we first deployed Docker to replace LXC and our legacy Puppet-heavy application configuration and deployment systems there really wasn’t any existing tool to manage this, thus we rolled our own, mainly a few Ruby scripts combined with a Puppet / Hiera / Mcollective driven workflow.

The main objective is to replace our legacy NFS file servers used to host uploads / attachments and static files for our web applications, while NFS(v4) performance is adequate, it is a clear single point of failure and of course, there are the age old stale mount problems should network interruptions occur.

Return Of The RSS

smcleod.net — Fri, 22 Sep 2017 03:00:00 GMT

Of all the tools for reading news and subscribing to software releases, I still find RSS the most useful.

I use Feedly to manage my rss subscriptions and keep all my devices in sync, but instead of using the Feedly’s own client, I use an app called Reeder as the client / reader itself.

Link: My Feedly RSS Feed

Feedly

RSS feed subscription management

Features:

Keyword alerts.
Browser plugins to subscribe to (current) url.
Notation and highlighting support (a bit like Evernote).
Search and filtering across large numbers of feeds / content.
IFTTT, Zapier, Buffer and Hootsuite integration.
Built in save / share functionality (that I only use when I’m on the website).
Backup feeds to Dropbox.
Very fast, regardless of the fact that I’m in Australia - which often impacts the performance of apps / sites that tend to be hosted on AWS in the US as the latency is so high.
Article de-duplication is currently being developed I believe, so I’m looking forward to that!
Easy manual import, export and backup (no vendor lock-in is important to me).
Public sharing of your Feedly feeds (we’re getting very meta here!).

Reeder

A (really) beautiful and fast iOS / macOS client

Tracking and managing your Postgres connections

CRAIG KERSTIENS — Mon, 18 Sep 2017 23:55:56 GMT

Managing connections in Postgres is a topic that seems to come up several times a week in conversations. I’ve written some about scaling your connections and the right approach when you truly need a high level of connections, which is to use a connection pooler like pgBouncer. But what do you do before that point and how can you better track what is going on with your connections in Postgres?

Postgres under the covers has a lot of metadata about both historical and current activity against a system. Within Postgres you can run the following query which will give you a few results:

SELECT count(*),
 state
FROM pg_stat_activity
GROUP BY 2;
 count | state
-------+-------------------------------
 7 | active
 69 | idle
 26 | idle in transaction
 11 | idle in transaction (aborted)
(4 rows)

Time: 30.337 ms

Each of these is useful in determining what you should do to better manage your connection count. All of these numbers can be useful to record every say 30 seconds and chart on your own internal monitoring. Lets break down each:

active - This is currently running queries, in a sense this is truly how many connections you may require at a time
idle - This is where you have opened a connection to the DB (most frameworks do this and maintain a pool of them), but nothing is happening. This is the one area that a connection pooler like pgBouncer can most help.
idle in transaction - This is where your app has run a BEGIN but it’s now waiting somewhere in a transaction and not doing work.

For idle as mentioned above it’s one that you do want to monitor and if you see a high number here it’s worth investing in setting up a pgBouncer.

For idle in transaction this one is a bit more interesting. Here what you likely want to do when first investigating is get an idea of how old those are. You can do this by querying pg_stat_activity and filtering for where the state is idle in transaction and checking how old those queries are. For ones that have been running too long you may want to manually kill them.

If you find that you have some stale transactions hanging around this could be for days, hours, or even just a few minutes you may want to set a default to kill those transactions.

To help with this Postgres has a nice feature of a statement_timeout. A statement timeout will automatically kill queries that run longer than the allotted time. You can set this at both a global level and for a specific session. To do this at the database level you’d run this with an alter database dbnamehere set statement_timeout = 60000; which is 60 seconds. To do so during a given session simply run set statment_timeout = 6000000;.

For idle in transaction that have been running too long there is its own setting setting that you can set in a similar fashion idle_in_transaction_session_timeout (on Postgres 9.6 and up). Setting both statement_timeout and idle_in_transaction_session_timeout will help with cancelling long running queries and transactions.

Keeping your connection limits in check should lead to a much healthier performing database and thus app.

Home Server Build

Nemo's Home — Sun, 17 Sep 2017 03:00:00 GMT

I’d been planning to run my own home server for a while, and this culminated in a mini-ITX build recently. The current build configuration is available at /setup/homeserver/.

In no particular order, here were the constraints:

The case should be small (I preferred the Elite 110, but it was unavailable in India).
Dual LAN, if possible (decided against it at the end). The plan was to run the entire home network from this directly by plugging in the ISP into the server.
Recent i3/i5 for amd64 builds.
Enough SATA bays in the cabinet for storage

The plans for the server:

Scheduled backups from other sources (Android/Laptop)
Run Kodi (or perhaps switch to Emby)
Run torrents. Transmission-daemon works. Preferably something pluggable and that works with RSS
Do amd64 builds. See https://github.com/captn3m0/ideas#arch-linux-package-build-system
Host a webserver. This is primarily for serving resources off the internet
- Host some other minor web-services
- A simple wiki
- Caldav server
- Other personal projects
Sync Server setup. Mainly for the Kindle and the phone.
Calibre-server, koreader sync server for the Kindle
- Now looking at libreread as well
Tiny k8s cluster for running other webapps
Run a graylog server for sending other system log data (using papertrail now, has a 200MB limit)

No plans to move mail hosting. That will stay at migadu.com for now.

I had a lot of spare HDDs that I was going to re-use for this build:

WD MyBook 3TB (external, shelled).
Seagate Expansion: 1TB
Seagate Expansion 3TB (external, shelled)
Samsung EVO 128GB SSD

The 2x3TB disks are setup with RAID1 over btrsfs. Important data is snapshotted to the other 1TB disk using btrfs snapshots and subvolumes. In total giving me ~4TB of storage.

Software

Currently running kodi-standalone-service on boot. Have to decide on a easy-to-use container orchestration platform. Choices as of now are:

Rancher
Docker Swarm
Shipyard
Terraform
Portainer

Most of these are tuned for multi-host setups, and bring in a lot of complexity as a result. Looking at Portainer, which seems well suited to a single-host setup.

Other services I’m currently running:

elibsrv. Running a patched build with support for ebook-convert
ubooquity for online reading of comics

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Project Updates

Nemo's Home — Sat, 16 Sep 2017 03:00:00 GMT

Over the last couple of years, I’ve been involved with lots of side projects, both online and offline. Some of them, I’ve written about on the blog, like my music visualizer project. A few of them, got their own project page, like the website for my niece (but no blog post) while some didn’t even get a mention. I thought I’d write about the many-many side projects I’ve started (and abandoned). You might also wanna visit the /projects page for the larger projects.

Home Server Build: Sep 2017 Built a home server, mostly as a HTPC but also as a learning exercise for managing services over Docker.
Sushi Go: Summer 2017 This is a work-in-progress conversion of Sushi Go (original), the popular card game by Gamewright into Ruby.
youtube-ripper: June 2017 Downloads music-compilations from YouTube and rips them into multiple tagged MP3 files.
cosmere-books: September 2017 Wrote a EPUB generator for multiple books in the Cosmere. Currently covers 4 different serializations at Tor.com. Also created a project page on all of my ebooks projects at /ebooks/
ideas: Ongoing I maintain a CC0 licensed list of personal ideas. Feel free to use.
spectrumyzer: May 2017 Created an animated wallpaper using spectrumyzer. Wrote a blog post about it.
google-sre: Feb/Sep 2017 EPUB generator for the Google SRE ebook. Started in February in Python. Gave up and redid it properly in September.
CodeChef Offline: March 2012 I attempted to make a offline repository for CodeChef problems. I spent some time in May 2017 upgrading the project with a cleaner scraper and a Jekyll base.
Hoshruba: June 2015 I wrote a script that scraped Tor’s serialized publication of the first book in Hoshruba series to generate EPUB and MOBI files. I would recommend the book if you are interested in reading what many would term the “original fantasy book”
HackerTray: December 2013 - I wrote a Linux PyGTK application that sits in your taskbar using Indicator Applet to show you the latest stories from Hacker News. Looking for a maintainer.
MagicMuggle: May 2017 I wrote a script to convert Magic Muggle (A Harry Potter fanfic about a muggle who accidentally gets into Hogwarts) books from their original reddit posts to EPUB and MOBI files.
Kerala IT Policy: March 2017 Attempted to transcribe the draft IT policies put up by the Government of Kerala. Lots of OCR followed by manual fixes. I stopped working on this when I realized that the government had actually put up a really nice website for this (with clear plaintext, not the bad PDF I was using as the source).
lightsaber: August 2015 I created a DNS based HTTP-3xx redirect service. Useful if you own a domain and you want it to be redirected, but don’t have a webserver with you. Made as part of the Django Hackathon organized by HackerEarth in Ruby.
HackerCouch: November 2015 My hack during hackbeach 2015. Created something best described as “couchsurfing for hackers”. Simple Jekyll/Ruby website hosted on GitHub Pages.

Better database migrations in Postgres

CRAIG KERSTIENS — Sun, 10 Sep 2017 23:55:56 GMT

As your database grows and scales there are some operations that you need to take more care of than you did when you were just starting. When working with your application in your dev environment you may not be fully aware of the cost of some operations until you run them against production. And at some point most of us have been guilty of it, running some migration that starts at 5 minutes, then 15 minutes in it’s still running, and suddenly production traffic is impacted.

There are two operations that tend to happen quite frequently, each with some straightforward approaches to mitigate having any noticable amount of downtime. Let’s look at each of the operations, how they work and then how you can approach them in a safer way.

Adding new columns

Adding a new column is actually quite cheap in Postgres. When you do this it updates its underlying tracking of the columns that exist–which is almost instant. The part that becomes expensive is when you have some constraint against the column. A constraint could be a primary or foreign key, or some uniqueness constraint. Here Postgres has to scan through all the records in the table to ensure that it’s not being violated. Adding some constraint such as not null does happen some, but is not the most common cause.

The most common reason for slowness of adding a new column is that most frameworks make it very simple for you to set a default value for the new column. It’s one thing to do this for all new records, but when you do this when an existing table it means the database has to read all the records and re-write them with the new default value attached. This isn’t so bad for a table with a few hundred records, but for a few hundred million run it then go get yourself coffee, or lunch, or a 5 course meal because you’ll be waiting for a while.

In short, not null and setting a default value (on creation) of your new column will cause you pain. The solution is to not do those things. But, what if you want to have a default value and don’t want to allow nulls. There’s a few simple steps you can take, by essentially splitting your migration up from 1 step to 4 migrations:

Add your new column that allows nulls
Start writing your default value on all new records and updates
Gradually backfill the default value
Apply your constraint

Yes, this is a little more work, but it doesn’t impact production in nearly the same magnitude.

Indexes

Index creation like most DDL operations holds a lock while it’s occurring, this means any new data has to wait for the index to be created and then the new writes flow through. Again when firsting creating the table or on a small table this time is not very noticable. On a large database though, you can again wait minutes to possibly even hours. It’s a bit ironic when you think about it that adding an index to speed things up can slow things down while it’s happening.

Postgres of course has the answer for this with CONCURRENT index creation. What this does is gradually build up the index in the background. You can create your index concurrently with: CREATE INDEX CONCURRENTLY. As soon as the index is created and available as long as you did what you were hoping to Postgres will swap over to using it on queries.

A tool to help

It’s a good practice to understand what is happening when you run a migration and its performance impact. That said you don’t have to manage this all on your own. At least for Rails there’s a tool to help enforce more of these as you’re developing to catch it earlier. Strong migrations aims to catch many of these expensive operations for you to have your back, if you’re on Rails consider giving it a look.

Have other tools or tips that can help with database migrations in Postgres? Drop me a note and I’ll work to add them to the list.

Working with Rust

Ben E. C. Boyter — Sun, 10 Sep 2017 00:59:08 GMT

For a while I have been wanting to play with a new programming language. Professionally I work with Java, C#, Python and JavaScript and so I went looking for something complimentary to them.

My first stop was was Go. I started by implementing a vector space search in it which you can see here https://github.com/boyter/golangvectorspace

While I liked the syntax (I still cannot think in Go), the libraries and the performance I realized that Go is close to a hybrid of Java and Python for me. It has the performance of Java with the Python style syntax. This is fine but I really wanted to push what I already am familiar with.

Postgres backups: Logical vs. Physical an overview

CRAIG KERSTIENS — Sun, 03 Sep 2017 23:55:56 GMT

It’s not a very disputed topic that you should backup your database, and further test your backups. What is a little less discussed, at least for Postgres, is the types of backups that exist. Within Postgres there are two forms of backups and understanding them is a useful foundation for anyone working with Postgres. The two backup types are

Physical: which consist of the actual bytes on disk,
Logical: which is a more portable format.

Let’s dig into each a bit more so you can better assess which makes sense for you.

Logical backups

Logical backups are the most well known type within Postgres. This is what you get when you run pg_dump against a database. There are a number of different formats you can get from logical backups and Postgres does a good job of making it easy to compress and configure this backup how you see fit.

When a logical backup is run against a database it is not throttled, this introduces a noticable load on your database.

As it’s reading the data from disk and generating (in layman terms) a bunch of SQL INSERT statements, it has to actually see the data. It’s of note that older Postgres databases (read: prior to 9.3) there were no checksums against your database. Checksums are just one tool for you to help check against data corruption. Because a logical dump has to actually read and generate the data to insert it will discover any corruption that exists for you.

This portable format is also very useful to pull down copies from production to different environments. I.e. if you need a copy of production data down on your local laptop pg_dump is the way to do it. Logical backups are also database specific, but then allow you to dump only certain tables.

All in all logical backups bring some good features, but come at two cost:

Load on your system
The backup contains data as of the time when it ran

Physical backups

Physical backups are another option when it comes to backing up your database. As we mentioned earlier it is the physical bytes on disk. To understand physical backups we need to know a bit more under the covers about how Postgres works.

Postgres, under the covers, is essentially one giant append only log. When you insert data it gets written to the log known as the write-ahead log (commonly called WAL). When you update data a new record gets written to the WAL. When you delete data a new record gets written to the WAL. Nearly all changes in Postgres including to indexes and otherwise cause an update to the WAL.

With physical backups what you require to be able to create a restore of your database is two things:

A base backup, which is a copy of the bytes on disk as of that point and time
Additional segments of the WAL to put the database in some consistent state.

A physical backup only requires a small amount of WAL to restore the database to some valid state, but this also gives you some new flexibility. With a base backup plus WAL you can start to replay transactions up to a specific point in time. This is often how point-in-time recovery is performed within Postgres. If you accidentally drop a table, yes… it happens, you can:

Find a base backup before you dropped the table
Restore that base backup
Replay wal segments up to roughly that time just before you dropped the table.

If you’re considering setting up physical backups, consider using a tool like WAL-G to help.

Logical vs. Physical which to choose

Both are useful and provide different benefits. At smaller scale, say under 100 GB of data logical backups via pg_dump are something you should absolutely be doing. Because backups happen quickly on smaller databases you may be able to get out without functionality like point-in-time recovery. At larger scale, as you approach 1 TB physical backups start to become your only option. Because of the load introduced by logical backups and the time lapse between capturing them they become less suitable for production.

Hopefully this primer helps provide a high level overview of the two primary types of backups that exist as options for Postgres. Of course there is much deeper you can go on each, but consider ensuring you have at least one of the two if not both in place. Oh and make sure to test them, an un-tested backup isn’t a backup at all.

xmlstarlet, xpath, and confusion

infrequent oscillations — Mon, 28 Aug 2017 12:06:08 GMT

In which we fail to account for XML namespaces.

You Donate $400/Year Thanks To The Best Business Trick Ever

Svedic.org — Fri, 11 Aug 2017 17:09:16 GMT

I’m going to tell you a story about one ingenious business model that the majority of people are not aware of. It costs average US household around $400 per year. To understand the model, you’ll need to understand three economic … Continue reading →

Migration to Ghost v1.0

infrequent oscillations — Tue, 01 Aug 2017 12:38:22 GMT

In which we start to get annoyed by Ghost.

Windows "vintage" default desktop colours

Subclassed — Mon, 31 Jul 2017 22:03:56 GMT

Note to self: these are the default desktop background colours for old Windows versions.

008080 (RGB 000-128-128) - Windows 95/98
3A6EA5 (RGB 058-110-165) - Windows 2000

(I know you love them, even if setting a wallpaper was the first thing you did after logging on a new PC. It's called nostalgia.)

The Sanity Test

infrequent oscillations — Thu, 27 Jul 2017 14:42:26 GMT

In which we are salty about some Perl.

Recent creations

infrequent oscillations — Mon, 24 Jul 2017 18:02:10 GMT

In which we summarise some craftings.

Why I moved from Fitbit to Apple Watch

infrequent oscillations — Mon, 24 Jul 2017 14:02:09 GMT

In which we get annoyed with Fitbit and defect.

Graphing performance as investigative endeavour

infrequent oscillations — Fri, 21 Jul 2017 16:32:24 GMT

In which we see if lap times can tell us about endurance.

Wanted: Collaborative Writer in Berlin

Svedic.org — Tue, 11 Jul 2017 14:17:13 GMT

“The advantage of collaborative writing is that you end up with something for which you will not be personally blamed.”—Scott Adams This is a unique job, for unique writers. The client is a well-off individual, the owner of a boring … Continue reading →

awk driven IoT

Linux Engineer's random thoughts — Wed, 05 Jul 2017 01:00:00 GMT

With a Raspberry Pi or other modern single-board computers, you can make very simple toys. I started with the hello world of interactive apps: the soundboard. But even then, I was too ambitious.

The soundpad

Since the main target was kids. I wanted a simple, screen-less toy that would teach …

Postgres Open Silicon Valley line-up: First take

CRAIG KERSTIENS — Sat, 01 Jul 2017 23:55:56 GMT

This year Postgres open and PGConf SV have combined to great a bigger and better conference right in downtown San Francisco. I’m obviously biased as I’m one of the co-chairs, and I know every conference organizer says picking the talks was hard, but I’m especially excited for the line-up this year. The hard part for me is going to be which talks do I miss out on because I’m sitting in the other session that’s ongoing. You can see the full list of talk and tutorial sessions, but I thought it’d be fun to do a rundown of some of my favorites.

How Postgres could index itself

Postgres indexing itself has long been on my wishlist. Andrew Kane from Instacart, and creator of PgHero has bottled up many learnings into a new tool: Dexter. I suspect we’ll get a look at all that went into this, how it works, and how you can leverage it to have a more automatically tuned database.

Scaling a SaaS Application Beyond a Single Postgres with Citus

Migration talks are all to common, from Postgres to MySQL from MySQL to Postgres, or from Dynamo to Postgres. But this one is a little different flavor from Postgres to sharded Postgres with Citus. Sharding into a distributed system of course brings new things to consider and think about, and here you’ll learn about them from first hand experience so hopefully you can avoid mistakes yourself.

Concurrency Deep Dive

This one looks to be a great under the hood look as well as likely very practical. It’ll cover MVCC which is really at so much of the core of how Postgres works, but then bring it up to what it means for things like locks. Best of all, this one like so many others comes with lots of real world experience from Segment.

Postgres window magic

Running PostgreSQL @ Instagram

Instagram is well known as one of the largest apps in the world. They optimized and changed their setup multiple times and probably scaled in about every way possible. Here we get to learn about all the various things you need in running at a truly astonishing scale.

Many many more

Of course there’s many more. Talks range from looks at new features, to how certain companies are using Postgres. We’ve got companies like Instacart and Instagram as mentioned giving talks, to Postgres core committers. Whether you want to learn about the inner workings of Postgres (which often hurts my brain) to how you can simply speed up your app you should find something you like, as long as you like Postgres that is. Take a look at the full list of sessions and we hope to see you there.

Before and After

infrequent oscillations — Wed, 28 Jun 2017 18:33:57 GMT

In which we tighten up a weave.

Design for searchcode server

Ben E. C. Boyter — Tue, 27 Jun 2017 11:18:05 GMT

A very brief update about the progress of searchcode server. Currently I am in the middle of reworking how the index is built and maintained. The idea being I want to add zero downtime index rebuilds which requires a blue/green index strategy. It is still very much in flux but the current design is to merge the indexer and searcher which should allow this to happen. I have been playing around with using an iPad as a production device these days and produced the following document.

Why Refactoring Isn't Popular

The Cranky Developer on Crater Moon Development — Thu, 22 Jun 2017 03:00:00 GMT

Refactoring is cleaning the kitchen while you cook.

Home Battery Systems – You may de-rate system capacity

Ben E. C. Boyter — Wed, 14 Jun 2017 11:06:12 GMT

A quick post. I was looking at the prices to install a home battery system the other day. During the sales process I was informed that for a modular system such as the one provided by Ampetus that if you don’t buy 2 or more batteries that

“you may de rate system capacity”

Sound’s impressively scary! Slightly worried I asked that it means. Thankfully what it actually means is that instead of getting 3kW delivery using two batteries that one battery matched with the inverter may only deliver 1.5kW. This is not a problem if you are still connected to the grid but it can mean that your inverter isn’t working at its full potential. So in short nothing to worry about.

Working with time in Postgres

CRAIG KERSTIENS — Thu, 08 Jun 2017 23:55:56 GMT

A massive amount of reporting queries, whether really intensive data analysis, or just basic insights into your business involving looking at data over a certain time period. Postgres has really rich support for dealing with time out of the box, something that’s often very underweighted when dealing with a database. Sure, if you have a time-series database it’s implied, but even then how flexible and friendly is it from a query perspective? With Postgres there’s a lot of key items available to you, let’s dig in at the things that make your life easier when querying.

Date math

The most common thing I find myself doing is looking at users that have done something within some specific time window. If I’m executing this all from my app I can easily inject specific dates, but Postgres makes this really easy for you. Within Postgres you have a type called an interval that is some window of time. And fortunately Postgres takes care of the heavy lifting of how might something translate to or from hours/seconds/milliseconds/etc. Here’s just a few examples of things you could do with interals:

‘1 day’::interval
‘5 days’::interval
‘1 week’::interval
‘30 days’::interval
‘1 month’::interval

A note that if you’re looking to remove something like a full month, you actually want to use 1 month instead of trying to calculate yourself.

With a given interval you can easily shift some window of time, such as finding all users that have signed up for your service within the past week:

SELECT *
FROM users
WHERE created_at >= now() - '1 week'::interval

Date functions

Date math makes it pretty easy for you to go and find some specific set of data that applies, but what do you do when you want a broader report around time? There’s a few options here. One is to leverage the built-in Postgres functions that help you work with dates and times. date_trunc is one of the most used ones that will truncate a date down to some interval level. Here you can use the same general values as the above, but simply pass in the type of interval it will be. So if we wanted to find the count of users that signed up per week:

SELECT date_trunc('week', created_at),
count(*)
FROM users
GROUP BY 1
ORDER BY 1 DESC;

This gives us a nice roll-up of how many users signed up each week. What’s missing here though is if you have a week that has no users. In that case because no users signed up there is no count of 0, it just simply doesn’t exist. If you did want something like this you could generate some range of time and then do a cross join with it against users to see which week they fell into. To do this first you’d generate a series of dates:

SELECT generate_series('2017-01-01'::date, now()::date, '1 week'::interval) weeks

Then we’re going to join this against the actual users table and check that the created_at falls within the right range.

with weeks as (
select week
from generate_series('2017-01-01'::date, now()::date, '1 week'::interval) week
)
SELECT weeks.week,
count(*)
FROM weeks,
users
WHERE users.created_at > weeks.week
AND users.created_at <= (weeks.week - '1 week'::interval)
GROUP BY 1
ORDER BY 1 DESC;

Timestamp vs. Timestamptz

What about storing the times themselves? Postgres has two types of timestamps. It has a generic timestamp and one with timezone embedded in it. In most cases you should generally opt for timestamptz. Why not timestamp? What happens if you move a server, or your server somehow swaps its configuration. Or perhaps more practically what about daylight savings time? In general you might think that you can simply just put in the time as you see it, but when different countries around the world observe things like daylight savings time differently it introduces complexities into your application.

With timestamptz it’ll be aware of the extra parts of your timezone as it comes in. Then when you query from one timezone that accounts for daylights savings you’re all covered. There’s a number of articles that cover a bit more in depth on the logic between timestamp and timestamp with timezone, so if you’re curious I encourage you to check them out, but by default you mostly just need to use timestamptz.

There’s a number of other functions and capabilities when it comes to dealing with time in Postrges. You can extract various parts of a timesetamp or interval such as hour of the day or the month. You can grab the day of the week with dow. And one of my favorites which is when we celebrate happy hour at Citus, there’s a literal for UTC 00:00:00 00:00:00 which is allballs(). If you need to work with dates and times in Postgres I encourage you to check out the docs before you try to re-write something of your own, chances are what you need may already be there.

Write Simple Code to Prevent Headaches Later

Ben Overmyer's Site — Mon, 05 Jun 2017 03:00:00 GMT

Consider the following code snippet:

someVar = false;

if ( anotherVar >= 1 ) {
 someVar = true;
}

Now, consider this rewrite:

someVar = ( anotherVar >= 1 );

Both snippets work as designed, so why is the second one better? It comes down to simplicity. The first snippet includes an if block. It would be possible for another programmer to add logic into that block that, while not modifying the final result of someVar, increases the complexity of the code. This makes it harder to maintain in the long run. The second snippet is very clear about what it does, and is more difficult to tack additional logic onto.

How to get Ctrl-arrow shortcuts working in RDP sessions on macOS / OSX

Subclassed — Thu, 18 May 2017 12:01:00 GMT

It looks like recent versions of the official Microsoft Remote Desktop client are a bit shy when it comes to sending CTRL to the remote Windows session. I discovered this because I use Ctrl-Arrow very often to move from word to word, and it just stopped working in RDP.

The solution is to make sure you don't have any overlapping OSX shortcut - typically, CTRL is used for Mission Control actions. Just disable them from System Preferences -> Keyboard -> Shortcuts -> Mission Control and you should be good to go. Any recent Apple keyboard will have real keys dedicated to those actions anyway.

(Come on Microsoft, show some guts - have an option to bypass Mac shortcuts entirely!)

Social Proof and the Bystander Effect

Beetle Space — Tue, 16 May 2017 10:00:00 GMT

Lately I’ve been reading the book Influence by Robert Cialdini. The chapter on Social Proof reminded me of an episode of This American Life I had heard some years ago [1].

The episode begins with a mailman delivering mail, when he sees:

And they were punching each other and …

How to identify software licenses using Python, Vector Space Search and Ngram Keywords

Ben E. C. Boyter — Mon, 08 May 2017 09:54:34 GMT

EDIT – I have since taken the ideas below improved them and released a command line application you can use to build software license reports https://github.com/boyter/lc/

The below is mostly a log of my thought process while building out some functionality that I wanted to add into searchcode server. I kept a record of progress and thoughts while doing this in the hopes that I get some sort of useful blog post out of it. It has been edited somewhat for clarity.

Building the perfect audio visualization

Nemo's Home — Mon, 01 May 2017 03:00:00 GMT

I made this as my animated wallpaper recently (Click to play/pause):

above video has a audio component, click at your own peril.

What follows is the story and the tech behind making this.

The Wallpaper

I have a long history of using custom wallpapers. This was my wallpaper from 2014-:

When I asked Vikalp to design a new one, I knew I wanted something that was slightly more softer. This is what he came up with, after a few iterations:

This wasn’t the final iteration, and both of us agreed that there was something missing.

Visualizations

I saw a colleague using cava and spent a bit of time trying out different visualization software. The ones that I tried out:

cava: works perfectly with i3, runs on a terminal. I couldn’t get it to work cleanly with transparency.
mildrop: Winamp’s legacy. This works great for parties, but is not really an everyday-use visualizer.
spectrumyzer: Worked with transparency, but limited to bars visualization.

I decided to go ahead with Spectrumyzer (This is the default config):

The Traffic Jam

The very same day, stuck in a traffic jam¹, I asked Vikalp for some color ideas on the visualization.

The obvious 2 were tried first:

It finally dawned on us to use the light blue variant with padding set to zero:

Here is one showing the actual positioning (set using the offsets):

Bezier Curves

With the padding set to zero, it already looked great. I ended up using this as my wallpaper for the next one week. Vikalp wanted to make the bars non-rectangular, and I spent some time figuring out how to make waveforms using bezier curves². The basic learning from my experiments were:

Cairo has methods for drawing cubic bezier curves.
Cubic bezier curves have 2 control points.
The control points must be symmetric (equidistant) as well as parallel to the origin points.
The parallel part ensures that the ending and starting line segments are always tangential giving a smooth joining.

This is roughly what you want to do when drawing waveforms:

If you are interested in playing around with Bezier curves, see Animated Bézier Curves. A Primer on Bézier Curves is a math-heavy explanation if you want to read further³.

The code I wrote picks the midpoints of the bars and then connects them using bezier curves:

# control point cords
# Make sure these are symmetric
c1x = rect_top_mid_x + 16
c2x = next_rect_top_mid_x - 16
c1y = rect_top_mid_y
c2y = next_rect_top_mid_y

I also had to make the number of bars configurable (this is default=64, which doesn’t look great):

Here is the complete final result in HD:

What I learned

Bezier curves are not magic.
Drawing pixels on screen and filling them was quite easy with Cairo and Python.
Coding is wizardry. The things that I take for granted every day (take a multi-page website and get useful tabular data out of it, for eg) are unthinkable for most people. The idea of doing water waves was something I knew would be possible before I even looked at the codebase.

If you’d like to replicate this setup, or build upon it, here is my spectrum.conf file. I also filed a PR (now merged!) to the spectrumyzer project adding support for curve based renders.

Sony World Junction - Where startup ideas are born. ↩
The Spectrumyzer codebase turned out to be fairly easy to understand. It was just cairo and pyGTK. ↩
A Primer on Bézier Curves was published on HN just a few days after I finished this project. ↩

Why use Postgres (Updated for last 5 years)

CRAIG KERSTIENS — Sun, 30 Apr 2017 23:55:56 GMT

Five years ago I wrote a post that got some good attention on why you should use Postgres. Almost a year later I added a bunch of things I missed. Many of those items bear repeating, and I’ll recap a few of those in the latter half of this post. But in the last 4-5 years there’s been a lot of improvements and more reasons added to the list of why you should use Postgres. Here’s the rundown of the things that make Postgres a great database you should consider using.

Datatypes, including JSONB and range types

Postgres has long had an open and friendly attitude for adding datatypes. It’s had arrays, geospatical and more for some time. A few years ago it got two datatypes worth thinking about using:

JSONB

JSONB is a binary representation of JSON. It’s capable of being indexed on with GIN and GIST index types. You can also query into your full JSON document for quick lookups.

Range types

While it didn’t arrive to the same fame as JSONB, range types can be especially handy if they’re what you need. Within a single column you can have a range from one value to another–this is especially helpful for time ranges. If you’re building a calendaring application or often have a from and to of timestamps then range types can let you put that in a single column. The real benefit is that you can then have constraints that certain time stamps can’t overlap or other constraints that may make sense for your application.

Extensions

It’d be hard to talk about Postgres without all the ecosystem around it. Extensions are increasingly quite key when it comes to the community and growth of Postgres. Extensions allow you to hook into Postgres very natively without requiring them to be committed back to the core of Postgres. This means they can add rich functionality without being tied to a Postgres release and review cycle. Some great examples of this are:

Citus

Citus (who I work for) turns Postgres into a distributed database allowing you to easily shard your database across multiple nodes. To your application it still looks like a single database, but then under the covers it’s spread across multiple physical machines and Postgres instances.

HyperLogLog

This is a personal favorite of mine that allows you to easily have close-enough distinct counts pre-aggregated, but then also do various operations on them across days such as unions, intersections, and more. HyperLogLog and other sketch algorithms can be extremely common across large datasets and distributed systems, but it’s especially exciting to find them pretty close to out of the box in Postgres.

PostGIS

PostGIS isn’t new, but it’s worth highlighting again. It’s commonly regarded as the most advanced geospatial database. PostGIS adds new advanced geospatial datatypes, operators, and makes it easy to do many of the location based activities you need if you’re dealing with mapping or routing.

Logical replication

For many years the biggest knock against Postgres was the difficulty in setting up replication. Originally this was any form of replication, but then streaming replication came along (this is streaming of the binary WAL or write-ahead-log format). Tools like wal-e help leverage much of the Postgres mechanisms for things like disaster recovery.

Then we had the foundation for logical replication in recent releases, though it still required an extension to Postgres so it wasn’t 100% out of the box. And, then finally we got full logical replication. Logical replication allows the sending of more or less actual commands, this means you could replicate only certain commands or certain tables.

Scale

In addition to all of the usability featuers we’ve seen Postgres continue to get better and better at performance. In particular we now have the foundations for parallelism and on some queries you’ll see much better performance. Then if you need even greater scale than single node Postgres (such as 122 or 244 GB of RAM on RDS or Heroku) you have options like Citus which was mentioned earlier that can help you scale out.

Richer indexing

Postgres already had some pretty powerful indexing before with GIN and GiST, those are now useful for JSONB. But we’ve also seen the arrival of KNN indexes and Sp-GiST and have even more on the way.

Upsert

Upsert was a work in progress for several years. It was one of those features that most people hacked around with CTEs, but that could create race conditions. It was also one of the few features MySQL had over Postgres. And just over a year ago we got official upsert support.

Foreign Data Wrappers

Okay, yes foreign data wrappers did exist many years ago. If you’re not familiar with foreign data wrappers, they allow you map an external data system to tables directly in Postgres. This means could could for example interact and query your Redis database from directly in Postgres with SQL. They’ve continued to be improved more and more from what we had over 5 years ago. In particular we got support for write-able foreign data wrappers, meaning you can write data to other systems from directly in Postgres. There’s also now an official Postgres FDW which comes out of the box with Postgres and it by itself is quite useful when querying across various Postgres instances.

Much more

And if you missed the earlier editions of this, please feel free to check them out. The cliff notes of them include:

Window functions
Functions
Custom languages (PLV8 anyone?)
NoSQL datatypes
Custom functions
Common table expressions
Concurrent index creation
Transactional DDL
Foreign Data Wrappers
Conditional and functional indexes
Listen/Notify
Table inheritance
Per transaction synchronous replication

Deconstructing an image

infrequent oscillations — Sun, 23 Apr 2017 17:29:41 GMT

The iOS apps and processing steps used to make something moderately interesting from photos of a kitten and a spider.

Ignorance

Beetle Space — Sun, 23 Apr 2017 10:00:00 GMT

There are two types of people: Those who are uncomfortable with their own ignorance, and those who aim to understand.

The former will be quick to fill the void with an explanation. The latter will not stop there, but will seek to test the explanation.

Most people are of the …

Reciprocity and the Golden Rule

Beetle Space — Sat, 22 Apr 2017 10:00:00 GMT

Lately I’ve been reading the book Influence by Robert Cialdini. In it he talks about the principle of reciprocity. We tend to repay, in kind, what someone else has given us - even when we did not want it, and even when we did not take it.

If someone gives …

Book Review (2016)

Nemo's Home — Sun, 02 Apr 2017 03:00:00 GMT

I tweeted this a while back about my reading progress in 2016, and thought I’d do a post about what I read.

Made a graph of my @goodreads reading progress in 2016. I only picked up the pace sometime in May, but managed to read ~10k pages this year.

— Nemo (@captn3m0) January 13, 2017

Continuing with the review tradition from last year, here are the top 3 books that I read in 2016:

The Library at Mount Char [review]
Binti (Hugo winner for Best Novella, SF)
Lolita

Other books that I enjoyed were “Bands of Mourning”, the 3rd book in Mistborn Era 2, and the Powder Mage Trilogy which I found hard to put down.

I decided to aim for 36 books in 2016 (as well as 2017 now), and crossed that nicely. I picked 36 because it corresponds to 3 books a month or 1 book every 10 days, which makes for a goal that is easily tracked. I count everything that goodreads might count as a book (which is both good and bad), but I stick to it. Far more important for me would be the page count, which I charted at the end of the year and I read ~830 pages a month, which I’m pretty happy with.

I’m hoping to read more technical books in 2017, and have made some progress on that front with re-reading SICP.

If you are interested in the script that generated the graph, you can find it on github.

Why is this GoLang solution faster than the equivalent Java Solution?

Ben E. C. Boyter — Fri, 31 Mar 2017 01:52:01 GMT

At work there is a tradition of a Friday quiz being posted by the winner of the previous week. I missed out on the most recent one due to having to duck off early to do my tax but the problem was rather an interesting one.

The challange itself is not as simple as you would initally think and taken from a 2015 IBM Ponder This https://www.research.ibm.com/haifa/ponderthis/challenges/May2015.html

Vulnerability Report: ACT Corp

Nemo's Home — Sun, 26 Mar 2017 02:00:00 GMT

ACT, for those who don’t know is one of India’s most popular broadband providers.

This is a very brief and concise summary.

ACT has a mobile application
That allows you to login and check your plan details, data usage etc
I’ve been wanting to build a command line application that lets me check the balance easily
I tried scripting their website, but it was too much javascript.
The mobile app uses an API to do the same
The API happens to have really bad auth
Got fixed almost 3 months after reporting this.

Request

curl https://myfibernet.actcorp.in/api/user/plandetails -H "Content-Type: application/json" -H "Authtoken: 2aee21dfb1ef77707c30f48ccc513ad60b74d1fc6a84d60ecc32323ab5941469" -H "Apiversion: 1.0" -H "Appversion: 32" -H "Devicetype: 1" -H "Deviceid: 68590327e3e0ca81" -H "Mobilenumber: 9999999999" -H "Mid: 8973808103928d98703e65c0106b7a9d4001886234afbc2d7ce6415b75f9c216" --data '{"username":"11111111"}'

The API responds back with the following:

{
  "code": 200,
  "status": true,
  "message": "Success",
  "data": {
    "plan_details": {
      "agreement_info": {
        "agreement_no": "XXXXXXXXXX",
        "promotion_code": "",
        "package_code": "ACTESS01M",
        "package_name": "",
        "agreement_startdate": "DD/MM/YYYY",
        "expiry_date": "",
        "status": "",
        "entity_code": "[]",
        "subscription_period": "[]",
        "payterm": "[]",
        "billingcycle_code": "[]",
        "contract_type": "ISP",
        "outlets": "1",
        "service_points": "",
        "package_tenure": int,
        "due_date": ""
      },
      "product_info": {
        "product_code": "",
        "product_desc": ""
      }
    },
    "plan_usage_info": {
      "service_id": "ACTESS01M",
      "service_name": "ACTESS01M",
      "outbyteslimit": 322122547200,
      "outbytesremaining": 153400581140,
      "outbytesused": 168721966060
    },
    "bill_info": {
      "accountno": "111111112233",
      "subscribername": "NAME",
      "phonenumber": "PHONENUMBER",
      "address": {
        "line1": "YUP",
        "line2": "THESE TWO LINES WERE FILLED",
        "line3": "",
        "district": "AND THIS",
        "city": "BANGALORE",
        "state": "KARNATAKA",
        "country": "India"
      },
      "billno": "10000001111",
      "billdate": "DD/MM/2016",
      "account_period": "01/MM/2016-30/MM/2016",
      "previous_due": "",
      "current_invoice_amt": "1234",
      "total_due": "0",
      "bill_due_date": "15 Nov 2016"
    }
  }
}

Some of these are empty fields, and some values that I didn’t understand I’ve replaced with [].

The fun part is that the request is actually a POST and contains the following data:

{"username":"11111111"}

I happen to have friends who also use ACT. I asked around for usernames, and just by changing this one parameter in the request, I could access the complete details of almost everyone else.

Almost everyone, because for certain cases, I get a valid empty response. Valid because it has the same schema, but empty because all values are empty strings. Don’t know what that happens consistently only for certain accounts. (One of these was a Hyd account, the other in BLR).

If you are interested I’m working on a simple API that lets you access the ACT API to check the same details. It would ask you for an OTP the first time you login, and then cache the credentials to let you check the balance easily.

~~I’ve reported this to ACT as soon as I found it. Will disclose after I’ve given them some time!~~

Update: This was reported and fixed by ACT after I managed to find a contact via an investor (really!).

Timeline

Date	Details
29 Nov 2016	Vulnerability Identified
29 Nov 2016	Email sent to ACT, no response
6 Dec 2016	Email sent with partial customer details to explain scope of the issue, no response
8 Dec 2016	Reminder sent, no response
20 Jan 2017	Another reminder with a writeup sent. I also set a deadline of 29th January (2 months since first contact). Also got in touch with CERT-IN. No response
23 Jan 2017	Accidentally an investor in ACT saw my tweet and responded over twitter. Send a writeup, along with the suggestion to take down the application
24 Jan 2017	ACT reports issue is fixed. I test and report back as fixed the next day
26 Mar 2017	Report published

However, the huge timeline involved here pretty much guarantees that if you are an ACT customer, your data is out there in the public.

CCTC Challenge VM

Nemo's Home — Sat, 25 Mar 2017 02:00:00 GMT

This is specifically about the contest held in 2011, 6 years ago. I’ve written about my experience during the contest on this blog.

More specifically, Round 2 of the contest was a pentesting scenario where we were only provided with a VM image and asked to test it and report any vulnerabilities that we found.

I recently found the VirtualBox images, and thought I’d share them as a easy intro to web security.

Instructions

Reach out to me for the VM image
Hack.
Credentials are student:student (username:password)
Open /cctc in your browser.

Rules

Only application and its serving components can be tested for vulnerabilities. The serving components include
- Webserver
- Operating System
- Any other services/files in the guest machine and guest operating system
Any vulnerability identified in any component outside the above mentioned ones, will not be used for evaluation
All participants should necessarily submit all the exploit codes/custom scripts written to identify the vulnerabilities in the system.
The deadline for the original challenge was 2 weeks, but you’re free to take as much time as you want. Feel free to publish a list of vulnerabilities you find.

The attached spreadsheet provides format of the report, challenges in scope, and the details to be filled out for each vulnerability identified.

The tasks to be performed are mentioned in the report. Each task consists of the following sections:

Vulnerability/Vulnerabilities - You need to write the description of the vulnerability
Root Cause(s) - What is the root cause of the vulnerability?
Approach adopted (Steps with screenshots) – Write the steps followed to exploit the vulnerability along with screenshot of the final screen and/or intermediate steps.
Remediation with sample code snippet – Write the remediation steps to address this vulnerability. Also, write the sample code if applicable.

Also attached is a step by step installation guide for application set-up.

Few points to be considered:

Challenges can be attempted and completed in any order.
Only the application and its serving components can be tested for vulnerabilities. All other components like VMware, if tested for security issues, would lead to disqualification.
I will not be responsible for the discovery/notification of any zero day vulnerability in any software. If any zero-day vulnerability is identified, it is the responsibility of the concerned participant to notify the vulnerability to the respective vendor as per vendor’s policy.

If you are really interested, you can find a copy of the report we submitted at /reports/cctc.

Thanks to Harshil and Shobhit for working alongside on this.

Getting started with JSONB in Postgres

CRAIG KERSTIENS — Sun, 12 Mar 2017 22:55:56 GMT

JSONB is an awesome datatype in Postgres. I find myself using it on a weekly basis these days. Often in using some API (such as clearbit) I’ll get a JSON response back, instead of parsing that out into a table structure it’s really easy to throw it into a JSONB then query for various parts of it.

If you’re not familiar with JSONB, it’s a binary representation of JSON in your database. You can read a bit more about it vs. JSON here.

In working with JSONB here’s a few quick tips to get up and running with it even faster:

Indexing

For the most part you don’t have to think to much about this. With Postgres powerful indexing types you can add one index and have everything within the JSON document, all the keys and all the values, automatically indexed. The key here is to add a GIN index. Once this is done queries should be much faster where you’re searching for some value:

CREATE INDEX idx_data ON companies USING GIN (data);

Querying

Querying is a little bit more work, but once you get the basics it can be pretty straight forward. There’s a few new operators you’ll want to quickly ramp up on and from there querying becomes easy.

For the most basic part you now have an operator so traverse down the various keys. First let’s get some idea of what the JSON looks like so we can have something to work with. Here’s a sample set of data that we get back from Clearbit:

{
"domain": "citusdata.com",
"company": {
"id": "b1ff2bdf-0d8d-4d6d-8bcc-313f6d45996a",
"url": "http:\/\/citusdata.com",
"logo": "https:\/\/logo.clearbit.com\/citusdata.com",
"name": "Citus Data",
"site": {
"h1": null,
"url": "http:\/\/citusdata.com",
"title": "Citus Data",
},
"tags": [
"SAAS",
"Enterprise",
"B2B",
"Information Technology & Services",
"Technology",
"Software"
],
"domain": "citusdata.com",
"twitter": {
"id": "304455171",
"bio": "Builders of Citus, the extremely scalable PostgreSQL database.",
"site": "https:\/\/t.co\/hKpZjIy7Ej",
"avatar": "https:\/\/pbs.twimg.com\/profile_images\/630900468995108865\/GJFCCXrv_normal.png",
"handle": "citusdata",
"location": "San Francisco, CA",
"followers": 3770,
"following": 570
},
"category": {
"sector": "Information Technology",
"industry": "Internet Software & Services",
"subIndustry": "Internet Software & Services",
"industryGroup": "Software & Services"
},
"emailProvider": false
}
}

Sorry it’s a bit long, but it gives us a good example to work with.

Basic lookups

Now let’s query something fairly basic, the domain:

# SELECT data->'domain'
FROM companies
WHERE domain='citusdata.com'
LIMIT 1;

 ?column?
-----------------
 "citusdata.com"

The -> is likely the first operator you’ll use in JSONB. It’s helpful to traverse the JSON. Though of you’re looking to get the value as text you’ll actually want to use ->>. Instead of giving you some quoted response back or JSON object you’re going to get it as text which will be a bit cleaner:

# SELECT data->>'domain'
FROM companies
WHERE domain='citusdata.com'
LIMIT 1;

 ?column?
-----------------
 citusdata.com

Filtering for values

Now with something like clearbit you may want to filter out for only certain type of companies. We can see in the example data that there’s a bunch of tags. If we wanted to find only companies that had the tag B2B we could use the ? operator once we’ve targetted down to that part of the JSON. The ? operator will tell us if some part of JSON has a top level key:

SELECT *
FROM companies
WHERE data->'company'->'tags' ? 'B2B'

JSONB but pretty

In querying JSONB you’ll typically get a nice compressed set of JSON back. While this is all fine if you’re putting it into your application, if you’re manually debugging and testing things you probably want something a bit more readable. Of course Postgres has your back here and you can wrap your JSONB with a pretty print function:

SELECT jsonb_pretty(data)
FROM companies;

Much more

There’s a lot more in the docs that you can find handy for the specialized cases when you need them. jsonb_each will expand a JSONB document into individual rows. So if you wanted to count the number of occurences of every tag for a company, this would help. Want to parse out a JSONB to a row/record in Postgres there’s jsonb_to_record. The docs are your friends for about everything you want to do but hopefully these few steps help kick start things if you want to get started with JSONB.

Deploying FreeBSD on Linode unattended in minutes

Notes on software development — Sat, 11 Mar 2017 02:00:00 GMT

I became a FreeBSD user over 2 years ago when I wanted to see what all the fuss was about. I swapped my y410p dual-booting Windows / Ubuntu with FreeBSD running Gnome 3. I learned a lot during the transition and came to appreciate FreeBSD as a user. I soon began running FreeBSD as my OS of choice on cloud servers I managed. So naturally, when I started working at Linode a year ago I wanted to run FreeBSD servers on Linode too.

Linode is a great platform for running random unofficial images because you have much control over the configuration. I followed existing guides closely and was soon able to get a number of operating systems running on Linodes by installing them manually: FreeBSD, OpenBSD, NetBSD, Minix3, and SmartOS to date.

Unofficial images come at a cost though. In particular, I became frustrated having to reinstall using the installer every time I managed to trash the disk. So over the past year, I spent time trying to understand the automated installation processes across different operating systems and Linux distributions.

Unattended installations are tough. The methods for doing them differ wildly. On RedHat, Fedora, and CentOS there is Kickstart. On Debian and Ubuntu there is preseeding. Gentoo, Arch, and FreeBSD don't particularly have a framework for unattended installs, but the entire installation process is well-documented and inherently scriptable (if you put in the effort). OpenBSD has autoinstall. Trying to understand each and every one of these potential installation methods was pretty defeating for getting started on a side-project.

A few weeks ago, I finally had the silly revelation that I didn't need to script the installation process -- at least initially. I only had to have working images available somewhere that could be copied to new Linodes. Some OSs / distributions may provide these images, but there is no guarantee that they exist or work. If I tested and hosted them for Linodes, anyone could easily run their own copy.

I began by running the installation process as normal for FreeBSD. After the disk had FreeBSD installed on it, I rebooted into Finnix, made a compressed disk image, and transferred it to an "image host" (another Linode in Fremont running an FTP server). Then I tested the reversal process manually to make sure a new Linode could grab the image, dd it to a disk, reboot and have a working filesystem and networking. (This transfer occurs over private networking to reduce bandwidth costs and thus limits Linode creation to the datacenter of the image host, Fremont.)

Then it was time to script the process. I looked into the existing Linode API client wrappers and noticed none of them were documented. So I took a day to write and document a good part of a new Linode Python client.

I got to work and out came the linode-deploy-experimental script. To run this script, you'll need an API token. This script will allow you to deploy from the hosted images (which now include FreeBSD 11.0 and OpenBSD 6.0). Follow the example line in the git repo and you'll have a Linode running OpenBSD or FreeBSD in minutes.

Clearly there's a lot of work to do on both this script and on the images:

Fremont datacenter has the only image host.
The script does not change the default password: "password123". You'll want to change this immediately.
The script does not automatically grow the file system after install.
The TTY config for these images currently requires you to use Glish instead of Weblish.
And more.

Even if many of these issues do get sorted out (I assume they will), keep in mind that these are unofficial, unsupported images. Some things will probably never work: backups, password reset, etc. If you need help, you are probably limited to community support. You can also find me with any questions (peaton on OFTC). But for me this is at least a slight improvement on having to run through the install process every time I need a new FreeBSD Linode.

Deploy FreeBSD and OpenBSD unattended on Linode https://t.co/j5A46ROqNM https://t.co/HSqrIvBMFj
— Phil Eaton (@phil_eaton) March 12, 2017

Go Time

Linux Engineer's random thoughts — Sun, 19 Feb 2017 01:00:00 GMT

For the Go 1.8 Release Party in Paris I gave a lightning talk on monotonic clocks. It's essentially a talk version of the Russ Cox's design document on monotonic clocks, which is really well written and sourced. You should go read it !

Here are the slides export and their …

Setup up ConcourseCI 2.6.0 behind Nginx with Self Signed Certificates on Ubuntu 16.04

Ben E. C. Boyter — Sun, 12 Feb 2017 23:56:58 GMT

[Concourse CI][1] is a very nice continuous integration server.

However for installs there are a few gotcha’s you need to keep in mind. Mostly these relate to how TLS/SSL works.

The first is that while it is possible to run concourse inside Docker I found this to cause a lot of issues with workers dying and not recovering. I would suggest installing the binarys on bare machines. When I moved from a docker cluser using Amazon’s ECS to a single t2.large instance not only were builds faster but it was a far more reliable solution.

Checkbot for Chrome: Public beta

Fri, 10 Feb 2017 02:00:00 GMT

Today I’m releasing the first public beta release of my new project: Checkbot for Chrome! Checkbot is a Chrome extension that crawls the pages on your site to suggest SEO, web speed and web security improvements. Checkbot aims to be a comprehensive website testing tool that gives you everything you need to identify, understand and fix website issues. Checkbot guides you in following web best practices by helping you fix problems such as broken links, duplicate page titles, insecure login forms, redirect chains and much more.

Comparing Elixir and Go

Brightball Articles — Tue, 07 Feb 2017 11:52:51 GMT

Elixir and Go have both grown significantly in popularity over the past few years, and both are often reached for by developers looking for high concurrency solutions. The two languages follow many similar principles, but both have made some core tradeoffs that affect their potential use cases. Let’s compare the two by taking a look at their backgrounds, their programming styles, and how they deal with concurrency.

Repository overview now in searchcode server

Ben E. C. Boyter — Mon, 30 Jan 2017 09:34:35 GMT

One feature that I have wanted for a long time in searchcode server was a page which would give an overview of a repository. I wanted the overview to give a very high look at the languages used, the total number of files, estimated cost and who would be the best people to talk to.

One thing that occurred to me when I started work was that it would be nice to calculate a bus factor for the repository as well. After all we all know that project managers do like to know who are the most critical contributors to any project and what the risk is.

Simple but handy Postgres features

CRAIG KERSTIENS — Sun, 08 Jan 2017 22:55:56 GMT

It seems each week when I’m reviewing data with someone a feature comes up that they had no idea existed within Postgres. In an effort to continue documenting many of the features and functionality that are useful, here’s a list of just a few that you may find handy the next time your working with your data.

Psql, and \e

This one I’ve covered before, but it’s worth restating. Psql is a great editor that already comes with Postgres. If you’re comfortable on the CLI you should consider giving it a try. You can even setup you’re own .psqlrc for it so that it’s well customized to your liking. In particular turning \timing on is especially useful. But even with all sorts of customization if you’re not aware that you can use your preferred editor by using \e then you’re missing out. This will allow you to open up the last run query, edit it, save–and then it’ll run for you. Vim, Emacs, even Sublime text works just take your pick by setting your $EDITOR variable.

Watch

Ever sit at a terminal running a query over and over to see if something on your system changed? If you’re debugging something whether locally or even live in production, watching data change can be key to figuring out. Instead of re-running your query you could simply use the \watch command in Postgres, this will re-run your query automatically every few seconds.

SELECT now() -
 query_start,
 state,
 query
FROM pg_stat_activity
\watch

JSONB pretty print

I love JSONB as a datatype. Yes, in cases it won’t be the optimal for performance (though at times it can be perfectly fine). If I’m hitting some API that returns a ton of data, I’m usually not using all of it right away. But, you never know when you’ll want to use the rest of it. I use Clearbit this way today, and for safety sake I save all the JSON result instead of de-normalizing it. Unfortunately, when you query this in Postgres you get one giant compressed text of JSON. Yes, you could pipe out to something like jq, or you could simply use Postgres built in function to make it legible:

SELECT jsonb_pretty(clearbit_response)
FROM lookup_data;

 jsonb_pretty
-------------------------------------------------------------------------------
 {
 "person": {
 "id": "063f6192-935b-4f31-af6b-b24f63287a60",
 "bio": null,
 "geo": {
 "lat": 37.7749295,
 "lng": -122.4194155,
 "city": "San Francisco",
 "state": "California",
 "country": "United States",
 "stateCode": "CA",
 "countryCode": "US"
 },
 "name": {
 ...

Importing my data into Google

This one isn’t Postgres specific, but I use it on a weekly basis and it’s key for us at Citus. If you use something like Heroku Postgres, dataclips is an extremely handy feature that lets you have a real-time view of a query and the results of it, including an anonymous URL you can it for it. At Citus much like we did at Heroku Postgres we have a dashboard in google sheets which pulls in this data in real-time. To do this simple select a cell then put in: =importdata("pathtoyourdataclip.csv"). Google will import any data using this as long as it’s in CSV form. It’s a great lightweight way to build out a dashboard for your business without rolling your own complicated dashboarding or building out a complex ETL pipeline.

I’m sure I’m missing a ton of the smaller features that you use on a daily basis. Let me know @craigkerstiens the ones I forgot that you feel should be listed.

A metaphor

infrequent oscillations — Sat, 07 Jan 2017 17:21:55 GMT

In which we dip our toe into politics.

Looking back on 2016, and forward to 2017

Ben Overmyer's Site — Sun, 01 Jan 2017 02:00:00 GMT

2016 was a rough year. We lost a lot of good people; the loss of Carrie Fisher in particular hit me hard. Politically, the year was disappointing. But 2016 was also a good year for me personally. Doors and eyes opened. 2017 should continue that trend, if I read the signs correctly.

Victories in 2016

I returned to Silver Gryphon Games. While this has not borne fruit yet, it marked a commitment that's important to me. I lost a lot of weight in the early part of 2016. I followed a paleo diet, practiced parkour, and started to see increased energy levels and a better self image. I learned what it meant to be a manager, but not until later in the year. For the first part of the year, I was a “manager” but didn't fully understand the difference between a technical lead and a manager. In November, it finally clicked: managers focus on the big picture. Technical leads bridge the gap between big picture and technical execution. That realization gave me new energy and a powerful sense of purpose.

Challenges in 2016

One of the biggest stressors for me in 2016 was personal finances. I moved out of my old apartment earlier than I should have. For two months, I was paying for two apartments. I misjudged my cash flow and missed payments on some things for the first time in over five years. Weight and diet were also challenges. While in the first quarter of the year I lost over thirty pounds, the last quarter of the year saw a slight reversal. I gained back ten of those pounds. My diet changed from paleo-ish to unrestricted. My drinking became even more of a problem in the latter half of the year. For the first time in my life, I had a couple days where I couldn't remember large chunks of the previous night. It feels like my memory generally has gotten worse, though I don't have data to support that, or even to suggest that the two are connected.

Goals for the New Year

I'm going to regain my budgetary sense from early 2016 and put significant effort towards paying down debt. This is my primary goal in 2017. I will return to a controlled diet. I'm cutting back on eating out. I love cooking, so this isn't that difficult for me. It's going to require paying attention, though. The big difficulty here is going to be work lunches, since I don't care for sandwiches or microwaved leftovers. Getting my drinking under control is going to be both simple and difficult. In 2016, I drank most often when at home playing video games. Since those two habits are so intertwined, I will need to find a way to sever the connection. In general, I believe I'll be able to succeed at my goals in 2017. I have great hope for the future.

Hacker News Personal Blogs 2017 | Top 100 Blogs

Migrating from Google (and more)

Contents

Email

Google Play Music

Google Keep

Phone

microG Core

UnifiedLP

Maps

Uber

Calendar/Contacts

Google Play Store

LastPass

GitHub

Sourcing developer marketing content

Turn emails into blogs

Beginner mindset with tickets

Make sales engineering obsolete

Training and Documentation don’t have to be siloed

The process

What’d I miss?

Top album picks for 2017

Father John Misty - Pure Comedy

HP 4951C Protocol Analyser

Guidance on performing retrospectives

Capture details immediately

Running the retrospective (the meeting part)

The important part

Sharing the details

Books I read in 2017

Man's Search for Meaning

And Then There Were None

Simply encrypt or decrypt a string using Boto3 Python and AWS KMS

Learnings from building my own home server

Learnings

Advice

MongoDB with forceful server restarts

SSHD on specific Interface

Wiki.js public pages

YAML config

Guest Access

Docker CA Cert Authentication

OpenVPN default gateway client side configuration

Server Side

Client Side

Should I buy Bitcoin?

searchcode plexus

Plexus “A combination of interlaced parts; a network.”

Flutter for React Native Devs in 30 Seconds

Serverless Machine Learning at Google

Are we human? Or are we reCAPTCHA?

Firebase Analytics in 30 Seconds

Medical Machine Learning in 30 Seconds

Adding Blog Here

Ready Player One

Six Easy Pieces

The Brothers Karamazov

How to root Nexus 7 (2012) and flash it to LineageOS, from Mac OSX

Talk - Clustered, Distributed File and Volume Storage with GlusterFS

Running terraform and docker on my home server

Learnings

TODO

Postgres - the non-code bits

PostgreSQL License

The people

The core team

Contributors

PostgreSQL the company

Engaging

Born to Run

Applying syntax in Sublime based on the first file line

How to build waifu2x command-line version on osx

BONUS: Qt GUI

Install and configure Qt and Qt-Creator

Compile waifu2x-converter-qt

Static Analysis: It's Simply Critical

Born a Crime

BBQ with a Dutch ICT/start up Delegation

The Crusades: The Authoritative History of the War for the Holy Land

Using Linux `perf` at Netflix