Hacker News Personal Blogs 2015 | Top 100 Blogs

My top 10 Postgres features and tips for 2016

CRAIG KERSTIENS — Tue, 29 Dec 2015 22:55:56 GMT

I find during the holiday season many pick up new books, learn a new language, or brush up on some other skill in general. Here’s my contribution to hopefully giving you a few new things to learn about Postgres and ideally utilize in the new year. It’s not in a top 10 list as much as 10 tips and tricks you should be aware of as when you need them they become incredibly handy. But, first a shameless plug if you find any of the following helpful, consider subscribing to Postgres weekly a weekly newsletter with interesting Postgres content.

1. CTEs - Common Table Expressions

CTEs allow you to do crazy awesome things like recursive queries but even the most simple form of them I don’t go a day without using. Think of a CTE or commonly known as with clause as a view inside the time that query is running. This lets you more easily create readable query. Any query that’s constructed that’s even 100 lines long, but with 4-5 CTEs is undoubtedly going to be easier for someone new to come in and understand than a 20 line query that does the same thing. A few people like writing SQL, but no one likes reading someone else’s so do them a favor and read up on CTEs.

2. Setup a .psqlrc

You setup a bashrc, vimrc, etc. Why not do the same for Postgres. Some of the great things you can do:

Setup pretty formatting by default with \x auto
Set nulls to actually look like something \pset null ¤
Turn timing on by default \timing on
Customize your prompt \set PROMPT1 '%[%033[33;1m%]%x%[%033[0m%]%[%033[1m%]%/%[%033[0m%]%R%# '
Save commonly run queries that you can run by name

Here’s an example of my own psqlrc:

\set QUIET 1
\pset null '¤'
-- Customize prompts
\set PROMPT1 '%[%033[1m%][%/] # '
\set PROMPT2 '... # '
-- Show how long each query takes to execute
\timing
-- Use best available output format
\x auto
\set VERBOSITY verbose
\set HISTFILE ~/.psql_history- :DBNAME
\set HISTCONTROL ignoredups
\set COMP_KEYWORD_CASE upper
\unset QUIET

3. pg_stat_statements for where to index

pg_stat_statements is probably the single most valuable tool for improving performance on your database. Once enabled (with create extension pg_stat_statements) it automatically records all queries run against your database and records often and how long they took. This allows you to then go and find areas you can optimize to get overall time back with one simple query:

SELECT
(total_time / 1000 / 60) as total_minutes,
(total_time/calls) as average_time,
query
FROM pg_stat_statements
ORDER BY 1 DESC
LIMIT 100;

Yes, there is some performance cost to leaving this always on, but it’s pretty small. I’ve found it’s far more useful to be on and get major performance wins vs. the small cost of it always recording.

You can read much more on Postgres performance on a previous post

4. Slow down with ETL, use FDWs

If you have a lot of microservices or different apps then you likely have a lot of different databases backing them. The default for about anything you want to do is do create some data warehouse and ETL it all together. This often goes a bit too far to the extreme of aggregating everything together.

For the times you just need to pull something together once or on rare occasion foreign data wrappers will let you query from one Postgres database to another, or potentially from Postgres to anything else such as Mongo or Redis.

5. array and array_agg

There’s little chance if you’re building an app you’re not using arrays somewhere within it. There’s no reason you shouldn’t be doing the same within your database as well. Arrays can be just another datatype within Postgres and have some great use cases like tags for blog posts directly in a single column.

But, even if you’re not using arrays as a datatype there’s often a time when you want to rollup something like an array in a query then comma separate it. Something similar to the following could allow you to easily roll up a comma separated list of projects per user:

SELECT
users.email,
array_to_string(array_agg(projects.name), ',')) as projects
FROM
projects,
tasks,
users
WHERE projects.id = tasks.project_id
AND tasks.due_at > tasks.completed_at
AND tasks.due_at > now()
AND users.id = projects.user_id
GROUP BY
users.email

6. Use materialized views cautiously

If you’re not familiar with materialized view they’re a query that has been actually created as a table. So it’s a materialized or basically snapshotted version of some query or “view”. In their initial version materialized versions, which were long requested in Postgres, were entirely unusuable because when you it was a locking transaction which could hold up other reads and acticities avainst that view.

They’ve since gotten much better, but there’s no tooling for refreshing them out of the box. This means you have to setup some scheduler job or cron job to regularly refresh your materialized views. If you’re building some reporting or BI app you may undoubtedly need them, but their usability could still be advanced so that Postgres knew how to more automatically refresh them.

If you’re on Postgres 9.3, the above caveats about preventing reads still does exist

7. Window functions

Window functions are perhaps still one of the more complex things of SQL to understand. In short they let you order the results of a query, then compute something from one row to the next, something generally hard to do without procedural SQL. You can do some very basic things with them such as rank where each result appears ordered by some value, or something more complex like compute MoM growth directly in SQL.

8. A simpler method for pivot tables

Table_func is often referenced as the way to compute a pivot table in Postgres. Sadly though it’s pretty difficult to use, and the more basic method would be to just do it with raw SQL. This will get much better with Postgres 9.5, but until then something where you sum up each condition where it’s true or false and then totals is much simpler to reason about:

select date,
sum(case when type = 'OSX' then val end) as osx,
sum(case when type = 'Windows' then val end) as windows,
sum(case when type = 'Linux' then val end) as linux
from daily_visits_per_os
group by date
order by date
limit 4;

Example query courtesy of Dimitri Fontaine and his blog.

9. PostGIS

Sadly on this one I’m far from an expert. PostGIS is arguably the best option of any GIS database options. The fact that you get all of the standard Postgres benefits with it makes it even more powerful–a great example of this is GiST indexes which came to Postgres in recent years and offers great performance gains for PostGIS.

If you’re doing something with geospatial data and need something more than the easy to use earth_distance extension then crack open PostGIS.

10. JSONB

I almost debated leaving this one off the list, ever since Postgres 9.2 JSON has been at least one of the marquees in each Postgres release. JSON arrived with much hype, and JSONB fulfilled on the initial hype of Postgres starting to truly compete as a document database. JSONB only continues to become more powerful with better libraries for taking advantage of it, and it’s functions improving with each release.

If you’re doing anything with JSON or playing with another document database and ignoring JSONB you’re missing out, of course don’t forget the GIN and GiST indexes to really get the benefits of it.

The year ahead

Postgres 9.5/9.6 should continue to improve and bring many new features in the years ahead, what’s your preference for something that doesn’t exist yet but you do want to see land in Postgres. Let me know @craigkerstiens

searchcode server

Ben E. C. Boyter — Tue, 29 Dec 2015 04:17:42 GMT

A month or so ago I started collection emails on searchcode.com to determine if there was enough interest in a downloadable version of searchcode. The results were overwhelmingly positive. The email list grew far beyond what I would have expected, and this was in the first month. As such I have been working in this downloadable version of searchcode which will probably be called searchcode server.

Progress has been reasonably straight forward consider that searchcode.com is written using mostly Python and searchcode server is mostly Java. The main reason for choosing Java is that I really wanted searchcode server to be a self contained application which could be downloaded and run without the configuration and setup of additional services.

Kettle, my personal dashboard app

Ben Overmyer's Site — Tue, 29 Dec 2015 02:00:00 GMT

Not too long ago, I got fed up with having to use a ton of different apps to do various things online. Also, having to deal with apps that almost-but-not-quite fit the bill for what I was trying to do was a bit frustrating. So, I set out to write my own personal dashboard web app to handle… well, everything. I'm calling it Kettle, mostly because that was the first noun that popped into my head that sounded vaguely appropriate. Does it make sense? Not really. It just sounds right, much like J.R.R. Tolkien loved the sound of the phrase “cellar door.” At the time of this posting, I've just finished the first iteration of the note-taking part of the app. It doesn't yet allow searching or deleting notes, but other than that, it's perfectly usable. You can check it out at the Kettle repository on GitHub.

App Use: Chordion

infrequent oscillations — Mon, 28 Dec 2015 16:34:20 GMT

Chordion is a combination arpeggiator / drum sequencer that supposedly “is a fun new way to make music on the iPad.” I think I’ve proven their “never hit a wrong note!” wrong. Maybe a few more practice runs will help. Listen on Soundcloud

Postgres 9.5 - The feature rundown

CRAIG KERSTIENS — Sun, 27 Dec 2015 22:55:56 GMT

The headline of Postgres 9.5 is undoubtedly: Insert… on conflict do nothing/update or more commonly known as Upsert or Merge. This removes one of the last remaining features which other databases had over Postgres. Sure we’ll take a look at it, but first let’s browse through some of the other features you can look forward to when Postgres 9.5 lands:

Grouping sets, cube, rollup

Pivoting in Postgres has sort of been possible as has rolling up data, but it required you to know what those values and what you were projecting to, to be known. With the new functionality to allow you to group various sets together rollups as you’d normally expect to do in something like Excel become trivial.

So now instead you simply add the grouping type just as you would on a normal group by:

SELECT department, role, gender, count(*)
FROM employees
GROUP BY your_grouping_type_here;

By simply selecting the type of rollup you want to do Postgres will do the hard work for you. Let’s take a look at the given example of department, role, gender:

grouping sets will project out the count for each specific key. As a result you’d get each department key, with other keys as null, and the count for each that met that department.
cube will give you the same values as above, but also the rollups of every individual combination. So in addition to the total for each department, you’d get breakups by the department and gender, and department and role, and department and role and gender.
rollup will give you a slightly similar version to cube but only give you the detailed groupings in the order they’re presented. So if you specified roll (department, role, gender) you’d have no rollup for department and gender alone.

Check the what’s new wiki for a bit more clarity on examples and output

Import foreign schemas

I only use foreign tables about once a month, but when I do use them they’ve inevitably saved many hours of creating a one off ETL process. Even still the effort to setup new foreign tables has shown a bit of their infancy in Postgres. Now once you’ve setup your foreign database, you can import the schema, either all of it or specific tables you prefer.

It’s as simple as:

IMPORT FOREIGN SCHEMA public
FROM SERVER some_other_db INTO reference_to_other_db;

pg_rewind

If you’re managing your own Postgres instance for some reason and running HA, pg_rewind could become especially handy. Typically to spin up replication you have to first download the physical, also known as base, backup. Then you have to replay the Write-Ahead-Log or WAL–so it’s up to date then you actually flip on replication.

Typically with databases when you fail over you shoot the other node in the head or STONITH. This means just get rid of it, completely throw it out. This is still a good practice, so bring it offline, make it inactive, but from there now you could then flip it into a mode and use pg_rewind. This could save you pulling down lots and lots of data to get a replica back up once you have failed over.

Upsert

Upsert of course will be the highlight of Postgres 9.5. I already talked about it some when it initially landed. The short of it is, if you’re inserting a record and there’s a conflict, you can choose to:

Do nothing
Do some form of update

Essentially this will let you have the typically experience of create or update that most frameworks provide but without a potential race condition of incorrect data.

JSONB pretty

There’s a few updates to JSONB. The one I’m most excited about is making JSONB output in psql read much more legibly.

If you’ve got a JSONB field just give it a try with:

SELECT jsonb_pretty(jsonb_column)
FROM foo;

Give it a try

Just in time for the new year the RC is ready and you can get hands on with it. Give it a try, and if there’s more you’d like to hear about Postgres please feel free to drop me a note craig.kerstiens@gmail.com.

Going from blog posts to full launches

CRAIG KERSTIENS — Sat, 26 Dec 2015 22:55:56 GMT

I recall extremely early stage where you’d build a feature, realize it was awesome, then the next day write a blog post for it. At some point you start to move from that to more coordinated launches. A larger coordinated launch allows you to reach a bigger audience, can lead to bigger deals, and help expand your overall market. But perhaps more importantly by the time you hit full launch you’ve message tested and ensured it’s going to resonate in the way you expect.

The process itself will both help amplify and validate/refine your message

This is often a more gradual process than a sudden single change, you’ll introduce new parts of this in time. And for many what an entire launch process looks like comes by trial an error, to help shorten that learning curve here’s key areas I pay attention for a launch and process followed by a rough timeline.

Product first

Making sure the product is in the right shape is key to any big launch. You don’t get a second shot and if the product isn’t in shape customers often won’t take a second look at it later. For this reason I strongly prefer to have your product locked and loaded before you even start talking launch times, or at least be in the bug clean up phase. This means you’ve built a feature, validated with alpha users or private beta, and are ready to open it up to the world.

If you have to set a launch date without the product or feature being already done allow padding. Sometimes it’s good for the team to know the padding, sometimes it isn’t. When you have extra time it’s not uncommon for your development to magically consume exactly that amount of time and still result in a small scramble towards the end.

A good driver I’ve found is needing to have it fully like to demo a few weeks out from the launch itself, such as during analyst pre-briefings.

Crafting your message

Every launch is an opportunity to tell your core message and value prop. If you miss this opportunity for focusing on a single narrow feature you’ve missed the biggest opportunity you had in a launch. You can’t relaunch your full product every time though–you do need some big improvements or feature that you can highlight, but you should still hit your core message.

First you should know that your feature solves some specific problem, you should know this from the alpha/beta testing and if it doesn’t solve this problem you’re not ready to launch. Yes, some people will launch a product before the product is completely there–this is common in a marketing driven company as opposed to a product/engineering driven company.

Your message should lead with the problem you’re solving, not the laundry list of features. The best launches lead with some broader thematic message, even better if it’s an altruistic world changing one. A rough example of this:

To the point to the product, and probably over generalized as boring:

Connectify brings a new way of taking your dumb devices at home and turning them into intelligent connected devices.

In contrast, broad thematic message, followed lightly by the product.

We live in a connected world, and with new connected devices there’s the opportunity not just give you more data but help you improve how you live your life. Connectify helps you at bringing the devices that matter together with ease.

Testing your message

You should treat your message just like a product, testing it gradually along the way. Once you’ve got some initial framing of it, test it internally, then with friendly customers or community members. Leading up to the launch I usually have a timeline and get all the content and communication rolling about 3 weeks out. I’ll give a bit of a timeline below but first some more around message testing

Analysts

If you regularly use any analysts you should absolutely use them to help with a launch. Several weeks out is a great time to test key messages with them, get feedback, and if you’re lucky you may even get them to provide a quote for the launch.

Keep in mind here a inquiry is an opportunity to test your message and get feedback. You should talk roughly half of the time here, they should be talking the other half. In contrast briefings before launch you should have your message fully baked and should simply be pitching your message and possibly demo-ing.

Friendlies

This may be more contentious, but at least at early stage sharing drafts with friendly community members is a great way to get feedback and refine your message. Here you should be especially concious of the request of their time and expect to have some delay before they get back to you. Being top secret about your message ahead of time won’t add much value to it being a home run, where as better ensuring it resonates will help it to be more successful.

Customers

Customers I call out as a separate bucket. Customers have less incentive to leak your news than friendlies, but also fall somewhere on the other spectrum of analysts. A key piece about customers is there is an opportunity for them to be a launch partner. And so on to that topic:

Launch partners

Press and others like seeing and knowing you have external validation. Similarly many see the benefit of being part of a launch, after all it’s more free press for them. For a launch partner there are various levels, though for most providing some quote is a pretty common level.

The best way to do this is talk to them about what they like about the feature/product and take a first stab at the quote for them from their feedback. Some may very much want to wordsmith their own which is fine, but minimizing the work required of them while–trying to hit something they’d say as well–as a message that flows can best be done by you taking a first pass.

Further there’s varying levels of value with quotes and references. In descending order:

Customers
Analysts
Community Members

The other details

During launch week I mostly want to be dotting i’s and crossing t’s, meaning: I want the product done. I want documentation done. I want the blog post finalized. I want to be in the mode of send internal announcements, prep internal teams, talk to media.

Prepping internal teams

Obviously the engineering and product people involved will be in the loop. But you need to notify many others some of which should have been in the loop already, some less so:

Support - There’s a new product surface area, support should be top of your list so they can field the tickets and questions that come in
Sales - Even if there is no price change or impact, new features allow sales to communicate value to customers

Timeline

Finally what’s the end to end timeline look like with all the little details. Here’s a rough one that’s fully built out. IF you’re smaller and don’t have a regular cadence of analysts in hand then just expect that doesn’t apply. IF your support team is the product and engineers maybe that’s lighter weight. Basically feel free to take out parts, but expect your process to grow to something of this size.

4 weeks out - Outline of blog post with key messages
Test that outline internally
3 weeks out - Start to get a rough draft in place
3 weeks out - Share internally and with friendlies. At this point you’re explicitely looking for message feedback. Tell people to not waste time on nitpicks of words or grammar, it will be 98% re-written by the time you’re done
2.5 weeks out - Analyst inquiries for message testing
2.5 weeks out - Start putting together product demo
2 weeks out - Start putting together documentation
2 weeks out - Start nailing down blog post for final messages
1 weeks out - Start to put final touches on blog post for grammar
1 week out - Analyst briefings
1 week out - Update support
3-5 days out - Stage blog post
3-5 days out - Stage new documentation
2-4 days out - Make sure PRs are ready or feature flags, in short the switch is there or live but not public
1-3 days out - Update sales
1-3 days out - Interall communication to all@
1-3 days out - Media briefings
LAUNCH DAY - Sit in a room and watch all the things, engage with twitter/HN/etc.

App Use: Keezy Drummer

infrequent oscillations — Sat, 26 Dec 2015 17:47:08 GMT

A quick noodle with Keezy Drummer recorded into the Zoom H4n, exported to Fission, then piped through Audio Hijack with the right channel duplicated into the left before saving to MP3. Listen on Soundcloud

Janathon, Dryathlon, Jamathon

infrequent oscillations — Sat, 26 Dec 2015 16:04:49 GMT

And a partridge in a pear tree. Signed up for Janathon (updates will probably be here). Signed up for Dryathlon (Sponsor me!). Decided to do an ad-hoc Jamathon (“Make and upload some musical output every day using one or two mobile apps”). (Yes, I’m going to be insufferable all month.)

App Use: Audioshare

infrequent oscillations — Sat, 26 Dec 2015 13:36:41 GMT

Not strictly a music-making app, [Audioshare] gets used for almost all of the things I upload to Soundcloud. But since I managed to accidentally make some “music” with it, I might as well blog about it (and steal the description from Soundcloud). There I was, playing something I’d just made in Soundscaper through Audioshare via Audiobus into Gliderverb when I had the urge to record what was happening since it was sounding ok.

App Use: TC-Performer

infrequent oscillations — Sat, 26 Dec 2015 13:33:27 GMT

Like a 2D theremin on acid, TC-Performer has a bunch of instruments (all digitally synthed by TC-11 - which is on the wishlist) controlled by multi-touch, movement, etc. Bit tricky to do coherent tunes but excellent for making this kind of swoopy droney chimey kind of space music. Listen on Soundcloud

App Use: Poly

infrequent oscillations — Sat, 26 Dec 2015 13:29:35 GMT

Some learning-curve twiddling with Poly. Should be good for generating driving rhythms as underlying texture. Listen on Soundcloud

App Use: Fieldscaper

infrequent oscillations — Sat, 26 Dec 2015 13:25:31 GMT

Recorded 1:45 of tap filling a bath using Fieldscaper. Chopped out a decent 45s segment and mangled it three different ways to make a kind of weird Dr Who 1970s sci-fi noisefest. Listen on Soundcloud

Insomniac Art

infrequent oscillations — Sat, 26 Dec 2015 13:06:09 GMT

Or “half an hour spent twiddling with various graphics apps on an iPhone whilst unable to sleep”. (Lots of 512x512 images make this a big post.) The original image Southwark Park, 2015-12-19, near the south-east gate, looking south-west. Waterlogue “Vibrant, medium, Auto”. BLACK “P400, Tura expired 2002” BLACK plus Waterlogue Combination of the last two. Decim8 These were all done via the random settings button. Etchings plus Union plus Pixlr Etchings was used to create a few variants of the BLACK version which where then combined in Union before being tweaked in Pixlr.

Pi-Hole for Ubuntu 14.04

Ben E. C. Boyter — Sat, 26 Dec 2015 04:24:32 GMT

Because of the fact that I personally work for an ad supported company and that searchcode.com is currently supported via third party advertising I tend to keep an eye on the state of ad blockers on the web.

Most people probably know about adblockplus and other browser extensions however there are other ways to block ad’s on ones network. One that I had previously read about was setting up your own Bind9 server on a server and adding custom rules to block them at a DNS level. Other the last week I had been playing around with this but since I am not a bind expert I was unable to get it working in a satisfactory way.

Shine a Spotlight On Those Photos

Beetle Space — Thu, 24 Dec 2015 10:00:00 GMT

A nearby photography club has annual county wide competitions. The judges select their favorite photos and display them in a mall for a month.

Several years ago, I took a friend to the mall to show him the photos. We had only a few minutes until closing time, and they …

App Use: Soundscaper

infrequent oscillations — Wed, 23 Dec 2015 11:30:52 GMT

The samples from Disquiet Junto 0207 soundscaped using the excellent SoundScaper (mostly) controlled by a KORG nanoKONTROL2. Listen on Soundcloud

App Use: Phase Rings, Dedalus

infrequent oscillations — Wed, 23 Dec 2015 11:25:31 GMT

A simple bit of noodling in Phase Rings (I think it was a Mixolydian scale but I can’t remember now - Always Takes Notes At The Time, Kids) through Dedalus for some texture. Listen on Soundcloud

App Use: Patterning, Dedalus

infrequent oscillations — Wed, 23 Dec 2015 11:18:45 GMT

A simple drum loop in Patterning filtered through Dedalus whilst I scrub through the various options to see what kind of aural mess I can make (hint: “A big one”). Good learning experience though. Listen on Soundcloud

Luigi review

Bytepawn - Marton Trencseni — Tue, 22 Dec 2015 01:00:00 GMT

I review Luigi, an execution framework for writing data pipes in Python code. It supports task-task dependencies, it has a simple central scheduler with an HTTP API and an extensive library of helpers for building data pipes for Hadoop, AWS, Mysql etc. It was written by Spotify for internal use and open sourced in 2012. A number of companies use it, such as Foursquare, Stripe, Asana.

Cargo Cult Data

Bytepawn - Marton Trencseni — Tue, 22 Dec 2015 01:00:00 GMT

Cargo cult data is when you're collecting and looking at data when making decisions, but you're only following the forms and outside appearances of scientific investigation and missing the essentials, so it doesn't work.

Disquiet Junto #207

infrequent oscillations — Mon, 21 Dec 2015 14:30:15 GMT

Screenshots of my effort for the Disquiet Junto #207 challenge. “M1 arpeggiated” - the underlying heavy drums “M2 bowed” - the gentle machine-like throbbing “M3 bowed” - the occasional background swell “M1 taped” - the initial drum-style intro “M2 arpeggiated” - extra texture “M3 shuffled” - the rolling trill (cf 2:40, 3:35)

App Use: Samplr #2, Addictive Synth

infrequent oscillations — Fri, 18 Dec 2015 14:05:27 GMT

App Use, Track 3 is more experimentation with Samplr. “Sherlock Holmes and the Voice Of Terror” provided the samples, Fission chopped them up, and Addictive Synth provided the arpeggiation samples. This was pretty tricky. (There was an earlier attempt at this but it was a fair old disaster. Even more repetitive and far too “chunky”.) It took many attempts to get the first sample working right - Samplr loops the sample when recording in Tape Mode and cutting it off whilst keeping within the beat of a bar was a nightmare.

App Use: Portable Dandy, Loopy

infrequent oscillations — Fri, 18 Dec 2015 13:48:46 GMT

Portable Dandy is a collection of samples you play in a virtual tape loop with a trio of effects. Quite fun but it has a proper problem as you can hear in thesoundtestroom review - certain sample and effect combinations will trigger horrible glitching. In an attempt to avoid this horror, I recorded the samples into [LPR][Loopy] and then sequenced them there. Where I discovered that Loopy is a right old bugger to work for this kind of thing.

App Use: Samplr, #1

infrequent oscillations — Fri, 18 Dec 2015 13:23:42 GMT

Samplr is a wizard’s toolbox of magical sound manipulation. In the hands of a master, it can produce some good stuff - “Do It Again”, “jamming with Samplr”, “SAMPLR - have some fun”. In the hands of a bumbling amateur, it makes … something? App Use, Track 1 is just a bunch of stock Samplr samples being randomly noodled through the various modes to see how it all works. It works as a mostly ambient track, I think.

Pay Down Mortgage or Invest?

Beetle Space — Fri, 18 Dec 2015 10:00:00 GMT

The following scenarios are ones many of us run in to:

Lumped Sum: I suddenly got $30,000. Should I put the money towards my mortgage, or should I invest it?
Periodic Payments: After all my monthly bills, I have $500 left over. Should I put it all towards my …

Ok, ok, I have a problem

infrequent oscillations — Thu, 17 Dec 2015 00:07:17 GMT

I already had a bunch of apps for making noises/musics but #november lead to - let’s be honest - a splurge. Which means it’s intervention time. The rule I’m not allowed to buy any more music/noise apps until I’ve made (and posted) something which each one. Since some of them are essentially filters or DAWs, combinations are allowed but should be minimal. Previously created content does not count for the purposes of this intervention.

2016 Adventures

infrequent oscillations — Tue, 15 Dec 2015 15:29:00 GMT

Things for 2016 {Preliminary list - will repost on 2016-01-01} (Stretch goals / PBs) [Caveats / Requirements] “unsupported” means carrying everything from the start - no refuelling from shops. Only rest stops are allowed. Movementercisers Janathon Juneathon Survive a full 6 hour race [with at least 42km] (no walking) (“unsupported”) Run more than 45km (48km) (50km) (56km) (no walking) (“unsupported”) Finish an official marathon (5:15) (5:00) (4:45) Half PB (2:00) (1:55) (1:50) 10km PB (50:00) (47:30) (45:00) Walk BAR-KNG at least twice (9:45) (9:30) [at least once “unsupported”] Walk SGR-KNG (15:00) (“unsupported”) Run BAR-KNG (6:30) (6:00) 7 more parkruns (+3 new places) All of Regents Park 10km series RBR ranking under 16 (15) (14) Randomerisers inktober blogtober noisevember

iSCSI SCSI-ID / Serial Persistence

smcleod.net — Mon, 14 Dec 2015 02:00:00 GMT

“Having a SCSI ID is a f*cking idiotic thing to do.”

- Linus Torvalds

…and after the amount of time I’ve wasted getting XenServer to play nicely with LIO iSCSI failover I tend to agree.

The Problem

One oddity of Xen / XenServer’s storage subsystem is that it identifies iSCSI storage repositories via a calculated SCSI ID rather than the iSCSI Serial - which would be the sane thing to do.

Tutorial: Dealing with Disk Space Errors in Linux

Ben Overmyer's Site — Thu, 10 Dec 2015 02:00:00 GMT

If you're seeing errors on a server (say, npm complaining about an ENOSPC error, or apt failing to install something) that suggest the hard disk is full, use this guide to resolve the issue.

Troubleshooting

All of the following commands assume you are logged in as a sudo user on a Linux machine. Some of the commands, like df -hT, won't work exactly as advertised on something like, say, Solaris.

Checking disk space

First, check the amount of available disk space.

df -hT

If one of the available drives (such as /dev/xvda1) shows as 100% full, then you'll need to clear some space. The first place to look is /var/log/, especially if nginx is writing log files to disk. Clear out all old logs (anything with a .gz file extension is fair game for deletion). If that doesn't free up enough space, the next place to check is the application directory for whatever app is on the server. If there are more than 3 releases on the server, you can safely delete the oldest ones (until you have at least 2 remaining). You can also remove all the files in /tmp/, although that should be a last resort.

Checking inode usage

If you've confirmed that the hard drive isn't full (or cleared enough space that it isn't), and you're still seeing “disk is full” type errors, then you almost certainly have too few inodes available on the server. This means, basically, that there are too many files on the server.

Checking inode usage

Run the following command to check inode usage:

df -i

You'll see something like the following:

Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             121K   409  120K    1% /dev
tmpfs            125K   479  124K    1% /run
/dev/vda1        1.9M  237K  1.7M   13% /
tmpfs            125K     2  125K    1% /dev/shm
tmpfs            125K     3  125K    1% /run/lock
tmpfs            125K    15  125K    1% /sys/fs/cgroup
tmpfs            125K     4  125K    1% /run/user/1000

The IUse% column is the one you're interested in. If IUse% is at or near 100%, then you have an inode usage problem.

Delete old kernels

Often, you can free up a ton of inodes if you delete old kernels. This is done with a pretty simple command:

sudo apt-get autoremove -y

Try that first. If that doesn't remove the old kernels, this page may help.

Script to check for directories with lots of files

This script will help find directories with large numbers of files. Put the following into a file list-files.sh in your home directory:

#!/bin/bash
# count_em - count files in all subdirectories under current directory.
echo 'echo $(ls -a "$1" | wc -l) $1' >/tmp/count_em_$
chmod 700 /tmp/count_em_$
find . -mount -type d -print0 | xargs -0 -n1 /tmp/count_em_$ | sort -n

Then make the file executable:

chmod +x ./list-files.sh

Finally, go to the root directory on the server and run the script:

cd /
~/list-files.sh

It may take a couple minutes to run, but it will present you with a list of all the directories with a number of files next to each, sorted in ascending order of number of files. Take a look at each of the high ones; that might give you an idea of what you need to clean up.

Some notes about good files to delete

Log files are safe to delete. So are old releases. Also check the /tmp/ directory; it can often be a hiding place for tons of files.

Postgres and Node - Hands on using Postgres as a Document Store with MassiveJS

CRAIG KERSTIENS — Tue, 08 Dec 2015 22:55:56 GMT

JSONB in Postgres is absolutely awesome, but it’s taken a little while for libraries to come around to make it as useful as would be ideal. For those not following along with Postgres lately, here’s the quick catchup for it as a NoSQL database.

In Postgres 8.3 over 5 years ago Postgres received hstore a key/value store directly in Postgres. It’s big limitation was it was only for text
In the years after it got GIN and GiST indexes to make queries over hstore extremely fast indexing the entire collection
In Postgres 9.2 we got JSON… sort of. Really this way only text validation, but allowed us to create some functional indexes which were still nice.
In Postgres 9.4 we got JSONB - the B stands for Better according to @leinweber. Essentially this is a full binary JSON on disk, which can perform as fast as other NoSQL databases using JSON.

This is all great, but when it comes to using JSON you need a library that plays well here. As you might have guessed it from my previous post this is where MassiveJS comes in. Most ORMs take a more legacy approach to how they work with the database, in contrast the other side of the world believes in document only storage way is the future. In contrast Postgres believes there is a time and place for everything, just like Massive, except it believes Postgres is the path just as I do.

Alright, enough context, let’s take a look.

Getting all setup

First go ahead and create a database, let’s call it massive, and then let’s connect to it and create our example table:

$ createdb massive
$ psql massive
# create table posts (id serial primary key, body jsonb);

Now that we’ve got our database setup let’s seed it with some data. If you want you can simple hop over to the github repo and pull it down then run node load_json.js to load the example data. A quick look at it, given an example.json file we’re going to iterate over it. For each record in there, we’re going to call saveDoc. Based on our table which has a unique id key and a body jsonb field it’ll simply save our JSON document into that table:

var parsedJSON = require('./example.json');
for(i = 0; i < parsedJSON.posts.length; i++) {
db.saveDoc("posts", parsedJSON.posts[1], function(err,res){});
};

If you want to just take a look at this github repo, once you create a database you can run node load_json.js to seed it.

Why JSON at all?

JSON data is all over the place, in many cases it’s fast and flexible and allows you to move more quickly. Yes, much of the time normalizing your data can be useful, but there is something to be said for expediency saving some data and querying across it. Querying across some giant document also used to be much more expensive, but now with JSONB and it’s indexes that can be extremely fast.

Querying

So how do we go about querying? Well it’s pretty simple with Massive, they provide a nice findDoc function to let you just search for contents of a specific key within the document. Let’s say I wanted to pull back all posts that are in the Postgres category, well it’s as simple as:

db.posts.findDoc({title : 'Postgres'}, function(err,docs){
console.log(docs);
});

The real beauty of this is if you added a GIN index (which will index the entire document) this query will be quite performant.

Just make sure to add your GIN index:

CREATE INDEX idx_posts ON posts USING GIN(body jsonb_path_ops);
CREATE INDEX idx_posts_search ON posts USING GIN(search);

But even better, with Massive it’ll automatically add these for you if you just start saving docs. It will automatically create the table and appropriate indexes, just doing the correct thing out of the box.

Full text and JSON

Cool, so you can do an exact look up. Which is great when you’re matching a category… which could be easily normalized. It’s great when you’re matching numbers, which also could likely reside in their own column. But what about when you’re searching over a large document, or a set of keys within some document which may require several joins, or indeterminate data structure, well you may want to search for the presence of that string at all. As you may have guessed this is quite trivial.

db.posts.searchDoc({
keys : ["title", "category"],
term : ["Postgres"]
}, function(err,docs){
console.log(docs);
})

Hopefully it’s pretty straight forward, but to be very clear. Call out the document table you want to search, then the keys you’ll want to include in the search, then the term. This will search for any place the contents that string are found in matching values for those keys.

Which will nicely yield the expected documents:

[ { link: 'http://www.craigkerstiens.com/2015/05/08/upsert-lands-in-postgres-9.5/',
title: 'Upsert Lands in PostgreSQL 9.5 – a First Look',
category: 'Postgres',
comments: [ [Object] ],
id: 2 },
{ link: 'http://www.craigkerstiens.com/2015/11/30/massive-node-postgres-an-overview-and-intro/',
title: 'Node, Postgres, MassiveJS - a Better Database Experience',
id: 3 } ]

In conclusion

While Massive isn’t perfect, its approach to storing queries in files, using the schema vs. having to define your models in code and the database, and it’s smooth document integration makes it a real contender as a better database library when working with Node. Give it a try and let me know your thoughts.

Apple Pay

infrequent oscillations — Tue, 08 Dec 2015 14:03:28 GMT

http://time.com/money/4139330/apple-pay-unused/ Survey: Only 20% of those that can use Apple Pay have used it Trustev surveyed 1,000 consumers who owned Apple Pay-compatible iPhones about their Apple Pay habits. (The iPhone 6, iPhone 6 Plus, iPhone 6s, and iPhone 6s Plus support Apple Pay, while older iPhones do not.) It found that about 80% had not used Apple Pay even once. Of the 20% who had, most (56%) only used it about once a week.

#noisevember

infrequent oscillations — Mon, 07 Dec 2015 14:50:26 GMT

The output In reverse order; unless otherwise indicated, things were created on iOS and recorded over Audiobus to Audioshare before exporting to Soundcloud. https://soundcloud.com/zimpenfish/tidal-noisevember (Tidal through Audacity) https://soundcloud.com/zimpenfish/much-indulgent-very-soundscape (SoundScaper) https://soundcloud.com/zimpenfish/just-beatwave-drums (Beatwave) https://soundcloud.com/zimpenfish/coconut-latte (Figure); https://soundcloud.com/zimpenfish/finsbury-park-bonus (Figure) https://soundcloud.com/zimpenfish/beyond-mere-indulgence (SoundScaper) https://soundcloud.com/zimpenfish/more-self-indulgence (FieldScaper) created from https://soundcloud.com/zimpenfish/before-self-indulgence (PixiTracker) https://soundcloud.com/zimpenfish/i-honestly-have-no-idea (Fieldscaper) https://soundcloud.com/zimpenfish/bass-player-mangled (Fieldscaper, creating from a recording of someone practicing a bass riff in the office kitchen) https://soundcloud.com/zimpenfish/mildly-delayed (SoundScaper, using one of the Bitwiz library tracks) https://soundcloud.

Study Notes In The Information Age

Beetle Space — Sun, 06 Dec 2015 10:00:00 GMT

It has been a long time since I was a college student. In those days I took all my notes in the classroom with a notebook. One notebook for each course.

Looking back at the experience, this had some down sides:

Some of what I wrote down was “useless”. Things …

Node, Postgres, MassiveJS - A better database experience

CRAIG KERSTIENS — Mon, 30 Nov 2015 22:55:56 GMT

First some background–I’ve always had a bit of a love hate relationship with ORMs. ORMs are great for basic crud applications, which inevitably happens at some point for an app. The main two problems I have with ORMs is:

They treat all databases as equal (yes, this is a little overgeneralized but typically true). They claim to do this for database portability, but in reality an app still can’t just up and move from one to another.
They don’t handle complex queries well at all. As someone that sees SQL as a very powerful language, taking away all the power just leaves me with pain.

Of course these aren’t the only issues with them, just the two ones I personally run into over and over.

In some playing with Node I was optimistic to explore Massive.JS as it seems to buck the trend of just imitating all other ORMs. My initial results–it makes me want to do more with Node just for this library. After all the power of a language is the ecosystem of libraries around it, not just the core language. So let’s take a quick tour through with a few highlights of what makes it really great.

Getting setup

Without further adieu here’s a quick tour around it.

First let’s pull down the example database from PostgresGuide

Then let’s setup out Node app:

$ npm init
$ npm install massive --save

Connecting and querying

Now let’s try to connect and say query a user from within our database. Create the following as an index.js file, then run with node index.js:

var massive = require("massive");
var connectionString = "postgres://ckerstiens:@localhost/example";
var db = massive.connectSync({connectionString : connectionString});
db.users.find(1, function(err,res){
console.log(res);
});

Upon first run if you’re like me and use the PostgresGuide example database (which I now need to go back and tidy up), you’ll get the following:

db.users.find(1, function(err,res){
^
TypeError: Cannot read property 'find' of undefined

I can’t describe how awesome it is to see this. What’s happening is when Massive loads up it’s connecting to your database, checking what tables you have. In this case though because we don’t have a proper primary key defined it doesn’t load them. It could treat id as some magical field of course like Rails used to and ignore the need for an index, but instead it not only encourages a good database design but requires it.

So let’s go back and create our index in our database:

$ psql example
$ alter table users add primary key (id);

Alright now let’s run our script again with node index.js and see what we have:

{ id: 1,
email: 'john.doe@gmail.com',
created_at: Thu Sep 24 2015 03:42:52 GMT-0700 (PDT),
deleted_at: null }

Perfect! Now we’re all connected and it even queried our database for us. Now let’s take a few more look at some of the operators.

Running an arbitrary query

db.run will let me run any arbritrary SQL. An example such as db.run("select 'hello'") will produce [ { ‘?column?’: ‘hello’ } ].

This makes it nice and easier for us to break out of the standard ORM model and just run SQL.

Find for quick look ups

Similar to so many other database tools find will offer you the most common quick look ups:

$ db.users.find({email: 'jane.doe@gmail.com'}, function(err, res){console.log(res)});
$ db.users.find({'created_at >': '2015-09-24'}, function(err, res){console.log(res)});

And of course there’s a where operator for multiple conditions.

Structuring queries in your application

While in the next post I’ll dig in deep to JSON, this is perhaps my favorite feature of Massive… It’s design for pulling out queries into individudal SQL files. Simply create a db folder and put your SQL in there. Let’s take the most basic example of our user email lookup and put it in user_lookup.sql

SELECT *
FROM users
WHERE email = $1

Now back in our application we can run this and pass in a parameter to it:

db.user_lookup(['jane.doe@gmail.com'], function(err,res){
console.log(res);
});

This separation of our queries from our code makes it easier to track them, view diffs, and even more so create very readable SQL.

Up next

So sure, you can connect to a database, you can query some things. There were a couple of small but more novel things that we blew through in here. First is the fact I didn’t have to define all my schema, it just knew it as it really should. The separation of SQL queries you’ll custom write into files is simple, but will make for much more maintainable applications over the long term. And best of all is the JSON support, which I’ll get to soon…

Quick update, and Hugo

Ben Overmyer's Site — Mon, 30 Nov 2015 02:00:00 GMT

Well, it's been about two months since my last post, which ironically was about my intention to write more frequently. To be fair, I wrote in my journal a number of times between then and now, but this is my first public post since then. Today I discovered a Go-based static site generator called Hugo that fascinates me with its speed and usability. I'm going to try using it to replace the current structure and content of benovermyer.com.

Documentation has a Cost and a Value

The Cranky Developer on Crater Moon Development — Sat, 28 Nov 2015 02:00:00 GMT

The cost of documentation is proportional to change frequency and distance from the code

Build vs. Buy, Part 5

The Cranky Developer on Crater Moon Development — Fri, 20 Nov 2015 02:00:00 GMT

Buy for parity, build for advantage

White Chocolate Protein Flapjacks

infrequent oscillations — Thu, 19 Nov 2015 09:45:25 GMT

Almost entirely a recipe I got from @chrismusther. Specific ingredients I used linked in [brackets]. Ingredients 75g Scottish Oats [Sainsbury’s] 75g Flax (Milled) [Linwoods Organic] 120g Whey Protein [Precision Engineered, Banana] 1 Apple (large) [Sainsbury’s 5 for £1] 300ml Apple Juice [Sainsbury’s Basics] 50g Mixed Fruit [Sainsbury’s Basics Dried] 30g Morello Cherries [Sainsbury’s Taste The Difference - Alternative] 20g Currants [Whitworths Juicy] 100g White Chocolate Chips [Sainsbury’s Cook’s] Method Put the oats, flax, whey protein, and chocolate chips into a bowl and mix loosely by hand to ensure a diverse community Grease up your silicon baking tray - 8" square is my preference Preheat your oven to about 150º / 305ºF / Gas Mark 2 Chop the apple as finely as you want the bits in your flapjacks but no finer than your sanity can allow Put the apple chunks, apple juice, and various fruits into a pan - precise ordering is irrelevant here Apply strong heat until the apple chunks go soft but before the juice really starts evaporating Pour the hot steaming fruit cocktail onto the aforementioned diverse oaty-flaxy-chocolatey-protein community and stir slowly but thoroughly This will create an astonishingly sticky mess that smells amazing - RESIST ALL TEMPTATION TO MASH YOUR FACE IN IT Carefully spoon the sticky mess into your baking tray, evening it out as much as you can (good luck with that) Place baking tray in oven and relax for 25-30 minutes until the top is firm Remove your delicious goodness from the oven and leave to cool before enjoying with a nice cup of tea Nutrition Based on the specific ingredients I used, MyFitnessPal reckons a 1/16th slice contains -

Build vs. Buy, Part 4

The Cranky Developer on Crater Moon Development — Thu, 19 Nov 2015 02:00:00 GMT

Customization and programmability of commerical and bespoke software

Why does the theatre cost ALL MY MONEY?

infrequent oscillations — Wed, 18 Nov 2015 14:26:08 GMT

In which we learn about the cost of theatre productions.

Build Vs. Buy, Part 3

The Cranky Developer on Crater Moon Development — Wed, 18 Nov 2015 02:00:00 GMT

Polist and quality comparisons in vendor vs in-house software

Build Vs. Buy, Part 2.

The Cranky Developer on Crater Moon Development — Tue, 17 Nov 2015 02:00:00 GMT

Upgrades and feature creep in commercial software

Build vs. Buy -- Responding to All-or-Nothing Approaches

The Cranky Developer on Crater Moon Development — Mon, 16 Nov 2015 02:00:00 GMT

Technology, cost, and quality in deciding build vs buy

Continuous Integration, Misapplied

The Cranky Developer on Crater Moon Development — Sun, 15 Nov 2015 02:00:00 GMT

Putting together an continuous-integration-style automated build and deploy halfway through the project is exponentially more effort than starting off with one. At some point in the project, it may even be that the effort outweighs the gain.

LiveReload for Chrome extensions with Gulp

Sun, 15 Nov 2015 02:00:00 GMT

LiveReload is a fantastic feature to add to your development workflow to increase your productivity. The general idea is that when a change occurs to a file that makes up your app, your app is automatically rebuilt and the app restarts itself. This saves you from manually rebuilding and restarting your app each time you make a change and generally makes development much more pleasant. There are LiveReload implementations available for many languages and frameworks such as tiny-lr which can be used with Gulp for web development.

HTTP/2 and SSL

infrequent oscillations — Fri, 13 Nov 2015 15:58:53 GMT

rjp.is is now HTTPS-only; and also supports HTTP/2 [FAQ] for future proofing. Here’s an owl to compensate for the entertainment-free nature of this post.

Fix Played Tracks with Empty Playcount in iTunes

Donat Studios — Thu, 12 Nov 2015 05:50:58 GMT

I have a quite large iTunes library, and I’m rather anal about keeping my metadata clean and up to date. In an earlier release of iTunes 12 there was a bug. Every so often after a song played, the Play Count would fail to increment, but the Last Played would update.

Searching my library, I found 74 tracks that had this issue.

I found this irritating and inaccurate, so I set forth finding a way to correct it.

I wrote the following simple AppleScript to set the playcount of such tracks to 1.

All you need to do is either paste the following into the 'Script Editor' or download it and run it. See video below if you need help running it once it’s open.

To view this video please enable JavaScript and upgrade to a web browser that supports HTML5 video

How to cluster and failover (almost) anything - An intro to Pacemaker and Corosync

smcleod.net — Mon, 09 Nov 2015 02:00:00 GMT

Slides

Failover Demo

Tiny Keep (#blogtober 31)

infrequent oscillations — Sun, 01 Nov 2015 01:08:55 GMT

THE END. Tiny Keep has the distinction of being the last. But sadly not the best. This is actual a fair reflection of me this evening. They’ve bizarrely decided to use normal* console controls - left stick does movement, right stick controls the camera. This makes it nigh-on impossible to control reliably, especially when you get to combat later. Managed to set myself and a bunch of enemies on fire. Oops.

No Brakes Valet (#blogtober 30)

infrequent oscillations — Sat, 31 Oct 2015 01:55:00 GMT

Despite the name, you do have brakes. I didn’t realise this for quite a few games… Easy enough - park the cars in the parking bays. Er, yeah. Don’t ram the explosives truck with another vehicle, btw. I cannot even.

searchcode local

Ben E. C. Boyter — Fri, 30 Oct 2015 09:56:37 GMT

I am going to copy the searchcode pitch itself below quickly before explaining it a bit further.

“searchcode offers powerful code search over billions of lines of open source code. Imagine what it could do with your private repositories.

There have been requests to offer a downloadable version of searchcode. Given enough interest a downloadable hostable version of searchcode will be offered. Register your email below to register your interest.

Hatoful Boyfriend (#blogtober 29)

infrequent oscillations — Thu, 29 Oct 2015 22:44:47 GMT

I’m a girl. At a school for pigeons. My best friend is a rock dove. Here’s my teacher. He’s a quail. OBVIOUSLY. Apparently I live in a cave. This might be a side plot about pudding. Or it might be just another thing which is making my brain bleed out. Oh ho ho, REFERENCES. “RAGE VOLTAGE” is an excellent metaphor but everyone knows that it’s really Amps that do the damage.

Why Docker?

Brightball Articles — Thu, 29 Oct 2015 02:00:00 GMT

Containers are not a new thing, but implementing them was always a little more complicated than it needed to be. Docker made great leaps in simplification of containers and set the world on fire from there. Let’s look at why.

Qora (#blogtober 28)

infrequent oscillations — Wed, 28 Oct 2015 21:46:15 GMT

Qora is a bit Sword & Sworcery but chunkier. (Excuse the window decorations - I’m having to do this in a coffee shop without FRAPS because Spectre is going to make me too sad to play any games when I eventually get home after TWO DAMN HOURS of RELENTLESS DRIVEL.) I’d like to point out here is that there’s no voice acting (rapidly becoming a pet hate of mine) in Qora - which means you can have lots of nice lovely textual conversations.

Prison Architect (#blogtober 27)

infrequent oscillations — Wed, 28 Oct 2015 02:07:09 GMT

I went into Prison Architect expecting a subtle, brooding, sneaky moral choice experience much like Papers Please (highly recommended, btw) … and was promptly set about the head by a two by four as the first mission was to build an execution chamber and then execute someone. Way to go with the subtle and brooding there, champ. It’s just grim. But hey ho, maybe that’s just the first “shocker” mission to loosen you up for some lovely subtle … what’s that?

Amnesia (#blogtober 26)

infrequent oscillations — Mon, 26 Oct 2015 23:42:14 GMT

Finally getting around to Amnesia and … meh. Games with voice acting really grind my weasels. It’s just such a waste of resources and effort and then all your dialog has to be plotted for voice and BLEUGH. It actually took a while to get to this point - it turns out that you need to turn off “Display Scaling for High DPI” or the screen will be zoomed and the mouse will jank about like Keith Richards double teaming meth with crack.

The Long Dark (alpha) (#blogtober 25)

infrequent oscillations — Mon, 26 Oct 2015 01:00:11 GMT

The Long Dark is like a first-person Don’t Starve - except prettier and more difficult. This is the only game I’ve seen which lets you pick metric (correct!) and 24 hour clock (correct!). Confusingly, even though the “Feels Like” went from -20º to -15º whilst I was playing, my temperature was heading downwards faster than everyone’s favourite* blue team. I do like a realistic inventory, though. You get some good scenery.

Gravity Ghost (#blogtober 24)

infrequent oscillations — Sat, 24 Oct 2015 23:26:26 GMT

Like one of those old Flash games where you had to propel your penguin between a bunch of planets to hit the black hole1, you have to leave one planet, get the star, and go through the door. This is why I keep my hair violently short - I’m fed up of being harassed by animal spirits for free rides. My luscious locks have snared me a space bunny!

A Wizard's Lizard (#blogtober 23)

infrequent oscillations — Fri, 23 Oct 2015 22:48:08 GMT

A Wizard’s Lizard – which sounds like a dirty, dirty euphemism – seems to be heavily inspired by another game. See if you can guess which one. PLOT. This screen, dark though it may be, contains all the clues you need to know about which game heavily influenced this one. It all seems to have gone a bit Dark Souls here - I died and now I’m unhollow? Or hollow-but-angel?

Antichamber (#blogtober 22)

infrequent oscillations — Thu, 22 Oct 2015 22:25:07 GMT

There’s probably some trick to Antichamber that I’m missing because the only non-obvious puzzle part I managed to figure out were the dreamcatchers and even then the wall had to hint heavily at it. Which is a shame because it looks lovely. Probably one for the philosophy students.

The Adventures of Van Helsing (#blogtober 21)

infrequent oscillations — Wed, 21 Oct 2015 22:07:44 GMT

Abysmal. Awful. Arse. Atrocious. I’m beginning to associate cut scenes and voice acting with games that should be put into the bin before they’re even made. Also - picking 1440x480 as my screen resolution and only offering 1366x768 as the best option (“This change will take effect when the game restarts”) is a guaranteed deletion. Sorry chaps, you got one chance and you blew it.

Naissancee (#blogtober 20)

infrequent oscillations — Tue, 20 Oct 2015 21:51:59 GMT

Naissancee is lovely in a style sense but since I’ve already lost the tutorial droid within the first 5 minutes, got stuck in a room I can’t get out of without dying -and- being returned to a point two steps before a level load, I’m not going to bother playing it any more. Just enjoy the pictures.

Crawl (#blogtober 19)

infrequent oscillations — Mon, 19 Oct 2015 21:11:10 GMT

I -think- Crawl is some kind of retro-styled take on Dark Souls but … I didn’t really have much of a clue what was going on. Maybe with more reading of instructions and tips it would make sense. Everyone worships Teok, right? Pretty sure I’ve just had my innards tenderised here. It does have a nifty death animation. And then there’s something about upgrading monsters based on how much wrath you have?

RocketsRocketsRockets (#blogtober 18)

infrequent oscillations — Sun, 18 Oct 2015 20:42:36 GMT

The visuals are nice but it zooms in and out like a pumping weasel is at the controls and seriously - if I’m a rocket ship, left means “turn left”, it NEVER means “fly left”. Shame, really, because the visuals are lovely.

Capsule (#blogtober 17)

infrequent oscillations — Sat, 17 Oct 2015 21:29:00 GMT

Capsule might be a contender for peak game of #blogtober. Atmosphere (literally), tension, and an actual story conveyed by recovered emails rather than garish cutscenes, uncanny valley facial animation, and terrible dialog. You have to get your capsule to the next waypoint without running out of oxygen or power because bad things happen when you run out of either. Your only help is a sonar ping that will (slowly) identify what’s around you.

Human Resource Machine (#blogtober 16)

infrequent oscillations — Sat, 17 Oct 2015 00:05:18 GMT

TL;DR: GLORIOUS. Buy it now. Even if you don’t like the game, Tomorrow Corporation deserve all the money they can get to keep making lovely sweet games for ME. I’m more than reasonably sure this is going to be Peak Game for #blogtober. After the surprising depth of Little Inferno, Human Resource Machine was a no-brainer pre-order and here it is. I think the third one bears a striking resemblance to my current form.

A Virus Named Tom (#blogtober 15)

infrequent oscillations — Thu, 15 Oct 2015 22:28:41 GMT

Remember Pipe Mania? Well, A Virus Named Tom is basically a tarted up Pipe Mania with a plot. See? Plot. I’m stuck here because all the “encrypted” (aka “hidden”) tiles are straight pieces and I can’t figure out how everything connects up. Plus the rotation of the pieces is done by dragging the little robot around the border and trust me, that gets pretty old pretty damned fast as a control mechanic.

Monitoring with Bosun

Brightball Articles — Thu, 15 Oct 2015 03:00:00 GMT

Bosun is a monitoring and alerting system developed by the good folks at Stack Exchange, then open sourced for the rest of us. It’s written in Go, meaning its monitoring agents can run anywhere that Go can drop a binary… which is just about everywhere. So what exactly does it do and how does it compare to the likes of New Relic, CloudWatch, Nagios, Splunk Cloud, Server Density, and other monitoring tools?

Dustforce (#blogtober 14)

infrequent oscillations — Thu, 15 Oct 2015 00:40:01 GMT

You have to sweep up leaves. Sadly without actually using your brush - a simple run through is sufficient to magically displace them into a nether dimension. The jumping and controls are horribly floaty and syrupy and laggy and ugh. AND it not only doesn’t have controller support by default, it let me accidentally assign the same joypad button (“pad0_dir2” is also a tremendously horribly label) to two different directions.

Torchlight (#blogtober 13)

infrequent oscillations — Wed, 14 Oct 2015 00:50:36 GMT

A hacky-slashy adventure RPG - kinda like Geneforge but with 90% of the budget spent on 3D modelling and voice actors instead of plot and story. Although it does have a rudimentary plot. It does look awfully pretty - but I can see myself playing Geneforge in preference to this just because it’s much more satisfying than this melange of mishmash.

Mirror Moon (blogtober #12)

infrequent oscillations — Mon, 12 Oct 2015 23:32:54 GMT

Baffling space exploratory puzzle something something. Possibly the tiniest minimalist title screen yet. Looking a bit like a low rent NES-inspired Elite Dangerous here. That’s quickly dispelled once you arrive on the first planet. Except I can’t really say anything about the next few screenshots since it’s probably spoilery. (I will say that I had to resort to walkthroughs because they do a bloody awful job of letting you know what you’re supposed to be doing.

How to get better at software development?

Nemo's Home — Mon, 12 Oct 2015 03:00:00 GMT

I often get a lot of queries from people asking me about how to get started with software development, and how to get better at it. My replies are almost reaching stock-level worthy of copy-paste now, so I thought I might as well write about it publicly.

What follows is a list of advice I’d give to any person who wants to write software for a living. A lot of it might apply across professions, and a lot of it is tailored to students in universtities. Not everything might apply in your case, YMMV. Take everything with a pinch of salt. Feedback is welcome.

Join a community.

Highly preferable if its an IRL (In-real-life) community rather than just a chatroom somewhere, but even those are preferable over nothing. Communities have this shared sense of learning, that you don’t enjoy anywhere else. Passive learning is something I talk a lot about, and it only happens because of chance interactions that happen in communities. Even online communities work fairly well, and by online communities I mean places like StackOverflow, AskUbuntu, ServerFault, HackerNews, subreddits etc. If you don’t have a physical community near you that you can join, maybe its time to start one?
Contribute to Open Source projects

It doesn’t have to be with your code, or even a large project. Even small javascript npm modules that you might think can be improved deserve some Pull Request love.
Write all code publicly

Your code not being public should be the exception, not the norm. I’ve found putting almost all my code on github fairly liberating. I keep all my OS configuration and a lot of other things on github.
Do tech talks

It doesn’t have to be at a big-name conference, but maybe at a small meetup around you. Good conferences will sponsor your tickets, and as a plus, you get to attend all the talks at that conference for free. Just make sure that you actually do know what you’re talking about, unlike a lot of talks that happen. The level of knowledge expected of a speaker is far more, and as a result if you are the one talking about something, you need to get better at it and understand it better, which is a great way of forcing yourself to learn something.
Stay Updated

Reading Hacker News is a fairly certain way of making sure of that. A person doing PHP development should be aware of things like Composer, HHVM, and perhaps the upcoming changes in PHP7 (They’re awesome). As a technologist, part of our job is to stay updated with trends (no matter how insane the JS framework wars sound). The code you will be writing 5 years from now will be in an entirely different framework than what you are using today. This doesn’t mean that you should start learning the ins and outs of every JS framework, but rather that you should be tangentially aware of developments happening in the space. (For eg, following stable updates of Rails even though you are not a Rails developer).
Learn more languages

I am a proud polyglot, and I very often realize that knowing more than one language changes your style and more importantly your thinking process significantly. For eg, a Ruby programmer will be fairly comfortable with the idea of metaprogramming compared to a PHP developer, and even more so when it might come to DSL (Domain Specific Languages). Similarly, knowing Haskell or Functional Programming in general teaches you a lot of things that you might re-use back in your JavaScript world.
This doesn’t happen unless you know more than one language. Moreoever, its always helpful to have JavaScript as your second language (if you are looking for one), because of its monopoly in front-end development. A lot of technologies (like CORS/JSONP) just don’t make sense unless you understand JavaScript.
Concepts Matter

I was asking people about good interview questions, and one that I really liked was “How do you write an HTTP server using sockets?”. A lot of developers are stuck in this moat of “programming = software development”. And you can’t get over that unless you start thinking in terms of concepts. This is not me trying to get people to become Architecture Astronauts, but me trying to get people to understand how things work. I’ve interviewed people who have no idea about how HTTP works, and in my opinion you can’t really be a web developer without knowing HTTP. A fairly good filter for good web developers is whether they know the ins-and-outs of HTTP. And HTTP is not a programming challenge, but rather a conceptual problem. Similarly, if you work in the frontend, and you don’t know what the Same Origin Policy is, I am not gonna hire you. (“Is it implemented on the browser or the server?” is a another good question). The point I’m trying to make is that you need to get a layer above your language’s standard library and understand how things work. Learning ActiveRecord is awesome, but do you understand how it works?
Ship Products

Doesn’t matter if they are small, or made in a hackathon. As long as its shipped, we’re cool. If its not, come back when you’ve shipped it.
Have side projects

This is slightly harder to do, but far more rewarding. Make sure that your side-project is not something you expect to make money out of, and that it has a fairly reasonable scope. Side projects are an excellent breeding ground for you to try out new technologies, and play around with new languages. Its a really good breakaway from work-things as well, on top of that.
Read technical books

As a start, I’d recommend everything that codinghorror has suggested here and here. There are a lot of good books listed on hackershelf.com as well. My personal favorite is Don’t Make Me Think, which is a book on Web Usability and something I think every developer and designer should be forced to read.

Thanks to Shashank Mehta for discussing these ideas with me and helping me frame this post.

Titan Attacks (#blogtober 11)

infrequent oscillations — Sun, 11 Oct 2015 18:56:33 GMT

Basically a fancy Space Invaders with upgradable weapons and savable hostages a la Defender. They’re confused. Points win prizes, not perfection. Silly aliens.

The Floor Is Jelly (blogtober #10)

infrequent oscillations — Sun, 11 Oct 2015 00:19:24 GMT

Puts me in mind of a weird cross between the creature innards levels of Loco Roco and Bugaboo The Flea. Simple - just bounce your way around the jelly floors to reach the curtains (or lift, later.) This is a cute level - to reach the curtains, you need to properly trampoline yourself with good timing to get a super jump. Just in case the player hasn’t realised this is possible, the little frog on the right is demonstrating.

Spy Chameleon (#blogtober 9)

infrequent oscillations — Sat, 10 Oct 2015 00:30:52 GMT

The screenshots tell pretty much the whole story and I’ve got to get up early. Enjoy.

FEZ (#blogtober 8)

infrequent oscillations — Fri, 09 Oct 2015 00:52:12 GMT

One of the big indies that I’ve somehow managed to avoid playing until now. I wonder if she has any kerosene Geezer has some cracking industrial strength drugs, it seems. Um, Lament Configuration, anyone? “We have such cubes to show you.” “This will seem like a memory of 2D!” etc. etc. Then the titular headwear arrives from the Lament giant Devil Square and Gomez can move 2D-wise through 3D. (It’ll make sense when you play the game.

HEAVY BULLETS (#blogtober 7)

infrequent oscillations — Wed, 07 Oct 2015 23:20:15 GMT

HEAVY BULLETS (or “superneonshootybang” as I’m now going to call it) is kinda DOOMy but with a twist - you have a revolver, 6 bullets, and the only reloads you get are when you retrieve the ones you’ve shot. There’s some plot about something going horribly wrong, yada yada. Amusingly, the graffiti seems to be from a random selection of anti-corporate & anti-rich slogans. All the neon lushness makes me happy.

SAN Intro

smcleod.net — Wed, 07 Oct 2015 03:00:00 GMT

Point Perfect (#blogtober 6)

infrequent oscillations — Wed, 07 Oct 2015 00:56:06 GMT

Ever wanted to fly a mouse cursor into space? Defeat hordes of alien creatures in a series of mini-games? Can you drag-n-drop the alien scum into the bin of destiny? Then Point Perfect is for you. You get three rounds of silliness - lowest score ignored - and they all revolve around manipulating your mouse cursor spaceship. You have no weapons - obvs, you’re flying a mouse cursor - and the only way to eliminate aliens is to highlight one (or more if you’re nimble) with your targeting skills and let the orbital laser take them out for you.

The Organ Trail (blogtober #5)

infrequent oscillations — Mon, 05 Oct 2015 19:55:33 GMT

No prizes for guessing what game this is parodying… And the intro is pitch perfect. Yes, he’s managed to break his arm driving a car down the road. Damn trails. Once you’re passed the intro (and gunfighting with HORRIFYINGLY BAD CONTROLS), you get to visit some cities… And yes, you have to contend with zombies stupid pop culture green monsters who bite the living. Apart from that, it’s your bog-standard Oregon Trail-alike (with HORRIFYINGLY BAD SHOOTING CONTROLS.

Race The Sun (#blogtober 4)

infrequent oscillations — Sun, 04 Oct 2015 18:55:46 GMT

Did you play 3D Deathchase as a child? But now you have a hankering after a thoroughly modern version to replicate that nauseating motion sickness? Welcome to [Race The Sun]. Ok, that’s the title screen and looks nothing like 3D Deathchase. Once you get going, however, … You have to avoid the things and pick up the other things and there’s moving things and jumpy things and you just keep going until you splatter your intestines over something or the sun goes down.

Geneforge (#blogtober 3)

infrequent oscillations — Sun, 04 Oct 2015 00:21:40 GMT

After reading the excellent blog of Jeff Vogel for many years, I eventually got around to buying The Geneforge Saga from GOG. It’s a charmingly old-worlde rustic RPG from the days when graphics cards cost as much as a small van and CPUs were the size of a small van. Fancy splash screen and astonishingly green menu screen for your edification. Before you start, you have to do the obligatory “assign points to categories” character creation screen.

Seeding a sharing-economy or platform company

CRAIG KERSTIENS — Fri, 02 Oct 2015 23:55:56 GMT

These days if you’re creating a company you likely hope to accomplish more with less people, two ways of doing this fall to: The sharing economy and creating a platform. It’s easy to see the case for this when you have such unicorns like AirBnB or Uber. The opportunity for each of those to compete against hotel chains or taxi services which each need to manage their own inventory is incredibly exciting and revolutionary. In a similar fashion platforms can offer much the same, Heroku’s platform and marketplace made it easier than ever for developers to click a button and get everything they needed years ago. It’s not just their code, it’s everything from Postgres to Mongo to Logging. Or take the app store as example. Smart phones weren’t a new thing when the iPhone came out, but it was only the saviest of users that had apps installed on their windows smartphone or blackberry. The app store made the iPhone different than any other phone by allowing others to build and improve it, turning the iPhone not into a phone but a platform.

Platforms and the sharing economy both let you get further than having to take on the costs of offering the equivilent all on your own. And while a great idea to venture into one of these two areas, starting them isn’t as trivial as simply deciding to. For both of these you have issues with having a two sided market, first you have to convince the providers to come along, then the customers or vice versa. As a result of this two sided market issue the easiest way to actually start is by bootstrapping it yourself – or faking it til you make it.

What are some good examples of faking this? I’m sure you can probably find some good stories going back about AirBnB or Uber, but let’s assume times were different then. Let’s take a look at a very recent example: Lugg which just launched in the latest batch of YC. Lugg is Uber for moving essentially, allowing you to on-demand request furniture moved from one place to another. Early on Lugg built their app, then waited for requests to come in, then the founders got in a truck and moved the furniture themselves. As a customer the founders are likely providing a great experience, without ever having to tip their hat at the ways their hacking the impression of being a large well oiled machine.

But what about a platform? Slack continues to grow like wildfire as the new medium for communication. These days there’s endless integrations for slack, and I expect they’ll continue to expand what a platform for communication looks like. But a year ago they were quite a ways from having people show up at their door to add an integration. Sure there were people using them, but to expect github/trello/asana to immediately build an integration for every new flavor of the week tool would be crazy. Yet, without these integrations slack wouldn’t be nearly as useful as it is today–and probably wouldn’t have seen the growth it’s seen. In the early days of a platform the easiest way to get these integrations and partners in place is to show up and build the work yourself. Slack carried the weight early of building these integrations, much as Heroku add-ons showed up at partners offices and help write the code to get them as a provider in the marketplace. And while both Slack and Heroku are larger companies now, it still holds true for smaller ones starting today. Blockspring, a company which aims to make web services available through spreadsheets, had to do very much the same thing building their initial integrations themselves. Now with their rapidly growing user base and already large collection of APIs they may be able to shift the model, but early on that wasn’t so much an option.

If you want to build a platform, start by creating the impression of one while still carrying the load yourself. Yes, move to a true platform as soon as you can, but don’t wait for others to show up before you go that route.

The Bridge (#blogtober 2)

infrequent oscillations — Fri, 02 Oct 2015 23:43:01 GMT

Day two brings The Bridge which brings the lovely charcoal sketch aesthetic to a world of Escher-like puzzles. There’s also a Braid-alike “turn back time” mechanic but I don’t know if it becomes relevant to later puzzles or is just a “BIG WHOOPSIE UNDO” button. Our hero has fallen asleep under an apple tree. You can see where this is going… After a mini-tutorial on the controls (which also serves to wake him up), you trudge off to find his house.

#blogtober

infrequent oscillations — Thu, 01 Oct 2015 23:50:42 GMT

Or “let’s try #septemblog again”. For each day in #blogtober, I’m supposed to be playing one of my many unplayed games that I’ve accumulated through things like Humble Bundle. First up is Dungeon Souls - a “2-D top-down hack n’ slash/RPG game”. (Not sure why FRAPS isn’t capturing the full glory of the title screen there.) Which is exactly what it says on the tin - you pick your bog-standard class of RPG adventurer,

Replacing Junos Pulse with OpenConnect

smcleod.net — Tue, 22 Sep 2015 03:00:00 GMT

In an attempt to avoid using the Juniper Pulse (Now Pulse Secure) VPN client we tried OpenConnect but found that DNS did not work correctly when connected to the VPN. This bug has now been resolved recently but has not made it’s way into a new build, in fact there have been no releases for 6 months.

Luckily the OpenConnect was not too difficult to build from source.

Build OpenConnect on OSX

Remove old openconnect and install deps

brew remove openconnect
brew install libxml2 lzlib openssl libtool libevent

Build openconnect

wget git.infradead.org/users/dwmw2/openconnect.git/snapshot/0f1ec30d17aa674142552e275bf3fac30d891b39.tar.gz
tar zxvf 0f1ec30d17aa674142552e275bf3fac30d891b39.tar.gz
cd openconnect-0f1ec30

LIBTOOLIZE=glibtoolize ./autogen.sh
PATH=/usr/local/opt/gettext/bin:$PATH
./configure
make
make install

To connect

sudo openconnect --juniper -u myusername www.myserver.com

If you’re comfortable with allowing admin users to run openconnect without entering a sudo password, add the following using sudo visudo:

Taking up the writing habit again

Ben Overmyer's Site — Sat, 19 Sep 2015 03:00:00 GMT

A few days ago, I migrated all of my blog posts from my last primary blog on WordPress.com to this blog. In the process, I read over a lot of old posts. I noticed something interesting: back in 2007 and 2008, I was writing blog posts with fair regularity. Now, it's a rarity. The blog posts back then were fairly journal-like in character. They told a story of what I was working on, how I felt on certain subjects, and described some of my adventures in games I was playing at the time. The blog posts I wrote in the recent past trend towards the same thing when they're more frequent. When I choose to try and write something useful, like the Laravel on Docker post, it takes a lot longer, a lot more effort, and is a lot more intimidating to start on. So as I sit here sipping on my orange pekoe black tea, listening to relaxing acoustic guitar, and writing these words, I'm thinking that I should allow myself to write journal-style posts again. This, if for no other reason than to ingrain that old writing habit. It'll be easier to write the big, important stuff if the simple stuff - the writing - comes naturally, right?

Heroku Data Links with Postgres and Redis

Brightball Articles — Wed, 16 Sep 2015 03:00:00 GMT

PostgreSQL has a great feature called Foreign Data Wrappers (FDW) that allows it to connect directly to outside systems. Although the setup can be a little complicated, once it’s available you can run queries with joins or subqueries against them, insert data, create views, etc. Heroku has dramatically simplified the process of using FDW with PostgreSQL and Redis thanks to Data Links. Let’s try it out.

SSD Storage - Two Months In Production

smcleod.net — Sun, 13 Sep 2015 03:00:00 GMT

Over the last two months I’ve been running selected IO intensive servers off the the SSD storage cluster, these hosts include (among others) our:

Primary Puppetmaster
Gitlab server
Redmine app and database servers
Nagios servers
Several Docker database host servers

Reliability

We haven’t had any software or hardware failures since commissioning the storage units.

During this time we have had 3 disk failures on our HP StoreVirtual SANs that have required us to call the supporting vendor and replace failed disks.

Get Going with Laravel on Docker

Ben Overmyer's Site — Fri, 11 Sep 2015 03:00:00 GMT

This post was updated May 14th, 2016 to significantly rework the tutorial. Docker is slowly taking over the world of web infrastructure. It makes working with multiple different services easy, and the problem of “works in dev, not in prod” goes away, since you have the same environment on your local machine as you do in the production infra. It also makes things like trying out web apps without deploying them to servers really easy. Ever wanted to just check out a personal demo of, say, WordPress or Ghost? Docker makes that simple. Docker can be intimidating to start out with. It's a complex beast, but once you've gotten it set up a couple times, it'll become second nature. This article will walk through the entire process of having a completely new and fresh Mac OS X environment all the way to running Docker container with your own Laravel application. It's a long tutorial, so grab a big mug of coffee and put on your favorite track and let's get to work. Note: This tutorial is for OS X. A lot of the pre-req stuff is different for Windows, so if you're running that OS, you might be better off using Docker Toolbox. Until the new Docker App comes out, anyway.

Prerequisites

The first step is getting some basic tools installed. OS X comes with git and Ruby installed, but we need to make a slight adjustment to our Ruby environment, and we'll need Homebrew to install some other things.

Homebrew

Let's start with Homebrew. Paste this into a terminal prompt:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Running that will install Homebrew. Next, we'll need Homebrew Cask:

brew install caskroom/cask/brew-cask

This is used to install binary applications that otherwise you'd use a GUI for.

Virtualbox

Let's install this via Homebrew Cask to make things easy.

brew cask install virtualbox

Enter Docker

Now we get to the meat of it - installing Docker and the other tools necessary to work with containers. We're going to be using a wrapper called dinghy for this. You could use the official Docker Toolbox, but the default VM that it provides, boot2docker, relies on vboxfs, which makes real-time updates (like gulp watch) really slow.

Install Docker Toolbox

When I originally wrote this post, Docker Toolbox made some questionable assumptions about how people were using Docker for development. Those have now been corrected, and it's safe to go ahead and install Docker Toolbox. You can get that here: https://www.docker.com/products/docker-toolbox

Docker is installed! Now what?

Now everything is all set up and ready to get container-ing. In the next section, we'll create a basic Laravel app to get a proper Docker workflow going.

Getting set up for Laravel

For this tutorial, we're going to use the Laravel framework. This will demonstrate how to use multiple containers while not requiring us to write a ton of code to get started.

Install Composer

PHP is installed by default with OS X, but you're going to need Composer also. Thankfully, it's a quick install:

curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer

Make sure that composer is in your PATH in your bash/zsh profile. Look for this line: ~/.composer/vendor/bin If it's nowhere in the file, you'll need to add something like this near the bottom of the file:

export PATH="$PATH:~/.composer/vendor/bin"

Install the Laravel CLI tool

This tool makes starting new Laravel projects much easier and faster:

composer global require laravel/installer

Start a new Laravel project

In a directory that makes sense for you (I use ~/Source/ for my code projects), run this shell command:

laravel new docker-app

This will create a new Laravel project called “docker-app” in a new docker-app directory. You're now ready to Docker-ize this app!

Docker-ize!

The next step is to create four files that we'll use to describe our container environment. These files are nginx-vhost.conf, supervisord.conf, Dockerfile, and docker-compose.yml. Each of these is at the root of your project. The contents of each file is as follows:

nginx-vhost.conf

We're using nginx as the web server for this app. We could use Apache, but for this exercise, we'll stick with nginx. The below file configures the virtualhost that will expose our Laravel app to the container.

server {
 listen 80;
 listen [::]:80 default ipv6only=on;
 root /srv/www/public;
 index index.php;

 # Disable sendfile as per https://docs.vagrantup.com/v2/synced-folders/virtualbox.html
 sendfile off;

 # Add stdout logging
 error_log /dev/stdout info;
 access_log /dev/stdout;

 location / {
 try_files $uri $uri/ /index.php$is_args$args;
 }

 location ~ .php$ {
 try_files $uri =404;
 fastcgi_split_path_info ^(.+.php)(/.+)$;
 fastcgi_pass unix:/var/run/php5-fpm.sock;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 fastcgi_param SCRIPT_NAME $fastcgi_script_name;
 fastcgi_index index.php;
 include fastcgi_params;
 }

 location ~* .(jpg|jpeg|gif|png|css|js|ico|xml)$ {
 expires 5d;
 }
 # deny access to . files, for security
 location ~ /. {
 log_not_found off;
 deny all;
 }
}

supervisord.conf

We're using Supervisor to run nginx and PHP-FPM. It makes things a little cleaner and easier. Below is the config file for Supervisor.

[supervisord]
nodaemon=true

[program:php5-fpm]
command=/usr/sbin/php5-fpm
autostart=true
autorestart=true
priority=5

[program:nginx]
command=/usr/sbin/nginx
autostart=true
autorestart=true
priority=10
stdout_events_enabled=true
stderr_events_enabled=true

Dockerfile

For a guide to the syntax you see below, check out the official Docker documentation for Dockerfiles.

FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update -y
RUN apt-get install -y software-properties-common
RUN add-apt-repository ppa:nginx/development
RUN apt-get update -y
RUN apt-get upgrade -y
RUN apt-get install -y supervisor nginx php5-fpm php5-cli php5-curl php5-gd php5-mysql php5-memcached php5-mcrypt

# Clean up to reduce container size
RUN apt-get remove --purge -y software-properties-common
RUN apt-get autoremove -y
RUN apt-get clean
RUN apt-get autoclean
RUN echo -n > /var/lib/apt/extended_states
RUN rm -rf /var/lib/apt/lists/*
RUN rm -rf /usr/share/man/??
RUN rm -rf /usr/share/man/??_*

# Configure php-fpm to not run as a daemon
RUN sed -e 's/;daemonize = yes/daemonize = no/' -i /etc/php5/fpm/php-fpm.conf
RUN sed -e 's/;listen.owner/listen.owner/' -i /etc/php5/fpm/pool.d/www.conf
RUN sed -e 's/;listen.group/listen.group/' -i /etc/php5/fpm/pool.d/www.conf

# Configure nginx to not run as a daemon
RUN echo "daemon off;" >> /etc/nginx/nginx.conf

# This next line lets nginx write to your working directory
RUN usermod -u 1000 www-data

# Configure nginx virtualhost
RUN rm -Rf /etc/nginx/conf.d/*
RUN rm -Rf /etc/nginx/sites-available/default
ADD ./nginx-vhost.conf /etc/nginx/sites-available/default.conf
RUN ln -s /etc/nginx/sites-available/default.conf /etc/nginx/sites-enabled/default.conf

# Add the application code into the container
ADD . /srv/www

# Configure Supervisor
ADD ./supervisord.conf /etc/supervisor/conf.d/supervisor.conf

# Fix permissions
RUN chown -Rf www-data:www-data /srv/www/

# Set our working directory
WORKDIR /srv/www

# Expose Ports
EXPOSE 80

# And finally, run the command to kickstart everything
CMD ["/usr/bin/supervisord"]

The above file is telling Docker how to build the container. It includes nginx and PHP-FPM, but it leaves database and cache store to other containers. The container's main job is to serve the Laravel application at /srv/www in the container on port 80 using nginx and PHP-FPM.

docker-compose.yml

This file is what Docker Compose will use to build our stack and link up the containers the app needs to function. It defines three containers - web, db, and cache. The “volumes” part of the web container's definition here will map your source code directory on your host to the container's app directory, which will let you update your container by editing files on your host without having to rebuild the container. Under the db container, you'll see the “environment” section. This is a list of environment variables that will be set in the MySQL container. The two blank ones will use the environment variable of the same name on your host to set them inside the container, which is far more secure than putting these values into version control! Just make sure that you set them! That'll happen in your bash/zsh profile, with lines like this:

export MYSQL_ROOT_PASSWORD=supersecurepassword1
export MYSQL_PASSWORD=anothersecurepassword2

Alright. Without further ado, here's the docker-compose.yml file:

web:
 build: .
 ports:
 - "8080:80"
 volumes:
 - .:/srv/www
 links:
 - db
 - cache
db:
 image: mysql:5.6
 environment:
  MYSQL_ROOT_PASSWORD:
  MYSQL_DATABASE: dockerapp
  MYSQL_USER: dockerapp
  MYSQL_PASSWORD:
cache:
 image: memcached:latest

Configure your Laravel app

Laravel 5.0 and later uses a .env file to configure the app. This is where you'll give your app the settings it needs to talk to the other services (in this case, memcached and MySQL).

.env

APP_ENV=local
APP_DEBUG=true
APP_KEY=dhBKg8eSVZ3tH5yuz6z40j13X13YWTJf

DB_HOST=db
DB_DATABASE=dockerapp
DB_USERNAME=dockerapp
DB_PASSWORD=anothersecurepassword2

CACHE_DRIVER=memcached
SESSION_DRIVER=memcached
QUEUE_DRIVER=sync

MEMCACHED_HOST=cache

MAIL_DRIVER=smtp
MAIL_HOST=mailtrap.io
MAIL_PORT=2525
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_ENCRYPTION=null

Run your new application container

Alright, we're just about ready for the finale. First we need to run Docker Compose against our app to get it running.

docker-compose build

This will build your container and fetch the other two from Docker Hub. It shouldn't take too long - the longest part will be downloading mysql and memcached containers. Then, put your containers online:

docker-compose up

Now try visiting http://192.168.99.100:8080 in your browser of choice. You might have to change the IP address to match that of your Docker machine (docker-machine ip default will give you that). You should see the Laravel welcome screen, and your work here is complete. Note: don't worry that your command prompt didn't come back. This runs in the foreground, so you'll need to use a new tab in your terminal app for other things. Or, you can use the daemon mode instead:

docker-compose up -d

Congratulations, you're now working with Docker!

Go Forth and Search

Ben E. C. Boyter — Thu, 03 Sep 2015 01:25:18 GMT

A very fast update. At the request of the excellent Lars Brinkhoff via GitHub I have added in the language Forth to be one of the supported languages inside searchcode.

An example search which shows this working would be the following https://searchcode.com/?q=forth&loc=0&loc2=10000&lan=181

I had to solve a number of interesting problems inside searchcode to support this change. For pragmatic reasons the way searchcode identifies what language any piece of code is written in is to run it though CLOC (Count Lines Of Code). Written in perl it does a reasonably good job of pulling out metadata for any given piece of code. However since my perl ability is poor at best submitting a patch to support forth was not going to be an option.

Exporting Documents from KnowledgeTree 3.7.0.2

Ben E. C. Boyter — Wed, 02 Sep 2015 01:13:04 GMT

I was recently tasked with exporting a large collection of documents from KnowledgeTree (KT) for a client. The collection was too large to use the download all functionality and too wide to attempt to export each folder individually.

I had played around with the WebDav connection that KT provides but it either didn’t work or was designed deliberately to not allow exporting of the documents.

I looked at where the documents were stored on disk but KT stores them as numbered files in numbered directories sans extension or folder information.

OS X Software Update Channels For Betas

smcleod.net — Tue, 01 Sep 2015 03:00:00 GMT

Set update channel to receive developer beta update

sudo softwareupdate --set-catalog https://swscan.apple.com/content/catalogs/others/index-10.11seed-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz

Set update channel to receive public beta update

sudo softwareupdate --set-catalog https://swscan.apple.com/content/catalogs/others/index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz

List available updates

sudo softwareupdate --list

Set update channel to receive default, stable updates

sudo softwareupdate --clear-catalog

Show current settings

defaults read /Library/Preferences/com.apple.SoftwareUpdate.plist

Write setting manually

defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL https://swscan.apple.com/content/catalogs/others/index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz

Decoding CAPTCHA's Handbook

Ben E. C. Boyter — Wed, 26 Aug 2015 01:49:31 GMT

Some time ago I wrote an article about Decoding CAPTCHA’s which has become what appears to be the first resource most people encounter when searching for information in the decoding CAPTCHA space.

I had continued to write about CAPTCHA’s over the years with posts scattered around the web. A while ago I started to consolidate all of my content on this blog and realised that I had considerably more CAPTCHA related articles then I thought. Some were in an unfinished or unpublished state. I had considered posting them all online but instead decided to polish it all up into a much better resource and publish it as a book.

Zero Downtime Deployment with AWS ECS and ELB

Brightball Articles — Tue, 25 Aug 2015 03:00:00 GMT

As development teams push farther toward continuous delivery, deploying updates to an application without disruption to users is constantly becoming a more sought-after practice. Amazon’s EC2 Container Service helps to make that easier than ever with tight Elastic Load Balancer integration.

Running three hours of Ruby tests in under three minutes

Ben E. C. Boyter — Tue, 18 Aug 2015 11:18:47 GMT

Recently the very cool hard working developers working on Stripe released a post about how they modified their build/test pipeline to reduce their test suite runtime from 3 hours to about 3 minutes.

The article is very much worth reading, as is the discussions that have come around it including those on Hacker News.

A few key takeaways,

For dynamic languages such as Ruby or Python consider forking to run tests in parallel

The Four Character Group Archetype

Tilting at Windmills — Sat, 15 Aug 2015 03:00:00 GMT

I've noticed that when a work of fiction focuses on a group of four characters, the roles those characters play within the group fall into a…

How to Use Heroku PGBackups

Brightball Articles — Tue, 11 Aug 2015 03:00:00 GMT

Backing up your data is one of the most critical activities for your application. Heroku PGBackups makes the entire experience pretty simple but comes with a lot of flexibility too, with a number of options for smooth restoration.

A Culture of Quality

Ben E. C. Boyter — Mon, 10 Aug 2015 11:17:33 GMT

The best working environment I had the pleasure to work in had a strong emphasis on testing and software quality in general. Product teams were encouraged to spend extra time ensuring that everything worked over shipping before it was ready. The transformation it went through was incredible. Having come from a culture very much wild west through to where it was. An example of the advantages this brought was that before adopting this culture website launches were a traumatic event. Teams would block out 48 hours stretches and work solid fixing bugs at go live. Very stressful for all involved and not a healthy working environment. Contrast to where things ended up where several websites were launched on the same afternoon without a hitch. Very impressive considering the scale of the websites being dealt with (several million uniques a day).

C# as a Language from old Google+ Post

Ben E. C. Boyter — Thu, 06 Aug 2015 01:37:38 GMT

The more I use C# as a language for writing things the more I am convinced that its approach really is the best language approach out there.

The unit test support is excellent which allows development speed to be just as fast as any dynamic language (Python, PHP, Perl).

The static typing catches so many issues before you get to runtime and allows sweeping changes without breaking things.

Unlike Java it has the var keyword (saves time and improves readability) and so many more useful functions which yes you can replicate but are just built in and work correctly.

Making army lists in Age of Sigmar

Ben Overmyer's Site — Mon, 03 Aug 2015 03:00:00 GMT

Making an army list in Age of Sigmar is an unusual process for those of us used to the traditional Games Workshop methods. In Warhammer Fantasy you decided on a points value, an army, and then tried to match a unit selection based on force organization rules and a balanced strategy to defeat as many types of opponents as possible. Those opponents would most often be similarly organized. In Age of Sigmar, however, there are only two ways to build for “strength:” by sheer force of numbers, or by maximizing keyword synergies. The sheer-force-of-numbers (a.k.a. pay-to-win) approach will not win you any friends, and may get you banned from games. The keyword synergy approach, though, is as yet an undiscovered country. Everyone is starting to pick up on the idea that you can take certain units to play on their collective strengths. As the community finds its feet in this new game, strong strategies are bound to come out.

For example, the Dark Elf Cold One Knights are somewhat lackluster by themselves. However, in the presence of a Dreadlord on Cold One they (and Cold One Chariots) reroll charge rolls and add 1 to Wound rolls for one of their attacks. It's combinations like this that will define Age of Sigmar army lists. While you can make all kinds of interesting setups, it's this system that's going to draw the tactical minds out of its players. I'm reminded heavily of Magic: the Gathering or Hearthstone here. Both of those games encourage players to build winning strategies on specific synergistic themes that amplify their benefits, while not discouraging taking completely odd selections for flavor or for backup strategies. For my part, I intend on running a lot of Dark Elves (or Exiles, as they're now called) as the core of my first army. There's a lot to love there.

Unit Tests Manage Complexity

The Cranky Developer on Crater Moon Development — Sun, 02 Aug 2015 03:00:00 GMT

Unit testing is more than a design tool, not just a practice that enhances quality, it is a tool for managing complexity.

Implementing AES

Krishna's blog — Thu, 30 Jul 2015 03:00:00 GMT

This article explains how to implement the Advanced Encryption Standard (AES) algorithm

A/B Testing

Ben E. C. Boyter — Mon, 27 Jul 2015 11:11:59 GMT

A/B testing is testing the comparison of two outputs where a single unit has changed. It is commonly used when when trying to increase conversion rates for online websites. Also known as split testing an example would be trying to increase user clicks on a specific button on a website. You may have a theory that red buttons work better then green. You would try out both against real users and see which one performs better.

A guide to analyst relations for startups

CRAIG KERSTIENS — Sat, 25 Jul 2015 23:55:56 GMT

When it comes to go to market and marketing there’s lots of pieces in a toolchest that all work together. One that comes a bit later, but if used properly (much like a PR agency) can be valuable is industry analysts. And while working with a PR agency can quickly start to become clear. How to work with analysts so it is productive on both sides can take a bit longer to figure out, or at least it did for me. Even before you do start working with them there’s the question of if or when should you. Here’s hoping this primer makes it a bit faster and easier for others.

What is an analyst

Apologies to all analysts, but of all parts of this post I might butcher this one

Analysts talk to a lot of companies, both the ones making products as well as the ones purchasing them. I’m not actually sure what the spread is I’d guess 80-20. A large output of this and other activities is creating various reports and rankings. Gartner’s Magic Quadrant is probably the most well known industry ranking. Much of what they create isn’t freely available for consumption so you likely don’t see the sheer volume of insights they put out.

Why would you engage with an analyst

So what do they do for you? There’s really two major buckets:

Help with sales/marketing - Given they’re informing and influencing buying decisions of businesses they can be one more person on your side. If a launch in Techcrunch makes business foo aware of product bar, then an analyst report or ranking can help sway a decision on whether to try bar vs. baz.

Consulting - The other major opportunity is for the analyst to give some form of guidance. In a larger company when you already have an established product they should absolutely be part of your launch process (more on that in a future post). They’re actively following your market and space, hopefully just as you are to some extent. They can offer an outside perspective and help with broad areas of focus and messaging.

More - In reality it’s as clean cut as above. They may be able to introduce you to good candidates for hiring. They may be able to introduce you to a large company interested in acquiring some capability which you have. They may be able to connect you with investors. All of these things can and do happen, but the above buckets typically are the primary drivers.

When?

First, engaging with analysts should always come after you have some confidence in the product, after you’ve started some marketing drumbeat, and after sales. In short don’t be in a huge rush here, you’ll get there, but don’t be in too big of a rush. As you start to get some attention and momentum it’s just as likely they’ll engage with you first as you reaching out. Also, marketing != sales, more on that in a future post.

But, let’s assume you’ve got a product which targets business (Analysts aren’t just for tech companies, though you’ll see the benefit here sooner if you’re say a database company than a HR product). Let’s also assume you’ve got some sales and have some good launches under your belt. As it starts to come up in sales calls if you’re in any industry reports or rankings that may be an indicator, if you’re hearing about other competitors having more validation in such reports. As a general rule of thumb once you’ve got inhouse PR they should be able to help guide and steer to the right time.

So how do you engage with an analyst

If you’re engaging in some form of report or article, that should start to be pretty self explanatory. They’ll send you a questionnaire, you fill it out. You go back and forth a little bit.

However, the majority of my interactions aren’t on those articles and reports, for ever one time I fill out lots of questions to help some report or ranking I have 20 calls with an analyst.

There are two primary calls you can have, an inquiry and a briefing.

Inquiries

Inquiry is just a fancy word for consulting call. An inquiry you will always be paying for.

A small detour here. The regularity and consistency in which you engage with an analyst makes a difference. They’re also people at the end of the day, so while firms have certain styles it’s even further multiplied by being very people driven. In your interactions you’ll have a different rapport with different people, it’s at a minimum important to be aware of this.

So back to an inquiry. Within an inquiry your goal is to pull back the curtain and give some backstage insights into what you’re doing and where you’re headed. This is typically under NDA and trust the NDA of an analyst. It’s worthwhile to be as candid as you can here, yes it feels weird, but you’ll get the most value. They’re not like that of a reporter looking for a scoop (not that you can’t trust reporters, just know if you say it, it’s on record). You don’t have to relish the entire call to one area, but areas of coverage are often:

Upcoming products and major releases you’re working on
Broader strategy and roadmap
Get input on what they’re seeing and hearing from customers

Briefings

The other type of call we have is a briefing. This is a little similar to that of a press briefing. You’ll get on the call, and walk through some upcoming launch or just give an update on your company and progress. The latter is more common if they’re unfamiliar with you or your product.

Analyst briefings are good to do earlier than your press briefings, compared to press they’re like a bike with training wheels. It’s best if you still maintain your balance–the ride will be smoother, but there’s a little less risk of completely toppling over. One key difference is you often have a powerpoint deck you get to walk through during an analyst briefing. I’ve found this is helpful for pacing and key messages, I used to be skeptical, but now very much feel it’s always worth doing.

Pro-tip: You can create a deck and use it for press too, no they won’t want to get on a gotomeeting, but you can send it over so they have more content later. BUT, more importantly you can also walk through it on your own screen if it helps with pacing.

Within a briefing you’ll have some ability to ask them questions at points. Does this resonate? Are you hearing similar? What are you seeing in the market? Don’t turn it into an inquiry, but knowing the parts that hit home for them allow you to refine your pitch for the next call.

Engaging - the tactical parts

“Analysts are pretty much paid to talk and write” - @cote. So expect that often when you occupy their time there’s a price to it. In terms of finding them it should be pretty easy, to know the list of ones in your space, you may see them quoted or reference in various media outlets. You may just naturally crop up in a report.

If you create a regular relationship with them you’ll have some contract of hours over the course of a quarter or year. At an early stage company this is often owned and manage by whomever runs your PR from an internal perspective.

Conclusion

If you’re about to engage with analysts for the first time or haven’t figured out how to get the most out of your interactions I hope the broad overview is helpful. If there’s some glaring parts you feel I’ve missed let me know @craigkerstiens. And for further reading/watching I’d encourage checking out the great talk from @cote in the Heavybit library.

As far as take-aways and a recap:

Don’t be too eager to jump in with analysts. They can absolutely provide value, but you have to put some time in before it really starts to pay off. It’s not an overnight change and takes building a rapport with them.
At the same time, analysts can be useful in many B2B areas not just tech ones.
When in an inquiry be open and as transparent as possible.
Powerpoint/Keynote/Google presentations are useful in briefings, even if it’s just for you to follow along.

iSCSI Benchmarking

smcleod.net — Fri, 24 Jul 2015 03:00:00 GMT

The following are benchmarks from our testings of our iSCSI SSD storage.

67,300 read IOP/s on a VM on iSCSI

(Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)
Per VM and scales to 1,000,000 IOP/s total

root@dev-samm:/mnt/pmt1 128 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=read
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
bs: 1 (f=1): [R] [55.6% done] [262.1M/0K /s] [67.3K/0  iops] [eta 00m:04s]

38,500 random 4k write IOP/s on a VM on iSCSI

(Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)
Per VM and scales to 700,000 IOP/s total

root@dev-samm:/mnt/pmt1 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
bs: 1 (f=1): [w] [26.3% done] [0K/150.2M /s] [0 /38.5K iops] [eta 00m:14s]

Raw device latency on storage units

Intel DC3600 1.2T PCIe NVMe

root@s1-san6:/proc  # ioping /dev/nvme0n1p1
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=1 time=104 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=2 time=83 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=3 time=51 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=4 time=71 us

SanDisk SDSSDXPS960G SATA

root@pm-san5:/proc  # ioping /dev/sdc
4.0 KiB from /dev/sdc (device 894.3 GiB): request=1 time=4.2 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=2 time=4.1 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=3 time=4.1 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=4 time=4.1 ms

Micron_M600_MTFDDAK1T0MBF SATA

root@pm-san5:/proc  # ioping /dev/sdf
4.0 KiB from /dev/sdf (device 953.9 GiB): request=1 time=157 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=2 time=190 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=3 time=65 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=4 time=181 us
```shell
## Latency on the a VM

- (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)

```shell
root@dev-samm:/mnt 127 # ioping pmt1/
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=1 time=0.6 ms
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=2 time=0.7 ms
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=3 time=0.7 ms

--- pmt1/ (ext4 /dev/xvdb1) ioping statistics ---
3 requests completed in 2159.1 ms, 1508 iops, 5.9 mb/s
min/avg/max/mdev = 0.6/0.7/0.7/0.1 ms
root@dev-samm:/mnt  # ioping pmt2/
4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=1 time=0.6 ms
4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=2 time=0.8 ms

--- pmt2/ (ext4 /dev/xvdc1) ioping statistics ---
2 requests completed in 1658.4 ms, 1470 iops, 5.7 mb/s
min/avg/max/mdev = 0.6/0.7/0.8/0.1 ms
root@dev-samm:/mnt  # ioping pmt3/
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=1 time=0.6 ms
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=2 time=0.9 ms
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=3 time=0.9 ms

A guide to PR for startups

CRAIG KERSTIENS — Tue, 21 Jul 2015 23:55:56 GMT

You’ve built your product and you’re now ready for your first major launch. Or you’ve been through a launch or two, but are looking to scale the process as you’re doing more launches and announcements. You really have two options: do it all on your own, or work with a PR agency. One frequent crossroad is that you’re not at the point of a full time PR person, but unsure what a PR agency can offer you; and, further what’s the best way to work with them so you’re getting the maximum value.

As I’ve talked to more startups lately, it’s become clear that effectively working with PR teams and the media is mostly learned by doing. Because there’s not much guidance out there, here’s an attempt at some basic guidelines.

On PR

First there’s two types here and they’re not mutually exclusive. In-house PR is a full time person or team that works within your company, here you’ll often have a pretty different experience. From my experience, in-house PR people tend to understand a company message and vision because they are living and breathing your company values every day.

The other alternative is hiring a PR agency. An agency will have several (sometimes hundreds!) of clients. The relationship that you’ll have with an agency is much different than in-house. You’ll use them just like you would a consultant or contractor. Most startups end up with the agency approach first, because of the perception of “more people working for a cheaper cost than hiring in-house.” However, it’s of note an agency doesn’t alleviate you of doing work, nor should you want them to handle all parts of it.

Messaging

An agency may offer to help with messaging, but take this somewhat lightly. I don’t doubt that some are very good at it, but in most cases I’ve found they don’t have the same amount of customer interaction as you as a founder or early employee would. Further, your vision of impact to the market and direction may be more distant than theirs. You should expect to own your messaging, just like you own your product.

Where they can heavily help is providing a lot of structured frameworks for helping you get to your messaging. Some pretty basic templates of standard questions for customers and partners can go along way in helping you actually uncover what they feel your value is.

On your key messaging/value prop, there’s two pieces I’ll drop in here. While I’d love to write another long post on it, I wonder when I’ll actually get it out. So the first is pitch the problem you’re trying to solve–Dave McClure talks about this as well as anyone. The second is don’t pitch features, pitch the use cases and solutions. Pitch what’s possible

Pitching

This is the number one area I’ve found that having PR makes a huge difference. In the world of reporting, different reporters have different beats (areas of coverage), styles, outreach preferences, and most importantly, different relationships with companies and people. Knowing all of this and how to pitch a story to them is key. Yes you can spend hours researching and creating a perfect story just for them, and do that again, and again and hopefully land some coverage. But I’d argue a bit: that’s not the best use of your time.

With a good PR person or agency you’ll be able to strike a mix of:

Here’s the outlets I want to be in and why (have a good reason for why).
Understanding the audience and readership.
What outlets you feel like your key customers are reading, and validate this with the agency.

From there, if you’ve found a good agency they already have relationships with your key journalists / publications. So if you have a compelling product, you just need to give them the right messaging of the particular launch or news.

What else to expect from your agency

A surprise for some is how the whole process works. The agency is going to be there on the phone with you. You’re not going to hang out over beers while pitching being chummy. The reporter is listening to multiple other pitches, it’s likely they had one right before you and right after. The agency is there listening, helping keep time and track of conversation for reporter fact-checking after the interview.

Hopefully they’re also keeping notes. They should be able to provide you with some high level notes of what message resonated with each reporter and what didn’t, what you covered, and what they asked. This is especially useful for future interactions.

Similarly you should get a briefing 1 pager ahead of time. You should be able to skim this, you don’t have to memorize. But it’ll include key things about recent articles written by the reporter, their beat, topics to dive into and ones to stay away from. If you can connect the dots, those notes from an initial call start to feed into the 1 pagers for future calls.

Onto the briefing

Of course it’s important to land the briefing in the first place, but just as important is getting it right. Coming into it, the reporter will have already gotten the high level pitch… It’s why they took the call. You’ll get a mixed bag of those that are open to teeing up the opportunity to those that want to get right to the news. Roll with what they prefer, but also don’t be afraid of trying to hit some of your key points.

Have your key messages ready

Sound bites help hugely here. Analogies, customer references, whatever you want to hit. Have it ready. Also if you’ve got a great sound bite that helps tell the story, it can make the reporter’s job easier. Just don’t swing too far into happy go lucky marketing land. It’s important to remember that you’re talking to a person. Have a conversation - don’t talk at them.

Go slow

It may seem obvious when you think about it, but as you’re talking the reporter is writing. Or at least you hope they are. Some do it by hand and type up notes late, some type right then and there. When you hear a pause it doesn’t always mean to keep going and it seldom means hurry up. Become extra comfortable with pauses. Check in if you’re going to fast, if they’re following, if they have any questions. I’ve had people bring me in a beer before because I’d had multiple cups of coffee through a few pitches, and they were trying to slow me down a bit. Know your pace, and then slow it down.

Questions

It’s okay if they don’t have a lot of questions, they may not. They may have none at all. Yes, pause, and give them a chance, or even ask if they have any. But don’t stress too much if they have no questions.

On the flip side of that - you’re PR person should have prepared a list of questions for you beforehand that the reporter could possibly throw your way. Be sure you’ve thought through and practiced all the Q&A scenarios before the interview so you aren’t caught off-guard when you’re in front of the reporter.

In conclusion

If it’s your first go around, don’t stress too much. Have the headlines you want in your mind and key messages, or better yet write them out. Personally I write key things on a whiteboard nice and large before I’m on the call. Finally once you’re all done, enjoy reading the coverage. But you’re not all done after you get some coverage look back, run a retrospective just like you would for a software project. What worked well, why did or didn’t something work. What can you improve next time.

*Full disclosure, this is based across interactions with a small sample size of different PR agencies and individuals. Mileage may differ heavily from PR firm to PR firm, but hopefully the above provides at least some roadmap for more clarity vs. flying blind. As always if you’ve got feedback/questions, feel free to let me know @craigkerstiens

Finally a special thanks to Paul Katsen for much of the inspiration on creating this post and to he and Katie Boysen for review

Five ways to avoid and control flaky tests

Ben E. C. Boyter — Tue, 21 Jul 2015 11:09:42 GMT

Having a reliable test suite should always be the goal in software development. After all if you can’t trust the tests then why bother running them at all? This is especially important in a shared coding environment and when running through Continuous Integration (CI).

1 - Test in Isolation

It may seem obvious but writing focused tests which do a single thing is one of the most effective ways to avoid them being flaky. Tests which do multiple things increases the chance for failure and can make the tests non deterministic. Always remember to test features and issues in isolation.

Delayed Serial STONITH

smcleod.net — Tue, 21 Jul 2015 03:00:00 GMT

A modified version of John Sutton’s rcd_serial cable coupled with our Supermicro reset switch hijacker:

This works with the rcd_serial fence agent plugin.

Reasons rcd_serial makes for a very good STONITH mechanism:

It has no dependency on power state.
It has no dependency on network state.
It has no dependency on node operational state.
It has no dependency on external hardware.
It costs less that $5 + time to build.
It is incredibly simple and reliable.

Essentially the most common STONITH agent type in use is probably those that control UPS / PDUs, while this sounds like a good idea in theory there are a number of issues with relying on a UPS / PDU:

Why Does Software Contain Bugs?

Ben E. C. Boyter — Mon, 20 Jul 2015 11:06:50 GMT

“Why does all software contain bugs?” this was a question recently asked of me. My response at the time was because all software is not perfect, but is this true?

Lets take a very simple example.

    public class Hello {
        public static void main(String[] args) {
            System.out.println("Hello World!");
        }
    }

The above example is arguably the simplest program that can be written using Java. It also happens to be the first program usually written by any Java programmer. It simply print outs the text “Hello World!” when it is run. Surely this program is so simple that it is perfect and therefore bug free?

No such thing as "real programming"

Brightball Articles — Mon, 20 Jul 2015 04:34:00 GMT

I read an article earlier today called The self-hating Web Developer that I found on Hacker News and it bothered me. It resonated with me as something that I professionally internalized over my career but it bothered me because Joseph encountered personal financial difficulty for both himself and his family due to the struggle. For that reason (and insomnia), I feel compelled to write this as reference to refer to for anybody else who may be struggling with the same thing.

HillHacks 2015

Nemo's Home — Mon, 20 Jul 2015 03:00:00 GMT

A little while back, I came across HillHacks, a conference in Dharamshala about “hacking and making in the Himalayas”. I was instantly hooked. It took a lot of scheduling troubles, but I decided to stay for the entire unconference, which started at 23rd May.

Its hard to describe the HillHacks experience in a single blog post. I met so many amazing people from all over the world. Learned a lot of different things. I Had a lot of fun teaching some other things. I helped organize some of the stuff, and managed to stay awake an entire night while participating in an CTF. And on top of that, got to eat delicious food.

HillHacks as an event, was divided into two segments:

An unconference (23rd May - 3rd June)
Main Conference (4-7 June)

A lot of people had arrived before me at the venue and taken care of the basic infrastructure. We had internet connectivity via two local ISPs. We had IPV6 connectivity via a tunnel in Belgium as well.

There were a lot of fun activities planned everyday: from unicycling to skateboarding and playing Cards against Humanity; it was a lot of fun living with so many strangers and trying to figure out ways to help.

I did a talk on SDSLabs, a quiz for everyone, and an introductory session on CTF contests. We then participated in a CTF organized in Germany as Team HillHacks. On the last day of the conference, I did a [talk][josd-talk] on “The Joy of Software Development”, which is a book I am working on.

For the first time in my life, I met people who actually use BSD. And to make it even more amazing, I met NetBSD Kernel developers, people on the BSD Security Team, and people who prefer OpenBSD over NetBSD (I’d never really cared for the distinction, as a Linux user)

We did a lot of hacks, including running an MPD Daemon and streaming it over IceCast. I also spent a lot of time cubing and teaching people how to solve Rubik Cubes. My times have also improved somewhat as a result. Thanks to trouble, I also learnt how to solve a MegaMinx.

As part of the School Outreach program (organized by the brilliant Tink), we taught kids about Codes and Ciphers, programming, speedcubing and lots of other things. The kids also performed in the final Gala Show giving us brilliant performances in 3 different plays (all 3 schools had their own plays).

I learned a lot of different things: how to start with Kernel Programming, DNSSEC, Retro Gaming. Thanks to a few dedicated volunteers, we even made a 8-inch Telescope that made staring at the night sky so much fun. We had a session on Typography, a story telling session in Malayalam (translated to English on the fly). I even learnt a bit of Emacs.

The list is so long, I don’t think I can do it justice in this single blog post.

The most amazing part was not the technical things, but the community itself. sva would often say that everyone of us has “sudo access on the conference” (geekspeak for full authority). Each of us helped organize it, any way we could. The community got together to setup the stage, tents, network and the entire infrastructure at HillHacks. Zainab even has a blog post on social cooking at Hillhacks.

As I sit here at the venue, it has been 2 weeks of fun and awesomeness here at HillHacks. I leave with lots of memories and hope to be here next year.

If this blog post interests you, be sure to check out hackbeach as well. We are doing a mini-conference around November in Kovalam.

Age of Sigmar breaks Warhammer Fantasy, but it's worth it

Ben Overmyer's Site — Sun, 19 Jul 2015 03:00:00 GMT

Age of Sigmar grabbed the Warhammer Fantasy rulebook, tore it apart, and threw the remains into the fire. I love the game that came out of that, even if there are some significant negatives to this event.

From simulationist roots to casual narratives

Warhammer Fantasy has long been the strategy-heavy cousin to the tactical-heavy Warhammer 40,000. The rules up until this point focused on maneuver and regiments, positioning and thinking ahead. Warhammer 40,000, on the other hand, has always been about skirmishes and characterful, tight battles. Where Fantasy relied on its movement trays and slower pace, 40k allowed for a lot of narrative to come through the gameplay. Age of Sigmar one-ups 40k in this regard. Yes, Age of Sigmar has a lot of holes in the rules. Someone who delights in using rules-as-written against his opponent is going to have a field day with Sigmar. But that's not what this iteration of the game is about. Age of Sigmar seeks to bring casual gaming to the fore. The rules explicitly encourage, via its Most Important Rule, house-ruling and collaborating with your opponent to set up the battle.

Play any army you come up with

One of my favorite things about the rules is its encouragement of creativity in army building. There are no points or required force organization. While this can obviously be abused by asocial types, for those of us that enjoy narrative battles, it's a godsend. I can mix Seraphons (Lizardmen) in with Stormcast Eternals and Exile Aelfs (Dark Elves), and that's perfectly legitimate. If I really wanted to, the rules allow me to field Chaos Warriors alongside Sylvaneth (Dryads) and Duardin (Dwarfs). I can't imagine the scenario where that would make lore sense, but you could do it. All of this makes for some fantastic battle and lore possibilities.

My God, it's beautiful

The miniatures in the starter box set are absolutely gorgeous, and make the collector in me very happy. The Lord Celestant in particular is a beautiful model. With his triumphant pose and intimidating Drakoth mount, he makes a fine addition to any army. The Khorgorath model on the Chaos side is really ****ing creepy. It's made of blood and skulls of its victims are pushing their way through its skin.

There's a free app for army building

This is a big one. I bought Army Builder ages ago to help me build 40k armies, but Games Workshop is notorious for never releasing anything for free. Privateer Press released a free app awhile ago for its Warmachine and Hordes games, and while you need to pay for your specific army, it does have the benefit of being a first-party app. Now, GW has released the Age of Sigmar app that includes an army builder and the Warscrolls for all existing models for free. The one thing that pisses me off about this app is that I don't get the Warscrolls for the starter box units, despite having shelled out $125 for said box. There's no way to redeem a product code or anything. The only way to get those Warscrolls in the app is to buy the Age of Sigmar book through it, which would be an additional $50. No thanks. But it's GW, so I'll give them points for the progress they did make.

...tournaments?

Yeah, the Age of Sigmar rules pretty much break any possibility of having tournaments without extensive house-ruling. That's a problem for people who're really into the game. I'll admit to being a little hesitant about this. I've never played in a 40k tournament, but I think it'd be fun to try. I can't imagine playing in an Age of Sigmar tournament, though. The rules aren't geared for that.

What about the people who enjoyed the old game?

Games Workshop's move to completely get rid of the old rules and cease supporting it in their stores is a bad one, despite the good game that Age of Sigmar is. The community that built up around WHFB is not so easily dismissed. For some people, their local Games Workshop store is the only place they could play the game they'd put a lot of time and money into, and I've heard rumors that GW stores are disallowing WHFB matches now. That's a terrible policy. Mantic Games has their Kings of War game, which keeps very closely to WHFB's roots. They just released a second edition, and have been inviting WHFB players left out in the cold to join their community. This is a great move on Mantic's part, and I really hope it works out for everyone. From what I've seen, Mantic cares a lot more about community than GW does.

So is it worth getting?

Yes. Especially since the rules are free, and the Warscrolls - the rules for individual units - are also free. You can get them on Games Workshop's site. In my opinion, the starter box is also totally worth the $125. It has wonderful minis, and the books are great too.

Migrated away from Heroku

Ben Overmyer's Site — Sat, 18 Jul 2015 03:00:00 GMT

Heroku made their free tier awful. That's fair enough, but I need more stability.

So, I've spun up a Digital Ocean droplet and learned how to deploy Rails apps without the help of a PaaS.

It's not intuitive for someone who's used to provisioning PHP web app servers, but after finally getting it working, it's actually a pretty cool way of doing things.

So yeah. Obelisk, the Ruby on Rails blog engine I've written to replace the PHP-based Monkblog engine, is now running on a server that I've provisioned from scratch. Yay!

searchcode the path to profitability

Ben E. C. Boyter — Fri, 17 Jul 2015 11:01:15 GMT

One of the things that has always bothered me about searchcode.com was that it never generated any money. Not a huge problem in itself as a side project, but the costs to run it are not insignificant due to the server requirements. I had looked into soliciting donations but I considered this highly unlikely to produce enough revenue to cover costs considering that sites such as gwern.net was unable to make enough to cover even basic costs through patreon (although since a recent HN post this has jumped from around $20 a month to over $150).

The benefit of testing for Developers, Managers and the Business

Ben E. C. Boyter — Thu, 16 Jul 2015 11:05:30 GMT

“Fixing regression bugs is analogous to digging a hole only to find the next day it has been filled in and having to dig it out again”

Ask any manager, developer or tester working on software without tests what the main pain points are. Nearly all the time the main one mentioned is dealing with regressions. This is bugs that were fixed a year ago which returned. Regression bugs cost the software industry billions of dollars a year. Worse still they are demoralising to everyone involved. Finding or fixing the same bug over and over causes you to start looking for new projects or new jobs.

Video - Cluster Failover Performance Demo

smcleod.net — Sun, 12 Jul 2015 03:00:00 GMT

AWS EC2 Instance Types to Use as Test Agents

Ben E. C. Boyter — Fri, 10 Jul 2015 11:03:10 GMT

When you are running test agents on AWS knowing what instance type to run as test agents (for TeamCity or otherwise) can involve a lot of trial and error. Not only can there be great savings to be made by picking the correct instance type you can speed up your builds and get test feedback back faster which can be far more valuable the cost of a few additional cents an hour.

Exploring Microservices Architecture on Heroku

Brightball Articles — Thu, 09 Jul 2015 02:17:02 GMT

Building an application with a microservice architecture is an excellent long-term decision if you can afford the increase in upfront time investment to do it properly. Heroku provides a platform that most developers know for simple deployment, but it also dramatically simplifies microservices architecture.

Issues with Google's Bug Prediction Algorithm

Ben E. C. Boyter — Wed, 08 Jul 2015 10:58:57 GMT

December 2011 the Google Engineering team published a blog post about bug prediction at Google. The topic caused quite a lot of discussion at the time over the internet on forums such as Hacker News and the Reddit programming sub-reddit.

How bug prediction works

In a nutshell the prediction works by ranking files against checking the file commit history and seeing how many changes have been flagged as bug fixes. Of course this means that code which was previously buggy will still appear in the list. This issue was also addressed in the post and the results were weighted over time to deal with this issue.

CentOS 7 and HA

smcleod.net — Tue, 07 Jul 2015 03:00:00 GMT

First some background…

One of the many lessons I’ve learnt from my Linux HA / Storage clustering project is that the Debian HA ecosystem is essentially broken, We reached the point where packages were too old, too buggy or in Debian 8’s case - outright missing.

In the past I was very disappointed with RHEL/CentOS 5 / 6 and (until now) have been quite satisfied with Debian as a stable server distribution with historicity more modern packages and kernels.

What is Usability Testing?

Ben E. C. Boyter — Mon, 06 Jul 2015 10:57:30 GMT

Usability tests are manual tests used to check that the user interface is understandable. The focus of the tests are to ensure that product meets its intended purpose. These sort of tests can be subjective and are usually impossible to automate. It is important to differentiate usability testing from simply showing an interface to someone and asking them “Do you understand how this?”. It is usually done by creating a scenario such as “Can you find and add this song to a new playlist” and observing the steps that the user takes to perform the task.

The One Question Every Prospective College Student Should Be Asking

Tilting at Windmills — Mon, 06 Jul 2015 03:00:00 GMT

Apologies for the clickbaity headline. That sort of thing isn't usually my style. I'll cut right to the chase. The question is "What…

Mutation Testing or How to Test Tests

Ben E. C. Boyter — Sun, 05 Jul 2015 10:56:03 GMT

Mutation testing is a technique used to verify that tests are providing value. Mutation testing involves modifying the given program in small ways. These could include changing boolean checks such as if a condition is True to being False. A mutated version of code is known as a mutant. For each mutant a the test suite is run against it. The tests when run over the mutant version should have a percentage of failure. Where a mutant is not caught additional tests can be written to cover these cases.

Who is Responsible for Software Quality?

Ben E. C. Boyter — Sat, 04 Jul 2015 10:54:08 GMT

In the beginning of my software development career I was interviewing for an intern position at Microsoft. I never did get the job but one think out of that interview process really stuck with. The second interviewer after the usual getting to know you chat aded me the following question. “On any given software project we have developers, software testers / quality assurance and managers involved. Who is responsible for the quality of the software?”. Being young and naive I confidently responded that the QA/testers were. After a long discussion artfully controlled by the interviewer I came to change my opinion. Below is the line of reasoning I went through with him.

Talks

Ben E. C. Boyter — Sat, 04 Jul 2015 09:12:08 GMT

A few of the talks I have given and the slide desks are included below.

Python @ searchcode.com sypy_searchcode.key (keynote) sypy_searchcode (powerpoint)
Decoding CAPTCHA’s for Fun and Profit DecodingCaptchas Reveal.js (futher details can be found http://www.boyter.org/2015/03/decoding-captchas-presentation/)
How AWS and Go Transformed a Public Broadcaster https://boyter.org/static/aws-go-archive-presso/
Sloc Cloc and Code - Can a crusty Go program outperform a well written Rust Project? https://boyter.org/static/gophercon-syd-presentation/ https://www.youtube.com/watch?v=jd-sjoy3GZo
DataEngBytes 2023 - Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100 https://boyter.org/static/dataenbytes2023/
GopherCon 2023 - Bloom Filters: Building a Cutting Edge Go Search Engine to Explore the World’s Source Code https://boyter.org/static/gophercon-syd-presentation-2023/

Heroku PostgreSQL vs Amazon RDS for PostgreSQL

Brightball Articles — Fri, 03 Jul 2015 14:04:23 GMT

PostgreSQL is becoming the relational database of choice for web development for a whole host of good reasons. That means that development teams have to make a decision on whether to host their own or use a database as a service provider. The two biggest players in the world of PostgreSQL are Heroku PostgreSQL and Amazon RDS for PostgreSQL. Here's a detailed comparison.

A story about Hubris and Integration Tests

Ben E. C. Boyter — Fri, 03 Jul 2015 10:51:07 GMT

Philip Dormer Stanhope, 4th Earl of Chesterfield (pictured) managed to embarrass me in front of my peers once. Sort of. In truth it was my hubris that caused the incident. Here is how it happened and what I learnt through the process.

In the summer of 2010 I was tasked with developing a new application where I worked. The requirement was fairly simple “We need a web application to upload a CSV”. Requirements such as this aren’t exactly conducive to a good outcome but I was confident that given the data required to upload it would be fairly easy to do.

How searchcode.com is Unit and Integration Tested

Ben E. C. Boyter — Wed, 01 Jul 2015 10:46:18 GMT

searchcode is a source code and documentation search engine. It allows users to is search over 20 billions lines of code and millions of API documentation. It at the time of writing gets over 300,000 unique visitors a month processes many millions of API requests.

What follows is how I am testing searchcode, the issues I hit and where I think I am getting the most value. As big believer in testing as a methodology of improving code quality, one of my goals when rewriting searchcode was to ensure that it had a comprehensive test suite.

Flaky Tests

Ben E. C. Boyter — Wed, 01 Jul 2015 01:50:04 GMT

A test is considered flaky or flakey if it if fails occasionally. Generally flaky tests are considered to be a bad thing and should be modified to ensure they work correctly every time. This is because a test that is not trustworthy will be ignored even when indicating real failure.

There are many situations that can cause to become flaky. Integration and acceptance tests are generally the tests in your test suite most likely to become flaky. They generally have more integrations across your software stack and as such there are more things likely to go wrong. We going to go through a few of the main reasons and go through in detail what you can do about a specific one.

Clean Testable Repository Data Access in C Sharp

Ben E. C. Boyter — Wed, 01 Jul 2015 01:44:13 GMT

Below is an implementation of an extremely clean data access pattern possible using C# and Entity Framework. It saves you the effort of mocking the database context as the code you end up writing is so simple it is all compile time checked.

The advantages of this are firstly that everything is very easy to test as you can perform all joins in your service layer with mocks of the repository. Secondly it makes your data layer stupidly simple allowing you to forgo writing many tests which would provide little value.

The Unsung Benefits of Software Testing

Ben E. C. Boyter — Tue, 30 Jun 2015 01:35:06 GMT

One benefit that is generally not talked about when discussing testing is the following. The feeling of productivity because you are writing lots of code.

Think about that for a moment. Ask any developer who wants to develop why they became a developer. One of the first things that comes up is “I enjoy writing code”. This is one of the things that I personally enjoy doing. Writing code, any code especially when its solving my current problem makes me feel productive. It makes me feel like I’m getting somewhere. Its empowering.

Testing In Software Engineering

Ben E. C. Boyter — Mon, 29 Jun 2015 01:38:23 GMT

Testing is software engineering is a method of providing information about code quality when developing a piece of software. The intent of writing and running tests is the enforce good software design and identify software bugs and defects. These defects can include specification/requirement errors as well as developer mistakes.

The general aim of software testing is the ensure that software meets the following goals,

Meets its requirements gathered before its design and implementation

Grouping Tests: Unit/Integration vs Fast/Slow Tests

Ben E. C. Boyter — Mon, 29 Jun 2015 01:36:47 GMT

There is a great deal of argument in the testing community over how to label tests. One camp likes to label tests using levels such that unit tests are in one group, integration in another and so forth. The other likes to label them based on how long they take to run ignoring what level they are in. Fast tests are those that run in milliseconds while slow take longer then this. The reason this is important is that when adopting a testing process slow and flaky tests (those which fail often) are the enemy. Slow tests tend to be run less often in the development process. A delay of a few seconds can seriously interrupt a developer or testers workflow. Also the more often your tests fail randomly the less confidence you are likely to have in them, ignoring genuine errors until they fail multiple times.

Sanity Testing

Ben E. C. Boyter — Mon, 29 Jun 2015 01:13:41 GMT

Software sanity tests are closely associated with smoke tests. They attempt to determine if is reasonable to continue with testing a given piece of software. The objective is not to test all functionality, but to determine if there is value in doing so. You can consider it a “Should I continue to test this?” check. Sanity tests differ from smoke tests as they exist to check if new functionality has been met and existing bugs have been resolved.

SSD Storage Cluster - Update and Diagram

smcleod.net — Wed, 17 Jun 2015 03:00:00 GMT

Due to several recent events beyond my control I’m a bit behind on the project - hence the lack of updates which I apologise for.

The goods news is that I’m back working to finish off the clusters and I’m happy to report that all is going to plan.

Here is the final digram of the two-node cluster design:

Plain text version available here

This was generated from the LCMC tool (beware - it’s java!).

Moving past averages in SQL (Postgres) – Percentiles

CRAIG KERSTIENS — Sun, 07 Jun 2015 23:55:56 GMT

Often when you’re tracking a metric for the first time you take a look at your average. For example what is your ARPU - Average Revenue Per User. In theory this tells you if you can acquire new user how much you’ll make off that user. Or maybe what’s your average life time value of a customer. Yet, many that are more familiar looking and extracting meaning from data median or a few different looks at percentiles can be much more meaningful.

And while you can very easily get the AVG in Postgres, with a small amount more effort you can report on percentiles as well. Window functions have been around for some time in Postgres. They allow you to order your result set over a certain group. The most basic example is if you want to order by date, but know which one falls at place 10 in order you can use a window function and project out the rank().

Beyond outputting the rank yourself and doing extra manipulation Postgres has some great utilities to make the most common uses even easier. Being able to compute things such as the perc 95 directly on the data, or lay out for every record in the result where it falls within a percentile is hugely useful. Let’s take a look:

Assuming you have a table called purchases, which has a total in it we could try:

SELECT id,
total,
ntile(100) OVER (ORDER BY total) AS perc_rank
FROM purchases

This would give us something like:

 id | total | perc_rank
----------|---------|-----------
264 | 12034 | 100
643 | 11830 | 100
...
...
304 | 751 | 95

What this would tell us is we have less than 5% of our purchases that have a total over 751. From here you can start to dig in and extract all sorts of different meanings, and by doing directly in SQL you’re closer to the data and have one less processing step.

Percentiles get even more fun with the ordered set functions that came out in Postgres 9.4. They even allow you to project out hypothetical values in certain cases. For now I’d encourage adding ntile to your toolbox anytime you’re analyzing average or medians it will make your world a bit better, and then consider exploring further on the ordered set functions

Thoughts on Writing

Nemo's Home — Sun, 07 Jun 2015 03:00:00 GMT

I have always wanted to be a writer. I think secretly us all reader-folk have that ambition. The joy of getting across your thoughts to another person without ever having met them is enormous.

Most of my writing time these days is spent over email, chat or my not-so-frequent blog posts. I tend to do a lot of research while writing, and it takes up a lot of time. As such, my writing output tends to be diminutive compared to what I’d like.

However, if you’ll go through my blog posts and emails, I write a lot about trivial things. Things that many people have already written about. Things that have probably been discussed to death, and where I have very little chance of actually coming up with something new.

Should I still go ahead and write about it?

This question has been bugging me for a while, especially as a blogger. I mostly write on technical topics these days. For instance, I have given talks on [Software Development], [UX Design], and even [Bitcoin]. I am nowhere close to being an authority on any of these things. Even in my specialized field of Web Development, there are so many things that I’m only barely aware about. So many things I am yet to even form my own opinions about. Topics I don’t even know exist.

When I go and read an article about Software Development from Joel Spolsky, or an article on Security by Bruce Schneier, or something on Startups by Paul Graham, or tptacek on Hacker News; I instantly sit back and take notice: I know their credentials and the fact that they are speaking authoritatively on the topic. However, what can I, a meager undergrad with almost zero experience, write on such topics. Why should I even try, when there are people hundreds of time better who understand these things a thousand times better than me.

In retrospect, this sounds quite similar to the Imposter Syndrome; and I’m not sure if this is exactly the same thing. I don’t get a feeling that I’m a fraud. I totally understand my own capabilities and successes, but the mere fact that there are people far better at what I’m doing is enough to dis-hearten me.

I’ve given this a lot of thought. A really good summary of my response is in the following answer by James Erwin, author of Rome Sweet Rome in a reddit AMA to a question asking for writer advice:

And if you’re going to write, write what you want to write. The odds against any creator are insane. If you’re going to devote months of your time, don’t let it be for an idea you think will sell. Odds are it won’t. Write something you want to write, or need to write. Write for yourself before anyone else. I’d rather read someone who is excited and passionate about what they want to say than someone who’s obviously trying to say what they think I want to hear.

— James Erwin

I write, despite all these doubts, for the following reasons:

Self-learning. A blog is an excellent way to keep track of your self-learning. Its amazing to come back a few years later and see the things you were struggling with before. Its equally amazing to do a trivial google search for an issue you face and find your own blog post or stackoverflow answer on the same.
Sharing Knowledge. Yes, there are people who might know it better, but that shouldn’t mean I should keep my knowledge to myself. That would go against all the values that I stand for.
Network Effect: Not in the strictest sense of the word, but my friend Shashank recently brought this up. I have a circle of people who know me and would vouch for my credentials. For the same reason, they are more likely to trust me as source, instead of a third person who they have no knowledge of.
I love writing. The mere process of putting words down is enchanting for me.

The next question that rises is: “What should I write about?”. Ruling out things I have no clue about, that still leaves a large number of topics I can cover. I am interested in UX Design, Hackers, Computer Security, Software Development, rationalism, skepticism, Free and Open Source movements, Political activism, Technocracy with a passing interest in several other fields such as cosmology and geek culture.

I am not going to pick one every day and write about something new. I don’t want to write something rubbish just for the sake of writing it. I ultimately want to write because I have something to say. It doesn’t have to be unique or ground-breaking. What matters is that I want to write about it.

A few days back someone contacted me on facebook asking me advice on getting started with web development. I get a lot of these queries, mostly over facebook, email, and quora. Our conversation went back and forth with me suggesting resources, and he getting exceedingly confused over whether he should use codecademy or udacity, or coursera or something else.

I have devoted a lot of time in my life to teaching people the nuances of these things. I have mentored many people, and acutely know the issues a beginner faces. In turn, I had an amazing mentor who taught me the importance of always learning things.

All of this lead me to realize one fact: I have been writing a lot about Software Development. Unfortunately, a lot of it is in private emails and chat. And I wanna write more about it, on a public medium.

So, I’m announcing the next thing I’m working on: a book called The Joy of Software Development. A few obligatory links:

The source code is available on GitHub.
Its licensed under the CC-by-SA 4.0 license.
The canonical url for the site is https://josd.captnemo.in/
Its hosted on the excellent GitBook platform, which automatically publishes each version as epub, mobi, and pdf as well.
Who the target audience for the book is.
You can file an issue for critique on GitHub

As expected, all the development, writing, and discussion on the book will be in the public, mostly on GitHub. I am writing this book, because I feel it needs to be written. I don’t expect it to be published, but that won’t stop me from writing it.

Artur Siekielski recently came across it, and wrote the following:

The book you’re writing looks very good! It’s filling a niche as I don’t think there are any modern books that focus on “bird’s eye view”, and I see it would be helpful for many programmers to refresh knowledge.

That gave me a bit of validation, as the only person who’d read it so far were my close friends.

If you wanna support its development, you can do one of the following:

Poke me on twitter or email and let me know you want to read it
Subscribe to the mailing list (I’ll send out updates there)
Watch or Star the repo on GitHub.
See the CONTRIBUTING file on github for contributing to the text.

Asking for Donations might sound weird to some. I don’t really need the money, but I think I’d get an additional sense of responsibility towards finishing it if people start giving me money. I will be donating the entire proceedings to EFF.

Coding Pet Peeves

The Cranky Developer on Crater Moon Development — Wed, 03 Jun 2015 03:00:00 GMT

Eight little things I like to nitpick about in code. Mostly applicable to Java, but some are language-neutral.

Numberphilia

Donat Studios — Tue, 02 Jun 2015 06:13:00 GMT

There's been a fair deal of hubbub in the community about version numbers, including what is and isn't Semantic Versioning, what qualifies as a breaking change and how to use version numbers.

Underscore.js released a second digit "minor update" that was actually a breaking update for some people. They argue it was minor because not very many people would be affected. The community however was angry.

On the opposite side of things, PHP Internals announced they will be skipping version 6 and numbering the next version of PHP 7 to avoid confusion with the never released UTF-16 native version. Again the community was angry.

Using Version Numbers Well

If you are semantically versioning your library, the version represents a contract of trust between you and your developers regarding the public API. The contract declares that nothing you change in a 'less than major' release should break their code. Ever.

Break the contract, you can end up breaking their trust. Break their trust and they'll look elsewhere.

Version Numbers SHOULD NEVER Be Branding

If you're using your version numbers as branding, you are doing it wrong. Version numbers should show a clear history of the public facing interface, not what you sell or advertise it as.

Microsoft of all people understands this. While their version numbers aren't truly semantic, they are aimed at developers - and highly out of sync with the marketing designation because the version numbers denote a true relationship.

You can have both a release number and a sales version. This is the way to go if sales demands control of the version number, such as where I work.

Increment Early, Increment Often

Breaking changes always need to be a first digit release. Always. It doesn't matter if it's not a common use case. It shouldn't matter if you're saving the number for a "woo big new release" version. A breaking change in a minor release violates trust and brings the quality of your software into question.

WHENEVER a behavior or signature of ANY explicitly public member of your API has changed whereas the previous behavior was anything other than bold face wrong it is a first digit release.

Numbers are Cheap

Numbers are cheap. Numbers are infinite. Never reuse numbers, ever. There is no reason.

PHP in my strong opinion did the right thing by skipping 6. They sidestepped ANY potential confusion, minor as it may have been. The argument that they caused confusion about "Where did 6 go" is childish. People will be investing time in the largest digit release, regardless of gaps. No one (save some overly cautious IT people) is going to be using PHP 5.6 and on seeing 7 released think "I'll upgrade to 6".

Trying to not have gaps in your version numbers makes as much sense as if Git were to use ordered numbers instead of hashes.

Version numbers should be a universally unique identifier for a specific set of code. If there is ever any form of contention over what a version number represents, increment. Don't think about it - just do it.

Conclusion

Always avoid confusion; don't ever be afraid to increment your major release number. Only good things can come from it.

Trust is maintained.

HtmlToWord is now WordInserter

Tom Forbes — Sun, 24 May 2015 23:57:06 GMT

I’ve released a redesign of my HtmlToWord library, specifically it now supports Markdown and multiple different ways to interact with Word. It’s now also been renamed to WordInserter to reflect this. Originally HtmlToWord was designed to take HTML input, process it and then insert a representation o...

Video - Storage Cluster Failover Demo

smcleod.net — Thu, 14 May 2015 03:00:00 GMT

A brief demonstration of the failover and recovery process on the storage clusters I’ve been building.

Upsert lands in PostgreSQL 9.5 – A first look

CRAIG KERSTIENS — Fri, 08 May 2015 23:55:56 GMT

If you’ve followed anything I’ve written about Postgres, you know that I’m a fan. At the same time you know that there’s been one feature that so many other databases have, which Postgres lacks and it causes a huge amount of angst for not being in Postgres… Upsert. Well the day has come, it’s finally committed and will be available in Postgres 9.5.

Sure we’re still several months away from Postgres 9.5 being released, anywhere from 3-6 months as a best guess. That doesn’t mean we can’t take a first look at this feature. Though before we get into it a few special call outs of thanks to Peter Geoghegan of the Heroku Postgres team for being the primary author on it, Andres Freund who recently just joined Citus Data for his heavy contributions, and Heikki Linnakangas as well for his contributions.

And now onto the exploration. Upsert is the common name, but if you’re unfamiliar upsert is essentially create or update – Create this new record, but if a conflict exists update it. Let’s take a practical example.

Assume you have a web scraper that imports product information into a table. Each product has a UPC code, title, description, and link. There’s a unique constraint on the UPC code. Now, if your web scraper tries to insert a new product, and a product with the same UPC already exists, you’d usually get an error. But you don’t want the query to fail, you’d want to update the existing product instead. Maybe with a new image, maybe a new description, whatever have you, but I don’t want it to blow up… I simply want to capture the new data and save it.

So before: Insert a record… Exception this violates a unique constraint… Let your app figure out what to do. protip: often applications would try to work around this, but you can run a chance of a race condition and duplicate records if there’s a conflict. TLDR; it’s not a perfect solution.

Now: Insert a record… There’s a unique constraint violation… Okay, let’s just update all the new record’s fields inside a single transaction

So enough explanation, here’s how it actually looks in the syntax:

INSERT INTO products (
upc,
title,
description,
link)
VALUES (
123456789,
‘Figment #1 of 5’,
‘THE NEXT DISNEY ADVENTURE IS HERE - STARRING ONE OF DISNEY'S MOST POPULAR CHARACTERS! ’,
‘http://www.amazon.com/dp/B00KGJVRNE?tag=mypred-20’
)
ON CONFLICT DO UPDATE SET description=excluded.description;

It’s been a long time coming for this, and it makes building applications that need this kind of behavior even easier. While it would have been great for this to be available years ago, kudos to Postgres and its community for taking the approach that is safe for your data. The result we have now both provides the desired behavior of create or update, and is performant without the risk of race conditions for your data.

Google Summer of Code 2015!

spinda's dreamwidth — Wed, 29 Apr 2015 06:57:27 GMT

My proposal for the Haskell's Google Summer of Code 2015 has been accepted! I'm super excited for this. There's a list of the other accepted proposals on reddit, and I've posted my full proposal as a Gist. I'll be working on embedding LiquidHaskell signatures in Haskell's native type system. Woo!

comments

More AUR Packages!

spinda's dreamwidth — Wed, 29 Apr 2015 04:02:02 GMT

New Packages

nemo-owncloud-git, for my port of the Nautilus ownCloud plugin to Nemo

Adopted Packages

gargoyle, an interactive fiction player
nemo-dropbox-git, Dropbox plugin for Nemo
nemo-fileroller-git, File Roller plugin for Nemo
jips, an IPS patcher

comments

Organizing Background Worker Queues

Brightball Articles — Thu, 23 Apr 2015 23:58:00 GMT

At work earlier today I ran across an issue where one of our application queues got backed up and it got me to thinking about how queues are organized in general. The TLDR answer: use urgency and intensity.

Go from a PHP Perspective

Brightball Articles — Fri, 17 Apr 2015 00:06:59 GMT

Here are the slides from my recent presentation to UpstatePHP in Greenville, looking at Go (Golang) from a PHP Perspective.

Talk - High Performance Software Defined Storage

smcleod.net — Wed, 15 Apr 2015 03:00:00 GMT

A high level talk from Infracoders Melbourne on 12/04/2015.

There’s also a low quality recording available here:

Building a high performance SSD SAN - Part 1

Filetree Listing

Ben E. C. Boyter — Wed, 15 Apr 2015 00:32:31 GMT

Just a quick update to searchcode. A few small tweaks here and there, but the largest is that there is now a file tree listing option which will show the file tree for any project. An example would be going to this file and then clicking the “View File Tree” button on the top left.

An example screenshot of the result of this is included below.

HP Support Solutions Framework Security Issue

Tom Forbes — Sun, 12 Apr 2015 02:50:23 GMT

After discovering the flaw in Dell’s System Detect software I looked into other similar software for issues. This post details two issues I found with the HP Product Detection software and explores the protections HP put in place. I’m also going to explain how they could be easily bypassed to allow ...

Ruby on Rails and PostgreSQL Class Slides

Brightball Articles — Mon, 06 Apr 2015 19:03:00 GMT

In August I taught a course titled Ruby on Rails and PostgreSQL - Intro to Advanced in Greenville over the span of 3 weeks. Here is the compilation of slides from the class.

Medium abuses nofollow

Nemo's Home — Mon, 06 Apr 2015 03:00:00 GMT

Update: Since I published this post, I have changed my opinion somewhat on the matter. This post is quite confrontational and I didn’t mean it to be that way. Medium is not wrong in this matter, but I still think we need to look for better solutions. I have since been working on a proposal/idea that would use machine learning to “solve” this problem, instead of side-stepping it.

I call medium a “mostly good” platform for lazy writers. A lot of people have written about its excellent typography, or it being the next “big publishing platform”. I’ve used medium in the past, and while it does have its benefits, I have stopped using it.

My primary reason was that I already have a blog, where I can control the entire experience. This is the same reason why New York Times does not start publishing articles on Medium.

The other reason is nofollow abuse.

Medium hosts more than 1M indexed pages. It has around 650k users currently by a conservative estimate. Rounding it to 700k to account for other users, collections, and other internal pages, it leaves us with around 300,000 articles on medium.

A basic tenet of the web is linkability. That is what Tim Berners Lee meant when he talked about HyperText:

HyperText is a way to link and access information of various kinds as a web of nodes in which the user can browse at will.

Over time, the web has evolved, and is now not just limited to human users, but to computers as well. This is an important consideration on which the web rests today. The biggest example of this is Google Search, which uses these links to “follow, spider, and index” the web. Google uses this linking information to build a citation index, which gives us the quality of a web page depending on the quality and number of sites that link to it.

If you know a bit or two about SEO, you might have heard of shady backlink techniques, which essentially amount to you getting links from an established site. This often takes the form of user-generated content such as comments and answers.

While fighting spam is important, it is far more important to make sure that web remains linked, that people are credited for the content they create. Medium hosts 300,000 articles published by half a million users, and yet none of these links back to external website, because of something called “rel=nofollow”.

When a link has a rel=nofollow attribute, search engines do not count it as a citation in their index. While this may be the right strategy for comments on a wordpress blog to prevent spam, this is not the right way to move forward if you want to “revolutionize the publishing industry”.

While medium is not as bad as some other sites in this regard (like quora, which even blocks the internet archive), it is very important because it portrays itself as a “publishing platform”. This means, medium is made up of articles, blog posts, with lots of outbound links compared to, for instance, StackOverflow answers (which solved this problem back in 2011).

If you publish content on medium, and provide relevant links for your readers, remember that these links are not considered as relevant by search engines.

Medium has said that this is not a top priority at the moment for them.

I understand completely. Handling spam would be a far more harder problem to solve than just blacklisting all outbound links. But we cannot go this way, if we want an open web. We need publishing platforms that cite content, and not blacklist it. This is why I write content on my own blog, and not on medium.

Protecting Users from Phishing and Fraud

Brightball Articles — Thu, 02 Apr 2015 01:47:23 GMT

This presentation covers my experiences combatting phishing and fraud using DMARC and assorted other techniques in a large eBay-like platform for a niche market...when the site previously did everything over direct user email...for over a decade. Good times.

Google Drive: Publicly hosting a static website

Wed, 01 Apr 2015 03:00:00 GMT

I frequently need to demo work in progress web sites and web apps to people I’m working with. I was looking for the most convenient way to do this and found that Google Drive was a good fit to letting me publicly share web development projects. Google Drive provides an easy way to store and share documents in the cloud. The Google Drive desktop app also conveniently keeps a folder on your machine in sync with your online Google Drive folder in the same way Dropbox does.

Dell System Detect RCE vulnerability

Tom Forbes — Mon, 23 Mar 2015 17:16:10 GMT

I recently discovered a serious flaw with Dell System Detect that allowed an attacker to trigger the program to download and execute an arbitrary file without any user interaction. Below is a summary of the issue and the steps taken to bypass the protections Dell put in place. Timeline: The issue wa...

Achieving Full Marks on Qualys SSL Labs

Dan Palmer — Mon, 23 Mar 2015 02:00:00 GMT

Qualys have become well known in the recent crop of SSL and TLS vulnerabilities as a first-responder with automated testing and validation, but scoring top marks on their SSL Labs test can be difficult. I explored what was required to score full marks.

Monk is now a Ruby app.

Ben Overmyer's Site — Sat, 21 Mar 2015 02:00:00 GMT

Well, it happened. It took a few days of work, but I taught myself enough Ruby on Rails to fully refactor Monkblog from PHP/Laravel to RoR.

The blog is now running on Heroku, and my next step will be to setup continuous deployment. I'm pretty excited about this development! I'm officially running a Ruby on Rails app in production that I wrote myself.

And of course, now that that's done, I'm working on improvements to the blog. The first order of business will probably be to make the writing screen look and feel more like Ghost's marvelous editor.

I also need to port all of my old content from wherever I can find it into this new blog. I lost several posts when my old web server died (long story), but I imagine I can restore them easily enough. Just need to find backups...

Blog post on recent talk

Nemo's Home — Fri, 20 Mar 2015 02:00:00 GMT

So I recently did a talk on Joy of Software Development. You can read more about the talk here (link includes slides and list of topics covered). This post is devoted to the references I’d promised to link to in the talk. Since it was an introductory talk, and I didn’t want to bore people to death, I decided to cover lots of topics at a shallow depth, instead of covering a few topics deeply.

This means that I need to post more material for people to follow up on. So, this is that reference blog post. Make sure you have a copy of the slides open as you go through these links.

Update: I also gave this talk (with a few updates) at GeekSkool in October 2015.

Software Development in general

Hiring in software industry is broken
Breadth First Learning
Recommended readings for developers by Jeff Atwood (codinghorror)
Ship v1, even if it sucks
Ship early, ship often
Never rewrite, always refactor by Joel Spolsky
The Joel Test to score a software company.

Software Security

Security is the opposite of obscurity
OWASP Top 10
Software Security course on coursera
CTFTime - See upcoming CTF contests
SmashTheStack - Learn buffer overflow attacks
Nebula Exploit Excercises
Backdoor - Security CTF platform for beginners by SDSLabs

Starting Advice

Agnostic Software Development

Free and open source development

Why Open Source misses the point of Free Software - by Richard Stallman
Open source definition
Choose a license
What is free software?
Why Free Software Movement is important
Mozilla mission statement

Version Control

Tests

During the talk, I decidedly used the term TDD incorrectly. TDD technically means going test first, but I used it as an introduction to testing in general. This was intentional. The links here will use TDD in the correct sense.

REST and APIs

Unix Philosophy

The epic Knuth vs McIlroy story
Zawinski’s Law
http://onethingwell.org/
Wikipedia article on the topic is surprisingly good
Good examples of pipes

Books

These are books i absolutely recommend every software developer to read, in order.

Other than these, I recommend reading Code Complete, Mythical Man Month, and everything by Jeff Atwood and Zach Homan, but only after you have read the above 2 books.

How to get better at Software Development?

This is just a small list of topics I cover in a recent blog post. This is only present in the updated version of the talk which I gave at GeekSkool. You can read the blog post here to look at the points I make.

Phew. That was a lot of links. If you are ever interested in learning more about software development, feel free to contact me. If you ever feel like chatting with me, I’m usually online at chat.sdslabs.co.

Updates to searchcode.com

Ben E. C. Boyter — Thu, 19 Mar 2015 00:11:33 GMT

Just a quick post to list some updates to searchcode.com The first is a slight modification to the home page. A while ago I received an email from the excellent Christian Moore who provided some mock-ups of how he felt it should look. I loved the designs, but was busy working on other issues. Thankfully however in the last week or so I found the time to implement his ideas and the result is far more professional to me.

Continuous integration for the Linux Kernel - Built within Docker

smcleod.net — Sun, 15 Mar 2015 02:00:00 GMT

Linux Kernel CI for Debian

Github: sammcj/kernel-ci

Those of us using technologies such as Docker and BTRFS or simply trying to gain a performance edge on the competition have a lot to gain from the features and performance of recent Kernel updates (especially from 3.18 onwards).

‘Enterprise’ Linux distributions such as RHEL & variants are concerningly out of date when comes to the Kernel. Many people seem to have forgotten what Linux is… Linux IS the Kernel.

Decoding Captcha's Presentation

Ben E. C. Boyter — Sun, 08 Mar 2015 22:41:13 GMT

A few days ago there was a lack of speakers for #SyPy which is the Sydney Python meet-up held most months and sponsored by Atlassian. I had previously put my hand up to help out if this situation ever came up and was mostly ready with a presentation about Decoding Captchas. I did not expect it to be so full that people were standing (largest crowd I had ever seen there). Thankfully it seemed to go over well and while I need to get more practice at public speaking I did enjoy it. A few choice tweets that came out of the end of the event,

Buxton's Rule

Nemo's Home — Sun, 08 Mar 2015 02:00:00 GMT

I consider myself a UX enthusiast. I consider that term to aptly describe my interest in UX. As I’m deeply involved in many UX and design decisions, I try to be well read on design and UX principles. While reading a discussion about iPhone prototypes on HN in June ‘12, I came across this comment:

Goes to show what it takes to achieve excellence: lots of trial and error. Produce at least 3 alternatives for every design decision (Bill Buxton agrees).

— mstuherl

It sounded so basic, yet often I see designers trying to defend their first design, because it seems good enough to them. No good design is ever born at the first step. Just like any other process, it takes multiple iterations to perfect it.

I recently got in touch with Morgan (mstuherl on HN), and thanked him for his comment. Here’s what he said when I told him I wanted to dub it mstuherl’s rule:

Hah! My name’s Morgan, so you can call it Morgan’s Rule if you like, but it comes from Bill, so Buxton’s rule would be more appropriate. His book Sketching User Experience contains yet more wisdom!

So thats what I’m calling it:

BUXTON'S RULE
Produce at least 3 alternatives for every design decisions.

scytheCTF and Updates

Nemo's Home — Fri, 27 Feb 2015 02:00:00 GMT

February has been an interesting month for me. I haven’t been programming a lot, but have definitely been writing a lot. I have got a few more upcoming projects as well, which I’d love to announce soon.

We recently held a short 8-hour CTF (scytheCTF) on Backdoor. I made two challenges for the CTF:

Both of these problems were rush jobs because of several reasons:

We didn’t have much time to set the problems.
We didn’t expect much participation in scytheCTF.
scytheCTF was a test CTF, just to figure out any issues with the internet launch of Backdoor.
scythe is also supposed to be beginner friendly, unlike our annual BackdoorCTF, which will include much harder problems.

I had a lot of fun with @kandoiabhi in setting the problems. It was also great seeing @DefConUA participate in such a small-scale contest.

Other than scythe, we recently had our annual SDSLabs trip to Rishikesh, which I enjoyed a lot. I also wrote a small post on my work setup.

Xen Orchestra Docker Image

smcleod.net — Thu, 26 Feb 2015 02:00:00 GMT

Docker config to setup XO which is a web interface to visualize and administrate your XenServer (or XAPI enabled) hosts

Github: sammcj/docker-xen-orchestra

Running the app

Updates are pushed to the Docker Hub’s automated build service:

https://registry.hub.docker.com/u/sammcj/docker-xen-orchestra

From Docker Hub

docker pull sammcj/docker-xen-orchestra
docker run -d -p 8000:80 sammcj/docker-xen-orchestra

Building

git clone https://github.com/sammcj/docker-xen-orchestra.git
cd docker-xen-orchestra
# Edit whatever config you want to change
docker build -t xen-orchestra .

See https://xen-orchestra.com for information on Xen Orchestra

Delete Government-Linked Certificate Authorities in OSX

smcleod.net — Mon, 23 Feb 2015 02:00:00 GMT

Inspired by http://zitseng.com/archives/7489

Source (Github)

WARNINGS

Do not run unless you understand what this is doing
The CA system is broken by design - This is not a fix for that
This is merely a band-aid for those interested or concerned about these root CAs

Usage

chmod +x delete_gov_roots.sh
./delete_gov_roots.sh

You’ll be prompted for your password as root access is required to delete system-wide root certs.

C# XML Cleaner Regex

Ben E. C. Boyter — Fri, 20 Feb 2015 00:41:47 GMT

One of the most annoying things I deal with is XML documents with invalid characters inside them. Usually caused by copy pasting from MS Word it ends up with invisible characters that you cannot easily find and cause XML parsers to choke. I have encountered this problem enough that I thought a quick blog post would be worth the effort.

As such here mostly for my own reference is a regular expression for C# .NET that will clean invalid XML characters from any XML file.

A product management blueprint

CRAIG KERSTIENS — Wed, 18 Feb 2015 22:55:56 GMT

I find myself having more conversations with startups – both small and large – about product management. I’ve blogged about some of the tools in my chest here but I haven’t talked much about my “blueprint” for product management, which I find myself laying out in many conversations over coffee. What follows is this process I’ve used a few times over with new teams to get product and engineering moving together, shipping in a predictable manner, and tackling bigger and more strategic projects.

Trust

I need to know how to work with my team, what their working styles are, and how we interact. This starts by simply interacting – specifically, outside of the office. I heard a similar opinion recently from Chris Fry (who ran engineering at Salesforce and Twitter) when he remarked something to the effect of: “you can tell a good PM from a bad one based on if he goes to drinks with his team.” Without getting hung up on whether it’s beers or coffee, it’s more about socialization with your team and time outside the office. My personal approach: expect a dinner invite over to my place when I take on running product for a new team.

Velocity

Once you’ve started to build some rapport, it’s time to get down to business. If being able to quickly commit and ship something isn’t a problem for you, then it’s easy to just assume this is working. In reality most teams I encounter that need PM support don’t have shipping nailed down. You probably already know if you fall into that category of feeling like you can commit and ship vs. not, so if you’re not able to do that a few tips:

There’s some projects that everyone wants to ship that’s been tried over and over, don’t tackle that first.
Shipping something is better than nothing. It doesn’t have to be the right thing.
Sometimes you don’t have to ship something to get velocity, you can launch things you already have
~~Kill scope~~ Test things earlier and more iteratively, the more you can validate or try something without requiring a large investment the more everyone feels better about the direction you’re heading.

The key here is to commit to projects, deliver, and move on. Your velocity depends solely on delivery, not tasks, not sprints, not projects, etc. If you haven’t shipped anything in a year, then your velocity for the year is zero. At a later point you should move from the focus on shipping anything to shipping the right things, it’s more important to ship 1 thing that moves the needle than 10 that don’t, but that’s a later concern.

Killing things

On the note of killing scope… I’ve heard it articulated at times, that some engineers are happy when certain PMs show up because it means less work for them. When you go over to an engineer’s desk are you creating more or less work? The answer should be less some large percentage of the time. If you can find a way to accomplish your goals with less effort, it’s always a win. Every project everywhere always needs more time or money, what’s more innovative is how you can help a project to ship without one of those two.

At a broader perspective than just scope – one of the biggest ways product can help engineering is by pushing harder for killing off features and the scope of a product. There’s a good test on if something is ready to ship: if you tell beta users you’re killing it and they yell at you that you shouldn’t kill it, then it’s ready to ship.

If you’ve already shipped things, but they’re not delivering value or not being used, kill them. It’s that simple, it may have been a great idea at the time, but either invest in making sure it’s used or kill it so you don’t have to maintain it.

Roadmap planning

Usually getting velocity and killing things takes 3-6 months to really take full effect. At this point a team feels like they’re not under a pile of technical debt, and they can commit to shipping projects. This is the point when product and engineering are melding and you can really start to have fun about where you’re headed. At this point I’ve seen a huge mix of where engineers are more actively or less actively engaged in this process. And the reality is this is everyone’s job to be thinking about where you’re headed as a company – at least that’s the case for any company that classifies itself as a startup.

My favorite tool for this is a team gridding exercise, you can read more about this here and here. This is often best conducted at an off-site where you have an opportunity for casual conversation which can foster broader thinking beyond the obvious bug fixes or smaller product improvements.

One item of note I’ve heard from teams that have done this or similar exercises is they still have trouble deciding what to do after the fact. The role of product is to get to that decision. The most important part is getting to a decision and not the perfect one, gather data, decide, revisit as you go along. All of this isn’t to say that it’s an arbitrary decision, customers, data all inform that as well as the effort to impact matrix exercise, but in the end a clear direction isn’t executed on consensus.

In conclusion

There’s really no end or done when it comes to the role and the work.

There’s always another milestone and the market is always moving around you. But once you’re able to execute predictably and think in an ordered sense about your roadmap, you’re in a position to be able to monitor and adapt to the market, and even more so experiment and shape the market yourself. At that point you have to keep doing it and then the hard part becomes finding ways of keeping a fresh perspective protip: customers are an important part of that equation

Have tips/tricks/practices that I completely missed here or that you disagree with? I’m always happy to talk with others so drop me a note craig.kerstiens@gmail.com.

Building a high performance SSD SAN - Part 1

smcleod.net — Mon, 16 Feb 2015 02:00:00 GMT

Over the coming month I will be architecting, building and testing a modular, high performance SSD-only storage solution.

I’ll be documenting my progress / findings along the way and open sourcing all the information as a public guide.

With recent price drops and durability improvements in solid state storage now is better time than any to ditch those old magnets.

Modular server manufacturers such as SuperMicro have spent large on R&D thanks to the ever growing requirements from cloud vendors that utilise their hardware.

Regular Expressions are Fast, Until they Aren't

Ben E. C. Boyter — Sun, 15 Feb 2015 23:36:40 GMT

TL/DR: Regular expressions are fast, until they aren’t. How I got a 20x performance by switching to string functions.

With the new version of searchcode.com one of the main things I wanted to address was performance. The previous version had all sorts of performance issues which were not limited to the usual suspects such as the database or search index.

When developing the new version one of the tasks listed in my queue was to profile search operations for anything slowing things down. Sadly I have since lost the profile output but observed that one of the main speed culprits is the format_results function inside the code model. For most queries I tried while it was the slowest operation it wasn’t worth optimising simply because its impact was so low. I did however keep it in the back of my mind that if there were any performance issues it would be the first thing to look at.

Direct-Attach SSD Storage - Performance & Comparisons

smcleod.net — Sun, 15 Feb 2015 02:00:00 GMT

Further to my earlier post on XenServer storage performance with regards to directly attaching storage from the host, I have been analysing the performance of various SSD storage options.

I have attached a HP DS2220sb storage blade to an existing server blade and compared performance with 4 and 6 SSD RAID-10 to our existing iSCSI SANs.

While the P420i RAID controller in the DS2220sb is clearly saturated and unable to provide throughput much over 1,100MB/s - the IOP/s available to PostgreSQL are still a very considerably performance improvement over our P4530 SAN - in fact, 6 SSD’s result in a 39.9x performance increase!

Search - A Journey of Delivery on a Budget

smcleod.net — Sun, 15 Feb 2015 02:00:00 GMT

“Search - A Journey of Delivery on a Budget”

Presented a Melbourne Search - July 2014 - “Search - A Journey of Delivery on a Budget” (Click for slides)

Talk - 24 Months

smcleod.net — Sun, 15 Feb 2015 02:00:00 GMT

The way we work at Infoxchange has changed greatly.

A retrospective journey into transforming Infoxchange’s technology and culture over the past 24 months - presented a Melbourne DevOps - December 2014

The Best Of - 2014 Edition

smcleod.net — Sun, 15 Feb 2015 02:00:00 GMT

At the end of every year I note down a summary of the best applications, hardware & websites I’ve enjoyed & depended on throughout the year (and often for some time before).

This post has long since been superseded. You can find the latest version of this post here.

Software / General Use

Fastmail - https://www.fastmail.com
Evernote - https://evernote.com
Reeder - http://reederapp.com
Keynote - https://www.apple.com/au/mac/keynote
Lastpass - https://lastpass.com
Plex - https://plex.tv
Calibre - http://calibre-ebook.com

Software / Geek Use

Sublime Text - http://www.sublimetext.com/3
Homebrew - http://brew.sh
DropSync - http://mudflatsoftware.com
Beets - http://beets.radbox.org
Textual - http://www.codeux.com/textual
XLD - http://tmkk.undo.jp/xld/index_e.html
Code Academy - http://www.codecademy.com
Exercism.io - http://exercism.io
Sickbeard + Headphones + Couchpotato + Sabnzbd - http://www.totalhtpc.com/ultimate-usenet-guide.html

Software / Mobile

Tweetbot - http://tapbots.com/software/tweetbot
Pushover - http://pushover.net
Lastpass - https://lastpass.com
Keynote - https://www.apple.com/au/ios/keynote
Reeder - http://reederapp.com/ios
Evernote- https://evernote.com
Plex - https://plex.tv
Backblaze - https://www.backblaze.com
WTF Podcast - http://www.wtfpod.com/app
MiniHack - https://itunes.apple.com/au/app/minihack-for-hacker-news/id631108846?mt=8
Uber - https://itunes.apple.com/au/app/uber/id368677368?mt=8
Goodreads - https://itunes.apple.com/au/app/goodreads-book-recommendations/id355833469?mt=8
Notify4M - https://itunes.apple.com/au/app/notify4m/id499161979?mt=8
Bandcamp - https://itunes.apple.com/au/app/bandcamp/id706408639?mt=8
Hype Machine - https://itunes.apple.com/au/app/hype-machine/id414315986?mt=8
Nuzzel (Only got onto this today) - https://itunes.apple.com/au/app/nuzzel-news-from-your-friends/id692285770?mt=8
Alien Blue - http://www.reddit.com/r/alienblue

Software / SysAdmin or DevOps Specific

Gitlab - http://gitlab.org
Gitlab-CI - https://about.gitlab.com/gitlab-ci
Dash - http://kapeli.com/dash
PostgreSQL (Makes my list every year & just keeps getting better) - http://www.postgresql.org
PGBadger - http://dalibo.github.io/pgbadger
Docker - https://www.docker.com
Consul - https://consul.io
Puppet (I couldn’t do my job as well without it) - http://puppetlabs.com
iTerm - http://iterm2.com
Nginx - http://nginx.org
Htop - http://hisham.hm/htop
Bonnie++ - http://linux.die.net/man/8/bonnie++
Openfire - https://www.igniterealtime.org/projects/openfire
Hiera-Eyaml - https://github.com/TomPoulton/hiera-eyaml
Rubinius - http://rubini.us
Puma - http://puma.io
XenServer - http://xenserver.org
ElasticSearch - www.elasticsearch.org
Logstash - http://logstash.net
FPM - https://github.com/jordansissel/fpm
PFsense - https://www.pfsense.org
Debian Jessie (Not quite released but the next great version of the best linux Distro IMO) - https://www.debian.org/releases/jessie
Check_MK Multisite - https://mathias-kettner.de/checkmk_multisite.html
PWSafe - https://itunes.apple.com/au/app/pwsafe-password-safe-compatible/id520993579?mt=12
Supervisord - http://supervisord.org

Websites

Last.fm - http://www.last.fm/user/sammcj2000
Feedly - https://feedly.com
HackerNews - https://news.ycombinator.com/news
Lucidchart - http://lucidchart.com/
MondoTunes (Might be a little biased here!) - http://mondotunes.org

Hardware

Das Ultimate 4- http://www.daskeyboard.com
Logitech Performance MX - http://www.logitech.com/en-au/product/performance-mouse-mx
15’’ Macbook Pro Retina - http://www.apple.com/au/macbook-pro
iPhone 6+ (because its bigger than bigger, or something) - https://www.apple.com/iphone-6
CuBox - http://www.solid-run.com/product/cubox-i4pro
Parani SD1000 Bluetooth Serial Adapter - http://www.senaindustrial.com/products/industrial_bluetooth/sd1000.php
SanDisk Extreme Pro 480GB SSD - http://www.newegg.com/Product/Product.aspx?Item=N82E16820171999

Other / Non-tech

All Day Socks - http://alldaysocks.com
Sennheiser Amperior On-Ear Headphones - http://www.head-fi.org/products/sennheiser-amperior-on-ear-headphones
O2 + ODAC - http://www.jdslabs.com/products/48/o2-odac-combo
Benchmark DAC1 HDR - http://benchmarkmedia.com/products/benchmark-dac1-hdr-digital-to-analog-converter
Bellroy Wallets - http://bellroy.com
Ink Shoes - http://www.inkshoes.it
Kindle Paperwhite - http://www.amazon.com.au/gp/feature.html?docId=3077740006
Nerf Jolt - http://nerf.wikia.com/wiki/Jolt_EX-1

Books

The Phoenix Project (Reread this year for the 3rd time) - http://itrevolution.com/books/phoenix-project-DevOps-book
Surely You’re Joking Mr Feynman! - https://www.goodreads.com/book/show/5544.Surely_You_re_Joking_Mr_Feynman_
The Dark Tower Series - https://www.goodreads.com/book/show/43615.The_Gunslinger
Snow Crash - https://www.goodreads.com/book/show/830.Snow_Crash

XenServer, SSDs & VM Storage Performance

smcleod.net — Sun, 15 Feb 2015 02:00:00 GMT

Intro

At Infoxchange we use XenServer as our Virtualisation of choice. There are many reasons for this including:

Open Source.
Offers greater performance than VMware.
Affordability (it’s free unless you purchase support).
Proven backend Xen is very reliable.
Reliable cross-host migrations of VMs.
The XenCentre client, (although having to run in a Windows VM) is quick and simple to use.
Upgrades and patches have proven to be more reliable than VMware.
OpenStack while interesting, is not yet reliable or streamlined enough for our small team of 4 to implement and manage.
XenServer Storage & Filesystems

Unfortunately the downside to XenServer is that it’s underlying OS is quite old. The latest version (6.5) about to be released is still based on Centos 5 and still lacks any form of EXT4 and BTRFS support, direct disk access is not available… without some tweaking and has no real support for TRIM unless you have direct disk access and are happy with EXT3.

Get internal and external IP address on Mac with Bash

Fri, 13 Feb 2015 02:00:00 GMT

Frequently while coding I need to know the internal and external IP address of my development machine. For example, when your machine is serving a website from a locally running web server, the internal IP address can be used to test the site from a smartphone browser connected to the same network. Likewise, the external IP address is useful when you want to make a local web server publicly accessible or when you need to whitelist your IP address in the firewall settings of an external server.

How I found a bug in HackerEarth

Nemo's Home — Thu, 12 Feb 2015 02:00:00 GMT

Source

I am not a competitive programmer. I love programming, but more so I love building things. As a result, I rarely participate in coding contests. Even when I do, I try to use languages like Ruby and Python just to see if I can do it my way, so to speak.

While trying a contest in Ruby, I realized that I could not use the ruby prime library. This is a standard library in Ruby for a long while, and HackerEarth platform runs on 2.1.1, which is quite new.

I reported this as a bug to HackerEarth in September ‘14. A quick reply from HE made me realize that they weren’t understanding the issue:

Modules like mathn or erb are part of standard library. They are available.

Try using, require ‘erb’require ‘mathn’ in code editor.
However 3rd party libraries are not available.

I decided to do some tests and check all standard libraries for their availability. For those unfamiliar with Ruby, this is how you load a standard library in ruby:

require 'prime'

Using the HackerEarth API, I was able to write some quick code that tested all expected libraries:

while read lib; do
  SOURCE="require%20'$lib'"
  echo "Testing $lib"
  curl -s -d "client_secret=API_SECRET&lang=RUBY&async=0&source=$SOURCE" http://api.hackerearth.com/code/run/ > $lib.json
done < libs.txt

Here libs.txt contains a list of all standard libraries. The above code is in bash, and makes use of curl. Parsing the results, I replied with the following:

Requiring the following libraries raises a missing error:

coverage - RE
extmk - RE
fiddle - RE
io/console - RE
json - RE
minitest - RE
minitest/benchmark - RE
minitest/spec - RE
mkmf - RE
objspace - RE
prime - RE
psych - RE
racc - RE
rake - RE
rdoc - RE
rexml - RE
rinda - RE
ripper - RE
rubygems - RE
tk - RE
win32ole - RE
xmlrpc - RE

HackerEarth admitted the issue (I posted code to replicate on github), and have since worked on it. I just ran the tests again, and only the following libraries are unavailable now:

curses - RE
dbm - RE
debug - RE
extmk - RE
gdbm - RE
generator - RE
iconv - RE
racc - RE
readline - RE
rexml - RE
rinda - RE
tk - RE
win32ole - RE

A few of these are understandable (win32 since HE platform runs on Linux, and tk, which is a graphical library). A few of these are unavailable in Ruby 2.1.1 (I copied the list of libs from the 2.1.3 docs).

Kudos to HackerEarth for fixing a bug that very few of their users would have faced. All the source code for this post can be found at github.

Note: This article was copied from HackerEarth because they are shutting down their notes platform.

I am offended

Nemo's Home — Fri, 06 Feb 2015 02:00:00 GMT

To start with, here’s a piece of art that is meant to offend you:

The above is an artwork by MF Hussain. Its was sold as an untitled work by Hussain to a private collector, but was named Bharat Mata later when it was sold in an auction.

Were you offended by looking at it? Maybe. Does it look vulgar and offensive to you? Perhaps. What should you do as a result?

CLOSE THE TAB!

Seriously, India. The right thing to do when you are offended is not to lynch a person to death, or to issue a fatwa against singing the national song. The right thing to do is this:

The right to get offended in India is a result of the way our constitution curbs the freedom of speech. However, these restrictions were not in the constitution that was passed when India was made a republic (26 Jan 1950). It was added as the First Amendment (ironic, I know), which passed in June 1951.

In the 18 months that passed between these two events, Indians had the right to absolute freedom of speech. I won’t go into the details of why both Nehru and Patel thought of bringing these restrictions (for the better of India), but needless to say, the reasons are no longer valid.

However, I found a curious piece of irony while researching this:

One of the prime opponents to the First Amendment restrictions was Syama Prasad Mookerjee, a long time RSS activist, founder of the Bharatiya Jana Sangh and widely regarded as the godfather of Hindu Nationalism.

And now today, 64 years later, these restrictions are getting enacted into even more draconian laws. One such law is Section 66A of the Information Technology Act.

Kapil Sibal, former union minister writes about it:

Allowing the government to regulate the internet is a recipe for disaster. Government being what it is, it would use such power to further its own ends.

However, he gives in to the diplomatic reasoning and writes further:

I know where I stand. I am for freedom of expression, but there are no absolutes in life. Limitless freedom contains within it the seeds of conflict. We must eschew conflict and embrace freedom, for peace and harmony.

Note that back in 2012, Kapil Sibbal had spoken in favor of Section 66A, citing it as a tool to protect women online. He seems to have reversed his stance since.

This is what Tushar Mehta, Additional Solicitor General has to say on the necessity of the act:

[…] every institution and every person right from the President can be subjected to criticism and it is people’s fundamental right to free speech and expression but such rights do not cover grossly offensive comments and posts on social networking sites.

AIB recently tested their rights by making a grossly offensive video and posting it on social networking sites. A lot of people were offended. The video was taken down as a result.

Did the people who were offended see the video? Yes, probably on YouTube. But the recordings are still floating around, and are available on torrents very easily.

You see, the internet is a resilient beast. You can’t control it, or bend it to your will. It does not run by your rules, and your sense of sensibility. It has no concept of right or wrong. It just is.

A nation where I am afraid to post critical views of the government or discuss events that might offend someone is not a nation worth living in.

The internet cannot be regulated. You might certainly think of it as possible, but we will always find a way.

If you are offended it is your problem, and frankly lots of things offend lots of people. - Salman Rushdie

I don’t think I will ever see the first amendment repealed in my lifetime. However, I’m gonna try my very best to get the Supreme Court to re-evaluate Section 66A as unconstitional and over-reaching.

If you want to let the government know of your thoughts on the matter, the Assistant Solicitor General (representing the Government in the case) can be reached at tusharmehta64@yahoo.com.

Internet Outrage Fatigue Syndrome (IOFS)

Tilting at Windmills — Wed, 04 Feb 2015 02:00:00 GMT

The 1980s were a scary time to be a kid. At least that is if you watched the evening news. Every night at 6 and 11 you'd be apprised of a…

Rebound Project

Ben E. C. Boyter — Sat, 31 Jan 2015 00:49:55 GMT

As I mentioned in the previous entry I had started work on a new project I called portfold. Built and released without fanfare I have quietly killed it before even the month is out. Why? I realise now that it was a rebound project similar to a rebound relationship. I had been getting a little down on searchcode.com and wanted to branch out to some new technology. Once done though the itch was scratched and now I am back to working on searchcode again.

Download a Git subdirectory from GitHub

Fri, 30 Jan 2015 02:00:00 GMT

Have you ever needed to download only a subdirectory from a GitHub project without downloading the entire repository? Surprisingly, there doesn’t seem to be a way of doing this easily using Git. A somewhat unexpected solution is to use SVN. First get the URL of the subfolder. GitHub URLs have the form: https://github.com/author/repo-name/tree/master/subdirectory Then replace /tree/master/ with /trunk/ to get: https://github.com/author/repo/trunk/subdirectory You can now download the subdirectory you’re interested in using SVN’s export command:

Are you a fighter pilot?

Nemo's Home — Wed, 28 Jan 2015 02:00:00 GMT

As part of a pre job interview for a position as a security consultant, I was asked this question. The interviewer expanded the question further as :

Given the choice between a luxurious journey in a passenger jetliner (flying business class) and a thrilling trip as a fighter pilot, which one would you choose?

My immediate reply was (without a single doubt): “I’ll take the fighter jet, thanks.”

Then the interviewer tried to dissuade me from my choice: “Its not as glamorous as it sounds. Its a terrible job flying a jet plane. There are lots of complications, you are literally defying death, and even the pay isn’t that good.”. He then spent quite some time explaining the luxuries and comforts that we take for granted in a passenger jet, and those that aren’t available in a fighter jet. “You can’t even piss properly”, he told me. “And there’s free booze on the Boeing.”

Me (after some deliberation and moment of self-doubt): I’d ultimately like to have my own private jet, but I’m willing to strap myself to a 300 million dollar plane just trying to get there. I’d take that over a passenger jet any day.

For those who didn’t get the analogy: He was trying to convince me to join a high risk job, where I’d be working late nights doing what I love. But it also means giving up tons of luxuries and comforts that I could get at other companies.

I’m sure that I’m the fighter jet kind of person, I’m just having difficulty deciding what jet I wanna fly. If you have an opening for a Full Stack Developer/Security Consultant, shoot me a mail.

Fix: WPML for WordPress causes theme to show multiple languages at the same time

Tue, 27 Jan 2015 02:00:00 GMT

The WordPress Multilingual (WPML) plugin offers a comprehensive solution for transitioning from a single language WordPress website to one that supports multiple languages. Once installed, you can add several translations of individual posts and custom posts from within the WordPress admin interface. When viewing a page for a specific language on your site, page content is then generated by pulling in posts for the current language. Problem: Theme and widgets showing multiple languages Many themes and plugins will work out of the box with WPML but a common problem is themes and plugins will display content for all languages instead of only the current one.

Bépo-android

Linux Engineer's random thoughts — Tue, 27 Jan 2015 01:00:00 GMT

This is a small project I recently released on github and Google Play. It aims at catering to the needs of people using the bépo layout, and wanting to use it for physical keyboards on Android.

Bépo is a french dvorak-like keyboard layout; it was designed by a community of …

Hiding visits to your own site in Google Analytics

Thu, 22 Jan 2015 02:00:00 GMT

When working on a site, it’s common to browse it yourself from multiple browsers, multiple devices and multiple locations for testing purposes. For analytics about your user behaviour to be useful, we need a way to exclude this internal traffic from Google Analytics reports. In this post I explain a method I found useful for hiding visits to your own site when you want to be able to test your site from any browser or any location.

submit-sitemap: Node.js module to submit sitemaps to search engines

Wed, 21 Jan 2015 02:00:00 GMT

I’ve just uploaded a small Node.js module for submitting/pinging sitemaps to search engines called submit-sitemap. This is meant to be used after you’ve made updates to your site to encourage crawlers to reindex your pages. The module currently submits your sitemap to Google and Bing. This saves you from having to use Google’s webmaster tools and Bing’s webmaster tools to submit your sitemap manually. For example, I’ve used this in the last stage of Gulp build scripts that generate and deploy static sites in the following way:

Never Buy Retail

Tilting at Windmills — Sun, 18 Jan 2015 02:00:00 GMT

In light of a particularly aggressive savings plan this year, I've been thinking hard about every purchase I make. In doing so I think I've…

Yu were mislead

Nemo's Home — Tue, 13 Jan 2015 02:00:00 GMT

I was eagerly awaiting the release of Yu Yureka, which has been widely hailed as a great budget phone by most reviews. I won’t go into the details of the phone, but rather the flash sale that took place on 13th Jan ‘15 (on amazon.in). Far from being a well-managed affair, the website was plagued with issues, and went down for everyone a couple of minutes before the sale.

Still, a few lucky people were able to buy the phone (sadly, I wasn’t one of them). Micromax said that they had to close registrations for the sale early and had around 3 lakh people lined up for the sale.

This is how the yuplaygod.com homepage looks right now:

Clearly, they had 10k units for sale, and one in every 300 people should have bought it, right? Wrong!

It seems Yu (the brand new subsidiary of Micromax) is not above lying. There were only 3000 devices on sale today, out of which only 2657 were claimed, after which the sale was shut down.

How do I know this? The way deals work on amazon is once you are on a deal page, the client keeps checking the deal status every few seconds so as to let you know as soon as its status changes. This deal status response does not only include the deal status code (say EXPIRED/SOLDOUT/AVAILABLE), but also includes the deal’s nitty details.

These details include:

totalCouponCount: 3000
claimedCouponCount: 2657
percentClaimed: 88
type: LIGHTNING_DEAL
title: Yureka
dealPrice: 8999
currentPrice: 12999

I’m not sure why Yu would try such a tactic (hype the device at low cost, overstate sales figures and then switch to a higher price), but it sure does not sound nice if you are one of the 3 lakh people who lined up to buy the device.

References

Since its my word against amazon, here’s a simple way to confirm the deal details for yourself:

Visit http://hurl.it/ (I don’t own this site)
Change the request method to “POST” from “GET”
Enter http://www.amazon.in/xa/dealcontent/v2/GetDealStatus where it says yourapihere.com
Click on “+ Add Body Button”
Paste {"requestMetadata":{"marketplaceID":"A21TJRUUN4KGV","clientID":"goldbox"},"dealTargets":[{"dealID":"ea9fef51","itemIDs":null}]} into the box that appears
Click on “Launch Request”
Scroll to the bottom to see the dealStatus

As an alternative, here is a permalink to the request.

Update: If you try to replicate the above steps, you will notice that the deal response is now blank. My guess is that the deal was deleted from the servers. However, the permalinks above should still work. I’m still waiting for any official word from either Yu/Amazon.

Here’s a better (edited) photo that Yu might wanna use:

Thanks to Shashank Mehta for the title suggestion.

The Content Singularity

Tilting at Windmills — Thu, 08 Jan 2015 02:00:00 GMT

Think back to the time before the Internet, if you're old enough. How did you get entertainment? Typically, you had to drive to a store pay…

Similar CSS Color Finder

Donat Studios — Tue, 06 Jan 2015 11:41:33 GMT

Update: CIEDE2000 support added to the existing CIE94 support.

It would seem that very similar but non-identical colors seem to pop up often in CSS files of any reasonable age, and I became sick of them. This started as a little script to help me find them in a stylesheet, and grew into this full-fledged tool.

This tool will scan CSS and CSS-like files like Sass, Less, etc and find colors similar to each other within a set tolerance.

This handles #hex, rgb, rgba, hsl, and hsla colors.

There is a command line version of this with Continuous Integration support available on GitHub.

Luminosity Masks in Darktable

Beetle Space — Tue, 06 Jan 2015 10:00:00 GMT

When I first got into photoediting, I started off with the free software GIMP. I was a student and couldn’t afford Adobe Photoshop. While GIMP had/has shortcomings, and as much as people like to denigrate it, it was (and likely still is), the best general purpose photoediting tool …

Thank You Pat

Nemo's Home — Sat, 03 Jan 2015 02:00:00 GMT

I’m a lazy reader. I’ll often start books and leave them halfway, often juggling 3-4 books at the same time. I read in sprints, often spending a few days just finishing lots of books followed by reading nothing for the next few weeks, perhaps. But that doesn’t mean I don’t appreciate good books. This is the story of how I found my favorite writer, and how I discovered the wonderful books that I enjoy so deeply.

I don’t write a journal regularly, but if I did, one of my favorite things to do with it is to figure out the little things that matter; to separate out the strands and understand the connections and the motivations behind where I am today

It is our choices…that show what we truly are, far more than our abilities.

Albus Dumbledore

What I love to do is figure out those tiny choices, and the reasoning behind them that led me to today.

I read Patrick Rothfuss’s “Name of the Wind” a long time back. It was an amazing book, which I found on a site called bestfantasybooks.com. In my defense, I was an avid Harry Potter fan at the time, looking for similar books, and I’d decided that reading the best fantasy would be good preparation before I could write my own.

As it so happens, “Name of the Wind” was a superb book. The kind that made me squirm in delight when I saw that the sequel was already out. So I did what any self-respecting book-lover would do: read it in a single stretch, screwing up my exams in the process. I didn’t sleep much for those few days (its a pretty long book), but I enjoyed every bit of it.

I started following Pat’s (hilarious) blog, where he often posted tidbits of his life, and came across his review of “The Alloy of Law” by Brandon Sanderson:

…
My last point is that Sanderson has now been added to a very short list of authors. Specifically, the list authors whom I wish to kill so that I might eat their livers and thereby gain their power.

Any author that Patrick wanted to kill sounded like a great one to read, so I started “The Alloy of Law”. I finished the book in a few short hours. Its paced excellently, and the action keeps on coming. I’d never really read much urban fantasy before, and despite me never having read Mistborn (Alloy is a sequel to the Mistborn trilogy) I was sucked in.

And here I am, counting down the next few days, waiting for the release of Firefight, Sanderson’s next book.

So, thank you Pat. Thanks for introducing me to my favorite writer. Thank you for writing those wonderful books, and keep on reviewing all that you love (I’m reading “Through the Woods” next).

If you are looking for a nice fat book to read next, I heartily recommend “The Name of the Wind” and “The Way of Kings”.

Your API is not RESTful

Dan Palmer — Sat, 03 Jan 2015 02:00:00 GMT

RESTful APIs are a popular thing, but is anyone really doing it properly? This post highlights some common flaws in RESTful APIs, and explains why it’s important that we improve them beyond the current standard.

New Beginnings

Tilting at Windmills — Fri, 02 Jan 2015 02:00:00 GMT

It's been a long time coming, but I think it's finally time for me to shutter GreaterDebater. Suffice it to say, the site never really…