Hacker News Personal Blogs 2013 | Top 100 Blogs

2013 Year in Review

Ben Overmyer's Site — Sun, 29 Dec 2013 02:00:00 GMT

A great deal happened in 2013 for me and my family. Sadly, though, I missed the mark on all six of my goals for the year. Despite that, I think 2013 can be counted a success in general.

My Goals for 2013

My first goal for 2013 was to release a computer game. For a variety of reasons, this never came close to completion. I overestimated my free time and, to be honest, my motivation. I ended up spending far more time developing tabletop games. I’ll get into that in a second. My second goal was to join a local developer group. Technically, I succeeded at this, since I started going to MNPHP meetups for awhile. However, because I didn’t sustain my attendance, I’m counting this one failed. There is promise here, though. My third goal was ambitious - take a vacation to Hawaii. Saving up the money to do so, especially amidst changing jobs, proved too difficult for me. My fourth goal, lead a yoga class, failed for more than just inability to do so. Most yoga schools these days seem to focus solely on the physical side of yoga, with no thought for the spiritual side. I had a moment of clarity when I realized that I couldn’t get to the level of proficiency in just a year to be qualified to teach both the physical and spiritual sides. I may revisit this goal some year in the future, but not in the near tense. My fifth goal was well on its way to success when I sabotaged it by re-incorporating Dicejockey into my main site. Though I wanted to reach 1,000 visitors per month for at least one month in 2013, my desire to pull in everything to one destination representing me was greater than my desire to continue a blog dedicated solely to MMORPGs and tabletop RPGs. My sixth and final goal for 2013, to collaborate on a project with my sister Tasha, went fairly well for the first half of the year. Unfortunately, I think I was too ambitious in the project we chose to work on, because my contribution would have been a large number of artworks that my skill level and free time did not allow for. Still, this goal was inspiring, and I’ll do something along these lines in 2014.

Major Happenings of 2013

I changed jobs twice this past year. The first time, I left Group 3 Marketing because I felt I was stagnating as a developer. I joined a startup called Sportsdigita and worked heavily on Symfony2 projects. While I certainly grew as a developer at Sportsdigita, the culture was damaging everything else about my life. In August, then, I left that company to work for the Minneapolis Star Tribune, the largest newspaper in Minnesota (and, possibly, the Midwest). The Star Tribune has proved to be a wonderful balance between interesting projects and good culture, and it doesn’t hurt that it pays well, too. My contributions to Silver Gryphon Games grew this year. My efforts on the web side increased, and my interest in this side of the business is definitely heating up. I released the first web tool for Ingenium, and it’s proved very useful - internally, anyway. I did more writing this year in previous years, releasing no less than three adventures for Ingenium. Two of those I did all the writing for, and the third I contributed a third of the final word count to. This also marked the first year where I managed a convention on my own, using the newly-minted “Booth 2.” All in all, 2013 was a successful year for me. I got back into tabletop gaming in a big way this year. I started what has become my longest-running D&D campaign ever. I started playing Warhammer 40k and Warmachine on a weekly basis with some good friends. I did a lot of miniature painting, and produced some works that I’m actually proud of, which hasn’t happened before. I also began writing the rules for my own war game, and that could bode well for Silver Gryphon Games, based on early playtest feedback. I met several new friends and strengthened relationships with others. I’ve been more social-minded this year than the past couple years. I’ve definitely gone to more conventions this year than in past years, also.

Closing Thoughts

Many positive things happened this year, but I’m disappointed that I wasn’t able to follow through with a single one of my goals. I grew as a developer, a gamer, and a person in general in 2013. Overall, despite the lack of goal completion, I think 2013 was a win for me. Here’s to an even better 2014!

Using Beyond Compare 4 Mac Beta with Git

Donat Studios — Fri, 27 Dec 2013 01:29:40 GMT

A little over a year ago I wrote a post about using Beyond Compare on Mac via Wine. A native version is now in beta and open to everyone! If you haven't tried Beyond Compare, I suggest you do. If you have you already know how awesome it is.

I've been testing the Alpha for many months now and thought I'd throw together some instructions for getting it going with git.

You can download the beta here: http://www.scootersoftware.com/beta.php

After you have the App installed, the next step is to install the command line tools. All other steps require this.

From the App, go to the "Beyond Compare" drop down menu and choose "Install Command Line Tools...". It will then prompt you for your password. After this, you bcomp file1.txt file2.txt

Using with Git CLI

The next step is to set it up as a git diff tool and merge tool. Git already has native support for it built in thanks to the Linux client having existed for years now.

This can be achieved simply by running the following commands.

$ git config --global diff.tool bc3
$ git config --global difftool.prompt false
$ git config --global merge.tool bc3

This simple setup will leave git diff and git merge functioning as usual, as well as allowing you to use Beyond Compare for the task by doing git difftool and git mergetool.

I highly recommend giving Beyond Compare a go as a merge tool next time you need to do a merge as it makes it wonderfully simple (when you understand whats going on with the 4 sections anyway).

Using with Tower 1.x

Update: Tower 2.0.6+ include built in support for Beyond Compare which the following instructions will interfere with.

Quit Tower
Navigate to ~/Library/Application\ Support/Tower
Create a CompareScripts folder if one does not already exist.
Download bcomp.sh and place it in the new CompareScripts folder.
Make it executable via chmod +x bcomp.sh
Download CompareTools.plist and place it in the ~/Library/Application\ Support/Tower directory. Rename / move any old version that might exist.

Using with SourceTree

Open SourceTree
Open "Preferences" from the SourceTree menu
Along the top bar, choose the "Diff" tab.
Under External Diff / Merge
1. for Visual Diff Tool choose Other then in the Diff Command enter /usr/local/bin/bcomp and for Arguments enter $LOCAL $REMOTE
2. For Merge Tool choose Other and in Merge Command enter /usr/local/bin/bcomp and for Arguments enter $LOCAL $REMOTE $BASE $MERGED

On GitHub

Nemo's Home — Wed, 25 Dec 2013 02:00:00 GMT

I am an Internet addict. And the website I’m most addicted to is called GitHub. GitHub is a social code hosting website that is totally awesome. Not your general run-of-the-mill awesome, but rather ass-kicking best-thing-in-the-world awesome. If GitHub was a ninja, it would be Po, defeating the evil clutches of SourceForge.

When Linux was released a couple of decades ago, it was hosted on a university FTP server. Today, the days of FTP are way past, and unless your code is hosted on a code-sharing website like GitHub, it is as good as dead. The last decade has seen an explosive grown in software and the open source movement. Things like GitHub, Linux, Android, Facebook have been made possible due to the combine efforts of millions across the globe following the Unix philosophy of doing one thing well.

Ahh, I digress. The point I’m trying to make is that it is very hard to explain to a layman how important GitHub has been to the software community. It has been used as a collaboration platform for writers, law makers, governments, programmers, and even musicians. People have used its issue tracking feature to even plan weddings. And above all, people adore GitHub. It is one of the few startups that have been accorded God status in the community.

The question here is what makes GitHub tick? As the largest hosting site for code, it obviously makes a huge impact just by virtue of being there. But its tide of features and innovative progression had made it a darling of all. For instance, their 1 click to fork feature has allowed people to contribute to any project so much easier.

GitHub has an amazing User Experience, making sure it works perfectly on all devices large or small. Their amazing support makes sure all of its users are happy as possible. Their regular meetups makes sure that GitHub is invested in the software community themselves. And their GitHub Store lets people invest back in GitHub by romoting them via Tees, stickers and even laptop sleeves.

My personal experience with GitHub has been overwhelmingly positive. People have stood beside GitHub even as they faced major issues, and for me that is indicative of a trust in GitHub that no money can buy. The GitHub API (which I’ve used more than once) lets developers create their own apps on top of GitHub which others can use to create even more awesome things.

For me GitHub is more than just a coding website. It is a testament to creativity and the Hacker Way, reminding me every day that anything is possible.

Testing a NAS hard drive over FTP

Linux Engineer's random thoughts — Fri, 20 Dec 2013 01:00:00 GMT

So I have this NAS made by my ISP, that does a lot of things; but recently, I started having issues with its behavior. Recorded TV shows had lag/jitter while replaying, and the same happened with other types of videos I put on it. I narrowed it down to …

Digging in with Foreign Tables

CRAIG KERSTIENS — Thu, 19 Dec 2013 22:55:56 GMT

I wrote a couple months back about exploring FDWs. Its become quite clear to me, despite still having ample room for improvement, they’re not getting enough attention. Foreign data wrappers are perhaps better thought of as a foreign table, or even better yet as a view into some remote data source. They don’t take care of auto-updating or syncing data, thats all up to you, but it gives you a straight forward mapping to work with remote data easier.

The first step in working with FDWs is getting them setup. I wrote an earlier post on how to do this manually. And if you’re on Heroku theres an even easier solution if you want to setup a mapping entirely from one DB to another. The pg-extras CLI plugin has a command fdwsql which will generate the SQL to map all the tables for you. To run it simply specify the prefix app and database:

heroku pg:fdwsql yourprefix APP::DATABASE_URL

This will generate a lot of SQL. From here you’ll want to connect to the database where you want those foreign tables to be visible. Then run all the SQL. This will create all the foreign tables, this will mostly look just like another view or table to you in \d.

Tips on working with them

For the most part you can work with your foreign tables just like any other view or table. You can insert into them, read from them, join against them. Though currently foreign tables have some performance limitations, such as when joining it may return a lot more data than you expect then join. To make your performance a bit more ideal you can follow a few basic principles.

Lets look at some example tables to highlight this:

> \d
users
todos

In this case users is local and the todos are a foreign table. Looking at each of the schemas we have something like you might expect:

> \d users
Table "public.users"
Column | Type | Modifiers
-------------+-----------------------------+-----------
id | integer | not null
email | text |
created_at | timestamp without time zone |
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"users_created" btree (created_at)
> \d todo
Foreign Table "public.todo"
Column | Type | Modifiers
-------------+-----------------------------+-----------
id | integer | not null
user_id | integer |
desc | text |
created_at | timestamp without time zone |
status | boolean |
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"todo_created" btree (created_at)

A Practical Exercise in Web Scraping

Pete Keen — Sun, 15 Dec 2013 02:00:00 GMT

Yesterday a friend of mine linked me to a fictional web serial that he was reading and enjoying, but could be enjoying more if it was available as a Kindle book. The author, as of yet, hasn't made one available and has asked that fan-made versions not be linked publicly. That said, it's a very long story and would be much easier to read using a dedicated reading app, so I built my own Kindle version to enjoy. This post is the story of how I built it.

Step 1: Source Analysis

The first step of any kind of web scraping is to understand your target. Here's what the first blog post looks like (with different content):

<h1 class="entry-title">The Whale</h1>
<div class="entry-content">
  <p>
    <a title="Next Chapter" href="http://example.com/the/next/chapter">
      Next Chapter
    </a>
  </p>
  <p>"And what tune is it ye pull to, men?"</p>
  
  <p>"A dead whale or a stove boat!"</p>
  
  <p>
    More and more strangely and fiercely glad and approving, grew the
     countenance of the old man at every shout; while the mariners
     began to gaze curiously at each other, as if marvelling how it
     was that they themselves became so excited at such seemingly
     purposeless questions.
  </p>
  
  <p>
    But, they were all eagerness again, as Ahab, now half-revolving in
    his pivot-hole, with one hand reaching high up a shroud, and
    tightly,   almost convulsively grasping it, addressed them
    thus:&mdash;
  </p>
  <p>
    <a title="Next Chapter" href="http://example.com/the/next/chapter">
      Next Chapter
    </a>
  </p>
</div>

After browsing around I found a table of contents, but since all of the posts were linked together with "Next Chapter" pointers it seemed easier to just walk those. The other interesting thing here is that there's a comment section that I didn't really care about.

Step 2: Choose Your Tools

The next stage of web scraping is to choose the appropriate tools. I started with just curl and probably could have gotten pretty far I knew the DOM futzing I wanted to do would require something more powerful later on. At the moment Ruby is where I turn to for most things, so naturally I picked Nokogiri. The first example on the Nokogiri docs page is actually a web scraping example, and that's basically what I cribbed from. Here's the initial version of the scraping function:

def scrape_page(url)
  html = open(url)
  doc = Nokogiri::HTML(html.read)
  doc.encoding = 'utf-8'

  content = doc.css('div.entry-content').first
  title = doc.css('h1.entry-title')

  next_url = ""

  content.search('a[title="Next Chapter"]').each do |node|
    next_url = node['href']
    node.parent.remove
  end

  {
    title: title,
    content: content,
    next_url: next_url
  }
end

Ruby has a built-in capability for opening URLs as readable files with the open-uri standard library module. Because of various problems with Nokogiri's unicode handling I learned about in previous web scraping experiences, the best thing to do is to pass a string to Nokogiri instead of passing it the actual IO handle. Setting the encoding explicitly is also a best practice.

Then it's a simple matter of using Nokogiri's css selector method to pick out the nodes we're interested in and return them to the caller. The idea is that, since each page is linked to it's successor we can just follow the links.

Step 3: The Inevitable Bugfix Iteration

Of course it's never that easy. Turns out these links are generated by hand, and across hundreds of blog posts of course there will be some inconsistencies. At some point the author stopped using the title attribute. Instead of using the super clever CSS selector a[title="Next Chapter"] I had to switch to grabbing all of the anchor tags and selecting based on the text:

content.search('a').each do |node|
  if node.text == "Next Chapter"
    next_url = node['href']
  end
  node.parent.remove
end

This works great, except that in a few cases there's some whitespace in the text of the anchor node, so I had to switch to a regex:

content.search('a').each do |node|
  if node.text =~ /\s*Next Chapter\s*/
    next_url = node['href']
  end
  node.parent.remove
end

Another sticking point was that sometimes (but not always) the author used non-ASCII in their URLs. The trick for dealing with possibly-escaped URLs is to check to see if decoding does anything. If it does, it's already escaped and shouldn't be messed with:

def escape_if_needed(url)
  if URI.unescape(url) == url
    return URI.escape(url)
  end
  url
end

Step 4: Repeat As Necessary

Now that we can reliably scrape one URL, it's time to actually follow the links:

task :scrape do
  next_url = 'http://example.com/the/first/chapter/'

  sh "mkdir -p output"

  counter = 0

  while next_url && next_url =~ /example.com/
    STDERR.puts(next_url)

    res = scrape_page(next_url)
    next_url = res[:next_url]
    title = res[:title].text

    File.open("output/#{sprintf('%04d', counter)}.html", "w+") do |f|
      f.puts res[:title]
      f.puts res[:content]
    end

    counter += 1

    sleep 1
  end
end

This is pretty simple. Set some initial state, make a directory to put the scraped pages, then follow each link in turn and write out the interesting content to sequential files. Note that file names are all four digit numbers so that the sequence is preserved even with lexicographical sorting.

Step 5: Actually Build The Book

At first I wanted to use Docverter, my project that mashes up pandoc and calibre for building rich documents (including ebooks) out of plain text files. I tried the demo installation first, but that runs on Heroku and repeatedly ran out of memory so I tried a local installation. That timed out (did I mention that this web serial is also very long?) so instead I just ran pandoc and ebook-convert directly:

task :build do
  File.open("input.html", "w+") do |f|
    Dir.glob('output/*.html').sort.each do |filename|
      f.write File.read(filename)
    end
  end

  STDERR.puts "Running conversion..."

  sh("pandoc --standalone --output=output.epub --from=html --to=epub --epub-metadata=metadata.xml --epub-stylesheet=epub_stylesheet.css input.html")
  sh("ebook-convert output.epub output.mobi")
end

Pandoc can take multiple input files but it was easier to manage one input file on the command line. The stylesheet and metadata xml files are lifted directly from the mmp-builder project that I use to build Mastering Modern Payments, with appropriate authorship information changes.

In Conclusion, Please Don't Violate Copyright

Making your own ebooks is not hard with the tools that are out there. It's really just a matter of gluing them together with an appropriate amount of duct tape and bailing twine.

That said, distributing content that isn't yours without permission directly affects authors and platform shifting like this is sort of a gray area. The author of this web serial seems to be fine with fan-made ebooks editions as long as they don't get distributed, so that's why I anonymized this post.

Quora answer about writing a search engine

Ben E. C. Boyter — Fri, 06 Dec 2013 03:31:30 GMT

The following I posted on Quora in response to the question “I am planning to make a small scale search engine on my local system, but I don’t know from where to start?”. It’s a reasonable answer so like my Mahalo one I thought I would make a copy for myself.

I agree with Wolf Garbe and that you are better off in your case starting with existing technologies, have a look at http://yacy.net/ and SphinxSearch as well. However if you are doing this to learn and not just deliver a product I can provide a few links for you.

Simple Git-backed Microsites

Pete Keen — Thu, 05 Dec 2013 02:00:00 GMT

A few days ago I built a new tool I'm calling Sites. It builds on top of git-backed wikis powered by GitHub's Gollum system and lets me build and deploy microsites in the amount of time it takes me to create a CNAME.

Something that I've wanted for a very long time is a way to stand up new websites with little more than a CNAME and a few clicks. I've gone through a few rounds of trying to make that happen but nothing ever stuck. Furthest progressed was a Rails app exclusively hosting Comfortable Mexican Sofa, a simple CMS engine. I never ended up putting any sites on it, though.

GitHub's Pages are of course one of the best answers, but I'm sticking to my self-hosting, built-at-home guns.

A Short Code Tour

The code is split up into four distinct parts:

viewer is a Sinatra app that presents wiki content as web pages. It also can serve static assets right from the wiki repo and caches everything in an in-memory LRU cache. If you have a file layout.erb it will wrap the pages in that layout, otherwise it'll pass the content straight to the browser.
manager is another Sinatra app that allows me to create new sites on the fly. Because a site is just a Gollum wiki and a Gollum wiki is just a git repo, it just has to create a git repo at the right place and do a redirect. It bakes in HTTP Basic Auth so other people can't create sites all willy nilly.
some small extensions to Gollum add basic auth and override the method Gollum uses to find the correct wiki repo. By default Gollum wants to be told exactly where the repo is in a class-level Sinatra setting, but that doesn't work when things are dynamic.
a Rack middleware ties it all together. The middleware has three jobs. If the incoming hostname maps to a CNAME that one of the sites has declared in a special wiki page named cnames, pass the request to the viewer app. Otherwise, either pass the request to Gollum for existing sites or to the manager to create a new site. The hard work of building the CNAME to site mapping is cached for a short period of time to minimize disk hits.

The neatest part about this setup is that, since sites are just git repos, I can clone the repo to my laptop and work with it in Emacs instead of directly editing in Gollum if I don't want to. This also lets me easily add asset files and layouts.

Demo Time

Here's what the manager app presents when you just go to sites.bugsplat.info:

I'm the only one that's ever going to be looking at this, so it doesn't really need to be anything fancy.

If you click on one of those links, you'll get the familiar Gollum interface:

To create a new site, just append its name to sites.bugsplat.info:

Notice the new little button. Clicking that creates the repo and sends you back to Gollum to populate the home page:

If you want to see how the demo site is put together, it's running here and the source is on GitHub here.

Installation

I built Sites to fit my infrastructure which is a delightful bastardization of 12 Factor so I haven't tried installing it elsewhere. I know it won't run properly on Heroku because it needs to be able to put the git repos in a persistent place, but it might work well as a Docker image. If you get it running somewhere please let me know and I'll link it here.

Go Binary Sizes Are Growing out of Control

Donat Studios — Tue, 03 Dec 2013 01:25:43 GMT

Update 2019-07-17: I have released a follow up post Go Binary Sizes Are Relatively Stable which speaks on what's happened in the time since this was published.

Note: This is not intended as a "Go Sucks" post. I love Go. I am not saying the developers are lazy or dumb or any of the things Reddit has implied. I am not implying I could build a better compiler. Rob Pike and the Go team are geniuses whom I look up to. The tone of this was intended to be a light "man I wish binaries were smaller" but that is not how Reddit seemed to take it. Also, the title is hyperbole and not meant to literally mean no one has control over it.

I started toying with Go about two years ago before the 1.0 release. I love the language, it meets me in a comfortable place between C and a scripting languages like PHP. I find it fun to write and an easy way to bust out performant code quickly. The binaries for a simple Hello, 世界 were large though, around 55KB. I figured this was something the Go team would work out later; it is a lot of space just to send a string to standard out.

With the release of Go 1.0 though, I noticed the binary size had grown slightly to 242KB. Discouraging but livable. I was quite hopeful with the 1.1 release, but it had actually nearly doubled in size coming out to a whopping 405KB. Then this morning I compiled it again in the newly release 1.2 and to my horror the ever simple program came out to 557KB.

What on earth is going on in that binary? I am no expert on compilers but I would imagine the vast majority of what is in there is completely unnecessary and could be optimized out. Changing the script to "Hello, World" rather than Go's flaunted unicode version has zero effect on binary size, as one would expect but I was hopeful.

For reference the complete source of my Hello, 世界 follows. Using fmt comes out even larger.

package main

func main() {
    println("Hello, 世界")
}

Here is a collection of binaries compiled in different versions of Go. Notice how imgavg grows from 1.9MB to 3.7MB. My applications are all small tools. I can only imagine the effect this has an already large application.

I really hope something is done about this. I would love to see tiny (<32KB) binaries from Go but right now that doesn't seem to be a priority for them. The fact that Hello, 世界 fills half a floppy means the language is not be useful for lighter or embedded environments.

As noted in the header there is an addendum to this:

Go Binary Sizes Are Relatively Stable

Simulating a Market in Ruby

Pete Keen — Mon, 02 Dec 2013 02:00:00 GMT

Trading markets of all kinds are in the news pretty much continuously. The flavor of the week is of course the Bitcoin markets but equity and bond markets are always in the background. Just today there is an article on Hacker News about why you shouldn't invest in the stock market. I've participated in markets in one way or another for about a decade now but I haven't really understood how they work at a base level. Yesterday I built a tiny market simulator to fix that.

Basic Concepts

tl;dr: a market for a commodity is two sorted lists, one of prices someone is willing to pay and another of prices is willing to accept.

A market exists to enable people to trade something, whether that be shares of stock or pork futures contracts or cryptocurrency tokens. In modern markets the fundamental core is called the order book. This is an open listing of offers to buy and sell a given commodity at some price. For example:

Pete has 10 shares of TSLA and is willing to sell them at $10 per share
Andrew would like to buy 10 shares of TSLA and is willing to pay $9.99 per share

The order book looks like this:

Buy	Sell
10 @ $9.99	10 @ $10.00

Emily can see the order book because it's public and open. She also has 10 shares of TSLA and decides to match Andrew's bid. She puts in a sell order at $9.99. Now the order book looks like this:

Buy	Sell
10 @ $9.99	10 @ $9.99
	10 @ $10.00

Hold on for a second. Why did Emily's offer bump Pete's down in the list? Each column of the order book is sorted by price and then by time. Buy orders are sorted highest price first, sell orders are sorted lowest first.

Trades only happen at the top of the order book. Because there's a match at the top of the order book the market executes the trade. Emily and Andrew exchange $9.99 for 10 shares of TSLA and Pete is left waiting for another suitor to come along and match his price.

This is how the notional price of an equity or contract is determined by the market. Each time a trade happens that price gets broadcast to the world as the price.

These orders are called limit orders because they say "buy TSLA for $9.99 but no more" or "sell TSLA for $10 but no less." In our example, if Fred didn't look at the book and decides to put in a buy for $10.05, he'll get his 10 shares at $10.00 from Pete.

Limit orders are the fundamental building block of a market. There are other order types but they're almost always built using one or more limit orders. One notable exception is a market order which orders a specific quantity at whatever the current market price is. (thanks for the corrections, minimax!)

Building a Simulation

Almost all of the above I learned by reading articles about basic trades and actually building a simulation. I decided to practice readme driven development and test driven development for this project, mainly because RDD helps me organize my thoughts and TDD helps me keep the programming going in the right direction. My first pass at the simulation was... well I guess you could say terrible. I completely misunderstood how the order book worked so I built this thing where any price match would execute a trade, not just at the top of the book. You can see that in the first working commit of my simulator.

After thinking about the problem really hard and reading about how order books are actually supposed to work I did some research and came across the algorithms gem. Among other awesome things, this gem includes several implementations of a data structure named the Red Black Tree. This structure keeps it's keys sorted during insert and removal, which is perfect for the order book. Each order book consists of a pair of these tree maps, one for buy and one for sell. The keys are the actual Order object and the value is just true, since we only really care about the keys.

The core of the simulation is submitting orders and checking to see if there's a match. Submitting is fairly trivial:

def submit_order(order)
  if order.order_type == :buy
    buy_map.push(order, true)
  else
    sell_map.push(order, true)
  end
end

Because the book is kept sorted, determining a match is also relatively straightforward. Here's the code in the book:

def match?
  return false if buy_map.size == 0 || sell_map.size == 0
  sell_map.min_key.match? buy_map.max_key
end

We just have get the top of each column in the order book and compare them. The tree map takes care that we can efficiently get both the min and max key.

Here is the implementation of Order#match?:

def match?(other)
  price_match = if order_type == :buy && other.order_type == :sell
    price >= other.price
  else
    price <= other.price
  end
  
  commodity == other.commodity &&
    quantity == other.quantity &&
    price_match
end

We first determine how to compare the price and then do some sanity checking on commodity and quantity. This simulator is limited to trading orders of exactly the same quantity but real markets can fulfill orders in more complicated ways.

Learning

So what did I learn yesterday afternoon? A few things. First, readme driven development and test driven development go hand in hand when building a project like this. Writing (and rewriting) the readme helped to clarify what I actually wanted to build, and writing tests both before and after building the implementation helped immensely with keeping the goal clear and my implementation correct.

Second, I gained a much better understanding of how markets work on a basic level. Actually getting in and building something seems to cement the ideas a whole lot better than just reading about them.

Making HackerTray

Nemo's Home — Thu, 28 Nov 2013 02:00:00 GMT

A few days back, I found the excellent HackerBarApp via Hacker News. Hacker News, for those of you who don’t know, is tech news website run by YCombinator. Hacker Bar was the simplest way of accessing HN stories that I’d ever seen. Unfortunately, it was only for Mac (made using rubymotion) and even though the source was available, it was of no use to me as a Linux User.

I decided to make a clone of Hacker Bar that would work on Linux. My first choice of the stack was node-webkit, an application framework that allows you to build cross-platform applications using HTML, CSS, JS, and modules from the node.js ecosystem. After reading a lot about node-webkit, I figured out that building this application in node-webkit (as it stands) would not be possible. Or rather, it would not work under Ubuntu and its derivatives because of lacking appindicator support in node-webkit. More details here.

The next obvious language and stack of choice was Python + Gtk. I’d already played a little bit with Gtk and Python some time back, so I knew the basics. But I’d never build a real application with PyGtk, just toys and small scripts. I found a basic skeleton app that was written for AppIndicator and modified it somewhat to form the base of HackerTray.

The next challenge I faced was keeping the check boxes always checked despite of any number of clicks after the first. That is, we don’t want any menu item to be “un-checked” at any moment. A basic idea is to do this (partial code):

def open(self, widget):
	if(widget.get_active() == False):
		widget.set_active()
	webbrowser.open(widget.url)

def addItem(self, item):
	#create a new CheckMenuItem (i)
	i.connect('activate', self.open)

However, this does not work as expected, because the widget.set_active() call also results in the activate event being fired, which ultimately calls open. This means on a click to an unchecked menuItem, the open function is called twice. This results in the browser opening the link twice.

As a workaround, I disabled the event handler in case it is a checked menuItem:

def open(self, widget):
	if(widget.set_active() == False)
		wiget.disconnect(widget.signal_id)
		widget.set_active(true)
		widget.signal_id = widget.connect('activate', self.open)
	webbrowser.open(widget.url)

def addItem(self, item):
	#create a new CheckMenuItem (i)
	i.connect('activate', self.open)
	i.signal_id = i.connect('activate', self.open)

The next thing I worked on was a persistent memory for the app. In a nutshell, I needed to make sure that the tick on an item remained there, even if the app was restarted. This meant writing a list of all the “viewed” items into a file. After looking at shelve for a bit, I just rolled my own implementation , based on storing the data into ~/.hackertray.json file.

After that I worked on packaging the app into a python package, so that it could be easily installed. The python packaging tutorial was an easy to use guide that let me create the package easily and push it to the Python Package Index. A few issues in the package were found, and were fixed quickly thanks to the pull request by @brunal.

After improving the README a bit, I posted about it on Hacker News, where it failed to get any traction. I re-tried with a link to the HackerTray website, and that fell flat as well. It was on the next day, when I posted it to HN for the third time, that it took off. After 50 or so upvotes, I found that my instance refused to run because it had hit the API Rate Limit on the excellent node-hnapi. I quickly pushed a fix that used a list of servers to hit as fallback in case it crossed the Rate Limits.

After a lot of feedback from HN, I started work on a node-webkit based clone of hackertray for Windows. I should be able to release it in a few more days, if nothing else crops up. Keep watching this space for info. If you have any queries, just file an issue on GitHub or contact me.

University Presentation

Tom Forbes — Wed, 27 Nov 2013 20:19:28 GMT

So I did a presentation on Information Security at University today. I think it went rather well, however I couldn’t show a couple of the demonstrations due to some SkyDrive files only being available online. That sucked because those were my best demonstrations, but overall I was happy. A few peopl...

PostgreSQL - It's kind've a nifty database

Brightball Articles — Wed, 27 Nov 2013 11:42:00 GMT

This is a presentation I recently gave to provide an overview of PostgreSQL and some of it's excellent features, including full-text search, multiple built in datatypes, data compression and extensions. Also, Morgan Freeman is narrating. You're welcome.

My Dimension in Rift

Ben Overmyer's Site — Wed, 27 Nov 2013 02:00:00 GMT

I started playing Rift again, after trying out Everquest 2 and Guild Wars 2 also. Until Everquest Next Landmark comes out, Rift may be my fantasy MMO of choice. We’ll see.

Hack things, acquire clothing.

Tom Forbes — Fri, 22 Nov 2013 18:59:42 GMT

I discovered an XSS flaw in a website a month ago and reported it to the owners. As a thank you they sent me a hat, a rather large american sized t-shirt and a pair of “DeFeet” socks (guaranteed to stay cooler and drier than any other brand). I didn’t expect them to ship something like that overseas...

Books I read in 2013

Krishna's blog — Tue, 19 Nov 2013 02:00:00 GMT

A list of books read in 2013, including fantasy, historical fiction, and science fiction.

The best Postgres feature you're not using – CTEs aka WITH clauses

CRAIG KERSTIENS — Mon, 18 Nov 2013 22:55:56 GMT

SQL by default isn’t typically friendly to dive into, and especially so if you’re reading someone else’s already created queries. For some reason most people throw out principles we follow in other languages such as commenting and composability just for SQL. I was recently reminded of a key feature in Postgres that most don’t use by @timonk highlighting it in his AWS Re:Invent Redshift talk. The simple feature actually makes SQL both readable and composable, and even for my own queries capable of coming back to them months later and understanding them, where previously they would not be.

The feature itself is known as CTEs or common table expressions, you may also here it referred to as WITH clauses. The general idea is that it allows you to create something somewhat equivilant to a view that only exists during that transaction. You can create multiple of these which then allow for clear building blocks and make it simple to follow what you’re doing.

Lets take a look at a nice simple one:

WITH users_tasks AS (
SELECT
users.email,
array_agg(tasks.name) as task_list,
projects.title
FROM
users,
tasks,
project
WHERE
users.id = tasks.user_id
projects.title = tasks.project_id
GROUP BY
users.email,
projects.title
)

Using this I could now just append some basic other query on to the end that references this CTE users_tasks. Something akin to:

SELECT *
FROM users_tasks;

But where it becomes more interesting is chaining these together. So while I have all tasks assigned to each user here, perhaps I want to then find which users are responsible for more than 50% of the tasks on a given project, thus being the bottleneck. To oversimplify this we could do it a couple of ways, total up the tasks for each project, and then total up the tasks for each user per project:

total_tasks_per_project AS (
SELECT
project_id,
count(*) as task_count
FROM tasks
GROUP BY project_id
),
tasks_per_project_per_user AS (
SELECT
user_id,
project_id,
count(*) as task_count
FROM tasks
GROUP BY user_id, project_id
),

Then we would want to combine and find the users that are now over that 50%:

overloaded_users AS (
SELECT tasks_per_project_per_user.user_id,
FROM tasks_per_project_per_user,
total_tasks_per_project
WHERE tasks_per_project_per_user.task_count > (total_tasks_per_project / 2)
)

Now as a final goal I’d want to get a comma separated list of tasks of the overloaded users. So we’re simply giong to join against that overloaded_users and our initial list of users_tasks. Putting it all together it looks somewhat long, but becomes much more readable. And as a bonus I layered in some comments.

--- Created by Craig Kerstiens 11/18/2013
--- Query highlights users that have over 50% of tasks on a given project
--- Gives comma separated list of their tasks and the project
--- Initial query to grab project title and tasks per user
WITH users_tasks AS (
SELECT
users.id as user_id,
users.email,
array_agg(tasks.name) as task_list,
projects.title
FROM
users,
tasks,
project
WHERE
users.id = tasks.user_id
projects.title = tasks.project_id
GROUP BY
users.email,
projects.title
),
--- Calculates the total tasks per each project
total_tasks_per_project AS (
SELECT
project_id,
count(*) as task_count
FROM tasks
GROUP BY project_id
),
--- Calculates the projects per each user
tasks_per_project_per_user AS (
SELECT
user_id,
project_id,
count(*) as task_count
FROM tasks
GROUP BY user_id, project_id
),
--- Gets user ids that have over 50% of tasks assigned
overloaded_users AS (
SELECT tasks_per_project_per_user.user_id,
FROM tasks_per_project_per_user,
total_tasks_per_project
WHERE tasks_per_project_per_user.task_count > (total_tasks_per_project / 2)
)
SELECT
email,
task_list,
title
FROM
users_tasks,
overloaded_users
WHERE
users_tasks.user_id = overloaded_users.user_id

CTEs won’t always be quite as performant as optimizing your SQL to be as concise as possible. In most cases I have seen performance differences smaller than a 2X difference, this tradeoff for readability is a nobrainer as far as I’m concerned. And with time the Postgres optimizer should continue to get better about such performance.

As for the verbosity, yes I could have done this query in probably 10-15 lines of very concise SQL. Yet, most may not be able to understand it quickly if at all. Readability is huge when it comes to SQL to ensure its doing the right thing. SQL will almost always tell you an answer, it just may not be to the question you think you’re asking. Ensuring your queries can be reasoned about is critical to ensuring accuracy and CTEs are one great way of accomplishing that.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Little Data: How do we query personal data?

Pete Keen — Tue, 12 Nov 2013 02:00:00 GMT

My wife and I recently moved from Portland, OR to Ann Arbor, MI. Among the cacophony of change that is involved with a move like that, we of course changed to the local utility company. Browsing around in their billing application one day I came across a page that showed a daily graph of our energy usage, supposedly valid through yesterday for both gas and electric. And it has a button that spits out a CSV file of the date, which means if I actually wanted to I could build my own tool to analyze our usage.

This got me thinking about all of the little databases out there that have data about me that I might actually consider useful. Useful, in this case, means that by looking some combination of the data I could draw some useful conclusions and take some sort of action. Here's a list of the ones that immediately came to mind:

Energy usage
LoseIt
Withings
RunKeeper
Bank records
Email

Some of the things in that list have easily accessed APIs, like RunKeeper and Withings. Some of them you can export the data manually to CSVs. And then some of them really should have an API but don't, like LoseIt. And then what if we could add more data? Things like how often we run the clothes dryer, how often the furnace or AC kicks in and for how long, whether we're in the house or not, etc.

None of this is Big Data. This is all Little Data. It's reminiscent of the Quantified Self movement, but it's more about combining existing data sets into a unified whole than synthesizing new data.

I've gone so far as to build isolated silos for some of these data sets. For example, I track my finances using ledger and a variety of home-grown tools. My email gets backed up hourly and that's provided some useful data occasionally. Even so, all of it is still in individual silos. Ideally I would be able to dump all of this into some kind of personal "data soup" that I could query with a unified interface and build analytics tools on top of.

At this point I have a bunch of questions and no answers. What shape does the data soup take? How does data get in, how would it get queried? What questions could I realistically answer? I'm going to keep thinking about this and hopefully you will too.

Restricting Thrift clients to specific IP addresses with Twisted

Tom Forbes — Mon, 11 Nov 2013 19:17:11 GMT

Apache Thrift is pretty awesome - you can build Twisted bindings for your Thrift interface file that work fantastically. There is one thing that took me a while to figure out: I want to restrict clients connecting to the service to a specific set of IP addresses stored in a database. There were thre...

Load a Github Gist with Composer

Donat Studios — Thu, 07 Nov 2013 04:36:45 GMT

Composer is amazing for pulling in packages, but what if you find a Gist that isn't Composer aware? Fear not, as Composer has the magical ability to pull in repositories not explicitly set up in Packagist by defining them in the "repositories" section.

But what if you wanted to load a single class from a git gist?

You can add the following repositories section to your composer.json file, adjusting name and url as necessary. The name should be a vendor/package (all lower) style name you make up for it, and url will be from the Clone this gist box on GitHub.

"repositories": [
    {
        "type":"package",
        "package": {
            "name": "turin86/wssoapclient",
            "version": "master",
            "source": {
                "url": "https://gist.github.com/5569152.git",
                "type": "git",
                "reference":"master"
            },
            "autoload": {
                "classmap": ["."]
            }
        }
    }
]

Then in your require section, add your selected name as follows.

"require": {
    "turin86/wssoapclient": "dev-master"
}

If the name of the class doesn't match the filename in a friendly autoload-able manner, you'll have to replace "classmap": ["."] with "files": ["ClassToAutoload.php"] replacing ClassToAutoload.php with the filename. More info on autoloading can be found here. Note that if you are experimenting with autoload settings you may need to remove your vendor directory and composer install, as updates to the autoload were not reflected for me after a composer update.

After this, just run composer update and voilà, you are loading a gist with Composer.

Embedded Linux Conference Europe 2013 notes

Linux Engineer's random thoughts — Wed, 30 Oct 2013 01:00:00 GMT

So I was in Edinburgh this year, and I took notes as I usually do. These are intended for personal consumption (do no expect LWN-style reports), but as more people were asking me to share them, I thought why not do it in public ?

Embedded Linux timeline

by Chris Simmonds …

Installing the PHP "memcached" Extension

Donat Studios — Wed, 23 Oct 2013 13:42:40 GMT

⚠️ Warning: These instructions are outdated

This article was written for OS X Mavericks and should not be followed on modern versions of macOS. It remains available for historical reference only.

For development on modern macOS, use a Homebrew-installed version of PHP. shivammathur/php is a good place to start.

Do not disable System Integrity Protection. Doing so weakens macOS security and may leave your system vulnerable to malware.

Please Note

This tutorial only covers the default PHP installation that ships with Mac OS X / macOS. If you have installed a new installation this does not cover you.

For those using brew, you can simply brew install the appropriate brew formula for the extension.

These directions are verified to work with Mac OS X 10.9 Mavericks through macOS 10.14 Mojave.

System Integrity Protection

If you are running OS X El Capitan or newer you will need to disable System Integrity Protection to modify system files and directories.

To disable System Integrity Protection, boot into recovery mode by restarting and then holding ⌘R as you hear the startup chime. Then start the Terminal from the Utilities menu.

Run the following command

$ csrutil disable

Then reboot. You are good to go.

If you wish to turn it back on, follow the instructions above but instead use:

$ csrutil enable

Click here for a full post on the topic.

The first step is to install the latest version of Xcode. On modern versions of MacOS this can be done from the App Store.

After this, install the Xcode developer tools.

$ xcode-select --install

Special note to Mojave users if the following steps fail, you may need to reinstall your Xcode header files. Instructions for that can be found here.

Next we will need to install the required dependencies before we can build the extension. If you are not already using Homebrew you should be.

$ brew install wget autoconf pkg-config libmemcached

You will also want to make sure you have PEAR installed; full instructions can be found here but can be summarized as follows:

$ wget http://pear.php.net/go-pear.phar
$ php go-pear.phar

Then in the /tmp directory we will create a folder we can work in.

$ cd /tmp
$ mkdir memcached-work
$ cd memcached-work

Next we will use pecl, part of pear, to fetch the current version of the extension.

The version of the memcached extension will vary so you will need to update the paths accordingly.

$ pecl download memcached
$ open memcached-{{version}}.tgz
$ cd memcached-{{version}}/memcached-{{version}}
$ phpize
$ ./configure
$ make
$ sudo make install

Finally you will need to add the following line to your php.ini

extension = memcached.so

You can verify your installation with the following:

$ php --info | grep memcached\\.

Depending on your setup now you may want to restart apache.

$ sudo apachectl restart

You should be all set to go!

How to: Disable macOS System Integrity Protection

Donat Studios — Wed, 23 Oct 2013 13:42:40 GMT

If you are running OS X El Capitan or newer you will need to disable System Integrity Protection to modify system files and directories.

To disable System Integrity Protection, boot into recovery mode by restarting and then holding ⌘R as you hear the startup chime. Then start the Terminal from the Utilities menu.

Run the following command

$ csrutil disable

Then reboot. You are good to go.

If you wish to turn it back on, follow the instructions above but instead use:

$ csrutil enable

My wishlist for a Star Wars Galaxies 2

Ben Overmyer's Site — Sun, 20 Oct 2013 03:00:00 GMT

Star Wars Galaxies 2 will most likely never happen, owing to the fact that SOE has publicly stated that it will never work on an MMO based on an external IP again. Whether that’s true or not, if somehow this project were to be conceived, here are the changes from the original that I’d personally want to see in a sequel.

Modern Graphics Engine

This is the most obvious one. My personal choice for SWG2’s engine would be Unreal Engine 4, owing to its extremely impressive technology. The “Infiltrator” demo video for UE4 shows that it would be perfect for the realistic art style of the Star Wars: Galaxies franchise, as compared to the more stylized art direction of Star Wars: The Old Republic.

A Return to Skill-Based Character Progression

Like most SWG veterans, I lamented the loss of the skill tree progression system when the New Game Experience took over. With SWG2, I’d want to see it return with a vengeance. Particularly, I’d like to see the depth and breadth of trees expand, allowing for greater freedom than even the original. Several aspects of the Star Wars universe were never fully explored in the first game, or were only given light treatment, such as slicing or smuggling.

More Player Races

The selection of available races in the original was good, allowing for most of the popular choices. I would like to see the Togruta, Chiss, and Devaronian races make a core appearance in SWG2. I’d also like to see droids as a player race, though that might be too complicated to balance against other races’ gameplay.

Bigger Worlds

The original SWG planets were pretty big, but they frequently felt empty. The expansion worlds had stronger design, but felt smaller. I’d want to see planets in SWG2 be more expansive than the original SWG planets, but have more points of interest throughout each. For example, the planets in the first SWG felt a lot like the same planet given a different skin each time. While I don’t want quests to provide a meta plot, I would like to see localized content that gives each world its own character.

Greater Freedom of Movement

Let’s face it, movement was pretty horrible in the original SWG. You couldn’t jump or fly over obstacles, and it was completely possible to walk straight up a sheer cliff face. While we don’t necessarily need the full parkour aspects of EverQuest Next, I do want the ability to jump over things, be prevented from walking up impossible surfaces, and be able to climb, swim, and otherwise move as a person should be able to move.

Atmospheric Vehicles

I know these were added to the original SWG, but it came only in the last few months of the game’s life. Given how important they are to the Star Wars universe in general, it would be a crime not to include these from the beginning in SWG2. I want to be able to bullseye womp rats in a T-16, over take a cruise over the highlands of Corellia in my own YT-1300. I also want shuttle transports to actually have real transit time, and not be a wait-and-teleport affair like they were in the original.

Player Factions

While it was great to have Imperial and Rebel affiliations in the original, to really ramp up the sandboxy nature of Star Wars Galaxies, the sequel would need to have player factions. This is particularly important if the game is set in a time period after the fall of the Empire, where a billion new factions would be vying for power. I see player factions being akin to massive alliances of Player Associations, rather than just an enormous PA. They’d also have to be able to influence planetary politics, like how Imperial and Rebel players were able to influence things towards the end of the original SWG.

A Player Exchange

Unlike the auction house format of current MMORPGs, I’d like to see something closer to the original vendor system in SWG. You’d be able to search available items on a local, planetary, or galactic level, as well as purchase them, but you’d need to physically travel to the vendor in order to pick up the item or items. This is important for two reasons: to encourage exploration and travel, and to support the possibility of player deliverymen - “FedEx” quests that actually matter. I want to have to travel to Tosche station to pick up some power converters.

Greater Visual Customization of Crafted Items

I liked that crafters could dye armor in SWG, but a sequel would need to take it to a new level. Item appearance should be further separated from item game mechanics than in the original. There is a vast variety of armor appearances in Star Wars, for example, and that should be reflected in SWG2. I don’t want just a dozen different kinds of armor in different colors. I want several dozen, if not more. If World of Warcraft can do it, so can SWG2. Involving the Player Studio could help defray the cost of developing new models and textures for components of items.

Vehicular Combat

If we can fly in the atmosphere, we should be able to fight in it also. I’d want to see hover tanks finally make an appearance, among other things. If PlanetSide 2 can balance heavy armor against infantry, Star Wars Galaxies can do it too. This list could be even longer, but the above are my biggest wishes for a Star Wars Galaxies 2. How about yours? What would you want to see in a sequel?

Hello, World!

Linux Engineer's random thoughts — Sun, 20 Oct 2013 01:00:00 GMT

Update : I updated the information below with the 2020 tech.

So I finally did it. You're reading it right now. My personnal website/blog.

I should be posting here about things that cross my mind as well as various projects I've been working on. And maybe even new projets I …

Post-mortem of a Dead-on-Arrival SaaS Product

Pete Keen — Sat, 19 Oct 2013 03:00:00 GMT

A little over a year ago I announced the launch of my latest (at the time) product named Marginalia. The idea was to be a sort of online journal. A cheaper, more programmer friendly alternative to Evernote. It never took off, despite my best intentions, and so a few months ago I told the only active user that I was going to shut it down, and today I finally took that sad action. This post is a short history of the project and a few lessons learned.

A Brief History

Marginalia actually started out pretty humbly. See, for a very long time I've been emailing ideas to myself and then forgetting that I did that and thus those ideas were just completely lost. I needed a better way to capture all of this so I could get it out of my head and into something more permanent. In late January 2012 I finally put together a simple Rails application that used a Mailgun incoming account to parse emails sent to a special address and add the Markdown-formatted body content to a new page. The app would reply to that email with the From: address set to a unique address. Anything emailed to that address would get appended to the note with a timestamp.

I used this simple app called "Ideas" for seven months and loved it to pieces. I captured a lot of ideas in there, along with daily work notes and other various things. At some point I started talking about it with other people and they encouraged me to try selling it, and after a lot of consideration I added subscriptions and billing and all of the other little things a real SaaS app needs. Then I wrote a blog post and got it on the front page of HN briefly, and then... nothing. A few signups for the original "pay once" subscription model, a handful of people signing up for the $5/mo plan and then never coming back, and otherwise no other traffic. I kept using it for awhile and then built a better mousetrap and moved on.

Lessons Learned

People want their privacy. I originally tried pitching Marginalia as a way to keep a programming journal, which ended up being a losing proposition. First of all, software developers don't want to spend money on tools when their free or already-paid-for editor is just as good. Second, keeping journals of any kind at an external service is something that most developers are not at all into because of the privacy implications.

Itched scratches frequently bleed money. Every month Heroku charged me $29 for SSL and a non-development-level database. Every month for more than a year, with zero income. It's not a lot of money but it's money from my web hosting budget that could have been going toward something more productive. I also spent an awful lot of money on paid advertising trying to get an audience with zero return.

Launching without an audience means nobody shows up. Building a product without an excited, engaged audience is one of those things that software developers tend to do, often and with gusto. It's so easy to build up this idea in your head and in your editor and just expect people to show up after you're done. It's something that I've done three times and it's something I will hopefully never repeat. For my latest product I started with a simple landing page with a Mailchimp signup form. Only after actually determining interest did I move forward with the plan.

Conclusion

The source is up on GitHub if you want to take a look at how it came together. There's some features, including on-page javascript evaluation and data blocks, that never got announced but which I still think were a good idea.

First post

Krishna's blog — Sat, 19 Oct 2013 03:00:00 GMT

A personal blog about mundane observations on reading, programming, and gaming, with occasional updates and vain attempts to drive traffic to the blog.

Adding tail-call optimization to Python

Tom Forbes — Tue, 15 Oct 2013 22:04:05 GMT

Tail-call optimization is a trick many languages and compilers use to avoid creating excess stack frames when dealing with recursive code like this:...

Tooling for Simple but Informative Emails

CRAIG KERSTIENS — Sun, 13 Oct 2013 23:55:56 GMT

Emails are one of my favorite methods of communicating with users. Its works as a quick test for product validation. It works well at one->some->many-> all. Its still highly effective even as much noise as we receive in our inboxes. Over the years I’ve tried a lot of email tools from custom built solutions, to newer entrants that help around drip actions (intercom.io and customer.io), to more “enterprise” tools such as Marketo. While I have varying opinions on all of those, I still find myself coming back to a simple one off script setup to deliver clear concise emails.

Getting the Data

The first step of any email is deciding what you want to do, but hopefully you know that already. The part that is usually a bit more effort is actually getting the list to send it to and formatting it appropriately. I usually opt for SQL. While the specifics of the query of course always vary it common follows a general structure:

WITH initial_data AS (
SELECT
email,
app_name,
information_about_app
FROM
users,
apps
WHERE users.id = apps.user_id
AND some_filter_to_limit_data
),
candidates_for_email AS ... --- likely to have additional CTEs
--- Finally I build up the list
SELECT email,
array_to_string(array_agg(data_for_email), '
') --- an important note is to add a newline or not here depending on how you wish to format it
FROM candidates_for_email
GROUP BY email;

The query structure you’ll want is first column email, second column whatever data you want to include in your email.

From here I usually create a dataclip of it. This makes it easy to allow my data to change over time. If I’m testing an email for data over the last 7 days I just come back in 7 days and I have new data. It also lets me easily share and iterate on the data. The nice part is there’s an easy way to click a button and get the data as a CSV which is what you want for sending.

Once you download the CSV you’ll want to remove the header line as its not needed for the script.

Sending the Mail

To actually send the email you’ll need this script, which is largely credited to @leinweber:

require 'mail'
require 'csv'
FILE = ARGV[0]
Mail.defaults do
delivery_method :smtp, {
address: 'smtp address',
port: 587,
domain: 'gmail.com',
user_name: 'craig.kerstiens@gmail.com',
password: ENV.fetch('EMAIL_PASSWORD'),
authentication: :plain,
enable_starttls_auto: true
}
end
def send_email(address, app)
mail = Mail.new do
to address
from 'Craig Kerstiens <craig.kerstiens@gmail.com>'
subject "Your email subject in here"
body generate_body(app)
end
end
def generate_body(app)
%Q(
Hi,
Your list of apps:
#{app}
Various email content in here...
)
end
CSV.parse(File.read(FILE)).each do |line|
address = line[0]
app = line[1]
m = send_email(address, app)
puts m.to_s
p m.deliver!
puts
puts
end

You’ll want to make sure to export the PW of your email provider with EXPORT EMAIL_PASSWORD=pw_here

You can easily download this script from off of Github’s Gist. I’d recommend using an email service provider other than Gmail in sending your emails such as mailgun as they’re built to handle sending a large amount of emails. Finally send your emails:

ruby email.rb nameofyourfile.csv

My Uni's timetable system sucks, so I built a better one.

Tom Forbes — Sat, 12 Oct 2013 02:05:09 GMT

tl;dr The timetable system sucks, so I made one that works Getting your timetable sorted at Uni has never been fun. In years 1 and 2 of my study the department posted a timetable for each year showing all modules and students were expected to remove the classes they did not take, which while not the...

Objective-C

Dan Palmer — Sat, 12 Oct 2013 02:00:00 GMT

If I mentioned that I like C, C++ or Python to other students on my course, or colleagues, there would be no reaction. There are things you can criticise about each one, but they are all very safe bets. When I tell people that I enjoy writing Objective-C however, they are confused and often quite hostile towards the language.

I am by no means an Objective-C expert, but I’ve been thinking through the reasons why I like it, so this is a random collection of reasons why I enjoy using the language.

Purchasing a £30,000 numberplate for the price of a bus ticket

Tom Forbes — Mon, 07 Oct 2013 04:40:33 GMT

Regtransfers.co.uk is a website that allows you to purchase customized numberplates for your car or motorbike. They boast a large number of famous clients and short numberplates are often on sale for upwards of £20,000 (the plate ABC 4 is up for £30,000). While playing with their site I discovered a...

Breaking out of secured Python environments

Tom Forbes — Wed, 25 Sep 2013 23:02:33 GMT

A week or so ago I was browsing /r/Python and I saw a link to a website called rise4fun.com , which is a Microsoft Research project that contains a lot of cool demos and tools that you can run in your browser. The demo I was linked to was a restricted Python shell that could be used to experiment wi...

DRY your Rails CRUD with Simple Form and Inherited Resources

Pete Keen — Sun, 22 Sep 2013 03:00:00 GMT

When you're writing a Rails application you usually end up with a lot of CRUD-only controllers and views just for managing models as an admin. Your user-facing views and controllers should of course have a lot of thought and care put into their design, but for admin stuff you just want to put data in the database as simply as possible. Rails of course gives you scaffolds, but that's quite a bit of duplicated code. Instead, you could use the one-two-three combination of Simple Form, Inherited Resources, and Rails' built-in template inheritance to DRY up most of the scaffolding while still preserving your ability to customize where appropriate. This lets you build your admin interface without having to resort to something heavy like Rails Admin or ActiveAdmin while also not having to build from scratch every time.

Inherited Resources consists of a base controller you can inherit from that implements all of the standard resourceful actions, plus a few convenience things for working with this controller. If you have a Book model, you could create a complete resourceful controller for it with the following code:

class BooksController < InheritedResources::Base
  protected
  def permitted_params
    params.permit(book: {:title, :author, :isbn})
  end
end

As you can see, Inherited Resources has automatic integration with Rails 4's Strong Parameters.

DRY up your CRUD

Rails 3.1 and further shipped with a thing called 'template inheritance'. This simply means that there's a default search path for templates that Rails will hunt through to find a suitable template. If there's no index.html.erb in app/views/books, for example, Rails will look in app/views/application because that's BooksController's base class. We can use this to construct default views for our CRUD controllers. First, let's make a base class for our controllers to inherit from:

class CrudController < InheritedResources::Base
  def attrs_for_index
    []
  end

  def attrs_for_form
    []
  end

  helper_method :attrs_for_index
  helper_method :attrs_for_form
end

And now we can set up default views. First, app/views/crud/index.html.erb:

<h1>
  <%= resource_class.to_s.pluralize %>&nbsp;
  <small>
  <%= link_to 'New', [:new, resource_class.to_s.downcase.to_sym] %>
</h1>
<table>
  <thead>
    <tr>
      <% attrs_for_index.each do |attr| %>
        <th><%= attr.to_s.titlecase %></th>
      <% end %>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <% collection.each do |resource| %>
      <tr>
      <% attrs_for_index.each do |attr| %>
        <td><%= link_to resource.attributes[attr.to_s], resource %></td>
      <% end %>
      <td><%= link_to 'Edit', [:edit, resource] %></td>
      </tr>
    <% end %>
  </tbody>
</table>

Inherited Resources gives you a few helpers that make these views really easy:

collection maps to the collection of objects in your controller. If it was BooksController, collection would return @books. This is only present in the index view.
resource_class returns the class of the resource your controller is managing
resource maps to @book and is available in every view except index.

Now we need a show view. Most of the time, all you want to see is a dump of the resource's attribute, which is exactly what we're going to do in app/views/crud/show.html.erb:

<h1>
  <%= resource_class %> <%= resource.id %>
  <small><%= link_to 'Edit', [:edit, resource] %></small>
</h1>
<table>
  <tr>
    <th>Key</th>
    <th>Value</th>
  </tr>
  <% resource.attributes.sort.each do |key, value| %>
  <tr>
    <td><%= key %></td>
    <td><%= value %></td>
  </tr>
  <% end %>
</table>

The edit and new views are super simple:

<h1>New <%= resource_class.to_s.titlecase %></h1>
<%= render 'form' %>

<h1>Editing <%= resource_class.to_s.titlecase %> <%= resource.id %></h1>
<%= render 'form' %>

Let's look at app/views/crud/_form.html.erb:

<%= simple_form_for resource do |f| %>
  <% attrs_for_form.each do |attr| %>
    <%= f.input attr %>
  <% end %>
  <%= f.button :submit %>
<% end %>

Here's where Simple Form really starts to shine. It auto-detects the proper form input to display by inspecting the attribute, so we don't have to do any work for a basic form. Of course, because we're using view inheritance, if you want to make a more complicated for you can just drop what you want into app/views/<controller>/_form.html.erb and it'll get picked up automatically. That also goes the same for any of the templates.

We should flesh out BooksController with overrides for those attr methods:

class BooksController < CrudController
  def attrs_for_index
    [:title, :author, :isbn]
  end

  def attrs_for_form
    [:title, :author, :isbn]
  end
end

Sorting, Paging, and Whatnot

You're probably thinking that this whole thing is great, but what if you need to, for example, sort or paginate the results on the index? Inherited Resources has you covered. Just override the collection method, like this:

class BooksController < CrudController
  def collection
    @books ||= end_of_resource_chain.order('created_at DESC')
  end
end

The end_of_resource_chain method gives you your resource relation after applying all of the other neat things that Inherited Resources can do. Check out the README for more details on that.

In the same vein, what if you want to limit the objects created and accessed by the rest of the CRUD actions? For example, let's say you want to limit the books available to the current user:

class BooksController < CrudController
  def begin_of_association_chain
    current_user
  end
end

begin_of_association_chain is where Inherited Resources starts out building objects. If you don't provide it, it defaults to the resource class.

There's a lot more you can do with Inherited Resources and Simple Form, like build controllers that deal with multiple nested resources and forms that have automatically populated select dropdowns. You should check them out.

Aboard the Nautilus

Nemo's Home — Sat, 21 Sep 2013 03:00:00 GMT

I’d done a post on this a long time back (2009), detailing what all softwares I use on a daily basis. This is an update to that post.

Since the last post, I’ve moved on to using Linux, using Elementary OS as my primary OS. Over the time period this post was written, I’ve shifted from using Cinnamon to Openbox and finally to elementaryOS’s pantheon as my Desktop Manager. I’m thinking of switching to Arch Linux, just to get a faster experience. I use Synapse as my application launcher, because its much faster than anything else out there.

For most of my web browsing needs, I rely on Google Chrome Stable and a daily build of Chromium (v31 as of now) for most of my work. I switch between them all the time. I use Firefox (stable) only to test out my projects from time to time.

The current editor I use is Sublime Text. It is everything you need, and much more. I’m still to get started using its build system, and its plethora of packages; but its still an excellent choice for a daily use editor. On the command line, I use Vim, git (with SCM Breeze), fasd and tig, which is an excellent git interface on the command line.

I listen to music on my own browser-based music player, called Muzi, YouTube and GrooveShark.

For my terminal needs, I use Gnome-Terminal. I use Byobu to manage my session, and often connect to it from other computers as well. Its an excellent multiplexer that fits in my workflow really well.

I use Imo.im on both the Desktop and my tablet to chat. I occasionally use ReText for editing markdown files. I use RedShift on my laptop and f.lux on my iPad to help me sleep better. I recommend it to everyone who is suffering from eye-strain or wants to sleep better.

On the browser, my most visited sites would be Hacker News (via hckrnews.com), WorkFlowy for managing my to-do list and GitHub on a daily basis for most of my projects.

Hardware

I own a old Nokia X3-02, and will be upgrading to a Firefox OS Phone soon enough. I use a Dell Inspiron 1545 as my personal machine. I also use an iPad 2 (with 3G) on a daily basis (mostly for reading). I also own a Dayan Zhanchi 3x3 and a 5x5 shengshou speed cube.

iPad Apps

The must have iPad apps for me are Chrome, imo, and iBooks. I have installed Mailbox alongside GMail, and haven’t used GMail since I installed it. I sometimes write stuff using Plaintext, and sketch using Paper. I read my RSS feeds using Newsify and Feedly.

Extensions

Chrome Extensions that I use on a daily basis include Chime for wonderful notifications (highly recommended), Ghostery for getting a tracker-free internet , HTTPS Everywhere to keep me secure, LastPass to manage passwords and Stylish for customizing the looks of various websites.

Dream Setup

My dream setup would consist of a lightweight Ubuntu Laptop that I can carry around that still has lots of processing power and battery life. I’ll prety much be satisfied by any high-end Android phone as long as it has a decent battery life.

Inspecting .NET applications with ILSpy

Tom Forbes — Thu, 19 Sep 2013 15:40:40 GMT

Every once in a while I come across an application that is so comically insecure that I feel the urge to blog about it. The application in question is a .NET application to manage care homes and provide a Medical Administration Record for residents. Staff login to the app using a username and passwo...

Introducing SingleBugs the Bug Tracker for Single Developers

Ben E. C. Boyter — Mon, 16 Sep 2013 04:50:08 GMT

Introducing the first beta release of SingleBugs, the bug tracker aimed at single/solo developers. Are you a solo developer? Do you find setting up Mantis/Fogbugs/Bugs.net/et.al too complex and a waste of your time? Do you want a single solution that easily syncs and backs up between your machines? Try SingleBugs.

Guaranteed to save you time and money setting up a bug tracker. Guaranteed to be the fastest bug tracker you have ever used.

Essential Tools for Starting a Rails App in 2013

Pete Keen — Sun, 15 Sep 2013 03:00:00 GMT

Over the past few years I've written a number of Rails applications. It's become my default "scratch an itch" tool for when I need to build an app quickly to do a task. Even though Rails is mostly batteries-included, there are a few tools that make writing new applications so much easier. This is my list of tools that I use for pretty much every new Rails project.

Edit: The discussion on Hacker News has some great gems that you should consider using as well.

Dotenv

Dotenv is a simple gem that loads environment variables from a file named .env in your project root into the ENV hash within Ruby. Getting configuration from the environment is one of the factors in 12 Factor Applications, and using a .env file for development eases the transition to deploying on Heroku. Or, if you're crazy like me, deploying on your own hardware using a nasty brew of Capistrano and Foreman.

Devise

Most Rails apps are going to need a way to authenticate users. You could write something yourself, but there are a lot of subtle security concerns that you have to take into account. By using an off the shelf product like Devise you're insulated from having to worry about that. Some people use AuthLogic, which is also perfectly fine.

Brakeman

There have been quite a few security vulnerabilities over the past year or so inside Rails, some of which are due to Rails themselves, but many are coding errors or best practices that, over time, have turned out to be not the best. Brakeman is a security scanner that looks at your code base for both categories of error and tells you if you're doing something wrong. I run Brakeman over my codebase as part of my test suite so I know immediately when I'm doing something that isn't quite right.

Rails Best Practices

In a similar vein to Brakeman, Rails Best Practices is a list of best practices that anyone can add to, vote on, and modify. They provide a scanner that looks for violations of these best practices and tells you about them. I also run this as part of my test suite, not because they're necessarily security focused, but hard-won experience has taught me that doing (most of) the things that RBP says to do leads to a more maintainable codebase. They provide a configuration file that you can tweak, in case the scanner starts warning on something that you don't think it should.

Simple Form

Much of what we do as Rails developers boils down to making simple CRUD forms to work with models. Much of this is going to be inside an admin interface that users never actually see so we want to get the job done as quickly as possible. Simple Form lets you write the simplest form declaration possible and bakes in a lot of useful things like error and validation handling. It's also compatible with a number of CSS frameworks like Zurb Foundation and Bootstrap. I tend to use Simple Form in lieu of an admin interface generator like ActiveAdmin, mostly because I haven't had much luck getting those to play with Rails 4.

Sidekiq

At some point every Rails application is going to need to do some background processing, especially if you're making server-side calls to other web services. These should always be done outside of a web request because Rule Number 1 is The network is unreliable (the PDF in the sources block is a great explanation of the problems of distributed computing, btw). I've explored a number of different background processing systems for Rails and the best that I've found is named Sidekiq. It uses less resources per worker than any of the rest and it is super easy to manage.

Disabling muting while typing in Google hangouts

CRAIG KERSTIENS — Thu, 12 Sep 2013 23:55:56 GMT

Google hangouts is awesome, its my preferred method for most audio/video calls these days. When running a group call I often dial into a separate phone if I have a better phone available for the group. It also got around the annoyance that when you are typing google automatically mutes you. This for most people is pretty subpar. While dialing in to the hangout can still be nice, you don’t have to do so to get rid of the annoying muting while typing. To fix such simply open up your terminal and run:

 defaults write com.google.googletalkplugind exps -string [\"-tm\"]

This clever hack discovered courtesy of @timtyrrell passed along to me by @mattmanning and @blakegentry

Diving into Postgres JSON operators and functions

CRAIG KERSTIENS — Wed, 11 Sep 2013 23:55:56 GMT

Just as PostgreSQL 9.3 was coming out I had a need to take advantage of the JSON datatype and some of the operators and functions within it. The use case was pretty simple, run a query across a variety of databases, then take the results and store them. We explored doing something more elaborate with the columns/values, but in the end just opted to save the entire result set as JSON then I could use the operators to explore it as desired.

Here’s the general idea in code (using sequel):

result = r.connection { |c| c.fetch(self.query).all }
mymodel.results = result.to_json

As the entire dataset was stored as some compressed JSON I needed to do a bit of manipulation to get it back into a form that was workable. Fortunately all the steps were fairly straightforward.

First you want to unnest each result from the json array, in my case this looked like:

SELECT json_array_elements(result)

The above will unnest all of the array elements so I have an individual result as JSON. A real world example would look something similar to:

SELECT json_array_elements(result)
FROM query_results
LIMIT 2;
json_array_elements

{“column_name”:“data_in_here”} {“column_name_2”:“other_data_in_here”} (2 rows)

From here based on the query I would want to get some specific value. In this case I’m going to search for the text key column_name_2:

SELECT json_array_elements(result)->'column_name_2'
FROM query_results
LIMIT 1;
json_array_elements
-----------------------
"other_data_in_here"
(1 rows)

One gotcha I encountered was when I wanted to search for some value or exclude some value… Expecting I could just compare the result of the above in a where statement I was sadly mistaken because the equals operator didn’t translate. My first attempt at fixing this was to cast in this form:

SELECT json_array_elements(result)->'column_name_2'::text

The sad part is because of the operator the cast doesn’t get applied as I’d expect. Instead you’ll want to do:

SELECT (json_array_elements(result)->'column_name_2')::text

Of course theres plenty more you can do with the JSON operators in the new Postgres 9.3. If you’ve already got JSON in your application give them a look today. And while slightly worse, if you’ve got JSON stored in a text field simply cast it with ::json to begin using the operators.

The Fizzbuzz Bug Tracker A Bug Tracker for Single Developers

Ben E. C. Boyter — Fri, 06 Sep 2013 08:22:26 GMT

This serves as the announcement of my new bug tracker product I am working on.

My needs are pretty specific and none of the existing bug trackers I have tried have meet the following goals.

Speed. Searching, adding projects/issues/comments should be instant. Any time waiting on the bug tracker is wasted time.
Outlook style view of projects/issues/comments. This should allow me to get an overview of how I am tracking.
Not require me to spin up a full web-server. I don’t want to have to install a full stack web-server just to track bugs.
Sync across all devices. I want it to just appear on every device I own without me having to worry about backups or keeping things in sync.
Be single user. I don’t want to have to login or perform user management.
Support multiple projects. I also want to be able to have multiple issues per project.

With that in mind let me introduce the Fizzbuzz Bug Tracker. Fizzbuzz is a single executable that works on Windows, Linux and OSX (once I have a machine to compile it on). A suggested workflow is to copy it onto your Dropbox folder (or equivalent) where it will be synced across all devices. Fizzbuzz has a strong emphasis on speed. Adding projects or issues, searching across projects and issues, drilling into projects or issues is near instant.

Collection of Letters for Neural Network OCR Training

Ben E. C. Boyter — Tue, 03 Sep 2013 08:59:42 GMT

I was looking for this on Google the other day and unable to find it. Essentially what I needed was a collection of images which are all the same size, but of different fonts so that I use them for training Neural Networks and test other OCR techniques. Since I couldn’t find any I thought I would upload my own collection.

I used the below images when working on my thesis. From memory over 20 different fonts and sizes were used to create about 200 examples of each letter. The full data set proved to be pretty accurate when it came to recognizing most examples of text I found on the web.

Adventures in Self Publishing

Pete Keen — Tue, 03 Sep 2013 03:00:00 GMT

Three months ago I decided to write my first technical book and it's earned me over $5,000 in the two weeks since launch day, so I thought I decided to share what I've learned.

I had been reading Nathan Barry's excellent book Authority and something about it inspired me. I started throwing around ideas, things that I knew well and that weren't well covered already, and I turned up Stripe. I know Stripe very well having used it for a bunch of projects in the past few years. I also know Rails, using it in most of those projects plus at my day job. I knew for sure that there were things about payment processing that weren't really talked about much in the 10 minute Stripe tutorials. Thus began my five month journey of writing and self-publishing Mastering Modern Payments: Using Stripe with Rails.

Why self publish?

For me, it wasn't even really a question. There's no way a traditional publisher would be interested in a tiny niche topic like this, and even if they were I wanted as much control as I could get for my first major writing project. Not to mention, the amount of money I would get from direct sales is vastly more than I would get from the same number of sales if I were getting royalties.

So, why not something like Leanpub? Again, it comes down to control. Leanpub provides a valuable service but I wanted to control the entire experience, from building the book files to the landing page all the way through to the sales experience. A great part of the value of the guide is that the Guide + Code edition comes with the code for the application that actually sells the book, and with Leanpub that wouldn't be possible.

Process and Tools

My writing process is pretty simple. I make an outline, at first just broad themes for chapters or sections. I step back, play with the order a little maybe, and then start adding more detail to the outline. At some point I get bored with that and start filling in chunks with prose. This is the same process I've used for most of my blog posts and it's been pretty successful.

Editing was a much bigger job for the guide than it has been for my blog, of course. I did a first pass by hand on actual paper with an actual red pen and then followed that up with another pass on the computer. Shortly after that I released the guide as preorders to my mailing list (more on that below) and they provided vast quantities of feedback, ranging from typos to bug fixes all the way through to high level feedback on the shape and intent of the guide. I can't thank those reviewers enough. They made it possible to have a polished product on launch day.

The tools I used for most of the writing process:

Emacs and Markdown mode. The book is entirely written in Markdown, except for a very few sections written in raw HTML. I use the excellent Markdown Mode for basically everything I write, since it basically just gets everything right.
Docverter. I use Pandoc and Flying Saucer via a local instance of Docverter to build the various formats of the guide.
Rake and Redcarpet. I wrote a few different custom Markdown renderers using Redcarpet that do things like highlight code samples with Pygments, check the syntax of all of the Ruby code samples, spider all of the links in the book, and finally render out the table of contents how I prefer it.
Gimp for the cover design.

I've open sourced the code that I used to build the book. It's very specific for Mastering Modern Payments and the process that I eventually settled on, but I'm hopeful that it will provide inspiration for others that might be contemplating the same kind of self-publishing venture. It includes the pdf template and style sheet I use, along with the Rakefile that contains all of the logic for building the guide in it's various formats.

Preorders

The first thing I did after actually deciding to write MMP was to put up a landing page and start collecting email addresses. On July 15th I started selling preorders, which in this case consisted of 30% off the final purchase price in exchange for an advanced copy of the guide along with regular weekly updates as people sent in feedback. This turns out to have been highly lucrative and extremely motivational. Preorder sales totaled over $3,000. Dozens of people were interested enough in this project to pay early. If you're developing an info product you should definitely consider some sort of preorder arrangement.

Numbers

I kept a simple journal throughout the main writing period and adding up all of that I wrote for about 70 hours. The total hours spent on the whole project is probably double that because I didn't count much of the development of the companion Rails application, editing the guide, writing emails to customers, nor of all of the other little tasks that go into a project like this.

As of yesterday, the guide has grossed $8,603 (this includes preorders):

99 copies of Just the Guide for $2,485
108 copies of Guide + Code for $5,341
3 copies of the Team license for $777

(note that these numbers don't add up to the straight price * copies amount because I've had both 10% and 30% discounts going at various times)

&&&&&

The biggest driver by far was HN traffic from two posts on launch day, August 15th. The second biggest driver was a link in Ruby Weekly. I purchased an ad on /r/rails and sponsored The Changelog podcast, neither of which directly generated sales but may have driven direct site visits and sales later on. There wasn't a whole lot of time to put thought into tracking visits from The Changelog so I don't have any direct numbers.

Since launch day the landing page has gotten 6,084 unique page views driving 136 sales for an overall conversion rate of 2.2%. Prior to launch day, the mailing list converted over 30% for preorders.

Lessons Learned

Get help sooner in the process

When I started this project I was all by myself which hampered editing and reviewing a lot. I didn't get my first comprehensive external review of the thing until after I had sold the first few copies. Next time I'll assemble a few reliable technical reviewers ahead of time so that they can start reading drafts before publication.
Write more guest blog posts

This is something that Nathan espouses in Authority and I just didn't have time to get to. I only had one guest blog post on launch day and it didn't generate much traffic at all, let alone sales. Next time I'll have at least three lined up, as well as having coverage in the relevant weekly newsletters and podcasts set up beforehand.
Write more relevant content for my email list

Early on I was sending interesting little Stripe and Rails things to my mailing list, but after preorders started it became product updates every time. I think at some point I started overselling them. For my next product I'm going to try to alternate product updates with interesting, relevant information that isn't directly related to the product.
Don't have so much going on while developing and selling

I had far too many things going on when I decided to launch. Here's an abbreviated schedule for launch day and the following:

2013-08-12 — Start packing for move
2013-08-15 — Launch day AND travel day on airplanes
2013-08-16 — Out of internet range, getting things set up for the wedding
2013-08-17 — Wedding Day
2013-08-18 — Travel back to Portland, get ready to move
2013-08-23 — Pack everything we own and start driving across the country
2013-08-28 — Attend a funeral
2013-08-29 — Sign a new lease in Michigan

I knew about the wedding and moving well ahead of time, so why did I pick August 15th? Lack of critical thinking, clearly.

Conclusion

I'd like to take the opportunity to thank my wife, who has put up with not only this project but all of my projects over the last two years. She's the most amazing, understanding person that I've ever met. I'd also like to thank Michael Buckbee, Andrew Culver, and a whole host of other people who kept me motivated and on track, right to the end.

This project has been an amazing experience. I've learned so much about how to build a product, how to build an audience, and how to make a success, and it's not even over. I'll be releasing periodic updates with corrections and new information since both Stripe and Rails are both active, changing products.

Saving Resources for Humans in PHP

Ben E. C. Boyter — Tue, 27 Aug 2013 03:23:43 GMT

One of the issues I have run into running searchcode.com is that a huge amount of time spent serving pages is serving them to bots (about 900,000 hits a day are from bots). I have been watching the load averages and they can spike to over 30.00 occasionally which is pretty bad for a 4 core/ht system. I don’t have any problems with bot’s really other then the fact that I cannot really control them. Sure you can specify crawl delays but its of limited use if the bot chooses to ignore it.

Mastering Modern Payments Is Out Today!

Pete Keen — Thu, 15 Aug 2013 03:00:00 GMT

I'm so proud to announce that Mastering Modern Payments: Using Stripe with Rails is officially launching this morning. Mastering Modern Payments is your guide to integrating Stripe with your Rails application and is packed with sample code and best practices that will make sure your integration works now and in the future.

I've collected for you the best resources from around the web and added my own experiences and stories to help you make your Stripe integration as robust as it can be. Click below to check it out!

Find Out More about Mastering Modern Payments

The Rule of Thirds - followup

CRAIG KERSTIENS — Tue, 13 Aug 2013 23:55:56 GMT

Several months back I wrote about how we do higher level, long term planning within the Heroku Postgres team. If you haven’t read the previous article please start there.

The exercise or rule of thirds is intended to be approximate prioritization and not a perfect science. Since that time I’m familiar with some teams both in and out of Heroku who have attempted this exercise with varying levels of success. We’ve now done this process 4 times within the team and after the most recent exercise attempted to take some time to internalize why its worked well, creating some more specifics about the process. Heres an attempt to provide even more clarity:

Gather data ahead of time

Its really common to have a list things to work on, but knowing the impact of those is commonly pure speculation. There may be some people that talk to customers, but even then its a subset of your actual customer base. Going into the exercise as much data you can have ahead of time on impact of features and specific problems helps. In our case we do this by:

Surveying current customers and users
Surveying attriters
Engaging with customer facing teams to hear trends
Input from external parties such as analysts on trends

Allow for casual discussion

We typically conduct our planning exercise at an offsite, this is a multi-day time of team bonding, planning, hacking. We intentionally schedule our planning excercise towards the end of the offsite. This allows us to have updates/presentations frmo the data we’ve gathered and from those that are customer facing. Presentations are meant to be short and direct, discussion can flow casually after. This gets a lot of people on the same page at a smaller level and reduces the problem of too many cooks in the kitchen come time for the actual exercise.

The rule of thirds

Creating the list

Coming to the exercise itself… We begin by everyone writing a list of their ideas individually, this is meant to be a list of the features we want to place on the grid. At this point theres no prioritizing of difficulty or impact. In addition each list while individually created does not have to contain items that only pertain to you, its more a comprehensive list of all the things you can think of that may be important to do.

Bucketing part 1

Once individual lists are created you can then collectively or designate one or two people to clean it up. We do this in two forms:

Removing duplicate items, which there should be several of.
Bucketing my a common/theme idea, this simply makes things more digestable

If you’re a big group of greater than 7 then it may be advisable to designate two people to do this exercise together. If a smaller group it can be manageable to coordinate collectively.

Bucketing part 2

Once you’ve removed dupes, identified themes, and removed excess items (depending on your team size you’ll find how many feels right - we aim an average of 5-6 per square for a team of 10) its then on to actually putting them on the grid. In the past we’ve done this a variety of ways but our most recent process seemed to be quiet efficient. We gave each item 60 seconds, at the end of that minute wherever the item was it was left there. This forced some quick discussion on impact and difficulty but in the end left us at a very good hit rate without taking multiple hours to complete the exercise.

Final pass

We intentionally design it so that low effort and high impact is on the top right corner. Finally once everything is on there we allocate names to the tasks, and put boxes around items we’re planning to do in the coming months. With boxes make it very clear of what we are doing as well as explicitly things we are not. The initials or names make it clear of how loaded down people are. If your name is on 3 tasks that are high difficulty, then you’re likely over allocated.

At this point things usually fall out pretty quickly and we emerge with some rough roadmap that in retrospect we’ve followed pretty accuately.

C# Vector Space Implementation

Ben E. C. Boyter — Tue, 13 Aug 2013 08:59:09 GMT

Since I am writing lots of Vector Space implementations in Go, Python etc… I thought I would add another one in C#. This one is a little more verbose then either the Python or Go implementations. The verbosity is mostly due to not using any of the nice C# LINQ functionality which would really reduce the size.

In any case here it is in case you are looking for a simple implementation of this useful class.

GoLang Vector Space Implementation

Ben E. C. Boyter — Mon, 12 Aug 2013 09:51:30 GMT

UPDATE – This is now actually available as a real Golang import with tests. Get it at https://github.com/boyter/golangvectorspace

I have mentioned this before somewhere but one of the first things I usually attempt to implement in any programming language I want to play with is a vector space. Its my own personal FizzBuzz implementation. It usually covers everything I need to know in a language (imports, functions, string manipulation, math functions, iteration, maps etc…) so I consider it a good thing to get started with.

The missing PostgreSQL documentation

CRAIG KERSTIENS — Wed, 07 Aug 2013 23:55:56 GMT

For a couple of years I’ve complained about the Postgres documentation and at the same time paraded it as one of the best sets of documentation I’ve encountered. In many ways the reason I veer towards Postgres as well as Python and Django is the quality of their documentation. If you need to find details about something its documented, and more importantly well and thoroughly documented.

In large part I came to Python by happenstance through Django, and Postgres through happenstance of an employer. Yet, Django was very little of an accident. The Django Tutorial got me a large part of what I needed to know and more excited about development than I had been in some time. Python has done some work at adding docs to make this even better, sadly its still very much needed for PostgreSQL.

Whats Missing in the Postgres Docs

Theres a huge variety of types of documentation, off the top of my head theres:

Reference docs (Postgres excels at this)
Onboarding (Postgres tutorial huh?)
Tailored guides (Postgres? I can haz? Nope… We don’t understand….)

Postgres is great if you know the name of what you’re looking for, but if you don’t you’re entirely left in the dark.

Understanding the power of Postgres

Postgres is good enough at performance, good enough at usability, and awesome at how powerful and flexible it can be. But all of this is entirely lost if you have to know the esoteric name of what you’re looking for.

What the hell is an hstore… In so many ways KVstore makes infintely more sense. In the same sense PLV8, I have to know not only what PL stands for but V8 as well, versus the JavaScript extension for Postgres.

I understand there are plenty of reasons why some of these things are the way they are, but its also limiting how great the broader perception is. Postgres externally is this hard to use DB, that well is just a database, versus giving developers a set of powerful and useful functions to make their lives better.

The Solution

Lets fix things, there are a ton of people that would love to know more about all things Postgres. This ranges from a good set of onboarding docs, to specific blog posts on topics that people are curious about. Just last week I got an email about improving the Postgres tutorial… Yes theres a tutorial hidden in the 2000 page set of documentation for Postgres. Its simply old, mostly uninteresting, and well just needs to be completely recreated. A great alternative would be a few tutorials/guides for:

Noobs to databases in general (Total 101 guide)
Building and architecting your application with Postgres (App Devs)
Administering and maintaining Postgres (DBAs)
SQL and reporting in Postgres (consumers of data, analysts, product people, marketing, etc.)

If jumping in and contributing to fixing the core tutorial isn’t your cup of tea because you don’t want to learn and write in SGML, send a pull request to postgresguide.com or do a [guest post on my blog](mailto:craig.kerstiens@gmail.com]. If thats too much effort please just let us know, what do you want to see - craig.kerstiens at gmail.com

A look at Foreign Data Wrappers

CRAIG KERSTIENS — Mon, 05 Aug 2013 23:55:56 GMT

There are two particular sets of features that continue to keep me very excited about the momentum of Postgres. And while PostgreSQL has had some great momentum in the past few years these features may give it an entirely new pace all together. One is extensions, which is really its own category. Dimitri Fontaine was talking about doing a full series just on extensions, so here’s hoping he does so I dont have to :)

One subset of extensions which I consider entirely separate is the other thing, which is foreign data wrappers or FDWs. FDWs allow you to connect to other data sources from within Postgres. From there you can query them with SQL, join across disparate data sets, or join across different systems. Recently I had a good excuse to give the postgres_fdw a try. And while I’ve blogged about the Redis FDW previously, the Postgres one is particularly exciting because with PostgreSQL 9.3 it will ship as a contrib module, which means all Postgres installers should have it… you just have to turn it on.

Let’s take a look at getting it setup and then dig into it a bit. First, because I don’t have Postgres 9.3 sitting around on my system I’m going to provision one from Heroku Postgres:

$ heroku addons:add heroku-postgresql:crane --version 9.3

Once it becomes available I’m going to connect to it then enable the extension:

$ heroku pg:psql BLACK -acraig
# CREATE EXTENSION postgres_fdw;

Now its there, so we can actually start using it. To use the FDW there’s four basic things you’ll want to do:

Create the remote server
Create a user mapping for the remote server
Create your foreign tables
Start querying some things

The setup

You’ll only need to do each of the following once, once you’re server, user and foreign table are all setup you can simply query away. This is a nice advantage over db_link which only exists for the set session. One downside I did find was that you can’t use a full Postgres connection string, which would make setting it up much simpler. So onto setting up our server:

# CREATE SERVER app_db
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (dbname 'dbnamehere', host 'hostname-here);

Next we’ll actually create our user mapping. In this case we’ll take the remote username and password and map it to our current user we’re already connected with.

# CREATE USER MAPPING for user_current
SERVER app_db
OPTIONS (user 'remote_user', password 'remote_password');

And finally we’re going to configure our tables. There were some additional pains here as there wasn’t a perfectly clean way to generate the CREATE TABLE. Sure you could pg_dump just that table, but overall it felt a bit cludgey.

# CREATE FOREIGN TABLE users
(
id integer,
email text,
created_at timestamp,
first_name text,
last_name text
)
SERVER app_db OPTIONS (table_name 'users')

Now we’ve got all of our local data, as well as remote data. For that report against two databases where you previously wrote a ruby or python script, ran a query, constructed another query, then executed it you can directly do in your database. We can simply query our new table - SELECT * FROM users LIMIT 5;

But the real power of foreign data wrappers goes well beyond just Postgres to Postgres. Having a defined contract in translating from one system to another, will really allow reinventing the way we work with data. This is especially true in large datasets where doing ETL on terrabytes of data takes longer than asking the questions of it.

While we’re waiting for more FDWs to be ready to use in production situations the Postgres FDW is a great start, though the Redis one is on its way. Even better is that it ships with standard installs of Postgres, meaning it will see more usage and help push them to advance further.

One final nicety, you’re not required to have ALL Postgres 9.3 DBs, just one that can then connect to the others, so go ahead and give it try :)

EverQuest Next Revealed

Ben Overmyer's Site — Sat, 03 Aug 2013 03:00:00 GMT

I’m writing this just a couple minutes after watching the EverQuest Next reveal video on Twitch. I couldn’t watch it live yesterday, but it was totally worth the wait. For an hour and a half, I was on the edge of my chair. The introduction was a beautiful piece of sand performance art, which I otherwise won’t talk about. The rest of the video was an introduction into what will probably be the single most revolutionary MMORPG to come out since 2003.

Dave Georgeson, director of development for EverQuest Next, started things off with a reveal of the art style. Let me tell you, this is a gorgeous game. The whole thing is very painterly, and is rendered using a modified version of the engine powering Planetside 2.

All of the races have been redone, and we were treated to concept art for several: the High Elf, the Dark Elf, the Human, the Dwarf, and the Kerran. The Dark Elf and the Kerran were probably the biggest departure from previous EQ designs. Kerrans are distinctly more lionesque, with pronounced lion’s heads instead of the old cat faces and huge builds. Dark elves look like standard dark elves, except they have draconic eyes and short horns. There’s a story behind that that I really want to hear.

The art style for characters is exaggerated, even slightly cartoony. I’m really OK with that, especially since they’re building SOEmote into the game from the beginning. It’s easy to see the expressions of the players themselves on their characters, and that’s marvelous for a roleplayer, but also for immersion and socializing in general.

Movement is fantastic. Characters slide, leap, vault, grab, and otherwise interact with the terrain in a very natural way. With EQN’s “active movement system,” players are free to really enjoy moving around in the world. There are items and spells that allow additional ways to move, like the Boots of the Zephyr in the video that allowed the players to float gently downward. Brilliant stuff.

Then, we got into the Four Grails, as Dave put it.

The Core Game

Grail 1 is the “Core Game.” This was the least impressive of the Grails, but it was still nice to see. Character evolution from a mechanical standpoint is based on collecting classes from around the world and mixing and matching abilities from them. It sounds like a hybrid of Guild Wars 2’s system and Rift’s system. Weapon choice matters, like it does in GW2.

Destructibility

Grail 2 is “Destructibility,” and this was the first mind-blowing revelation. The entire game world is built of voxels, despite looking natural and perfectly designed. It’s fully destructible, even though it heals over time. It’s like if SOE took a look at Minecraft and applied that on a grand scale. The way it works, too, is gorgeous. You can definitely see the aftermath of epic battles in the environment that it occurred in.

A Life of Consequence

Grail 3 is “A Life of Consequence.” Besides the standard “every action you take matters” spiel, we were introduced to the idea of emergent AI in an MMO. SOE will create mob behavioral patterns - likes, dislikes, etc. - and then release mobs of that type into the world. They will live, react, and die according to how they’re programmed to behave. This means it’s entirely possible to have roving bands of orcs that actually decide to set up camp in a particular area - choose their own spawn point, as it were. It’s also possible to drive them out.

Permanent Change

Grail 4 is “Permanent Change.” For the most part, Dave talked about the Rallying Cry system, which is a little like a server-wide public quest that lasts 2-3 months each… and has permanent effect on the game world. These Rallying Cries are promised to be different on every server, and that along with the Life of Consequence and Destructibility Grails ensure that no two servers will have the same environment and history, let alone community.

EverQuest Next Landmark

As if all this wasn’t enough, then they got into the EverQuest Next Landmark game that’s coming out later in 2013. Landmark sounds basically like Minecraft on an epic scale, except that one continent in each Landmark server will have EverQuest Next art direction enforced… and creations from those areas will make it into EverQuest Next as real, lasting landmarks.

Player Studio

On top of that, SOE’s Player Studio will find a very interesting real money economy implementation for people who enjoy building things in Landmark; you’ll be able to sell not only finished works, but components. The example that Dave gave was of a player creating a nice crenelated tower. People could buy that tower, then make castles from it and sell those castles, and the player who created the tower would get royalties based on how many of that tower were used in the castle design for every castle sold. It’s a fascinating system.

Basically, EverQuest Next sounds like the game I’ve been waiting for my entire gaming life. I really want to hear about how the crafting system works, and more about all the races, but… it sounds awe-inspiring. EverQuest Next takes player involvement in the game to a level never before seen.

EverQuest Next - Landmark Focuses on Construction and Exploration, Is Free-to-Play

EverQuest Next + Jeremy Soule = Geek Delight

EverQuest Next Redefines Next-Gen MMOs Enhanced by Zemanta

Postgres Dollar Quoting

CRAIG KERSTIENS — Fri, 02 Aug 2013 23:55:56 GMT

After my most recent post on documenting your database I had a colleague and friend chime in:

{% blockquote @danfarina https://twitter.com/danfarina/status/362007008079126528 %} @craigkerstiens You may want to mention for another post the generality of dollar quoting: it’s not just for CREATE FUNCTION. {% endblockquote %}

Luckily I was able to convince him to create the post. You can read a bit more on him below, but without further adieu here’s a bit on dollar quoting within Postgres:

Postgres supports two forms of entry of data literals into the system. One is the familiar single-quote:

=> SELECT 'hello';
?column?
----------
hello
(1 row)

This format is problematic when one might be using single quotes in the textual string.

Postgres also supports another way to enter data literals, most often seen in CREATE FUNCTION, but can be profitably used anywhere. This is called “dollar quoting,” and it looks like this:

=> SELECT $$hello's the name of the game$$;
?column?
------------------------------
hello's the name of the game
(1 row)

If one needs nested dollar quoting, one can specify a string, much like the ‘heredoc’ feature seen in some programming languages:

=> SELECT $goodbye$hello's the name of the $$ game$goodbye$;
?column?
---------------------------------
hello's the name of the $$ game
(1 row)

This can appear anywhere where single quotes would otherwise be, simplifying tasks like using contractions in database object comments, for example:

=> CREATE TABLE described(a int);
=> COMMENT ON TABLE described IS $$I'm describing this,
including newlines and an apostrophe in the contraction "I'm."$$;

Or, alternatively, entry of literals for types that may include apostrophes in their serialization, such as ’text’ or ‘json’:

=> CREATE TABLE json(data json);
=> INSERT INTO json(data) VALUES
($${"quotation": "'there is no time like the present'"}$$);

Security

Even though dollar quotes can be used to reduce the pain of many quoting problems, don’t be tempted to use them to avoid SQL injection: an adversary that knows one is using dollar quoting can still mount exactly the same kind of attacks as if one were using single quotes.

There is also no need, because any place a data literal can appear can also be used with parameter binding (e.g. $1, $2, $3…), which one’s Postgres driver should support. Nevertheless, for data or scripts one is working with by hand, dollar quoting can make things much easier to read.

About the Author

Daniel Farina is a long time colleague and friend, having worked together at 5 different companies. He’s part of the Heroku Postgres team as the resident tuple groomer, and the creator of WAL-E.

As is always the case if you have articles you’d like to see created or if you’re interested in doing a guest post please feel free to drop me a line craig.kerstiens at gmail.com. And if you have articles you feel are helpful to others in the Postgres world drop me a note as well for including them in Postgres Weekly.

Automatically inline Python function calls

Tom Forbes — Thu, 01 Aug 2013 18:54:18 GMT

Edit: Code is here on GitHub Calling functions in Python can be expensive. Consider this example: there are two statements that are being timed, the first one calls a function that returns an integer while the second one calls a function that returns the result of a second function call which return...

Clean Repository Data Access in C#

Ben E. C. Boyter — Thu, 01 Aug 2013 02:52:14 GMT

Mostly as a self reference here is an extremely clean data access pattern possible using C# and Entity Framework. It saves you the effort of mocking the database context as the code you end up writing is so simple it is all compile time checked.

Essentially you define a very simple class which provides a single method for getting data (although you may want a save data method too) and make sure you add an interface to make unit testing/mocking easier.

Documenting your PostgreSQL database

CRAIG KERSTIENS — Mon, 29 Jul 2013 23:55:56 GMT

Just a few days ago I was surprised by what someone was doing with their database, and not in the typical horrifying travesty against mankind. Rather, it was a feature that while familiar with I’d never seen anyone fully take proper advantage of - COMMENT or describing tables. Postgres has a nice facility for you to provide a description for just about anything:

Table
Column
Function
Schema
View
Index
Etc.

The specific use case was a database acting as a datamart pulling in data from multiple sources to be able to report against disparate data. Over the years I’ve seen this occur really one three ways, the first is that a limited set of people, typically one person, have knowledge over all the datasources and thus far the sole individual responsible for creating reports and answering questions of the data. The second, is wide open access to anyone that wishes for it. In this case you often have people asking questions of the data, and because they don’t understand the relationships coming up to entirely wrong conclusions. The final approach is to create some external documentation, entity relationship diagrams, data dictionaries, etc. This last one often works okay enough, but often suffers from lack of updates and being too heavyweight.

A better solution, and all around good process is simply documenting clearly within the database itself. Simply comment each table and column, just as you would outside of your DB then it can be quite clear when inside the database working interactivly:

COMMENT ON TABLE products IS 'Products catalog';
COMMENT ON COLUMN products.price is 'Current price of a single item purchased';

While an obvious example above naming even the most mundance columns can help create more accurate reports. Then of course when you want to inspect your DB its quite clear:

\d+ users
# \d+ users
Table "public.users"
Column | Type | ... | Description
------------+-----------------------------+-...-+-----------------------------------------
id | integer | ... | auto serial pk
first_name | character varying(50) | ... | required first name of user
last_name | character varying(50) | ... | required first name of user
email | character varying(255) | ... | email address of account
data | hstore | ... | mix of data, city, state, gender
created_at | timestamp without time zone | ... | when account was created, not confirmed
updated_at | timestamp without time zone | ... | time any details were last updated
Indexes:
"idx_user_created" btree (date_trunc('day'::text, created_at))
Has OIDs: no

But it doesn’t necessarily have to stop there. Which actually brings me to one other item, you should be commenting your SQL just the same. SQL comments can be done easily by just starting a line with --, or you can have it at the end of the line with further info. Here’s a nice example:

-- Query aggregates all project names that have open past due tasks grouped by email
SELECT
users.email,
array_to_string(array_agg(projects.name), ',')) as projects # Aggregate all projects and separate by comma
FROM
projects,
tasks,
users
-- A user has a project, which has tasks
WHERE projects.id = tasks.project_id
-- Check for tasks that are due before now and not done yet
AND tasks.due_at > tasks.completed_at
AND tasks.due_at < now()
AND users.id = projects.user_id
GROUP BY
users.email

You comment your code, why shouldn’t you comment your database?

Gigablast Aquired and Code Posted

Ben E. C. Boyter — Mon, 29 Jul 2013 08:58:30 GMT

Interestingly it seems that Matt Well’s search engine Gigablast has been acquired by Yippy.com [1]1 [2]2 [3]3 (demo here http://demo.yippy.com/). Gigablast has always been one of my favorite search engines simply because it is so interesting. Started by a single guy, with an interesting blog and being one of the last true new indexes of the web it was always worth a look. While its sad to see it go this way I am happy that Matt presumably has been able to cash out on his creation. Well done to him. I must admit Gigablast had been going downhill for a while and this might explain why ProCog appeared and then vanished so quickly.

Reawakening the Starborn

Ben Overmyer's Site — Sat, 27 Jul 2013 03:00:00 GMT

About a year and a half ago, I started a project to create a sci-fantasy sandbox MMORPG. It was called Starborn: Universe. I went through the trouble of getting a HeroEngine/HeroCloud environment and began learning the toolset. Design documents popped up in my Documents folder and on the dev blog I started for the project. Then, I got busy with other things, and Starborn: Universe went dark. Until now. I’m dusting off the design documents, but Starborn: Universe is as much of an experiment as it is a game. See, my theory is that online player-driven games can start with very little and grow over time. The idea of the “Lean Startup” can apply just as much to an MMORPG as it can to an iOS app. The trick is that MMORPGs are hard to produce. They have a lot of moving parts and require a lot of resources on the part of the developer and operator. So how does one or two people accomplish this monumental feat? By starting small, and iterating quickly. If you’re curious as to what I have in mind, head over to the Starborn: Universe devlog. That’s where I’ll be writing about Starborn.

CoffeeScript's Scoping is Madness

Donat Studios — Fri, 26 Jul 2013 01:19:44 GMT

CoffeeScript is a programming language that compiles to JavaScript. Its mission is to "expose the good parts of JavaScript in a simple way". I've heard a lot of interest in it lately from various channels.

After a coworkers departure, I inherited maintenance on our deployment bot, and it was written in CoffeeScript. While trying to make some small modifications, I ended up getting unexpected issues I didn't understand. Things I did not touch were suddenly breaking. I did not understand until I investigated the generated JavaScript.

It ends up CoffeeScripts approach to scoping of variables is to not scope variables. There is no shadowing. There is no var, let nor const equivalent, there is simply no way to explicitly define scope.

Variables in CoffeeScript are scoped to their outermost definition, regardless of whether the name was already in use elsewhere. When you use a variable of a name, its scope now includes all inner scopes, trampling any existing definitions of the variable in those scopes.

Let's examine the following examples:

CoffeeScript

test = (x) ->
  y = 10
  x + y;

alert test 5

Generated JavaScript

var test;

test = function(x) {
  var y;
  y = 10;
  return x + y;
};

alert(test(5));

Now let's make one small change and define a variable y in the outer scope, outside the function.

CoffeeScript

y = 0;
test = (x) ->
  y = 10
  x + y;

alert test 5

Generated JavaScript

var test, y;

y = 0;

test = function(x) {
  y = 10;
  return x + y;
};

alert(test(5));

The Problem

Notice the highlighting indicating the scope of the y variable. By defining y in the global scope all subsequent y's are also now that global. You may no longer have a variable named y in your code, at any deeper scope. The outer y is the only y.

Creating a global of a name given overrides all subsequent definitions.

In JavaScript var, let, and const keywords define the scope of a variable. Leave it out and it scopes downward until it hits a var definition of that variable or the global level. This means that inner scopes inherits from outer scopes unless it defines its own scope of the variable. CoffeeScript on the other hand offers you no method to scope a variable. Scoping is entirely left to the lowest level use of a variable name.

This behavior is dangerous and makes code very difficult to maintain. The problem is twofold. Firstly, when writing a function, you need to be aware of all shallower scopes variable definitions, as not to trample them. Secondly, when altering code at any scope but the outermost, you need to be aware of all deeper scopes variables as to not trample any of them.

The only "safe" way to write CoffeeScript is to assume all variable names are global. In practice they are.

There have been proposed solutions, such as a Go-lang style := for inner scoping, but these have not been accepted as of this writing. The creator of CoffeeScript, Jeremy Ashkenas, when questioned about it by Armin Ronacher on Twitter replied:

Not gonna happen ;) Forbidding shadowing altogether is a huge win, and a huge conceptual simplification.

This is not true for applications of any scale larger than toy. It makes development more complex, having to check every inner scope before using a new variable name. This is not simpler. The entire behaviour of all inner scopes can change with a single bad variable defintion.

Finally

CoffeeScripts goal of "expos[ing] the good parts of JavaScript in a simple way" was missed. Instead of improving JavaScript's scoping which can be kind of daunting to learn, they actually made it worse. If you want to improve JavaScript make every variable local until explicitly defined global, as is the behavior in the vast majority of languages.

JavaScripts "global unless defined as local" behavior is weird, but manageable. CoffeeScript's "local until defined elsewhere globally" behavior on the other hand is simply difficult to predict and makes it easy to accidentally change the entire meaning of code even when you entirely understand the behavior.

It is a bad behavior, period.

Update

I've a fight on Reddit here.
I've irked Reginald Braithwaite, the author of CoffeeScript Ristretto, so badly he's giving away 100 copies of his book for free!
- The author has upgraded me to Not Wrong™!

SSDs are awesome, buy one.

Tom Forbes — Thu, 25 Jul 2013 22:54:58 GMT

I recently brought a Samsung 840 Series Pro 256GB 2.5 inch SATA Solid State Drive and its easily the best PC hardware purchase I have ever made. Before I purchased it I was thinking about replacing my laptop as it was getting pretty sluggish, despite having decent specs even 2 years after I purchase...

Why I still recommend Windows

Nemo's Home — Sun, 21 Jul 2013 03:00:00 GMT

Even though I am a long time Linux user, and a big time fan of the many Linux distros that I’ve tried out over time, I still go around recommending Windows to people who ask me for advice. The only exception I make is when the person in question is a developer, in which case I try to convert them to the Church of Linux.

The main reason I recommend windows to non-developers is because it is a far better operating system than most Linux distributions (for general public). Now, before you bring out your pitchforks, hear me out.

The first and foremost reason that I give is that Windows sports a far better integration across all its services. Nautilus/Nemo in the Linux world do not reach same level of integration that Windows Explorer does. For instance, just look at the way the “Send To” feature works in explorer. To add a folder to the send to entry, you just have to add a shortcut to that folder inside the special “Send To” folder. On nautilus, the equivalent would be going about installing an extension, and editing a configuration file by hand.

Or take a look at how the “Play All” feature in Explorer. Or the “Libraries” feature in Windows 7. Or the simple way that you handle file sharing in Windows. Even though Linux has (arguably better) Samba support for files sharing, you have to go about editing a handful of files to make it work. I personally find apache easier to configure to just share files one way.

Games are another reason. Even though Steam is available on Linux, all of the non-Valve triple-A titles are missing on Linux. Even though I continue to buy and play the Humble Bundles that offer Linux as a platform, I’m reminded of the stark reality every day when my friends ask me if I’ve played a recent title such as Call of Duty, Metro, NFS or even Swapper.

The next peeve that comes to my mind is the ridiculous driver support. It has been improving since a long time, but its still not there. Even Ubuntu needs to fetch proprietary drivers for my WiFi support on Broadcom, which needs an internet connection in the first place. This means I need to find out a LAN network connection to even start using Ubuntu. Similarly the pain I’d to go through to install drivers for Ralink network drivers on a friend’s laptop was immense. I can never use circular scrolling or touchpad zoom on my laptop in Linux because there are still no drivers available anywhere for it. And don’t get me started on UEFI boot issues. No matter what people believe or pretend, hardware support is just not good enough to be relied on in the Linux world. As a side note, I haven’t synced my iPad in Ubuntu since I shifted to iOS 5, and apple driver support on Linux will remain abysmal forever just because iTunes will never be released for Linux. The last version of iOS that had music sync support (via libimobiledevice) was iOS 4.0 (released 3 years ago in June 2010).

Next I want to point put out the upgrade pain that everyone has to go through. It’s like a constant rite of passage, which turns a Linux noob into an actual user. I am yet to do an Ubuntu upgrade which went smoothly and didn’t break a thing; and I’ve been upgrading my Ubuntu since 10.04 was released. The upgrade tag on askubuntu is chock full of horror stories.

Another thing that frustrates me to no end is that the Ubuntu Dash, and the GNOME Overview are both slow as hell. I’m currently using Cinnamon, which is faster than both of these, but still an order of magnitude slower than the Windows Start Menu. Synpase is better, but cannot be set as the deafult.

I was using Windows 7 on my cousin’s laptop these last few days and I remembered the favourite app that I used to no end: Everything. It is the quickest file search I’ve ever used. The alternatives, in the Linux world are synapse, zeitgeist, and plain old locate command. The only issue is that I’ve to manually run updatedb manually, while Everything was always up to date, using the NTFS File Journal. To this date, I am yet to find a good enough alternative to Everything.

It is true that Windows lacks many of the good things that Linux distros provide, such as the excellent package management support, POSIX compatibility, and the plethora of tools we get on the command line; but at the same time, it is also a better operating system for most of the masses. I’ll continue to recommend Windows to all my non-developer friends till “The year of Desktop Linux” arrives.

Returning to Skyrim

Ben Overmyer's Site — Sat, 20 Jul 2013 03:00:00 GMT

The last time I posted about Skyrim, it was shortly after release. You can read the first in the series here. At the time, I was playing on the Xbox 360 version. This time around, I’ve purchased the Legendary Edition for PC. I’m also playing a Destruction mage instead of the heavily armed and armored warrior I focused on in the 360 version.

Besides taking a completely different character route this time around, I’m also focusing much more on quests other than the main line. I own a small house that I’ve built but not yet furnished, gotten a fair ways through the College of Winterhold quest line, and continue to light up vampires that seem to crop up all the time in areas I’m traveling to.

I heard so many awesome things about mods for the game that I just had to get the PC version. That, and I couldn’t take screenshots in the 360 version, which was infuriating for someone who blogs about games. Today, I read an article someone reblogged on Reddit about a mod called Falskaar. Falskaar is unique in that it was produced chiefly by a 19 year old who declined to go to university in order to produce the mod. It took him a full year, and he’s aiming to get a job at Bethesda by proving his dedication.

By all accounts, it’s a damned good mod. I look forward to installing it when I finish with vanilla Skyrim.

If you’re interested in hearing further about my adventures in Skyrim, by all means, let me know in the comments. Otherwise, next week I’ll be writing about a topic near and dear to my heart…. sci-fantasy sandbox MMORPGs.

DNS: The Good Parts

Pete Keen — Fri, 19 Jul 2013 03:00:00 GMT

Frequently I come across confusion with domain names. Why doesn't my website work? Why is this stupid thing broken, everything I try fails, I just want it to work!! Invariably the question asker either doesn't know what DNS is or doesn't understand how something fundamental works. More generally, people think that DNS is scary or complicated. This article is an attempt at quelling that fear. DNS is easy once you understand a few basic concepts.

What is DNS

First things first. DNS stands for Domain Name System. Fundamentally it's a globally distributed key value store. Servers around the world can give you the value associated with a key, and if they don't know they'll ask other servers for the answer.

That's it. That's all there is to it. You (or your web browser) ask for the value associated with the key www.example.com and get back 1.2.3.4.

Basic Exploration and Fundamental Types

The great thing about the DNS is that it's completely public and open so it's easy to poke around. Let's do a little exploring, starting with this domain, petekeen.net which I am hosting on a machine named web01.bugsplat.info. Note that you can run all of these examples from an OS X or linux command line.

First, let's look at a simple domain name to IP address mapping:

$ dig web01.bugsplat.info

The dig command is a veritable Swiss Army knife for querying DNS servers and we'll be using it quite a bit. Here's the first part of the response:

; <<>> DiG 9.7.6-P1 <<>> web01.bugsplat.info
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51539
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

There's only one interesting thing in here. We asked for one record and got exactly one response. Here's the question we asked:

;; QUESTION SECTION:
;web01.bugsplat.info.		IN	A

dig defaults to asking for A records. A stands for address and is one of the basic fundamental types of records in the DNS. An A record holds exactly one IPv4 address. There's an equivalent record for IPv6 addresses named AAAA. Next, let's look at the answer our DNS server gave us:

;; ANSWER SECTION:
web01.bugsplat.info.	300	IN	A	192.241.250.244

This says the host web01.bugsplat.info. has exactly one A address: 192.241.250.244. The 300 is called the TTL value, or time to live. It's the number of seconds that this record can be cached before it needs to be checked again. The IN component stands for Internet and is meant to disambiguate between the various types of networks that the DNS historically was responsible for. You can read about those in IANA's DNS Parameters document (thanks for the correction, mcmatterson!)

The rest of the response tells you things about the response itself:

;; Query time: 20 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Fri Jul 19 20:01:16 2013
;; MSG SIZE  rcvd: 56

Specifically, it tells you how long it took for your server to respond, what that server's IP address is (192.168.1.1), what port dig asked (53, the default DNS port), when the query completed, and how many bytes the response contained.

As you can see, there's an awful lot going on in a single DNS query. Every time you open a web page your browser makes literally dozens of these queries to resolve the web host, all of the hosts where external resources like images and scripts are located, etc. Every single resource involves at least one DNS query, which would involve an awful lot of traffic if DNS wasn't designed to be heavily cached.

What you probably can't see, however, is that the DNS server at 192.168.1.1 contacted a whole chain of other servers in order to answer that simple question of what address does web01.bugsplat.info map to. Let's run a trace to see all of the servers that dig would have to contact if they weren't already cached:

$ dig +trace web01.bugsplat.info

; <<>> DiG 9.7.6-P1 <<>> +trace web01.bugsplat.info
;; global options: +cmd
.			137375	IN	NS	l.root-servers.net.
.			137375	IN	NS	m.root-servers.net.
.			137375	IN	NS	a.root-servers.net.
.			137375	IN	NS	b.root-servers.net.
.			137375	IN	NS	c.root-servers.net.
.			137375	IN	NS	d.root-servers.net.
.			137375	IN	NS	e.root-servers.net.
.			137375	IN	NS	f.root-servers.net.
.			137375	IN	NS	g.root-servers.net.
.			137375	IN	NS	h.root-servers.net.
.			137375	IN	NS	i.root-servers.net.
.			137375	IN	NS	j.root-servers.net.
.			137375	IN	NS	k.root-servers.net.
;; Received 512 bytes from 192.168.1.1#53(192.168.1.1) in 189 ms

info.			172800	IN	NS	c0.info.afilias-nst.info.
info.			172800	IN	NS	a2.info.afilias-nst.info.
info.			172800	IN	NS	d0.info.afilias-nst.org.
info.			172800	IN	NS	b2.info.afilias-nst.org.
info.			172800	IN	NS	b0.info.afilias-nst.org.
info.			172800	IN	NS	a0.info.afilias-nst.info.
;; Received 443 bytes from 192.5.5.241#53(192.5.5.241) in 1224 ms

bugsplat.info.		86400	IN	NS	ns-1356.awsdns-41.org.
bugsplat.info.		86400	IN	NS	ns-212.awsdns-26.com.
bugsplat.info.		86400	IN	NS	ns-1580.awsdns-05.co.uk.
bugsplat.info.		86400	IN	NS	ns-911.awsdns-49.net.
;; Received 180 bytes from 199.254.48.1#53(199.254.48.1) in 239 ms

web01.bugsplat.info.	300	IN	A	192.241.250.244
bugsplat.info.		172800	IN	NS	ns-1356.awsdns-41.org.
bugsplat.info.		172800	IN	NS	ns-1580.awsdns-05.co.uk.
bugsplat.info.		172800	IN	NS	ns-212.awsdns-26.com.
bugsplat.info.		172800	IN	NS	ns-911.awsdns-49.net.
;; Received 196 bytes from 205.251.195.143#53(205.251.195.143) in 15 ms

The DNS is arranged in a hierarchy. Remember how dig inserted a single . after the hostname we asked for before, web01.bugsplat.info? Well, that . is pretty important and stands for the root of the hierarchy. The root DNS servers are run by various companies and governments around the world. Originally there were only a handful of these servers but as the Internet has grown more have been added, so that now there are notionally 13. Each one of these servers, however, has dozens or hundreds of physical machines hiding behind a single IP.

So, at the top of the trace we see the root servers, each represented by an NS record. An NS record maps a domain name, in this case the root, to a DNS server. When you register a domain name with a registrar like Namecheap or Godaddy they create NS records for you.

In the next block you can see that dig randomly picked one of the root server responses and asked it for the A record web01.bugsplat.info. Which root server? Let's ask!

$ dig -x 192.5.5.241

; <<>> DiG 9.8.3-P1 <<>> -x 192.5.5.241
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2862
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;241.5.5.192.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
241.5.5.192.in-addr.arpa. 3261	IN	PTR	f.root-servers.net.

The -x flag tells dig to do a reverse lookup on the given IP address. The DNS responds with a PTR record which maps an IP with a hostname, in this case f.root-servers.net.

Getting back to our original query, the F root server responded with another set of NS servers, this time the ones responsible for the info top level domain. dig asks one of these servers for the A record for web01.bugsplat.info, gets back another set of NS servers, and then asks one of those servers for the A record for web01.bugsplat.info. and finally receives an actual answer. (thanks for the corrections, colmmacc!)

Whew! That would be a heck of a lot of traffic, except that almost all of these entries are cached for a long time by every server in the chain. Your computer caches too, as does your browser. Most of the time DNS resolution will never touch the root servers because their IP addresses hardly ever change. The top level domains com, net, org, etc, are also generally heavily cached.

Other Types

There are a few other types that you should be aware of. The first is MX, which maps a domain name to one or more email servers. Email is so important to the functioning of the Internet that it gets its own record type. Here are the MX records for petekeen.net:

$ dig petekeen.net mx

; <<>> DiG 9.7.6-P1 <<>> petekeen.net mx
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18765
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;petekeen.net.			IN	MX

;; ANSWER SECTION:
petekeen.net.		86400	IN	MX	60 web01.bugsplat.info.

;; Query time: 272 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Fri Jul 19 20:33:43 2013
;; MSG SIZE  rcvd: 93

Note that an MX record points at a name and not an IP address.

The other record type that you should be familiar with is CNAME which stands for Canonical Name and maps one name onto another. Let's look at the response we get for a CNAME:

$ dig www.petekeen.net

; <<>> DiG 9.7.6-P1 <<>> www.petekeen.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16785
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.petekeen.net.		IN	A

;; ANSWER SECTION:
www.petekeen.net.	86400	IN	CNAME	web01.bugsplat.info.
web01.bugsplat.info.	300	IN	A	192.241.250.244

;; Query time: 63 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Fri Jul 19 20:36:58 2013
;; MSG SIZE  rcvd: 86

The first thing to notice is that we get back two answers. The first says that www.petekeen.net maps to web01.bugsplat.info. The second gives the A record for that server. One way to think about a CNAME is as an alias for another domain name.

Why CNAME is Messed Up

CNAMEs are incredibly useful, but they have one very important gotcha: if a CNAME exists for a particular name, that is the only record allowed for that name. No MX, no A, no NS, no nothing. This is because the DNS substitutes the CNAME's target for its own value, so every record valid for the target is also valid for the CNAME. This is why you can't have a CNAME on a root domain like petekeen.net, because you generally have to have other records for that domain like MX.

Querying Other Servers

Let's say for sake of argument that you messed up a DNS configuration. You think you've fixed the problem, but you don't want to wait for the cache to expire to see. With dig you can actually query one of a number of public DNS servers instead of your default server like this:

$ dig www.petekeen.net @8.8.8.8

The @ symbol followed by an IP address or hostname tells dig to query that server on the default DNS port. I use this a lot to query Google's public DNS servers or Level 3's sort-of-public servers at 4.2.2.2.

Common Situations

In this last section we'll talk about some common situations that web developers find themselves in.

Redirect bare domain to www

Almost always you'll want to redirect a bare domain like iskettlemanstillopen.com to www.iskettlemanstillopen.com. Registrars like Namecheap and DNSimple call this a URL Redirect. In Namecheap you would set up a URL Redirect like this:

The @ stands for the root domain iskettlemanstillopen.com. Let's look at the A record for that domain:

$ dig iskettlemanstillopen.com
;; QUESTION SECTION:
;iskettlemanstillopen.com.	IN	A

;; ANSWER SECTION:
iskettlemanstillopen.com. 500	IN	A	192.64.119.118

That IP is owned by Namecheap and is running a small web server that just serves up an HTTP-level redirect to http://www.iskettlemanstillopen.com:

$ curl -I iskettlemanstillopen.com
curl -I iskettlemanstillopen.com
HTTP/1.1 302 Moved Temporarily
Server: nginx
Date: Fri, 19 Jul 2013 23:53:21 GMT
Content-Type: text/html
Connection: keep-alive
Content-Length: 154
Location: http://www.iskettlemanstillopen.com/

CNAME to Heroku or Github

Notice in the screenshot above that there's a second row defining a CNAME. In this case www.iskettlemanstillopen.com maps to an application running on Heroku. You'll have to set up Heroku with a similar domain mapping, of course:

$ heroku domains
=== warm-journey-3906 Domain Names
warm-journey-3906.herokuapp.com
www.iskettlemanstillopen.com

Github is similar, except that the mapping lives in a file called CNAME at the root of your pages, as described in their documentation.

Wildcards

Most DNS servers allow you to set up DNS wildcards. For example, I have a wildcard CNAME set up for *.web01.bugsplat.info that maps to web01.bugsplat.info. That way I can host arbitrary things on web01 and not have to create new DNS entries for them every time:

$ dig randomapp.web01.bugsplat.info

;; QUESTION SECTION:
;randomapp.web01.bugsplat.info. IN	A

;; ANSWER SECTION:
randomapp.web01.bugsplat.info. 300 IN CNAME	web01.bugsplat.info.
web01.bugsplat.info.	15	IN	A	192.241.250.244

Wrap Up

Hopefully this gives you a good beginning understanding of what DNS is and how to about exploring and verifying your configuration. Just remember that you can always ask the DNS questions and generally get back answers. The Internet standards (RFCs) that define DNS are:

There are a few more interesting RFCs as well, including 4034 about a standard named DNSSEC and 5321 which talks about DNS as it relates to email. These are all fascinating reads if you want more background information

This article is featured in Hacker Monthly issue 42.

List of useful CAPTCHA Decoding Articles

Ben E. C. Boyter — Fri, 19 Jul 2013 02:46:43 GMT

This website ranks quite high in most search engines for the search term “captcha decoding” or some permutation of it. As such here are a collection of useful links if you are looking into doing such a thing. If any more come up I will be sure to update this post.

http://www.boyter.org/decoding-captchas/

Shameless self promotion but this link is why this page ranks so highly. Its an article I wrote some time ago about how to go about decoding a simple CAPTCHA. There is full source code and the principles can be applied to 90% of CAPTCHA’s out there. For the record it only came about because a colleague bet me that I couldn’t decode his websites CAPTCHA which was the one used in the article. Of course I waited till he changed it before publishing.

Displaying a processes output on a web page with Websockets and Python

Tom Forbes — Mon, 15 Jul 2013 21:40:38 GMT

A few days ago a colleague of mine asked me how you would pipe the standard output of a process into a browser. I hacked around for a few hours and came up with a websockets based solution (using Twisted and Autobahn.ws ) that you can see below ( Your browser needs to support WebSockets, sorry IE9 a...

Announcing: Mastering Modern Payments: Using Stripe with Rails

Pete Keen — Mon, 15 Jul 2013 03:00:00 GMT

Over the past few years I've put together a number of projects that use Stripe and their Ruby API to collect payments and manage subscriptions. I've learned quite a bit about how to effectively use the things that Stripe provides to my best advantage. Two months ago I decided that I would like to share that knowledge and so I started working on a guide to integrating Stripe with Rails and today I'd like to announce that Mastering Modern Payments: Using Stripe with Rails will be available on August 15th, 2013.

Mastering Modern Payments covers the gamut of Stripe integrations, from the simplest using checkout.js all the way through custom payment forms, subscriptions, background workers and more. It's my first ebook and I couldn't be more excited.

My intended audience is Ruby developers who know Rails well but need some help with their first Stripe integration or want help figuring out how to make their existing integration better. If that's you, you should sign up for the mailing list to get updates and announcements leading up to launch day as well as a 10% off coupon code and a free chapter named State and History which covers how to easily keep an audit trail on your sales records.

Find Out More about Mastering Modern Payments

About

Ben E. C. Boyter — Thu, 11 Jul 2013 07:02:47 GMT

I’m a software developer by trade who after years of leaving articles and blog posts all over the place has finally gotten around to consolidating them all into this single site.

I am a professional software engineer who has been working in development with a focus on testing for over 15 years. In that time I have played an integral part in developing and implementing testing methodologies for all sorts of projects including,

Decoding CAPTCHA’s

Ben E. C. Boyter — Thu, 11 Jul 2013 06:00:02 GMT

Looking for a practical guide to CAPTCHA decoding? All About CAPTCHA’s. This eBook will teach you how to identify weaknesses and exploit CAPTCHA’s from beginning to end. Buy now using Leanpub

Most people don’t know this but my honors thesis was about using a computer program to read text out of web images. My theory was that if you could get a high level of successful extraction you could use it as another source of data which could be used to improve search engine results. I was even quite successful in doing it, but never really followed my experiments up. My honors adviser Dr Junbin Gao http://csusap.csu.edu.au/~jbgao/ had suggested the following writing my thesis I should write some form of article on what I had learnt. Well I finally got around to doing it. While what follows is not exactly what I was studying it is something I wish had existed when I started looking around.

hstore vs. JSON - Which to use in Postgres

CRAIG KERSTIENS — Wed, 03 Jul 2013 23:55:56 GMT

If you’re deciding what to put in Postgres and what not to, consider that Postgres can be a perfectly good schema-less database. Of course as soon as people realized this then the common comes a question, is hstore or JSON better. Which do I use and in what cases. Well first, if you’re not familiar check out some previous material on them:

If you’re already up to date with both of them, but still wondering which to use lets dig in.

hstore

hstore is a key value store directly within your database. Its been a common favorite of mine and has been for some time. hstore gives you flexibility when working with your schema, as you don’t have to define models ahead of time. Though its two big limitations are that 1. it only deals with text and 2. its not a full document store meaning you can’t nest objects.

Though major benefits of hstore include the ability to index on it, robust support for various operators, and of course the obvious of flexibility with your data. Some of the basic operators available include:

Return the value from columnfoo for key bar:

foo->'bar'

Does the specified column foo contain a key bar:

foo?'bar'

Does the specified column foo contain a value of baz for key bar:

foo@>'bar->baz'

Perhaps one of the best parts of hstore is that you can index on it. In particular Postgres gin and gist indexes allow you to index all keys and values within an hstore. A talk by Christophe Pettus of PgExperts actually highlights some performance details of hstore with indexes. To give away the big punchline in several cases hstore with gin/gist beats mongodb in performance.

json

JSON in contrast to hstore is a full document datatype. In addition to nesting objects you have support for more than just text (read numbers). As you insert JSON into Postgres it will automatically ensure its valid JSON and error if its well not. JSON gets a lot better come Postgres 9.3 as well with some built in operators. Though if you need more functionality in it today you should look at PLV8.

Which to Use

So which do you actually want to use in your application? If you’re already using JSON and simply want to store it in your database then the JSON datatype is often the correct pick. However, if you’re just looking for flexibility with your data model then hstore is likely the path you want to take. hstore will give you much of the flexibility you want as well as a good ability to query your data in a performant manner. Of course much of this starts to change in Postgres 9.3.

Shipping with Stripe and EasyPost

Pete Keen — Tue, 02 Jul 2013 03:00:00 GMT

Let's say that instead of running a Software as a Service, you're actually building and shipping physical products. Let's say quadcopter kits. People come to your website, buy a quadcopter kit, and then you build it and ship it to them. It takes you a few days to build the kit, though, and you would rather not charge the customer until you ship. Traditionally Stripe has been focused on paying for online services but recently they added the ability to authorize and capture payments in two steps. In this post we're going to explore billing with Stripe and shipping with EasyPost with separate charge and capture.

Step 1: Calculate Shipping

EasyPost makes it really simple to calculate shipping rates. Just take the customer's shipping address and create an EasyPost::Shipment object:

from_address = EasyPost::Address.create(
  name:    'Pete Keen',
  street1: '618 NW Glisan Ave',
  city:    'Portland',
  state:   'OR',
  zip:     '97211',
  country: 'US',
  email:   'pete@petekeen.net'
)

to_address = EasyPost::Address.create(
  name:    params[:to_name],
  street1: params[:to_street1],
  city:    params[:to_city],
  state:   params[:to_state],
  zip:     params[:to_zip],
  country: params[:to_country],
  email:   params[:email],
)

parcel = EasyPost::Parcel.create(
  length: 10,
  width: 10,
  height: 6,
  weight: 30,
)

shipment = EasyPost::Shipment.create(
  to_address: to_address,
  from_address: from_address,
  parcel: parcel
)

@rates = shipment['rates']

Display those rates to the customer, have them pick the one they want, and then move on to the next step.

Step 2: Authorize

Charging a card and just doing the authorization step are remarkably similar. Just get the user's credit card info with stripe.js or checkout.js and then make the charge with the capture parameter set to false:

charge = Stripe::Charge.create(
  card:   params[:stripeToken],
  amount: 10000 + (@selected_rate['rate'].to_f * 100).to_i,
  currency: 'usd',
  capture: false
)

This will authorize a charge of $100 plus the shipping rate for up to seven days. EasyPost gives back shipping rates as decimal strings, so to give it to Stripe we have to convert it to a number, then to cents, and finally to an integer. Save shipment.id and charge.id to your database, you'll need them later.

Step 3: Build the Product

You're on your own here. Just make sure to get everything done in 7 days, because after that Stripe will release the funds from the customer's credit card and the charge object won't be valid anymore.

Step 4: Ship It!

Now you're at the point where you're ready to ship. All you have to do now is purchase the shipping label and tell Stripe to capture the charge:

shipment = Shipment.retrieve(saved_shipment_id)
label = shipment.buy

charge = Stripe::Charge.retrieve(saved_charge_id)
charge.capture

label_url = label['label_pdf_url']
# display the url and print it

Now just print off the PDF, tape it to your box, and take the package to the UPS or FedEx store or the post office and drop it off. Wait a few weeks and search YouTube for quadcopter videos and you'll be sure to see one of your kits, gracefully flying around.

Conclusion

Stripe is awesome for processing credit cards. EasyPost is the easiest way to buy shipping known to man. Combine them and you have yourself a simple way to sell and ship physical products in the US and worldwide.

10 Years Ago, in an MMO far, far away

Ben Overmyer's Site — Sat, 29 Jun 2013 03:00:00 GMT

Ten years ago nearly to the day, Star Wars: Galaxies launched to the world. As a remembrance of the adventures I had there, I’m writing now about the game as it was in those days and the community that surrounded it. As you read this, you may wish to listen to the playlist on YouTube that starts with the above video. It’s a wonderful collection of the music from the game, familiar audio glitches included.

The features page for the first SWG website

One of the things I remember reading first about Star Wars: Galaxies was an interview with lead world artist John Roy about procedural planet generation, and how he would discover features of the landscape that were new even to the designers. That got me really excited about exploring these worlds that I’d only read about or seen on the big screen. I’m not sure how often this technique is employed in modern MMORPGs. My impression is that the vast majority of current worlds are slowly crafted by hand by artists, which is why we have so many beautiful vistas but so few boring areas to contrast them with. If all we ever see is beauty, we start to tire of it and look for something better.

I miss the rolling hills of Corellia where my first SWG toon “grew up.” Another thing that I truly miss about Star Wars: Galaxies is the wait times at the starports. You could only travel to another planet if you bought a ticket for a starship and waited at a starport for one. They arrived in ten minute intervals, which means frequently large groups of players would congregate at the main starports while waiting to travel to another planet. It made for some great conversations, and new friends.

A starport in Star Wars: Galaxies

Something else I miss about Star Wars: Galaxies was the grind. As a crafter who played before the concept of “practice mode” was introduced, I would sit there and grind out components to level up my skill for hours. When I ran out of materials or empty space in my inventory, I’d make a backpack, shove the extra items in there, and head out to my personal harvesters for more materials. This was before anyone was able to build medium or heavy harvesters, so there were still vast fields of personal harvesters dotting the landscape.

A personal mineral extractor

The grind, at least for SWG crafting, meant long periods of sitting there doing the exact same motions over and over again. This sounds really boring, but frequently after about five minutes of doing this, I would hit a kind of altered state of mind - a vaguely pleasant feeling of just “being.” The only other time I’ve ever felt that is when practicing Zen meditation, and I sorely miss it.

Before vehicles were introduced, everyone had to walk everywhere. Shuttles and starports were jammed full of players on their way someplace. Player cities didn’t exist yet, but there were always massive collections of player houses exactly 1 km outside of every major NPC city. It made those cities feel a lot like real cities - you would always be passing by other players on your way either into the city core for transport, into the suburbs for shopping or socializing, or out into the wilderness for adventuring and hunting. Every city had its own distinctive feel, too, on my home server of Chilastra.

Corellia had some beautiful vistas

Coronet on Corellia was where I first landed, and it’s where I spent the first few months of the game. It felt very rugged, like a city on the fringe. Theed, on Naboo, was where the entertainers and image designers hung out. It was definitely a “pretty” city. Mos Eisley was… well, a desert city. Because of all the difficult, unbuildable terrain surrounding Mos Eisley, it wasn’t nearly as dense as the others. I think the players more nostalgic for A New Hope landed there. The other cities, like Restuss on Rori and the Imperial Outpost on Talus, had much smaller and more focused populations. Later, when they allowed players to build on some of the more advanced planets like Dantooine and Lok, more interesting communities started springing up.

In the end, Star Wars: Galaxies really was the best MMORPG experience I’ve ever had. The early months, even with all the bugs, network issues, and missing content (capes!?), are permanently etched in my memory as a wonderful time. Rest in peace, SWG.

Pivoting in Postgres

CRAIG KERSTIENS — Thu, 27 Jun 2013 23:55:56 GMT

Earlier today on an internal Heroku group alias there was a dataclip shared. The dataclip listed off some data grouped by a category, there was a reply a few minutes later with a modification to the query that used the crosstab function to pivot directly in SQL. There were immediately several reactions on the list that went something like this:

While a mostly simple function in Postgres (there are a few rough edges), it really is all too handy. So here it is in action. Taking some data that looks like

row identifier, in this case date
category grouping, in this case OS
value

Given a really basic query that generates some sample data it may look something like this:

SELECT generate_series AS date,
b.desc AS TYPE,
(random() * 10000 + 1)::int AS val
FROM generate_series((now() - '100 days'::interval)::date, now()::date, '1 day'::interval),
(SELECT unnest(ARRAY['OSX', 'Windows', 'Linux']) AS DESC) b;

You get results that look like:

But of course this isn’t overly helpful in comparing day to day overall. You can do so on a OS by OS basis, but its annoying enough as is. The easy solution is to simply use a pivot table on your data. Most people at this point would pull it up into Excel or Google Docs, or you can do it directly in Postgres. To do so you’ll first enable the extension tablefunc:

CREATE EXTENSION tablefunc

Then you’ll use the crosstab function. The function looks something like:

SELECT *
FROM crosstab(
'SELECT row_name, category_grouping, value FROM foo',
'SELECT category_names FROM bar')
AS
ct_result (category_name text, category1 text, category2 text, etc.)

Lets see it an actual action. Given the same query we used to generate fake data we can actually pivot on it now directly in PostgreSQL:

SELECT *
FROM crosstab(
'SELECT
a date,
b.desc AS os,
(random() * 10000 + 1)::int AS value
FROM generate_series((now() - ''100 days''::interval)::date, now()::date, ''1 DAY''::interval) a,
(SELECT unnest(ARRAY[''OSX'', ''Windows'', ''Linux'']) AS DESC) b ORDER BY 1,2
','SELECT unnest(ARRAY[''OSX'', ''Windows'', ''Linux''])'
)
AS ct(date date, OSX int, Windows int, Linux int);

And see some results:

Have fun analyzing your data directly in your DB now. And as always if you have feedback/questions/requests please feel free to drop me a line craig.kerstiens@gmail.com

Javascript Functions for PostgreSQL

CRAIG KERSTIENS — Tue, 25 Jun 2013 23:55:56 GMT

Javascript in Postgres has gotten a good bit of love lately, part of that is from Heroku Postgres recently adding support for Javascript and part from a variety of people championing the power of it such as @leinweber (Embracing the web with JSON and PLV8) and @selenamarie (schema liberation with JSON and PLV8). In a recent conversation it was pointed out that it seems a bit of headache to have to create your own functions, or at least having an initial collection would make it that much more powerful. While many can look forward to PostgreSQL 9.3 which will have a bit more built in support for JSON a few functions can really help make it more useful today.

These are courtesy of Will Leinweber. For each of the following functions I’ll highlight an example of using it as well. To get an idea of the data its being run on:

select * from example;
data
--------------------------------------------
{"name":"Craig Kerstiens", +
"age":27, +
"siblings":1, +
"numbers":[ +
{"type":"work", +
"number":"123-456-7890"}, +
{"type":"home", +
"number":"456-123-7890"}]}
(1 row)

get_text

CREATE OR REPLACE FUNCTION
get_text(key text, data json)
RETURNS text AS $$
return data[key];
$$ LANGUAGE plv8 IMMUTABLE STRICT;

Then using the function:

select get_text('name', data) from example;
get_text
----------------
Craig Kerstiens
(1 row)

get_numeric

CREATE OR REPLACE FUNCTION
get_numeric(key text, data json)
RETURNS numeric AS $$
return data[key];
$$ LANGUAGE plv8 IMMUTABLE STRICT;

Then using the function:

select get_numeric('siblings', data) from example;
get_text
----------------
1
(1 row)

json_select

create or replace function
json_select(selector text, data json)
returns json as $$
exports = {};
(function(a){function z(a){return{sel:q(a)[1],match:function(a){return y(this.sel,a)},forEach:function(a,b){return x(this.sel,a,b)}}}function y(a,b){var c=[];x(a,b,function(a){c.push(a)});return c}function x(a,b,c,d,e,f){var g=a[0]===","?a.slice(1):[a],h=[],i=!1,j=0,k=0,l,m;for(j=0;j<g.length;j++){m=w(b,g[j],d,e,f),m[0]&&(i=!0);for(k=0;k<m[1].length;k++)h.push(m[1][k])}if(h.length&&typeof b=="object"){h.length>=1&&h.unshift(",");if(u(b))for(j=0;j<b.length;j++)x(h,b[j],c,undefined,j,b.length);else for(l in b)b.hasOwnProperty(l)&&x(h,b[l],c,l)}i&&c&&c(b)}function w(a,b,c,d,e){var f=[],g=b[0]===">"?b[1]:b[0],h=!0,i;g.type&&(h=h&&g.type===v(a)),g.id&&(h=h&&g.id===c),h&&g.pf&&(g.pf===":nth-last-child"?d=e-d:d++,g.a===0?h=g.b===d:(i=(d-g.b)%g.a,h=!i&&d*g.a+g.b>=0));if(h&&g.has){var j=function(){throw 42};for(var k=0;k<g.has.length;k++){try{x(g.has[k],a,j)}catch(l){if(l===42)continue}h=!1;break}}h&&g.expr&&(h=p(g.expr,a)),b[0]!==">"&&b[0].pc!==":root"&&f.push(b),h&&(b[0]===">"?b.length>2&&(h=!1,f.push(b.slice(2))):b.length>1&&(h=!1,f.push(b.slice(1))));return[h,f]}function v(a){if(a===null)return"null";var b=typeof a;b==="object"&&u(a)&&(b="array");return b}function u(a){return Array.isArray?Array.isArray(a):b.call(a)==="[object Array]"}function t(a,b,c){var d=b,g={},j=i(a,b);j&&j[1]===" "&&(d=b=j[0],j=i(a,b)),j&&j[1]===f.typ?(g.type=j[2],j=i(a,b=j[0])):j&&j[1]==="*"&&(j=i(a,b=j[0]));for(;;){if(j===undefined)break;if(j[1]===f.ide)g.id&&e("nmi",j[1]),g.id=j[2];else if(j[1]===f.psc)(g.pc||g.pf)&&e("mpc",j[1]),j[2]===":first-child"?(g.pf=":nth-child",g.a=0,g.b=1):j[2]===":last-child"?(g.pf=":nth-last-child",g.a=0,g.b=1):g.pc=j[2];else{if(j[1]!==f.psf)break;if(j[2]===":val"||j[2]===":contains")g.expr=[undefined,j[2]===":val"?"=":"*=",undefined],j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0])),(!j||j[1]!=="(")&&e("pex",a),j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0])),(!j||j[1]!==f.str)&&e("sex",a),g.expr[2]=j[2],j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0])),(!j||j[1]!==")")&&e("epex",a);else if(j[2]===":has"){j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0])),(!j||j[1]!=="(")&&e("pex",a);var k=q(a,j[0],!0);j[0]=k[0],g.has||(g.has=[]),g.has.push(k[1])}else if(j[2]===":expr"){g.expr&&e("mexp",a);var l=o(a,j[0]);j[0]=l[0],g.expr=l[1]}else{(g.pc||g.pf)&&e("mpc",a),g.pf=j[2];var m=h.exec(a.substr(j[0]));m||e("mepf",a),m[5]?(g.a=2,g.b=m[5]==="odd"?1:0):m[6]?(g.a=0,g.b=parseInt(m[6],10)):(g.a=parseInt((m[1]?m[1]:"+")+(m[2]?m[2]:"1"),10),g.b=m[3]?parseInt(m[3]+m[4],10):0),j[0]+=m[0].length}}j=i(a,b=j[0])}d===b&&e("se",a);return[b,g]}function s(a){if(a[0]===","){var b=[","];for(var c=c;c<a.length;c++){var d=r(d[c]);b=b.concat(d[0]===","?d.slice(1):d)}return b}return r(a)}function r(a){var b=[],c;for(var d=0;d<a.length;d++)if(a[d]==="~"){if(d<2||a[d-2]!=">")c=a.slice(0,d-1),c=c.concat([{has:[[{pc:":root"},">",a[d-1]]]},">"]),c=c.concat(a.slice(d+1)),b.push(c);if(d>1){var e=a[d-2]===">"?d-3:d-2;c=a.slice(0,e);var f={};for(var g in a[e])a[e].hasOwnProperty(g)&&(f[g]=a[e][g]);f.has||(f.has=[]),f.has.push([{pc:":root"},">",a[d-1]]),c=c.concat(f,">",a.slice(d+1)),b.push(c)}break}if(d==a.length)return a;return b.length>1?[","].concat(b):b[0]}function q(a,b,c,d){c||(d={});var f=[],g,h;b||(b=0);for(;;){var j=t(a,b,d);f.push(j[1]),j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0]));if(!j)break;if(j[1]===">"||j[1]==="~")j[1]==="~"&&(d.usesSiblingOp=!0),f.push(j[1]),b=j[0];else if(j[1]===",")g===undefined?g=[",",f]:g.push(f),f=[],b=j[0];else if(j[1]===")"){c||e("ucp",j[1]),h=1,b=j[0];break}}c&&!h&&e("mcp",a),g&&g.push(f);var k;!c&&d.usesSiblingOp?k=s(g?g:f):k=g?g:f;return[b,k]}function p(a,b){if(a===undefined)return b;if(a===null||typeof a!="object")return a;var c=p(a[0],b),d=p(a[2],b);return l[a[1]][1](c,d)}function o(a,b){function c(a){return typeof a!="object"||a===null?a:a[0]==="("?c(a[1]):[c(a[0]),a[1],c(a[2])]}var d=n(a,b?b:0);return[d[0],c(d[1])]}function n(a,b){b||(b=0);var c=m(a,b),d;if(c&&c[1]==="("){d=n(a,c[0]);var f=m(a,d[0]);(!f||f[1]!==")")&&e("epex",a),b=f[0],d=["(",d[1]]}else!c||c[1]&&c[1]!="x"?e("ee",a+" - "+(c[1]&&c[1])):(d=c[1]==="x"?undefined:c[2],b=c[0]);var g=m(a,b);if(!g||g[1]==")")return[b,d];(g[1]=="x"||!g[1])&&e("bop",a+" - "+(g[1]&&g[1]));var h=n(a,g[0]);b=h[0],h=h[1];var i;if(typeof h!="object"||h[0]==="("||l[g[1]][0]<l[h[1]][0])i=[d,g[1],h];else{i=h;while(typeof h[0]=="object"&&h[0][0]!="("&&l[g[1]][0]>=l[h[0][1]][0])h=h[0];h[0]=[d,g[1],h[0]]}return[b,i]}function m(a,b){var d,e=j.exec(a.substr(b));if(e){b+=e[0].length,d=e[1]||e[2]||e[3]||e[5]||e[6];if(e[1]||e[2]||e[3])return[b,0,c(d)];if(e[4])return[b,0,undefined];return[b,d]}}function k(a,b){return typeof a===b}function i(a,b){b||(b=0);var d=g.exec(a.substr(b));if(!d)return undefined;b+=d[0].length;var h;d[1]?h=[b," "]:d[2]?h=[b,d[0]]:d[3]?h=[b,f.typ,d[0]]:d[4]?h=[b,f.psc,d[0]]:d[5]?h=[b,f.psf,d[0]]:d[6]?e("upc",a):d[8]?h=[b,d[7]?f.ide:f.str,c(d[8])]:d[9]?e("ujs",a):d[10]&&(h=[b,f.ide,d[10].replace(/\\([^\r\n\f0-9a-fA-F])/g,"$1")]);return h}function e(a,b){throw new Error(d[a]+(b&&" in '"+b+"'"))}function c(a){try{if(JSON&&JSON.parse)return JSON.parse(a);return(new Function("return "+a))()}catch(b){e("ijs",b.message)}}var b=Object.prototype.toString,d={bop:"binary operator expected",ee:"expression expected",epex:"closing paren expected ')'",ijs:"invalid json string",mcp:"missing closing paren",mepf:"malformed expression in pseudo-function",mexp:"multiple expressions not allowed",mpc:"multiple pseudo classes (:xxx) not allowed",nmi:"multiple ids not allowed",pex:"opening paren expected '('",se:"selector expected",sex:"string expected",sra:"string required after '.'",uc:"unrecognized char",ucp:"unexpected closing paren",ujs:"unclosed json string",upc:"unrecognized pseudo class"},f={psc:1,psf:2,typ:3,str:4,ide:5},g=new RegExp('^(?:([\\r\\n\\t\\ ]+)|([~*,>\\)\\(])|(string|boolean|null|array|object|number)|(:(?:root|first-child|last-child|only-child))|(:(?:nth-child|nth-last-child|has|expr|val|contains))|(:\\w+)|(?:(\\.)?(\\"(?:[^\\\\\\"]|\\\\[^\\"])*\\"))|(\\")|\\.((?:[_a-zA-Z]|[^\\0-\\0177]|\\\\[^\\r\\n\\f0-9a-fA-F])(?:[_a-zA-Z0-9\\-]|[^\\u0000-\\u0177]|(?:\\\\[^\\r\\n\\f0-9a-fA-F]))*))'),h=/^\s*\(\s*(?:([+\-]?)([0-9]*)n\s*(?:([+\-])\s*([0-9]))?|(odd|even)|([+\-]?[0-9]+))\s*\)/,j=new RegExp('^\\s*(?:(true|false|null)|(-?\\d+(?:\\.\\d*)?(?:[eE][+\\-]?\\d+)?)|("(?:[^\\]|\\[^"])*")|(x)|(&&|\\|\\||[\\$\\^<>!\\*]=|[=+\\-*/%<>])|([\\(\\)]))'),l={"*":[9,function(a,b){return a*b}],"/":[9,function(a,b){return a/b}],"%":[9,function(a,b){return a%b}],"+":[7,function(a,b){return a+b}],"-":[7,function(a,b){return a-b}],"<=":[5,function(a,b){return k(a,"number")&&k(b,"number")&&a<=b}],">=":[5,function(a,b){return k(a,"number")&&k(b,"number")&&a>=b}],"$=":[5,function(a,b){return k(a,"string")&&k(b,"string")&&a.lastIndexOf(b)===a.length-b.length}],"^=":[5,function(a,b){return k(a,"string")&&k(b,"string")&&a.indexOf(b)===0}],"*=":[5,function(a,b){return k(a,"string")&&k(b,"string")&&a.indexOf(b)!==-1}],">":[5,function(a,b){return k(a,"number")&&k(b,"number")&&a>b}],"<":[5,function(a,b){return k(a,"number")&&k(b,"number")&&a<b}],"=":[3,function(a,b){return a===b}],"!=":[3,function(a,b){return a!==b}],"&&":[2,function(a,b){return a&&b}],"||":[1,function(a,b){return a||b}]};a._lex=i,a._parse=q,a.match=function(a,b){return z(a).match(b)},a.forEach=function(a,b,c){return z(a).forEach(b,c)},a.compile=z})(typeof exports=="undefined"?window.JSONSelect={}:exports)
return JSON.stringify(
exports.match(selector,
data));
$$ LANGUAGE plv8 IMMUTABLE STRICT

Then using the function:

select json_select('.name nth-child(1)', data) as name, json_select('.numbers', data) as phone
from example;
name | phone
--------------------+------------------------------------------------------------------------------------------
["Craig Kerstiens"] | [[{"type":"work","number":"456-123-7890"},{"type":"home","number":"123-456-7890"}]]
(1 row)

javascript injection attack

create or replace function
js(src text) returns text as $$
return eval(
"(function() { " + src + "})"
)();
$$ LANGUAGE plv8;

Have any others you feel are essential when starting to work with JSON? Let me know craig.kerstiens@gmail.com. Beyond that give JSON and JavaScript a try inside your database.

The Joys of UNIX Keyboards

Donat Studios — Wed, 19 Jun 2013 02:10:51 GMT

I fell in love with a dead keyboard layout.

A decade or so ago while helping a friends father clean out an old building, we came across an ancient Sun Microsystems server. We found it curious. Everything about it was different from what we were used to. The command line was black text on a white background, the connectors strange and foreign, and the keyboard layout was bizarre.

We never did much with it; turning it on made all the lights in his home dim, and our joint knowledge of UNIX was nonexistent. It sat in his bedroom for years supporting his television at the foot of his bed.

I never forgot that keyboard though. The thought that there was this alternative layout out there seemed intriguing to me.

I am ruined for all other keyboards…

Mac is the main platform at my new job. I found myself unhappy with Apple keyboards; they are flat, bland, and completely unsatisfying to type on. I dug out an old AppleDesign keyboard, and used that for several months but I still needed something more.

I read about Happy Hacking Keyboard, and after some deliberation decided to give it a try. I thought I could learn to accept most of the layout changes, but the lack of arrow keys on the "Professional" model made me squirm a little. Luckily there is a "Lite" version that includes arrow keys.

I purchased the Happy Hacking Keyboard Lite2.

The Happy Hacking Keyboard much like the Sun Microsystem keyboard I found years earlier features a UNIX layout. They designed it with typing at a terminal in mind.

A Better Layout

For those unfamiliar with UNIX keyboards, there are a few distinctions.

The Control key moved to the Caps Lock keys usual position. This makes it much easier to hit common commands than its usual location, often without ever leaving home row. I can't imagine why anyone thought Caps Lock should have had this prominent of placement.
The Meta (◆ / ⌘ / Windows) key is in its rightful place next to the space bar, as on a Mac keyboard.
The Backspace key moved down a row to be above the Enter key. This change is my favorite change. Having a small reach, normally my hands have to leave home row to hit backspace. but on a UNIX board I have no issue hitting it with my pinky with the rest of my fingers remaining firmly on home row.
Escape is also in reach of home row, placed where you would usually find ~. I could have done without this change, but it is fine.
The ~ and \ have moved to the usual location of the Backspace. Working exclusively on UNIX systems now, I don't mind the \ being harder to reach. I could understand a Windows user finding this irritating.

The HHKB is a minimal version of the UNIX keyboard. To this end the F Keys are mapped to Fn + 1, 2, 3, etc. The HHKB encourages you to stay on home row; you can hit any key on the keyboard without leaving it. I find my typing accuracy and speed to have improved thanks to its use.

The HHKB features one more change I don't care for. The Backspace key by default performs Forward Delete. Luckily changing this is easy by flipping a toggle switch on the back of the board.

Closing

I recommend this keyboard to anyone who spends a lot of time at the terminal. You are able to do more of what you need without ever leaving home row, every important key is within reach.

Elsewhere

Discussion on Hacker News
Mentioned on BSD Now Podcast: Episode 333

Making eBooks with Docverter

Pete Keen — Tue, 18 Jun 2013 15:00:00 GMT

I've been writing my guide to integrating Stripe with Rails using markdown, as with most textual projects that I work on. Every chapter is a markdown-formatted file living in a git repo, synced-on-save to my git server and S3 using SparkleShare. When I want to peek at the rendered version I use little previewer app running on a VM on my Mac mini that I talked about previously.

A good eBook needs a PDF version, of course. Awhile back I wrote an open-source service named Docverter that can render XHTML to PDF using a library named Flying Saucer, among other things. All you have to do is pipe in the HTML and other related files and you get back a rendered, self-contained PDF file. There are a few non-trivial aspects to this, of course, because HTML is not primarily intended for printable output. The W3C has worked up a whole CSS module for page-related styles but it's not the most readable document. There's a few simple-ish things that you can do to your document to make it look nice, though.

Here's the simplest HTML to PDF renderer:

require 'docverter'

Docverter.base_url = 'http://c.docverter.com'

html = <<HERE
<html>
  <head>
    <title>Test Document</title>
  </head>
  <body>
    <h1>Test Header</h1>
    <p>This is some text</p>
  </body>
</html>
HERE

File.open("out.pdf", "w+") do |f|
  f.write(Docverter::Conversion.run do |c|
    c.from    = 'html'
    c.to      = 'pdf'
    c.content = html
  end)
end

The HTML document is very simple, as is the conversion. Docverter::Conversion.run takes a block which yields a Docverter::Conversion object that can be set with any options Docverter supports. Most basically, you have to specify from, to, and content. If you run this program you'll get a file named out.pdf.

Fonts

The first non-trivial thing that one would want to do is customize fonts. Flying Saucer knows quite a bit of CSS, including @font-face. All you have to do to customize fonts is to download the font as a ttf and modify the above program to look like this:

require 'docverter'

Docverter.base_url = 'http://c.docverter.com'

html = <<HERE
<html>
  <head>
    <title>Test Document</title>
    <style type="text/css">
      @font-face {
        font-family: 'Droid Sans';
        font-style: normal;
        font-weight: 400;
        src: url('droid_sans.ttf');
        -fs-pdf-font-embed: embed;
        -fs-pdf-font-encoding: Identity-H;
      }
      body {
        font-family: 'Droid Sans';
      }
    </style>
  </head>
  <body>
    <h1>Test Header</h1>
    <p>This is some text</p>
  </body>
</html>
HERE

File.open("out.pdf", "w+") do |f|
  f.write(Docverter::Conversion.run do |c|
    c.from    = 'html'
    c.to      = 'pdf'
    c.content = html

    c.add_other_file 'droid_sans.ttf'
  end)
end

A few interesting things are going on here. First, @font-face declares the font. font-family must match the name the font file specifies. Second, -fs-pdf-font-embed and -fs-pdf-font-encoding must match the values given above or the embedding won't work. src is the filename of the file, which we add to the conversion using add_other_file, which takes a path. Drop droid_sans.ttf in the same directory as the script and run it again. Notice that the PDF is now in Droid Sans, which is pretty pleasant.

Footers

Most documents longer than a page are going to need page numbers. Adding page numbers to a PDF with Docverter is very much not trivial. You need to combine the powers of CSS Paged Media running elements and generated content to generate properly formatted footers with page numbers. Here's the HTML source:

<html>
  <head>
    <title>Test Document</title>
    <style type="text/css">
      @font-face {
        font-family: 'Droid Sans';
        font-style: normal;
        font-weight: 400;
        src: url('droid_sans.ttf');
        -fs-pdf-font-embed: embed;
        -fs-pdf-font-encoding: Identity-H;
      }
      body {
        font-family: 'Droid Sans';
      }
      div.page_footer {
        display: block;
        text-align: center;
        font-family: 'Droid Sans';
        position: running(footer);
      }
      div.page_footer .page_number:after {
        content: counter(page);
      }
      @page {
        @bottom-center {
          content: element(footer);
        }
      } 
    </style>
  </head>
  <body>
    <div class="page_footer"><span class="page_number"></span></div>
    <h1>Test Header</h1>
    <p>This is some text</p>
  </body>
</html>

Three things to note here. First, the new div with class page_footer has to come before the rest of the body content because it gets moved into place as Flying Saucer renders. If it's at the bottom it won't exist when Flying Saucer tries to render the page and so nothing gets rendered at all.

Second, notice position: running(footer). CSS Paged Media introduces the running() which tells the renderer to stick the content that's currently selected, in this case div.page_footer, into a slot named footer. We set up the page using the @page selector. Inside there is another selector named @bottom-center, which specifies the center section at the bottom of the page. The content attribute with an element() value tells the renderer to take the content from the slot named footer and use it to populate the section. Note that we could have named the footer slot anything. The named slots are in their own namespace separate from ids and classes.

Finally we get to actually setting up the page number. There's a default counter named page which we put after the span with class page_number using CSS generated content.

Page Breaks

PDFs are effectively pre-printed documents, so CSS Paged Media gives you a few different facilities for controlling page breaks. For example, if I wanted a page break before every H1 element I could say this:

h1 {
  page-break-before: always;
}

If, instead, I want to break after a certain element, like a closing paragraph or something, I could do this:

p.closing {
  page-break-after: always;
}

Sometimes you have elements that you want to not break across pages if at all possible. In Mastering Modern Payments there are dozens of code examples that, if I let Flying Saucer break at the natural spots, would have one or two lines on one page and the rest of the 10-line sample on the other. CSS Paged Media lets you control that, too, with page-break-inside. Here's the rule I use:

code {
  page-break-inside: avoid;
  orphans: 0;
  widows: 0;
}

This says to avoid inserting page breaks inside a code block. The orphans option controls how many lines are allowable at the bottom of a page inside a block and widows controls how many are allowed at the top of the next. By setting both to 0 I'm saying that I don't want any page breaks at all. Flying Saucer will ignore me if it's not possible, if for instance I have a code block that spans more than an entire page.

Making nice-looking PDFs with HTML source is not trivial. It would probably end up being easier to just drop the raw text inside Apple's iBooks Creator and style it that way, but I like a challenge. It's already looking pretty nice, and with a little more work I think I can have professional-grade PDF output. Docverter also knows how to do Markdown to ePub and Mobi so I should be set for book production. Now to finish writing.

Page Viewer, a Simple Markdown Viewer

Pete Keen — Sat, 15 Jun 2013 03:00:00 GMT

For various projects including Mastering Modern Payments I've found it really useful to be able to view the Markdown source rendered as HTML but I don't really care about editing it online. I put together a little gem named page_viewer which renders Markdown files like this:

page_viewer has some convenience features. First, it dynamically renders the markdown files for each request. This means that if the files change underneath there's no cache to refresh or anything. Second, code blocks are highlighted with Pygments. You'll need a working python installation but that's it. Third, it integrates with Docverter for on-the-fly PDF conversion.

It's also easy to subclass to do what you want. For example, the site where I'm developing my guide subclasses PageViewer::App to add some routes that render the whole guide as one page with a table of contents, and another one that renders the whole thing as a PDF.

I use an instance page_viewer to view my personal wiki nicely rendered on a web page. My wiki repo is hosted on my private gitolite instance with a local clone. Each time I push changes, which happens pretty frequently with Sparkleshare, gitolite clones the repo to a directory somewhere else on the machine. This happens to be the same directory that page_viewer is pointing at.

Installation is pretty simple. Create a new project and include the page_viewer gem in the Gemfile:

source :rubygems

gem 'page_viewer'

Configure and run PageViewer::App inside config.ru:

require 'page_viewer'

PageViewer::App.set :page_root, '/path/to/some/markdown/files'

use Rack::Auth::Basic, "Restricted Area" do |username, password|
  username == ENV['USERNAME'] && password == ENV['PASSWORD']
end

run PageViewer::App

:page_root needs to contain the path to the markdown files you want to render and is required. Every file in that path ending in .md will be renderable.

Is this going to be useful for other people? Probably not, since it's really specific to my own needs. If you have a dynamically changing set of markdown documents you want rendered fresh all the time, though, this may be the ticket.

Explaining your PostgreSQL data

CRAIG KERSTIENS — Thu, 13 Jun 2013 23:55:56 GMT

I’ve written a bit before about understanding the output from EXPLAIN and EXPLAIN ANALYZE in PostgreSQL. Though understandably getting a grasp on execution plans could probably use some more guidance. Yet, this time around I’m taking a bit of a cop out and highlighting a few tools instead of documenting myself, which I’ve done in a talk I’ve frequently given Postgres Demystified.

Getting at the Data

The first small thing you can do is actually retrieve the data in JSON form. By adding in (format json) right after your EXPLAIN or EXPLAIN ANALYZE command it’ll as you’d expect return it in JSON. To give an example:

# EXPLAIN SELECT * FROM users LIMIT 1;
QUERY PLAN
--------------------------------------------------------------
Limit (cost=0.00..0.03 rows=1 width=812)
-> Seq Scan on users (cost=0.00..1.50 rows=50 width=812)
(2 rows)

Then in JSON format:

EXPLAIN (format json) SELECT * FROM users LIMIT 1;
QUERY PLAN
------------------------------------------------
[ +
{ +
"Plan": { +
"Node Type": "Limit", +
"Startup Cost": 0.00, +
"Total Cost": 0.03, +
"Plan Rows": 1, +
"Plan Width": 812, +
"Plans": [ +
{ +
"Node Type": "Seq Scan", +
"Parent Relationship": "Outer",+
"Relation Name": "users", +
"Alias": "users", +
"Startup Cost": 0.00, +
"Total Cost": 1.50, +
"Plan Rows": 50, +
"Plan Width": 812 +
} +
] +
} +
} +
]
(1 row)

While its on my list to build some interesting apps by pulling in the JSON input, others may be equally as interested in taking advantage of this data in its JSON form. If you take a shot at building something with this output, as always I’d love to hear about it - craig.kerstiens@gmail.com

Despez

Of course if you’re itch isn’t in better tools for Postgres, you may just want to have a solution that works today. While its not perfect, one of the best ones out there is Dezpez’s explain tool. You can take any execution plan and paste it in and get some better visual representation of the result. You can also share them as well.

Book Review: The Box

Pete Keen — Sun, 09 Jun 2013 03:00:00 GMT

As a kid something about the Millennium Falcon always bothered me. See, everybody knows that cargo travels around the world, and presumably around the galaxy, in containers. You know, the big steel boxes that you see on TV where murders and drug deals are always happening. If Han Solo's ship is supposedly some awesome super duper fast freighter, where do the containers go? I guess on some level the fact that Solo was a smuggler sort of registered, but still you'd think that the tea and stuff (that's what smugglers move according to the school history books) would just be hidden in the containers.

After reading The Box) by Marc Levinson I'm relatively certain the Falcon would be considered a breakbulk tramp freighter. "Breakbulk" meaning traditional non-containerized freight, "tramp" for the fact that Han and Chewie are tramps. Well, and they travel around without a fixed schedule or ports of call, picking up freight wherever and whenever they can. But mostly because they're tramps.

History Lessons

The Box presents a comprehensive history of the shipping container and of shipping in general, from the days of sailing ships, gallantly making their way from India to Britain for the tea trade, all the way to modern day when huge container ships carrying over fifteen thousand twenty foot long standard containers ply the seas between Asia and Europe.

One of the most interesting sections of the book for me was the exhaustive description of how, exactly, a breakbulk freighter is crammed to the gills with individual items of every imaginable description. It turns out that longshoreman (people who work the docks and load the ships) used to have to know a lot of details of how to pack a ship and what things can be crammed where in order to maximize the limited amount of storage capacity.

The Container

Malcolm McLean’s fundamental insight, commonplace today but quite radical in the 1950s, was that the shipping industry’s business was moving cargo, not sailing ships.

The origin of the standardized international shipping container is almost, but not entirely, the story of a man named Malcolm McLean. Before McLean containers had been tried on and off for decades, both in the United States and in Europe, on trucks and rails and ships. There had never been dedicated through-service, though, where a container could be packed at the shipper's factory and arrive at the destination without ever being opened.

A 25-ton container of coffeemakers can leave a factory in Malaysia, be loaded aboard a ship, and cover the 9,000 miles to Los Angeles in 16 days. A day later, the container is on a unit train to Chicago, where it is transferred immediately to a truck headed for Cincinnati. The 11,000-mile trip from the factory gate to the Ohio warehouse can take as little as 22 days, a rate of 500 miles per day, at a cost lower than that of a single first-class air ticket. More than likely, no one has touched the contents, or even opened the container, along the way

The road to the standard intermodal container took quite a few twists and turns, however. Every shipper had their own standard at first, none of them compatible with each other. Eventually a committee convened by the ISO decided on dimensions and shippers slowly adopted them. The advent of the giant containerships in the 70's and 80's spelled the end of the non-standard container sizes. Originally there were two sizes, both 8 feet 6 inches high and 8 feet wide, one being 20 feet long and the other 40 feet long. Eventually a few more sizes were added, but the core standards were still the same. The key innovation that standardization added was the method by which containers are connected together to form stacks on ships, the twistlock.

One thing I was astounded by was just how fast the container changed the shipping industry. Over the course of fifty years, containerization eliminated thousands of jobs worldwide, eliminated entire ports while creating new ones where there was just a tiny dock. It opened up entire countries to new forms of trade and ruined the economies of others. The Box tells this whole compelling story, really a story of unintended consequences. Kind of like how Han Solo went from being a tramp to a general in a galaxy-spanning republic.

Build a Better Warrior in Rift

Ben Overmyer's Site — Sat, 08 Jun 2013 03:00:00 GMT

I’ve been playing Rift a lot lately, and the more I play, the better I understand the ins and outs of being a Warrior. This post has a few of my thoughts on the subject of tuning warrior builds. The class is really versatile, but focuses heavily on two aspects - tanking and dealing damage. That much is obvious. What is less obvious is that there are several ways to do each of these. My personal preference is for dealing respectable amounts of damage to multiple enemies while being able to take a lot of punishment before going down. My favorite ability for staying in the fight is No Permission to Die. It’s in the fifth tier of the Warlord soul, and it can be a godsend. If you die within ten seconds of triggering it, you are resurrected at 50% of your maximum health. Properly timed, it can really save your bacon. Also, it’s not on global cooldown, so it’s really hard to NOT time it right if you have a good sense of your combat situation. My next favorite ability is Cornered Beast, which is the 32 point unlock for the Champion soul. It hits up to 6 enemies for a ton of damage over time AND generates Attack Points over time, making it a powerful threat-building ability. Awesome for tanks, and for area effect damage. Along with Warlord and Champion, picking the Reaver soul allows for a ridiculous amount of survivability. It has a number of abilities that boost armor, endurance, and resistances. A balanced mix of the three souls, like the build linked here (Champion (32) / Warlord (23) / Reaver (21)), will give you a decent AoE tanking playstyle. You won’t win any DPS races, but it will keep you in the fight for a good long time.

How and why I'm not running my own DNS

Pete Keen — Fri, 07 Jun 2013 03:00:00 GMT

A few months ago I posted about how I run my own DNS servers using my virtual private servers and tinydns. Well, it turns out that's not a great idea, for a few reasons. First, because if I mess up I'm entirely shut out of my servers. I tried to turn off a service on them the other day and accidentally turned of the tinydns service instead and it took me ages to get back in. Running DNS on the same machines that handle email and web hosting for almost every piece of my online presence is just way too fragile.

Buy the entire collection of DNS articles as a nicely formatted ebook.

Second, and really more importantly, this happened:

At some point last week I started seeing a very large number of queries for various domains coming into one of my servers. There were thousands of them a minute, supposedly coming from just a handful of machines. I couldn't figure this out, and the only solution I could reasonably come up with was to turn off logging. That's totally unacceptable, and if I couldn't figure this out I probably shouldn't be running DNS for myself after all. After consulting with a few very helpful people from the Internet we determined that my server was being used as part of a reflection attack originating at a known bad actor used for various command and control botnet nastiness. It was time to switch as soon as possible. Thankfully I had the TTL for my NS records set very low, otherwise this would have been a much more painful process.

Route 53

Before I decided to go the route of self-hosting I had investigated a few different providers. The most promising of these was Amazon's Route53, with the small snag that there was no easy out of the box solution to dynamic DNS for my home machine. When it came time to switch I figured that a solution would present itself and charged forward with manually switching a few domains over.

At some point after moving my most important domains and turning the DNS service off for good on my VPSs I came up with this little script:

#!/usr/bin/python

ZONE_ID = "ZXXXXXXXX"
DOMAIN_NAME = "dynamichost.example.com."

from boto.route53.connection import Route53Connection
from boto.route53.record import ResourceRecordSets
import requests
import sys

ip = requests.get("http://ip.example.com").text

conn = Route53Connection()

response = conn.get_all_rrsets(ZONE_ID, 'A', DOMAIN_NAME, maxitems=1)[0]
old_ip = response.resource_records[0]

if ip == old_ip:
    sys.exit(0)

changes = ResourceRecordSets(conn, ZONE_ID)

delete_record = changes.add_change("DELETE", DOMAIN_NAME, "A", 60)
delete_record.add_value(old_ip)

create_record = changes.add_change("CREATE", DOMAIN_NAME,"A", 60)
create_record.add_value(ip)

changes.commit()

This runs every minute on a VM running on the Mac mini in my living room. It uses requests to get my external API from a tiny webservice running on one of my servers, and if it's changed from what Route53 thinks it is, it uses boto to delete and recreate the A record. There's a bunch of public services out there that provide your external IP but I wanted to run my own. Of course.

Here's the code for the webservice:

run lambda do |env|
    [200, {"Content-Type" => "text/plain"}, [Rack::Request.new(env).ip]]
end

I think I learned a valuable lesson about limits with this whole "host everything all the time" exercise. Namely, that DNS is best left to the professionals, just like outgoing email. Route53 isn't the cheapest provider around but they promise 100% uptime and have a very nice easy to work with API. I very much recommend them.

Cancer

Pete Keen — Thu, 06 Jun 2013 03:00:00 GMT

"Thank you for calling the Bergs and the Keens. We're not home right now...". Mom hasn't changed the outgoing voicemail message yet. John's voice is still there, encouraging anyone who wants to leave a message. It's the last vestige of him, other than what we hold in our memories. The last bit of him that's still real, and even that's just an echo that will fade with time.

Note: this is a deeply personal entry. If you're here for the programming stuff you may want to skip it.

Update August 29th 2013

February 2006

A few weeks prior my stepdad John almost ran his pickup truck into the side of a semi-truck on the highway. That's how these things get noticed, you see. The months, years, of waking up at 5am and not being able to sleep because of the pounding headaches just become part of the routine, but as soon as something catastrophic happens, that's when people sit up and take notice. I must have been at school when Mom called and told me that John had to have some tests done because they think he had brain tumor. After reviewing the MRI his brain surgeon announced that John had a large tumor, almost the size of a golf ball, embedded in his cerebrum. He was to get have it removed as soon as possible.

My sisters and I sat in the uncomfortable waiting area for 12 hours while his surgeon opened up John's skull and removed the tumor. We came back every day for two weeks while he recovered, first in a normal recovery room and then in the ICU when things turned sour for awhile. The first clue was when the nursed asked him if he knew where he was and he said "I'm going to be skating soon". The winter Olympics were going on in Italy that month and my mom and him were avid fans.

Eventually, with time, he got better. The MRI scans became less frequent. They talked about having to go to Madison for the day less and less. Things went back to normal. The tumor they pulled out of his head was benign. He never quite talked the same after that, and his temper was maybe a little bit hotter than it used to be, but by and large he was back to the John we knew and loved.

February 2012

I was standing in the living room when my phone rang. It was Mom, calling me for our weekly Sunday chat. I could tell she was a little upset, so I asked her what was going on, and she says "Well, John's in the hospital. We went to the ER last night because of some back pain he's been having for a few days and they did a CT scan and found cancer in his kidney. He's having surgery tomorrow. How was your week?". She was clearly in shock. I don't really remember my reaction, but I know I was crying and couldn't talk and had to yell at her to stop talking for a second.

My sister was on a plane that afternoon from Pennsylvania and I was on the next available flight out of Portland on Monday morning. I landed in Milwaukee and drove straight to the hospital, where I found Mom and my sisters waiting in the waiting area. The surgery went fine and he was in recovery. They took out one of his kidneys but they weren't sure what was going on. They knew, or at least they had a pretty good idea, but they didn't want to say without running a bunch of tests. Again, we were at the hospital every day, sitting with him, willing him to get better. After a few days we finally got a diagnosis: stage IV metastatic renal carcinoma in his kidneys and lungs. As soon as he was healed from surgery he'd be starting chemo.

Summer 2012

John and Mom came to visit my then-girlfriend Emily and me for a few days. John was in between chemo drugs, since the first one stopped working eventually. We went up to Snoqualmie Falls for the afternoon, took some great photos, and had generally a pretty good weekend. They were supposed to take the train back from Seattle to Chicago, but John wasn't really feeling well enough to go that long so they ended up flying back.

October 31st 2012

Emily had been looking forward to handing out Halloween candy for about a week. It's a thing that they used to do in her family and she's pretty sentimental about these kind of things, so she roped me into it. She picked up two huge bags of mixed goodies a few days prior, and then at the appointed hour on the appointed day we took the camp chairs down to the front porch and set up, waiting for kids to show up.

An hour passed. None did. Not a single one.

Emily's phone rang. It was her aunt, which was pretty weird. Her aunt doesn't normally call, she just sends care packages in the mail filled with candy and bottles of BBQ sauce. Her aunt tells her that her mom is in the hospital, about to undergo emergency surgery to remove part of her colon. She's had a blockage for awhile and the doctors decided that it had to come out, right then and there. They think it's cancer.

Emily was on a plane the very next day and stayed for more than a week, one of the longest times we've been apart since we started going out in 2009. We talked on the phone every night. She came back but without any news about what kind of cancer her mom has. Eventually we find out it's stage IV metastatic colon cancer.

November 28th 2012

While Emily was visiting family I had started to develop some pain in my groin. I attributed it to over-use while she was away, but we were pretty concerned nonetheless. Emily made me promise that if the pain didn't go away, that I was to see a doctor and get it checked out. Well, the pain went away for a few days and I forgot about it, but then came back with a vengeance. I made an appointment the next morning to see a GP, which happened a few days later. He looked around, felt around, it was pretty embarrassing for both of us I think. He said "I'm pretty sure it's one of two things, neither of which is cancer, but let's get an ultrasound just to make sure." The next day I was in the ultrasound lab, and then I was back at work and trying not to freak out.

The next day I got a call from my doctor. He said that he put in a call to the urology department and they would put their best surgical oncologist on the job and that I would be taken care of. A few days later I finally got an appointment and went in. About two weeks after that I was in surgery. Mom flew out to help Emily take care of me while I recovered and stayed about a week. That's the longest she could stand to leave John, who was steadily getting worse.

December 26th 2012

My first day of chemo was the day after Christmas. About a week prior my diagnosis came back: stage IIa metastatic testicular germ cell cancer. Merry Christmas, everybody! Emily and I went in to the clinic at 8am to get familiar with the setup and for me to get my first treatment. It was... uneventful? I guess I expected it to be kind of dramatic, but really it was no big deal. My chemo schedule was one week of 5 days straight followed by two weeks of one day, repeated three times. Nine weeks. The first we were ok but by week seven I was a slug, barely able to make it into the treatment chair before falling immediately asleep. I got nauseous just getting in the car to go to the clinic. I'm proud to say I didn't throw up once the entire time. I didn't actually go the whole nine weeks. Toward the end my platelet count was too low to get treatment but my medical oncologist was adamant that skipping the last two treatments was no big deal. In March my doctors pronounced me tentatively cured.

April 19th 2013

Mom called while I was out to lunch with coworkers. She said that John was rapidly getting worse. I already had bought a ticket for May and she told me not to change it. Yet. The next day at noon she called again and said that I had to get home as quickly as I could. I changed my ticket, threw clothes in my backpack, and Emily and I were on the road to the airport in 10 minutes so I could catch the last flight of the day to Wisconsin. I didn't speed from Milwaukee to Milton, since it was dark and rainy, but I didn't dawdle either. When I got there Mom and my sisters were all there in the living room with John, watching movies and taking turns holding his hand while he lay in his hospice bed, breathing slowly. At 8am the next morning he passed with all of us around him. He didn't give up, right to the end. Mom said he had been so happy that I was better.

The funeral was that Friday. We spent the week gathering photos and figuring out the arrangements. I'm told it was a nice service, but I don't really remember much of it. Emily was able to be there for a few days which was helpful. John's ashes came back to the house that Wednesday. I won't tell you what him and Mom worked out for when she eventually passes, but I will say it's simultaneously the most romantic and the saddest thing I've ever heard.

June 4th 2013

I had my three-month follow up appointment. My blood markers were completely clear and the doctor said that my chest X-ray was so perfect she could put the scan in a medical textbook. I'm cured. Modern medicine works. Hallelujah.

At this point we don't know what's going to happen with Emily's mom but we're hoping for the best. Emily and I are getting married next year and hopefully everyone will be there, but life might get in the way. I guess we'll find out.

A very good friend of mine asked me on more than one occasion "Have you ever really thought about it?" "It" being death. At the time I didn't really have an answer for him. My mind was generally on other things back then. Girls, computers, cars, moving halfway across the country, etc.

Now, though, I can confidently say that my own mortality has crossed my mind once or twice. If there's a lesson to be drawn from my story it's this: live life how you want for as long as you can, because it may be shorter than you're expecting.

Epilogue

Emily's mom passed away peacefully in her sleep on August 24th, exactly one week after Emily and I got married in an intimate family-only ceremony.

Blog Relocation

Pete Keen — Sun, 02 Jun 2013 03:00:00 GMT

After a lot of thought and deliberation I've decided to retire bugsplat.info as my blog address. It's served me well for about six years, but the word "bugsplat" has recently gained some other unrelated connotations, the latter being the most unsavory. The other big reason is that at this point I would like people to associate my work with my actual name, not some other name that you would only know was related if you knew me already.

That said, any @bugsplat.info email addresses you have will continue to work. Any links to bugsplat.info will also continue to work through the magic of HTTP 301 Moved Permanently. Read on to see how I set that up, because it's kind of interesting.

HTTP has a whole bunch of different status codes. Pretty much everyone knows about 404 Not Found, of course. Other important codes are 200 Ok, which is what servers respond to requests they can handle along with the content at that address. 401 Authorization Required is another interesting one. That's what triggers a browser login box that you might see from time to time.

301 Moved Permanently (and its little brother 302 Moved Temporarily) are used to tell your browser "hey, the content that you're looking for is over at this other location". Google also uses these redirections when generating search results, which is why they're so important to get right.

Along with changing the domain of this I wanted to change the URL format of blog posts. I'm pretty tired of having the date in the URL itself. It's long and redundant and kind of ugly. Some small changes to the application code took care of generating those URLs, but what to do about the redirects? I installed a new app at bugsplat.info running this code:

require 'rack/rewrite'

use Rack::Rewrite do
  r301 %r{^/payment-integration.html}, "http://www.petekeen.net/mastering-modern-payments"
  r301 %r{^/\d{4}-\d{2}-\d{2}-(.*)$}, "http://www.petekeen.net/$1"
  r301 %r{^(.*).html$}, "http://www.petekeen.net$1"
  r301 %r{^(.*)$}, "http://www.petekeen.net$1"
end
run lambda { |env| [200, {"Content-Type" => "text/plain"}, ["Hello. The time is #{Time.now}"]] }

This app uses rack-rewrite, a small Rack middleware that emulates Apache's mod_rewrite but using a Ruby DSL. There's four specific rewrite instructions here. The first one rewrites the URL for my guide to using Stripe with Rails to a URL matching its actual title, which should help for SEO purposes. The next one sends old blog posts with the date and with .html at the end to the new site, sans date and suffix. The last two rules are generic catch-alls. Oh, and that little lambda thing is just there so the middleware has something to attach to, it's never actually called.

I think I caught all of the links on the site itself, but if you notice any weirdness with this new setup, please let me know.

Postgres Indexing - A collection of indexing tips

CRAIG KERSTIENS — Thu, 30 May 2013 23:55:56 GMT

Even from intial reviews of my previous post on expression based indexes I received a lot of questions and feedback around many different parts of indexing in Postgres. Here’s a mixed collection of valuable tips and guides around much of that.

Unused Indexes

In an earlier tweet I joked about some SQL that would generate the SQL to add an index to every column:

# SELECT 'CREATE INDEX idx_'
|| table_name || '_'
|| column_name || ' ON '
|| table_name || ' ("'
|| column_name || '");'
FROM information_schema.columns;
?column?
---------------------------------------------------------------------
CREATE INDEX idx_pg_proc_proname ON pg_proc ("proname");
CREATE INDEX idx_pg_proc_pronamespace ON pg_proc ("pronamespace");
CREATE INDEX idx_pg_proc_proowner ON pg_proc ("proowner");

The reasoning behind this is guessing whether an index will be helpful can be a bit hard within Postgres. So the easy solution is to add indexes to everything, then just observe if they’re being used. Of course you want to add it to all tables/columns because you never know if core of Postgres may be missing some needed ones

As included with the pg-extras plugin for Heroku you can run a query to show you all unused indexes. On Heroku simply install the plugin the run heroku pg:unused_indexes to show the size and number of times an index scan has been used. On a non Heroku Postgres database you can run:

# SELECT
schemaname || '.' || relname AS table,
indexrelname AS index,
pg_size_pretty(pg_relation_size(i.indexrelid)) AS index_size,
idx_scan as index_scans
FROM pg_stat_user_indexes ui
JOIN pg_index i ON ui.indexrelid = i.indexrelid
WHERE NOT indisunique AND idx_scan < 50 AND pg_relation_size(relid) > 5 * 8192
ORDER BY pg_relation_size(i.indexrelid) / nullif(idx_scan, 0) DESC NULLS FIRST,
pg_relation_size(i.indexrelid) DESC;
table | index | index_size | index_scans
---------------------+--------------------------------------------+------------+-------------
public.grade_levels | index_placement_attempts_on_grade_level_id | 97 MB | 0
public.observations | observations_attrs_grade_resources | 33 MB | 0
public.messages | user_resource_id_idx | 12 MB | 0
(3 rows)

Costs of Indexing

There are really a couple of primary costs when it comes to indexing your data. The first is the overall size of the index. Indexes take size on disk, fortunately in most cases disk is pretty cheap. If you’re limited on disk size and not on your current performance then its pretty clear the trade-off you want to take. If you do need to get the size of your index you can do that by running:

# SELECT pg_size_pretty(pg_total_relation_size('idx_name'));

The harder trade off to look at is the cost in terms of throughput. As your data comes in there’s a cost for maintaining that index as the data within it has to be computed. If you’re doing crazy regex’s in your index then you can expect this to have an impact on your throughput.

Composite Indexes vs. Multiple Indiviual Indexes

A composite index is an index that includes multiple columns. Given an example table of purchases:

# \d purchases
Table "public.purchases"
Column | Type | Modifiers
-------------+-----------------------------+-----------
id | integer | not null
item | integer |
quantity | integer |
color | integer |

You might want to add an index on item and quantity together. You can do this with:

CREATE INDEX idx_purchases_item_quantity_color ON purchases (item, quantity, color)

From now on if you included item and quantity in a query its likely it would use this index just as it would if you used item, quantity and color. If you have a large varied set of data within each of these such an index can prove very useful. The caveat is that if you’re querying against only quantity and color then this index is useless, it must include the item column.

In contrast if you have three individual indexes Postgres may combine these or simply use one that would be the most efficient out of the three.

CREATE INDEX idx_purchases_item ON purchases (item);
CREATE INDEX idx_purchases_quantity ON purchases (quantity);
CREATE INDEX idx_purchases_color ON purchases (color);

Of course in this case if you query any individual column it would use the index if appropriate.

What Else

What else do you want to know about Postgres Indexing? Drop me a line craig.kerstiens at gmail.com or hop over to Postgres Guide and read a little there or even contribute some articles of your own.

Design for Failure: Processing Payments with a Background Worker

Pete Keen — Thu, 30 May 2013 03:00:00 GMT

Processing payments correctly is hard. This is one of the biggest lessons I've learned while writing my various SaaS projects. Stripe does everything they can to make it easy, with quick start guides and great documentation. One thing they really don't cover in the docs is what to do if your connection with their API fails for some reason. Processing payments inside a web request is asking for trouble, and the solution is to run them using a background job.

The Problem

Let's take Stripe's example code:

Stripe.api_key = ENV['STRIPE_API_KEY']

# Get the credit card details submitted by the form
token = params[:stripeToken]

# Create the charge on Stripe's servers - this will charge the user's card
begin
  charge = Stripe::Charge.create(
    :amount => 1000, # amount in cents, again
    :currency => "usd",
    :card => token,
    :description => "payinguser@example.com"
  )
rescue Stripe::CardError => e
  # The card has been declined
end

Pretty straight-forward. Using the stripeToken that stripe.js inserted into your form, create a charge object. If this fails due to a CardError, you can safely assume that the customer's card got declined. Behind the scenes, Stripe::Charge makes an https call to Stripe's API. Typically, this completes almost immediately.

But what if it doesn't? The internet between your server and Stripe's could be slow or down. DNS resolution could be failing. There's a million reasons why this code could take awhile. Browsers typically have around a one minute timeout and application servers like Unicorn usually will kill the request after 30 seconds. That's a long time to keep the user waiting just to end up at an error page.

The Solution

The solution is to put the call to Stripe::Charge.create in a background job. This example is going to use a very simple background worker system named Sucker Punch. It runs in the same process as your web request but uses Celluloid to do things in a background thread.

First, let's create a job class:

class StripeCharger
  include SuckerPunch::Worker

  def perform(event)
    ActiveRecord::Base.connection_pool.with_connection do
      token =  event[:token]
      txn = Transaction.find(event[:transaction_id])

      begin
        charge = Stripe::Charge.create(
          amount: txn.amount,
          currency: "usd",
          card: token,
          description: txn.email
        )
        txn.state = 'complete'
        txn.stripe_id = charge.id
        txn.save!
      rescue Stripe::Error => e
        txn.state = 'failed'
        txn.error = e.json_body
        txn.save!
      end
    end
  end
end

Again, pretty straightforward. Sucker Punch will create an instance of your job class and call #perform on it with a hash of values that you pass in to the queue, which we'll get to in a second. We look up a Transaction record, initiate the charge, and capture any errors that happen along the way.

Transaction in this case is a simple ActiveRecord object with just a few attributes, just enough to capture what Stripe gives us:

class Transaction < ActiveRecord::Base
  attr_accessible :stripe_id, :state, :amount, :error, :email
end

Sucker Punch needs to know about our job class, so let's tell it in an initializer:

SuckerPunch.config do
  queue name: :payments_queue, worker: StripeCharger, workers: 10
end

Now for the controller that ties it all together:

class TransactionsController < ApplicationController

  def create
    txn = Transaction.new(
      amount: 1000,
      email: params[:email],
      state: 'pending'
    )
    if txn.save
      SuckerPunch::Queue[:payments_queue].async.perform(
        transaction_id: txn.id,
        token: params[:stripeToken]
      )
      render json: txn.to_json
    else
      render json: {error: txn.error_messages}, status: 422
    end
  end
  
  def show
    txn = Transaction.find(params[:id])
    raise ActionController::RoutingError.new('not found')
      unless txn

    render json: txn.to_json
  end
end

The create method creates a new Transaction record, setting its state to pending. It then queues the transaction to be processed by StripeCharger. The show method simply looks up the transaction and spits back some JSON. On your customer-facing page you'd do something like this:

function doPoll(id){
    $.get('/transactions/' + id, function(data) {
        if (data.state === "complete") {
          window.location = '/thankyou';
        } elsif (data.state === "failed") {
          handleFailure(data);
        } else {
          setTimeout(function(){ doPoll(id); }, 500);
        }
    });
}

Your page will poll /transactions/<id> until the transaction ends in either success or failure. You'd probably want to show a spinner or something to the user while this is happening.

With this setup, you've insulated yourself from problems in your connection to Stripe, your connection to your customer, and everything in between.

This is an excerpt from my guide Mastering Modern Payments: Using Stripe with Rails.

Postgres Indexes – Expression/Functional Indexing

CRAIG KERSTIENS — Wed, 29 May 2013 23:55:56 GMT

Postgres is rich with options for indexing. First you’ve got a variety of types, and beyond that you can do a variety of things with each of these such as create unique indexes, use conditions to index only a portion of your data, or create indexes based on complex expressions or functions. In cases where you commonly use various PostgreSQL functions in your application or reporting you can get some great gains from this.

Let’s take a look at a really simple case. Given a basic user table:

# \dt users
Table "public.users"
Column | Type | Modifiers
------------+-----------------------------+-----------
id | integer | not null
email | character varying(255) |
created_at | timestamp without time zone |

You may commonly want to run a report against it showing your signups by date. Let’s say you do this by running the query:

SELECT
count(*),
date_trunc('day', created_at)
FROM
users
GROUP BY
2;

If you’re commonly using date_trunc('day', created_at) for grouping, filtering, or projecting it out you can get some great gains by creating an index on this:

# CREATE INDEX idx_user_created ON users(date_trunc('day', created_at));

Of course you can go beyond the built in functions of Postgres and use more complicated functions you create yourself. For example if you have JSON stored within PostgreSQL, have PLV8 enabled, and want to create a Javascript function to parse and return the text for a given key:

# CREATE OR REPLACE FUNCTION
get_text(key text, data json)
RETURNS text $$
return data[key];
$$ LANGUAGE plv8 IMMUTABLE STRICT;

Of note in the above function is IMMUTABLE and STRICT. Immutable specifies that the function given the same inputs will return the same result. Strict means that if you send in NULL values you’ll get a null result.

Given some example data inside your JSON field:

{
"name": "Craig Kerstiens",
"location": "San Francisco",
"numbers": [
{
"type": "work",
"number": "123.456.7890"
},
{
"type": "home",
"number": "987.654.3210"
}
]
}

If you wanted to return just the name you could index on:

# CREATE INDEX idx_name ON users(get_text('name', json_data));

Or even combine with built ins for a case-insensitive version:

# CREATE INDEX idx_name ON users(lower(get_text('name', json_data)));

Indexes like all of the above can be useful when you’re filtering on something that postgres can take advantage of. In most cases any conditions with the exception of a LIKE beginning with a % work for this. With Postgres 9.2 even a count(*) in certain cases can take advantage of the index because of index only scans.

Whether you’re looking to take advantage of all the power of Javascript with JSON or another procedural langauge – or simply speed up a basic report using built in functions expression indexes can give you some great benefits.

My SQL Bad Habits

CRAIG KERSTIENS — Sun, 26 May 2013 23:55:56 GMT

I’m reasonably proficient at SQL – a coworker when pseudocoding some logic for him pointed out that my pseudocode is what he thought was executable SQL. I’m fully capable of writing clear and readable SQL – which most SQL is not. Despite that I still have several bad habits when it comes to SQL. Without further adieu heres some of my dirty laundry so hopefully others can not make the same mistakes.

Order/Group by Column Numbers

When quickly iterating on a query its a lot less typing to put the column number as the thing you want to order by. Here’s a quick lightweight example:

SELECT
email,
created_at
FROM
users
ORDER BY 2 DESC
LIMIT 5;

This gives me my last 5 users that have signed up for my site. Of course as soon as I have this I may want to add some data to it, like their first name so I can send them a welcome email. I quickly alter the query to:

SELECT
email,
first_name,
created_at
FROM
users
ORDER BY 2 DESC
LIMIT 5;

And now I have 5 users that have signed up ordered by their first name. Sure its obvious when you have 1 column you’re ordering by, but when you have GROUP BY 1, 2, 3, 4, 5, 6 which is actually open in one of my tabs currently its a bit more confusing….

Though if you really want to have some fun, share a query with someone that looks something like this:

SELECT
email as "3",
first_name "2",
created_at "1"
FROM
users
ORDER BY "1", "3" DESC
LIMIT 5;

Implicit Joins

I seldom use the syntax INNER JOIN. Instead I simply put the two tables in my where clause and ensure I have a where condition. The problem with ensuring I have a where condition is sometimes I don’t, especially when you’re dealing with 3 tables.

SELECT
email,
product.name,
product.price
FROM
users,
orders,
items
WHERE users.id = orders.user_id
AND orders.id = items.order_id

Is less clear (especially when dealing with 5-6 tables) than the alternative:

SELECT
email,
product.name,
product.price
FROM users
INNER JOIN orders on users.id = orders.user_id
INNER JOIN items on orders.id = items.order_id

Lack of comments

I comment my SQL far less than I comment my code, yet it can be done just as easily. For example I have this in one of my queries:

SELECT convert_from(CAST(E'\\x' || array_to_string(ARRAY(
SELECT
CASE
WHEN length(r.m[1]) = 1
THEN encode(convert_to(r.m[1], 'SQL_ASCII'), 'hex')
ELSE substring(r.m[1] from 2 for 2)
END
FROM regexp_matches(url_here, '%[0-9a-f][0-9a-f]|.', 'gi') AS r(m)
), '') AS bytea), 'UTF8');

While this has its own issues theres no documentation around what this actually does, in contrast:

--- DECODES url ---
SELECT convert_from(CAST(E'\\x' || array_to_string(ARRAY(
SELECT
CASE
WHEN length(r.m[1]) = 1
THEN encode(convert_to(r.m[1], 'SQL_ASCII'), 'hex')
ELSE substring(r.m[1] from 2 for 2)
END
FROM regexp_matches(url_here, '%[0-9a-f][0-9a-f]|.', 'gi') AS r(m)
), '') AS bytea), 'UTF8');

Comments also work well inline at the end of a line.

Large Manually Generated Lists

A lot of times in working with some specific data set I’ll manually or automatically generate a list that I want to filter. A common example is filtering out staging/dev environments. I’ll often manually search and prune the list, then save that result for the queries I’m going to build going forward. This is a bit of effort but still feels reasonable the downside is it results in something like:

SELECT
foo
FROM
bar
WHERE
bar.id NOT IN (34723, 42735, 32321, 47205, 20375, 30261, 26194, 109371, 9313, 6351, 20184, 50273, 34735, 39854, 23954, 25323, 23405, 30528, 50182, 29340, 47659, ... and the list goes on)

SQL is meant to be reasonable for containing some level of logic. Data changes, hard coding keys is going to bite you at some point, spend the extra effort and re-use something thats clear.

What else

I’m sure theres plenty more; I suspect within a few minutes of sitting down with someone they could point out some other bad habits. While I know mine at least some of mine I still often know the trade-off. What are yours? I’d love to hear to document them for others so hopefully they can prevent developing the same bad habits. Let me know; craig.kerstiens@gmail.com

Book Review: Tubes: a Journey to the Center of the Internet

Pete Keen — Sat, 25 May 2013 03:00:00 GMT

Andrew Blum took his Internet service completely for granted until one day a squirrel chewed through a cable in his neighborhood, cutting off his access and sending him around the world, attempting to answer the question: What exactly is the Internet? What is it made of? Where is it, really? He documents his travels in the book Tubes: a Journey to the Center of the Internet.

Blum starts out describing a map of the internet (note: not this other map of the internet) made by a company named TeleGeography, which shows an incredibly intricate web of little lines connecting major cities around the globe. These lines are fiber optic cables, running over land and under oceans, hooking up with each other in giant data centers in places like Ashburn, VA and London, UK. Blum goes to these interconnect points and talks with the people who run them, the people who started them back in the early 1990s when the Internet was just getting going. The book concludes with a trip to Prineville, OR to see one of Facebook's newest data centers, an example of where all the data that traverses these fiber optic lines ends up.

Prior to reading Tubes I had a pretty good understanding of some of these topics. For example, Neil Stephenson (author of some of my favorite books) wrote an amazing article for Wired Magazine in 1996 where he travels around the world following the building of FLAG, a state of the art (for the time) 26,000 kilometer undersea cable. Tubes did have a lot to say about how the early days of the commercial internet came about, though, especially with regards to the early Internet exchange points in Washington DC, San Jose, and London. Also, I had no idea that there was such a huge facility in Ashburn run by a company named Equinix. Blum goes into a lot of great detail about the history of these companies and the people that run them.

Overall I think Blum does a good job communicating that the Internet really isn't a thing so much as an idea, very much made up of the people who design and build out the fiber optic links and the things they link together. The fiber is almost an afterthought, a means to an end. If you're at all interested in how the Internet came to penetrate our lives over the last two decades, you should check out Tubes.

New Blog Design

Pete Keen — Fri, 24 May 2013 03:00:00 GMT

For the past three years this blog has has featured the same basic design. Header, sidebar, content, footer. Simple, classic, kind of ... homely. Today I decided to give a new design a whirl, and if you're reading this on the website you can see what's changed.

Essentially I've removed the sidebar in favor of a clean header and footer. I've also condensed several pages onto a comprehensive About page. The colors are basically the same, although I've brightened up the background a bit. The biggest change by far has been switching from some random two-column css layout I found somewhere on the internet to a design based on Twitter Bootstrap. I used a tool called Bootstrap ThemeRoller to customize the colors and fonts and then made a few remarkably small changes to the markup and here we are.

See any weirdness or bugs, or want to offer some criticism, constructive or not? Email me or Tweet at me.

Mounting a Magic Trackpad on a Kinesis Advantage Keyboard

Pete Keen — Fri, 17 May 2013 03:00:00 GMT

Update: The Magic Trackpad 2 fits in the same spot but you might have to use a different sticky material. See these tweets for details.

Way back in early 2009 my wrists started acting up. As a professional software developer only two years into my young career, this was super concerning. I asked around at the office and my coworkers told me I should get a Kinesis Advantage Pro keyboard, which aside from being super goofy looking, seems to have cured a lot of people of their RSI symptoms. I looked into it and eventually broke down and bought one, and then another one because the first developed some sticky keys but I didn't want to part with it to send it back for service.

Using a mouse with the Kinesis is a little bit tiring, though. It's a much larger keyboard than you might be used to, so the reach from keyboard to mouse is in kind of a weird direction and your mouse might be in a weird spot. I've been using an Apple Magic Trackpad for the past few years and after awhile I got sick of the reach and stuck it in the obvious place, the giant blank spot between the keywells of the Kinesis, like so:

It's not an exact fit, though, so I added a little shim made of two wrist rest pads stuck together:

This worked alright but as time went by the trackpad started slipping around and I had to adjust it every few seconds. I started dreaming about designing and 3D-printing a bracket to hold the trackpad in place so it wouldn't move around but wasn't permanently attached to the keyboard in case I need to replace it. That's expensive and tedious and probably not that fulfilling in the end, so I investigated a bunch of other options and finally hit upon the Heininger 1065 CommuteMate DashGrip. It's intended to be slapped onto a car dashboard so you can perch your sunglasses and keys (wait aren't they in your ignition?) and spare change and 40oz Slurpee on your dash, without worrying that they'll scatter everywhere when you take a corner too hard.

The DashGrip is basically a rectangle of sticky gel-like material that's surprisingly difficult to photograph:

In this photo I've already cut it to size and attached it to the trackpad. The other half is still in its original shipping wrapper. It's easy to cut down to size with scissors, and with it the trackpad fits snuggly but very securely in its designated spot. By snug I mean very strong. Strong like ox:

The coolest part is that it doesn't leave any adhesive residue on the keyboard or trackpad. I'm not really sure how this stuff works but it seems pretty magical. Here's a side view of what it looks like put together:

There's two caveats to this otherwise great solution. First, it's not going to work for you if you hate tap-to-click. The switches for clicking are actually inside the two front feet of the trackpad, which in this setup are not really in a clicking kind of mood since they're embedded in soft sticky gel. Relatedly, if you want to use the trackpad away from the keyboard it's going to be kind of a pain, because of how super strong the bond is (see photo above). I happen to love tap-to-click, though, and don't use the trackpad away from the keyboard often enough to care.

If you can live with those two things, I suggest you try out this setup, especially if you already have a Magic Trackpad and a Kinesis. It's pretty awesome.

Distributed Personal Wiki

Pete Keen — Fri, 10 May 2013 03:00:00 GMT

For as long as I can remember I've been trying to find a good way to keep personal text notes. Recipes, notes, ideas, that kind of thing. Things that aren't really suited to blogging. Along the way I've used (and stuck with) PmWiki, DocuWiki, TiddlyWiki, and most recently I built my own sort-of-pseudo-wiki Marginalia.

Lately, though, it's been kind of a drag to use a web-based application just to write down some work notes. Having sort of an obsession with Markdown I decided to just start keeping notes in Markdown-formatted files in a directory. Of course, files that aren't backed up are likely to disappear at any moment, so I naturally stuck them in a git repository and pushed to my personal git server. But then, how do I deal with synching my work and home machines? I guess I'll manually merge changes...

Yeah, that lasted about 10 minutes. I had a whole setup baked up that tied together a rake script and an OS X LaunchAgent that watched a directory and everything, but the merging is of course the hardest part.

I went hunting for alternatives. I even briefly considered trying out Evernote, but that didn't really meet with my self-hosting ideals. Dropbox was also a non-starter because, again, not self-hosted. Then I came across SparkleShare and my eyes lit up. SparkleShare uses git as its transport mechanism, can sync with any git repository, and automatically watches a given directory for changes.

SparkleShare is trivial to set up. Download the package, drag the app to /Applications, and run it. It'll create a directory at ~/SparkleShare and then ask you to add a hosted project. Just create a git repo somewhere (or not, it can sync to any directory that it can talk to via ssh), point SparkleShare at it, and you're done. Do this across every machine you want this share on and whenever you add, delete, or modify a file in that directory it'll get synced to all of the other machines automatically.

So, that covers the sync and backup strategy. What about the wiki part? I've been using Emacs markdown-mode for several years to get syntax highlighting when writing blog posts and such, and unbeknownst to me in a recent release Jason added Wiki links and keyboard shortcuts to follow them. I upgraded my dotfiles to the latest version of markdown-mode and then bam, wiki in a git repo.

There are two more things that I want to do that would make this system work really well. First, I want to set up Gollum on one of my servers and point it at the git repo that SparkleShare is syncing, so that I can have a web interface and pretty formatting when I want it. The nice thing is that markdown-mode and Gollum use the same syntax for wiki links.

Second, I want to replicate the send-an-email-to-create-a-note functionality that Marginalia has. I think I can do this with a tiny CGI script hooked up to Mandrill or Mailgun's incoming email processing system. All it has to do is drop the message text into the SparkleShare-synced directory with a filename based on the subject.

Learning Ruby on Rails: Why? How? Pitfalls?

Brightball Articles — Thu, 09 May 2013 19:00:00 GMT

I've always been a proponent of the "right tool for the job" approach to programming. Different languages are well suited for different situations. Over the past 2 years I've spent a great deal of time with Ruby on Rails after coming from a background of PHP, Java and Perl. Here's how I got started and some of the lessons I learned along the way.

CX – Conference Experience (Facilitating Communication)

CRAIG KERSTIENS — Mon, 29 Apr 2013 23:55:56 GMT

Following up on my earlier post about CX or Conference Experience – I’m going to dig in a bit on how you get good conversation to happen. In the past two years I’ve been to nearly 20 conferences, I’ve been to conferences with great talks, with great parties, with great swag, and hands down my favorite conferences have always been a result of great conversation. With the number of talks that are recorded and immediately available online after, what can I say I’m a hallway track guy.

I’ve seen a number of conferences intentionally design around this concept, in some ways the unconference is purely a hallway track conference. I’ve also seen conferences that weren’t clearly planned for this and have pulled off some of the best situations where people turn their phones off and engage in real conversation.

Breaks

Breaks are always a hard part to balance. Every conference organizer I’ve talked to felt they had the perfect mix for breaks. If you’re designing for convesation then you need to allow time for conversation to actually happen. Whether a multi-track or single track conference 15 minutes for breaks doesn’t give you the time to actually engage in conversation. If multitrack its time enough to grab a drink then head to the next room. If single track its time to have the entire room leave and then come back.

Even 30 minute breaks fall into this category. At a multitrack conference I’m usually saying hi to enough people during a 30 minute break that I don’t have the chance to really get deep into a conversation. If I do it usually results in skipping the next talk.

Staggered Talks

Locations for Conversations

Having places for people to get away an talk is critical, in a venue that only has 1 main room and no convenient places for a quick coffee or beer it becomes very hard for conversations to happen. If its on the agenda to facilitate such things thats great, but going a step further and allowing for it to be facilitated naturally is even better. On more than one occasion I’ve been mid conversation wtih someone, we cut it off agreeing to pick it back up at the next break only to never locate them again for the rest of the conference. I’d have just as much preferred to continue the conversation there, but sadly there was in the middle of a talk room with no where else feasible to go.

Water

This one caught me by surprise… The event was PyCodeConf in Miami and it was one of the evening activities – a pool party with food, drinks, and mariachi band. Early into the evening, inevitably someone was thrown into the pool (by friends in a general good nature), but thus ruining a lovely iPhone. As a result I’m pretty most people, just as I did, went to their rooms but their cell phone away, or set it near their stuff. Much of the rest of the evening was an entirely internet and twitter free evening. Causing lots of uninterrupted conversations to happen and resulted in a pool full of geeks.

Welcoming noobs

The big difference between just assuming this will happen and actively working to facilitate it is who gets involved in the conversation. For a first time presenter there’s a wealth of nerves about talking. For a first time attendee there’s a wealth of knowledge to soak up. First time attendees may not feel as comfortable interjecting them into a conversation, approaching presenters, or talking to someone they’ve been following for years – yet they have just as much to add as anyone else.

Creating places where more conversation is happening helps ease this and build a better community.

CX – Conference Experience (People)

CRAIG KERSTIENS — Mon, 29 Apr 2013 23:55:56 GMT

Following up on my earlier post about CX or Conference Experience – I’m going to dig in a bit on getting the right people there. There’s a lot of different ways to approach this from having a good ratio of:

Designers to Developers
Females to Males
Noobs to well known community members
Overall lack of suits

While this is by no means a how we can solve all of the above problems, theres some bas

Talks

Quality talks are obviously important. A conference should at whatever cost ensure that the talks are good. This can happen a variety of ways, but regardless of the method you should ensure talks are worthwhile; since this is largely what most people are paying to attend for.

Invite-only

Perhaps the easiest way to do this is via invite only to speakers you know will do a good job. The downside ot this of course is that you must already know enough people to fill out a good agenda, and also you limit the ability for others to contribute.

Coaching/Practicing

This is not an either/or option with any of the other pieces for getting a good agenda. Having presenters do a trial run can ensure a minimum level of quality, and working with them to coach them can help make the talk even more effective. Its likely that the organizers know the audience as well as anyone, so no one better than them to help with this. Of course this is a time sink for both parties, but can give good returns. Of course this could be done independent of a specific conference such as through speakup.

At a very least getting a quick run through, outline, or something of that nature can ensure that a presenter doesn’t fill a 45 minute talk slot with only 5 minutes of content.

Open CFP

Likely the most common approach to getting speakers is having an entirely open CFP. Its typical then that either the organizers or a speaker selection commitee then discusses and makes selections. This can usually work to have a nice balance of experienced and known to be good speakers and newer less experienced speakers that can have great potential as well.

Open Open CFP

Going even further is an open CFP where all talks are published after the CFP then voted on. While this does a great deal to ensure transparency, it doesn’t necessarily improve ensuring theres a great line up of speakers. Being able to write an interesting talk proposal is an entirely separate process from delivering an interesting talk.

There’s a common question lately for both the open CFP and open open ones one whether to do blind review of the talks. The logic here is an attempt to be entirely fair, versus having some bias. This is an understandable goal, but can come at the expense of quality. Truth be told; I’m not entirely sure of a way to balance this.

Choosing

There’s definitely not a one size fits all. If you’re goal is to provide a good list of talks then you should keep that in mind in how you decide your talks. If you’re goal is to pull others in then it should be shaped differently. Zach Holman recently talked a bit about this and had some interesting ideas to minimize risk for new speakers.

The Drawback to Web Frameworks

Brightball Articles — Mon, 29 Apr 2013 20:37:00 GMT

Web frameworks are great, don't get me wrong here. They provide a structure and consistency across projects that will transcend developers over the life of a system while dramatically simplifying the code base amongst other wonderful side effects. But what's the downside?

"Make it faster."

Brightball Articles — Sun, 28 Apr 2013 05:22:00 GMT

I'm obsessed with performance tuning. It's an itch that can never fully be scratched. A sickness that can never be cured. Here's the story of how I caught the bug.

CX – Conference Experience (Talks)

CRAIG KERSTIENS — Sat, 27 Apr 2013 23:55:56 GMT

A couple of weekends ago I had the great opportunity to attend lessconf. It was an all around great conference, and as a result of the greatness I ended up having a conversation with a few people around conference experience. I must give much of the credit to Swift, as he mentioned he’d already been thinking alot about this since Waza. In general it feels like there’s a few key themes that any conference should focus on, then a lot of small things that can really push it over the top. Here’s a few:

In general the key areas for any great conference are:

Talks
Great people
Ensuring communication happens
Bonus points

Digging in deeper on the first area…

Talks

Invite-only

Coaching/Practicing

At a very least getting a quick run through, outline, or something of that nature can ensure that a presenter doesn’t fill a 45 minute talk slot with only 5 minutes of content.

Open CFP

Open Open CFP

Choosing

The Impossible Assignment

Brightball Articles — Thu, 25 Apr 2013 20:12:00 GMT

I had the opportunity to visit the class of one of my legendary former professors yesterday and got to share a classic story about him...the time he gave us an impossible assignment.

Create Better Villains for your Adventure

Ben Overmyer's Site — Sat, 20 Apr 2013 03:00:00 GMT

Villains are the mainstay of any good adventure. With very few exceptions, there is always some form of evil mind plotting dark things in the background. Whether this is a mindless mob of creatures that just happens to be devouring the locals, or a masterfully devious court magician manipulating a king to sinister ends, you can’t get away from the guys.

This blog post aims to assist you in building memorable villains that are more than just numbers on a sheet of paper. If you’re a player and not a game master, the character building advice in here might help you too, but this is specifically geared towards the folks weaving the story behind the encounters.

The Nature of a Villain

While it’s fairly obvious that a villain exists to ruin the lives of good people, a truly memorable villain has much more than that going on. He has his own motives and desires. He may be acting in accordance with his own morals, or he might not care at all.

Most frequently, a villain’s goals shouldn’t be an end, but a way to reach some other starting point. The kobold king isn’t sending raiding parties out to nearby villages because he feels like it; he’s weakening the nearby human presence so his people can expand outward from their all-too-small cave system.

The depth of a villain’s forward thinking depends on how intelligent you want him to be. Don’t restrict a villain’s planning just because he’s a minor NPC in your adventure. Players will inevitably do something you don’t expect, and if they just happen to latch onto your minor villain, you’ll then have a direction to take that in.

Journey of the Hero

The player characters will only grow stronger as a result of having a characterful villain as their foil. You can consider your adventure a success when not only does it give your PCs the magical loot of awesomeness they’ve been waiting for, but it also encourages them to reconsider their characters’ actions and motives.

Joseph Campbell’s Hero with a Thousand Faces is a fantastic guide to setting up an epic story. His writings on the “road of trials” in particular are relevant to creating worthwhile villains for the players.

A player character that’s just starting out usually has weak stats, basic equipment, and almost zero personality. She might have a single personality quirk or cliché that her player jots down to guide her role-playing, but rarely will players take the time to build a full story for their characters. After all, that’s what they’re hoping you will do.

As the PC encounters and interacts with your adventure’s many challenges, locations, and characters, she will grow in both power and personality. If her player is into role-playing, this will happen sooner rather than later, and with more depth. She will need a good villain, though, to balance that development against.

Brainstorming

Creating a villain should start with looking at what you want to achieve with the adventure and with the campaign at large. An adventure isn’t a set narrative that you can just write and expect to stay the way you wrote it. The players will do things you don’t expect and interact with the environment in ways that force you to adapt the adventure to fit their actions. As such, your villain should support your goal with the adventure, rather than just play to the script.

Jot down a few notes about your adventure’s premise. Is it a classic battle of good versus evil? Is it a day in the life of an adventurer? Is it an exploration of the unknown? Think about your favorite movies or books that fit the theme you’re going for. What twists in those stories really got your blood pumping?

Potential for Loss

Think about what would really hurt the player characters. I don’t mean in terms of hit points lost or character sheets thrown away – I mean tragedies that specifically target each PC’s hopes, dreams, and beliefs. Whether the players are into role-playing or not only determines whether you have more or less fodder for this; it’s still possible to make a tactical gamer sit back in his chair with a thud when his character loses something dear to him. It usually just changes the context.

Build something of that potential loss into each villain of the adventure. You don’t need to include every player character in every adventure, but you shouldn’t focus on only one player character either. Also, choose a handful of NPCs and target their potential for loss also. It helps build a living world when you include characters other than the players’ in your adventures.

Potential for Gain

Saruman was originally a hero, but gave in to temptation.

Now think about what the villain can offer the player characters. Temptation plays on base emotions; use it. For tactical gamers, this could be a particular piece of loot that they want desperately. For role-players, it might be a bit of information about a relative they’ve been searching for. For every player, write down at least one thing that the villain might have the ability to provide.

As with the potential for loss, also make sure that some of your NPCs stand to gain something by working with the villain. Even if that incorruptible priest of all that is good can’t possibly be turned to the dark side, the players might not know that, and that creates tension. Tension is what every great story (and adventure) is built on.

Villains are Story Generators

Other than trying to provoke emotional tension from players through the potential for loss or gain, a good villain should also act as a way to move the story along. The best stories, whether they’re explosion-ridden Hollywood blockbusters or 1500-page literary masterpieces, are about the heroes changing in some way over the course of the story. Villains are almost always the catalyst for that change.

If your players are craving action, send a wave of minor baddies against them. Let them know that the villain made this happen, whether directly or indirectly. Weave a spider’s web with your villain at the center. Even if he is not of the evil genius persuasion, he’s still fully capable of planning ahead.

Summary / TL;DR Version

Keep your villains close to your players’ hearts. Tension is a great story device, so use it. Tempt the PCs. Hurt the PCs. Make a list of the villain’s goals and aspirations.

And, above all else, remember: villains are not just stat blocks, they’re people too.

Heroku's Acquisition 2 Years Later

CRAIG KERSTIENS — Thu, 18 Apr 2013 23:55:56 GMT

Just over two years ago, Heroku was acquired. I was around and peripheral to this just before the acquisition and came on board only barely after. While there is a large group of people that have been there longer than I have (several for 4+ years) I’m still commonly asked how things have changed, how things work, and other questions of that nature. I wrote about some of these processes over a year ago in the months after joining Heroku around our hiring, our teams, and how we work. Many of these things haven’t changed, and yet almost always at a conference I’m asked how are things different since being acquired.

Here’s my personal take (and while I don’t typically include this – to be safe, this is not an official Heroku view of what’s changed).

Are things different since the acquisition?

Heroku in many ways operates like a wholly owned subsidiary or as an independent business within Salesforce. We still have our own office space, our own IT (if you can even call it that), and in general entirely own workflow. I have a salesforce email account that automatically forwards to my Heroku google apps account, and that’s about as much as I know about it. In fact, we’re currently preparing for our new office and its planned to be our home for somewhere between the next 5-10 years.

At the same time some things do flow through Salesforce, some of those things are logistical some strategic. First the perhaps most unfortunate part – hiring. This has not really changed how we hire, but rather once you are coming on board theres more paperwork. In general this is the most painful part that instead of 1/2 pieces of paper theres a few to sign. At the same time some of these come with some greater gains such as benefit, etc. To be honest I dont fully recall what all of the paper work is, but either way there is a bit more of it. The other area we really see this effect is expense accounts. All expenses flow through concur, which in my opinion is a pretty good solution. I’ve used many worse expense solutions and seen few if any better.

Do they influence the product?

Not really

In general we at Heroku aim to have some broader alignment around what we’re trying to accomplish. Salesforce believes in helping their customers become a customer company. We believe that developers are worth of great experiences. Salesforce believes in ensuring its customers are successful. Heroku believes that developers should focus on adding value of their customers not just keeping the lights on. In all of these things its important to have alignment. Not for the company we’re building for the next six weeks or six months, but for the next six years. Given both Salesforce’s and Heroku’s worldview I believe what we’re trying to accomplish is in good alignment.

But does Salesforce influence our product roadmap? No, we aim to listen to our customers problems and build to solve those problems and through that improve the way software is delivered. At the time Salesforce acquired us, the pieces of Cedar were already in motion; including Procfile support, logplex, and other pieces. Later was fully productized Cedar and which was a goal we’d long had – making other languages such as Node, Java, and Python available. This wasn’t driven because Salesforce wanted Java, but because we saw value in delivering the same value of the platform to other communities.

Have we changed?

Sure, we’ve changed, but I’d surmise very little if at all due to Salesforce. We’ve grown from a 20 person company to now over 100. We were a primarily local team and now have people all over. Company offsites get a bit harder to coordinate with 100 people, and finding a date that absoluely everyone can make it is nearly impossible. Many of our changes are more strictly a change of growth than they are because of Salesforce.

I dont get it, why acquire you then?

Because we’re aligned in a much bigger vision, we can both work together long term to accomplish our goal.

The internet is changing the world; software is everywhere.

I believe Salesforce truly understands this. Heroku does as well (though, at the time of the acquisition I fully suspect many Herokai held their breath to see how it would all go). Six weeks went by without trying to ‘change us’, then six months, and here we are over two years later. In many ways so many of us have started to better understand that Salesforce truly understands the above.

If you understand the above, then you know that delivering software and improving that process gives any business a competitive advantage. This is at the core of the value we aim to provide. E.g. If you ranked all companies in terms of how strong they aligned with Heroku, Salesforce might be at the very top.

Profiling Django templates with Django-Debug-Toolbar

Tom Forbes — Thu, 18 Apr 2013 23:46:58 GMT

My last post about the speed of Django’s templating language caused a bit of a stir and it was clear that people didn’t really have a clue how long the templates were taking to render in their applications. Enter Template-timings Template-timings is a panel for Django-debug-toolbar (which everyone s...

Using array_agg in Postgres – powerful and flexible

CRAIG KERSTIENS — Wed, 17 Apr 2013 23:55:56 GMT

In almost any application it’s common to want to aggregate some set of values together, commonly in a comma separated form. Most developers do this by running a query to get much of the raw data, looping over the data and pushing it into a set, appending each new value to the appropriate key. Hopefully, it’s not a surprise that there’s a much better way to do this with PostgreSQL.

Postgres has a flexible and robust array datatype that comes with a variety of functions. Even without taking advantage of the array datatype in your application, you can still take advantage of some of the functions to get the functionality you need. Lets take a look at an example schema and use case.

An example

Given a project management application, you may have users who have projects that have tasks. An example piece of functionality might be to send an email with a list of all projects that have tasks that are past their due dates of completion. Your schema might look something like this:

 # \d users
Table "public.users"
Column | Type | Modifiers
------------+-----------------------------+-----------
id | integer | not null
email | character varying(255) |
...
# \d projects
Table "public.projects"
Column | Type | Modifiers
------------+-----------------------------+-----------
id | integer | not null
user_id | integer | not null
name | character varying(255) | not null
...
# \d tasks
Table "public.tasks"
Column | Type | Modifiers
--------------+-----------------------------+-----------
id | integer | not null
project_id | integer | not null
completed_at | timestamp without time zone |
due_at | timestamp without time zone |
...

To get a list of all projects that have tasks that haven’t been completed, you would start with something like:

SELECT
projects.name
FROM
projects,
tasks
WHERE projects.id = tasks.project_id
AND tasks.due_at > tasks.completed_at
AND tasks.due_at > now()

This would give you a list of projects which you could then easily join this with users:

SELECT
users.email
projects.name
FROM
projects,
tasks,
users
WHERE projects.id = tasks.project_id
AND tasks.due_at > tasks.completed_at
AND tasks.due_at > now()
AND users.id = projects.user_id

At this point you’ve got everything you need to pull this up into Ruby, Python, or other language of your choice and then build the full set. However if this is thousands or even hundreds of results you’ll be spending more time than necessary, grouping this data for a sensible email. With 3 other small changes you can have this already formatted for you to immediately send of in an email. The first is using a handy function called array_agg which will aggregate items and then you can format them how you wish. The second is just ensuring you’re grouping correctly. Finally you’ll want to unnest the array so it formats the data in a clean way for you.

Looking at it all put together:

SELECT
users.email,
array_to_string(array_agg(projects.name), ',')) as projects
FROM
projects,
tasks,
users
WHERE projects.id = tasks.project_id
AND tasks.due_at > tasks.completed_at
AND tasks.due_at > now()
AND users.id = projects.user_id
GROUP BY
users.email

This would give you a nice clean result of projects that have overdue tasks that you could then send to the user in an email:

 email | projects
---------------------------+-------------------
craig.kerstiens@gmail.com | blog, timetracker
craig@heroku.com | foo, bar, baz

Building a Search Result Extract Generator in PHP

Ben E. C. Boyter — Wed, 17 Apr 2013 02:23:56 GMT

During some contracting I was doing recently there was a requirement to implement some search logic using only PHP. There are no issues with that but it turns out I couldn’t find a decent extract generator handy as usually one would just plug into the search engines provided version to do this.

Off the top of my head I could only think of one example I was aware of which lives in Sphider (for the record it lives in searchfuncs.php from line 529 to 566). Sadly it has a few issues. Firstly the code is rather difficult to understand, and more importantly it usually has accuracy issues. A quick search turned up this link http://stackoverflow.com/questions/1436582/how-to-generate-excerpt-with-most-searched-words-in-php on StackOverflow. The second answer looked promising but its even more difficult to understand and a bit of profiling showed some performance issues will all of the regex going on in there.

Scaling Evangelism – Creating Advocates

CRAIG KERSTIENS — Tue, 16 Apr 2013 23:55:56 GMT

The first area I tend to think about when it comes to developer marketing is around advocates. A simple definition of an advocate is someone that speaks/writes in favor of some thing.

Creating advocates is a means of creating more of myself to go out and talk loudly and ideally effectively. However, they do take time to cultivate and it is a hard item to track. But by cultivating advocates I’m able to scale what would otherwise be a bottleneck of my own personal time.

Mentorship

The people in your group of 3 you’ll have nearly daily communications with, even if quick. These are the first people you think to work and collaborate with. It’s the case with all, but especially so here that there may be some strong mutual benefit. With both the groups of 3 and 12 you’ll be looking to:

Co-author content with
Collaborate on projects
Help amplify their content

Helping them Succeed

For a few years I’ve followed a similar process to this. Essentially creating advocates is done through first making them successful, then empowering to talk about their success. I’ve often done this by giving people a private channel to me – through IM, SMS, Phone, Skype, or even in person. Sure its easy enough to get my work email address, its a pretty easy one to guess, and even at that address I do aim to respond to every request. These channels in reality don’t make me more available; but they do allow a different type of communication to happen.

My personal approach has found it more out of place to ask how someones weekend was over email – IM or Skype make this easy. Broader questions of how someones actually doing in life generally are much easier to do over a coffee or a beer. This allows for a deeper relationship; at the same time I can ensure everyone I interact with is successful. It’s easier to ensure someones successful if they’re willing to talk to you, actually connecting with people enables this.

Communicate

Once someones been successful theres usually some value in their story. This doesn’t need to come in a marketing-ese approach though. Instead contribute back the value of what you did to the broader community with clear steps how they can get the same benefit. For many developers they feel talking about their own success is a bit of imposter syndrome. While this isn’t always the case; if it is you can help to work with someone to highlight the value in what they’ve done. This could be as simple as some encouragement to actually helping review drafts of blog posts.

Of course once someones started talking about what they were able to accomplish help them celebrate the success. Yelling from the rooftops only helps to further their efforts and give them the confidence to do it all over again.

Conclusion

To create advocates you don’t have to write a list of people you wish to shoot for, but realizing there are limiitations on your time can help you be more intentional. Knowing that where you spend your time and who you spend it with has a cost can allow you to be more explicit about ensuring you’re providing value to them, from there it should naturally run its course. Want to discuss this more? Email me at craig.kerstiens at gmail.com.

Doing Marketing (for developers) Differently

CRAIG KERSTIENS — Fri, 12 Apr 2013 23:55:56 GMT

As developers we can tend to be a fickle bunch; especially in certain open source communities. We like to see intelligence and systems applied to things, hope for a better world through making things open, and appreciate when others relate to our world as we often attempt to relate to theirs. Marketing is often a loaded word when it comes to developers – leaving mixed feelings about webinars, email campaigns, and the like.

A quick clarification in terms of marketing I’m referring to product marketing, but of a technical product. This is based on the idea that developers are the new kingmakers and when marketing a product to them it needs to be done differently.

While many of the above steps can add immense value, they often miss when it comes to developers. By changing these processes only slightly there can be much more efficiency in connecting with and actually delivering the value you hope for to developers.

Email Marketing

This is an area that there are more and more companies already starting to solve the right problems. Companies like intercom.io and customer.io are allowing for better tracking of what users are doing and notifying them at the right time instead of via massive impersonal campaigns. Developers build systems that take into account similar factores every day, you should be doing the same.

Emails that are sent at the wrong time or miss on what someone is trying to do can often do more damage towards building a rapport than good. Pushing content to solve a recent issue or problem at a relevant time should be table stakes for any email thats sent.

Community Engagement

Whether at conferences or in online collaborations developers are more social than so many give them credit for. We appreciate when things are done to improve the greater good particularly for developers, but often for the world as well. From supporting individual developers or projects financially such as through gittip or certain kickstarter projects all the way to employing developers to work full time on open source.

Being Better Than Webinars

After doing a webinar for a particular audience I had a Heroku co-worker come up to me. This individual has contributed many ideas that have become core functionality such as the Cedar stack and is generally open minded and listens very well. He came up to me and genuinely asked, “so what is a webinar?” I explained its like a live webcast with an opportunity for questions at the end. He was quiet for minute and responded, “Why can’t we just say that?”

There are some words that simply send off the wrong signals. Webinar rather clearly is one of those. As a community we’ve shown that we have a desire for this kind of content. The conference community has continued to grow steadily, and while they are many parts to every conference a pretty common one is the talk track itself. A talk at a conference is essentially a live in person webinar, of course the experience is often a bit richer.

Instead of the term webinar what is wrong with online office hours or webcasts with an opportunity for questions? Even doing the same thing with a different name is underdelivering on what we’re capable of.

I welcome hearing from others directly at craig.kerstiens@gmail.com on what they’re doing to engage with developers thats working. Not only working in the sense of eyeballs or dollars, but in adding value, in improving communities, in make the world better for developers. If there’s enough interest would be happy to post more detail around things I’ve seen work and what I hear works for others.

More holes than swiss cheese

Tom Forbes — Thu, 11 Apr 2013 23:24:30 GMT

tl;dr Never trust user input. Note: Before I published this I contacted the owner of the site and he has since replaced it with a new and more secure version When I browse the net I often see the hallmarks of a security issue with websites that I visit, and little alarm bells go off in my head. I wa...

Profession Talents for Ingenium

Ben Overmyer's Site — Sat, 06 Apr 2013 03:00:00 GMT

This post presents a prototype profession mechanic we’re working on for Ingenium 2nd Edition. We’re looking at giving each profession its own Profession Talent to add a little more variety to the game.

The Priest profession from first edition was pretty…. plain, and there was nothing priestly about her, other than role-playing ques. So with this new mechanic, she gets a Talent that reinforces her role as a messenger of the gods.

The priest profession from first edition was presented like this:

Priest

Starting Talent Pool: Mystic

Attributes: +1 Magical Aptitude

Speed: 3

Hit Points: 1d8

Advancement: Healing Arts

Starting Gold: 1d4 x 5

Priests are those divine focal points who channel their gods’ will into physical manifestations. Their magic is not based on arcane rules and laws; rather, they are blessed as Chosen and receive gifts of power for as long as they serve their gods.

The flavor text stands in Second Edition, but there are a number of changes. Here’s the new Priest block:

Starting Talent Pool: White Magic

Attribute Bonus: +1 Magical Aptitude

Hit Die: d6

Talent: Exorcism

Exorcism

The Priest has the divine gift and knowledge to pull a demon out of a possession victim. She makes a Willpower roll against the demon’s Challenge Level, and if the total is higher, the demon vacates the possessed body as a cloud of roiling black smoke. The victim then returns to his original stats, and if this causes any damage he has sustained to exceed his normal maximum Hit Points, he dies instantly. If his Willpower was at least half of the demon’s, he remembers everything he did while possessed. A failed exorcism increases the Challenge Level of the target demon by 1.

About Profession Talents

Every profession gets a unique Talent with this mechanic, and one that scales with the character’s power. She will never out-level her profession Talent.

Also, by adding “Profession Talents,” we add another option for expansion down the road.

Unlocking Hillsborough

Dan Palmer — Sat, 06 Apr 2013 02:00:00 GMT

I’m writing this on the train home from Rewired State’s latest event: National Hack the Government Day 2013 (event summary page). It was another great event with the same friendly atmosphere that goes along with so many (especially Rewired State’s) developer events. My friend Elliot and I won in one of the categories, and so this post is mostly about what we did, how we did it, and why we think it’s important.

Why I Blog

CRAIG KERSTIENS — Sun, 31 Mar 2013 23:55:56 GMT

I blog because I’m lazy. There’s more too it though: In any given day I may explain something to someone, the first time I do this I make a bit of a mental note. The second time I do this, especially within a short time frame I make a physical note of this in the form of the title of a blog post. Once I’m already to a second time of doing this its almost inevitable I’ll continue repeating myself – and its valuable to others.

Becoming replaceable

My goal in doing this is actually to become heavily replaceable. Keeping information locked away means I’m the only one capable of doing it. Making myself highly replaceable means I can continue to work on new and interesting things.

Lots can be shared

I don’t recall exactly where I read it, I believe it may have been James Governor, but any email that doesn’t contain proprietary info or trade secrets should be blogged. I fully expect the sentiment around this is something to the effect that anything that contains good ideas/learning/processes can also be helpful to others.

Distilling information

For every time I explain something I often linger about one piece too long, hit something that doesn’t need to be discussed, or miss something entirely. Putting an idea or guide together in written form you re-read it, which is hard to do when simply verbally explaining. If you’ve video taped yourself before for public speaking practice you know this is an awkward but valuable experience. Its much easier to practice as an exercise in writing.

Economies of scale

While in some ways I am absolutely lazy when I blog I get economies of scale I couldn’t otherwise have. On a given day on a high day I may have 15 one on one conversations. Every post has the opportunity to become a one on one conversation.

One on one conversation is my preferred method instead of in a large group. In a related area I’ve been having an ongoing conversation wth Ryan Daigle on whether blog posts should have comments. I opt for having you reach out to me directly via email, he believes comments should be available. Putting this in words crystalizes that to me its just relative to how you prefer to interact.

Summary

As with any other post, this is one conversation I’ve now had a few times over. Hopefully this gives some basis to why I feel others should contribute content as well. Whether you want make your self dispensible, refine your thoughts, or reach new economies of scale I believe its a worthwhile exercise for many. If you have other reasons you feel its valuable as always please reach out.

Being Defiant: Finishing Out Rift's Free Tier

Ben Overmyer's Site — Sun, 31 Mar 2013 02:00:00 GMT

Today is the last day of March and the last day I’ll be writing about the MMORPG Rift as it applies to tabletop RPGs. If you’ve missed any of these posts, you might want to check them out. Go ahead, I’ll wait.

This post is a retrospective and has little to do with tabletop RPGs. If you’re just looking for new ways to experience your favorite RPG, come back next week – I’ve got something very interesting planned!

Generally, it takes me about one month to get bored of an MMORPG. I’ll steep myself in the lore, powerlevel up to about 75-90% of endgame, get involved in crafting, and then… burn out. The novelty factor wears off, and I stop playing.

Most of the time, it’s not a conscious decision to stop playing. I’ll just go a day without playing the game, since I find something else I want to do more that one day. Then two days go by. Then three. Before I realize it, it’s been over a week since I last logged in, and I have no intention of going back.

A rare handful of MMORPGs keep my interest for longer than that. The list so far has been Star Wars: Galaxies and World of Warcraft. Now, I’m adding Rift to the list.

I originally reinstalled the game because it was going to be free and I’d heard good things about Dimensions. So, I did, and started this blog series to tie the game to tabletop gaming. It just clicked. The novelty factor has worn off, and I’m still playing. That bodes well for its longevity, and I’m going to resubscribe.

The Dimensions are awesome. The fishing is weirdly addictive. And playing an MMO with a great and friendly community is refreshing.

If you’re interested in trying it out, jump on the Faeblight server and look me up. I’ll be playing either my Eth cleric Amun or my Kelari mage Sidious.

Being Defiant: Currencies Beyond Gold and Silver

Ben Overmyer's Site — Sun, 24 Mar 2013 02:00:00 GMT

In Rift, there are five ways to acquire new armor and weapons. You can take it from the corpses of monsters you kill. You can run through dungeons, collecting new pieces from the tougher monsters and bosses. You can run from rift to rift, defeating the planar invaders and getting new gear from the rift rewards. You can craft it or purchase it from other players who craft gear. You can also use special currencies earned through various means to purchase new gear from special NPC merchants.

In existing tabletop RPGs, there are really only three ways to get gear – buying it from merchants, looting bodies, and finding it lying around in dungeons.

Many MMORPG loot mechanics won’t work in a tabletop RPG because the gear just appears in the player’s inventory when certain conditions are met. In a tabletop game, gear rarely just appears. I’ve touched on crafting in tabletop RPGs before, so this time around I’ll focus on special currencies.

Rift’s Currency for Planar Invasions

In games like Rift and World of Warcraft, you receive units of these currencies for achieving feats of excellence. For example, in Rift, you earn “planarite” and “sourcestone” for defeating planar invaders. Sourcestone is much rarer than planarite, each unit of its most common version being worth 50 planarite. It’s only awarded for helping defeat special rift events like full invasions.

While there’s only one version of planarite, there are multiple types of sourcestone, each being dropped by progressively more difficult planar invaders. In order to get the higher sourcestones, you must defeat higher-level invaders.

All of these can be traded in for impressive, rare gear. The best items cost a combination of sourcestone and planarite.

How to Deal with Special Currencies in Tabletop Games

In tabletop RPGs, by comparison, you’re looking at a single currency – gold, credits, silver, whatever. In some games there are multiple types of currency but only one “track.” For example, in Dungeons & Dragons, you have copper, silver, electrum, gold, and platinum, but they’re all just higher denominations of each other. This currency is accepted everywhere from all merchants.

To make sense, a unique currency needs to be widespread in use and creation, and to make sense in a setting, the powers-that-be need to allow that currency to exist alongside the official currency of the realm. That last part is historically difficult to come by, but in fictional worlds, we can gloss over that fact.

The best reason to include a currency like this is to gate powerful gear behind difficult tasks. It also provides a way to uniquely theme gear and work in factional alliances and reputation.

You could make up a large order dedicated to service to a particular god, for example. Instead of just being all priests or all paladins, this order could include a variety of skilled craftsmen who produce special gear. The craftsmen may only part with the gear in exchange for a special currency that only the order uses. Higher-ranking members of this order may be giving the PCs quests, and in gratitude for their help, rewards them with some of the order’s currency.

This also offers role-playing opportunities, oddly enough. Possession of particular currencies could trigger interesting reactions from NPCs. Maybe they’re associated with the producer of the currency. Maybe they’re afraid of the producer. Maybe it’s more complicated than that.

If it makes sense for your setting, try adding a special currency and unique rewards for that currency to your next adventure. It just might take your campaign in a very interesting direction.

Bird Watching

Beetle Space — Fri, 15 Mar 2013 09:00:00 GMT

A friend of mine recently took his bird watching hobby to another level. Armed with a 400mm lens, a scope and a sturdy tripod, he and his wife scour the local area for birds. The absence of leaves makes it easier to find them. [1]

I met up with them …

Prioritizing and Planning within Heroku Postgres

CRAIG KERSTIENS — Wed, 13 Mar 2013 22:55:56 GMT

Over a year ago I blogged about Heroku’s approach to Teams and Tools. Since that time Heroku has grown from around 25 people to over 100, we’ve continued to iterate and find new tools that work for how we do things. For many of the product management and software engineering books I’ve read I’ve yet to find something that helps a team priorize in a fashion I that feels right.

One process emerged nearly a year ago from within the Heroku Postgres team and is now followed by many others. Within a team this process is now commonly conducted each 6 months. Lets take a look at how this process looks

It Starts with Ideas

Hopefully having ideas of things to work on isn’t a problem, if it is just go spend some time with customers – listen to their problems, see how they use the product, then come back and write down the ideas. For most teams this is simply an excercise of thinking back and writing it down. Some teams at Heroku have resorted to keeping running backlogs of things they’d like to do this. We do this by keeping a Trello board which columns for:

New ideas
Ponies
Stallions

Ponies and Stallions are things that would be great to do, however a sizeable amount of work must be done on them and we’re not currently tackling them. Ponies are less sizeable and likely to get done not in coming weeks but perhaps in coming months up to a year. Stallions are great but large effort and may or may not get done but in the category of things we would like to be able to do.

Once you’ve got your ideas whether in your head on a backlog we begin by writing them out typically on sticky notes or index cards.

Laying it all out

From here we create a simple grid:

The grid has two axis. One is for impact the other for difficulty. At this point we aim to lay out every idea that we’ve already written down into a quadrant. Commonly this is done at team offsites where the team is free of distractions and able to devote appropriate time to it. Being able to accomplish this in one sitting with the team is important to having cohesion around the result.

A plan

At this point hopefully its quite obvious what you want to tackle. If there’s anything in the top right it should be an easy win for something for you to focus on. From we often transcribe this into a powerpoint/keynote document and highlight things that we will definitely aim to accomplish in the next 6 months as well as things we’re intentionally not working on. This leaves us with an artifact of both things we will work on and explicit things we wont work on.

What works for us

In general we try to have more work than we can tackle to ensure we’re constrained in a good form and not wasting idle time of people. Ensuring we’re selective about the things we’re working on and that we’re working on the right things works for us. We’ve found this simple exercise valuable for many teams to plan and ensure we’re working on those right things. Of course this may not work for everyone but for our goals and culture aligns well for us.

If you’ve got simple but unique techniques that work for your team as always would love to hear about them – craig.kerstiens@gmail.com

Just how slow are Django templates?

Tom Forbes — Wed, 13 Mar 2013 20:17:38 GMT

Edit 2: I made a Django debug toolbar panel that profiles your Django templates and all their components. You can find it here: https://pypi.python.org/pypi/django-debug-toolbar-template-timings Edit: It appears that Django does have some form of template caching. The graphs have been updated to inc...

Fixing Database Connections in Django

CRAIG KERSTIENS — Thu, 07 Mar 2013 22:55:56 GMT

If you’re looking to get better performance from your Django apps you can check out Pro Django, PostgreSQL High Performance, or read some my earlier posts on Postgres Performance. All of these are of course good things to do – you can also start by correcting an incredibly common but also painful performance issue, that until 1.6 is unaddressed in Django.

Django’s current default behavior is to establish a connection for each request within a Django application. In many cases any particularly in distributed cloud environments this is a large time sink of your response time. An example application running on Heroku shows a typical connection time of 70ms. A large part of this time is the SSL negotiation that occurs in connecting to your database, which is a good practice to ensure security of your data. Regardless, this is a long time in simply establishing a connection. As a point of comparisson its commonly encourage that most queries to your database are under 10ms.

An example that highlights this in a small lightweight application shows the bulk of a request time being within a connection displayed by New Relic:

One option to remedy this is by running a connection pooler on your Database side such as Pgpool or PgBouncer. In fact Ask the Pony already highlighted these potential gains. While running an external DB they’re essentially testing the benefits of conncetion pooling. This is an obvious gain and can be in a much more lightweight format.

Connection Pooling in Django

As Django establishes a connection on each request it has an opportunity to both pool connections and persist connections. There are two major options for pooling, each works quite well with Django and provides some dramatic improvements. While the first request may take the 70ms of connection time, subsequent requests show absolutely no connection time since the connection already exists. This is highlighed by these two comparissons of before and after in actually the times it grabs a connection:

Clearly theres plenty of value to having a persistent connection or a pool within Django itself. As of today theres a few options for that:

Django-PostgresPool

The first Django-PostgresPool is created by kennethreitz. As in general I’d encourage the use of dj_database_url you can easily begin using his package (once installed) with:

import dj_database_url
DATABASE = { 'default': dj_database_url.config() }
DATABASES['default']['ENGINE'] = 'django_postgrespool'

An important thing to note is if you’re using South you’ll also want to setup the adapter for it:

SOUTH_DATABASE_ADAPTERS = {
'default': 'south.db.postgresql_psycopg2'
}

djorm-ext-pool

The second option djorm-ext-pool is created by niwibe. Once you’ve installed djorm-ext-pool you then add it to your INSTALLED_APPS within your settings.py. From here then you can setup your pool:

DJORM_POOL_OPTIONS = {
"pool_size": 20,
"max_overflow": 0
}

django-db-pool

The third and final option is django-db-pool. You can set it up with:

DATABASES = {'default': dj_database_url.config()}
DATABASES['default']['ENGINE'] = 'dbpool.db.backends.postgresql_psycopg2'
DATABASES['default']['OPTIONS'] = {
'MAX_CONNS': 10
}

Gotchas

Each of these does work with recent versions of Django, though in some cases there are gotchas. If using a prodution worthy python web server such as Gunicorn or uwsgi and running with gevent or eventlet some edge cases can present themselves. Regardless of potential gotchas it is worth attempting this and of course providing feedback to maintainers and the community as you find those.

The future

Django more recently has directly started to address these issues of large costs of establishing a connection. The first major step here is this patch from Aymeric. You can find more dicussion around this particular patch here. Essentially with this patch which will hit in Django 1.6 developers then get a persistent connection which will help reduce the time. If you’re interested in trying the 1.6 master you can do this by adding it to your requirements.txt as:

https://github.com/django/django/archive/master.zip

At this point it does not introduce pooling which could allow even more gains, though I’m sure if there’s enough need it’ll be on a roadmap at some point. Though, as it stands today before 1.6 your best bet is one of the above options.

Simple database read scaling without sharding in rails

CRAIG KERSTIENS — Wed, 06 Mar 2013 22:55:56 GMT

In an earlier post I provided a high level overview of sharding. Sharding while a very solid approach to scaling capacity versus simply only relying on vertical scaling can also be a time intensive one. Additionally in some cases certain sites may only need extra capacity for a short lived period of time. Fortunately theres a nice middle ground alternative for scaling capacity that works well in quite a few cases. It even has a benefit that can potentially in place of sharding.

This method results in scaling your reads to replica databases, you can do this on Heroku by taking advantage of followers. A follower is a read only database on Heroku Postgres that receives asynchronous updates of your data usually only lagging a very few commits behind. This means you can write all of your data to the leader (main) database, and then read from another.

While you can arbitrarily do this there’s some major benefits to doing it based on the models. This is because Postgres maintains a cache on each instance its running on. Though you may have the same dataset, Postgres maintains frequently accessed or queried data in the cache giving you better performance. For more on this you can read earlier posts on PostgreSQL Performance.

Setting it up with Rails

With a follower database created you can begin adding support for this to your application. The first thing is to add the gem to your Gemfile:

gem 'ar-octopus', :require => "octopus"

Then of course to install it with bundle install. Now we can actually begin to add the code needed to have specific models access the follower.

octopus:
shards:
shard_sqlite:
adapter: sqlite3
database: db/db_one.sqlite3
pool: 5
timeout: 5000
shard_pgsql:
adapter: postgresql
username: postgres
password:
database: db_two
encoding: unicode

class Project < ActiveRecord::Base
octopus_establish_connection(:adapter => "sqlite3", :database => "db_one")
end

Finding fried chicken with C-Sharp, Mono and ServiceStack

Tom Forbes — Sat, 02 Mar 2013 23:57:06 GMT

I think I just solved a very first world problem - Where to find the nearest takeaway that sells fried chicken. When you are stumbling out of Spiders at 1am in the morning and craving some SFC goodness the last thing you want to do is wander the backstreets of hull in the cold looking for an open ta...

Why Code Search is Difficult

Ben E. C. Boyter — Fri, 01 Mar 2013 01:26:36 GMT

I was chatting with a colleague the other day and he was asking me why code search is a difficult problem. After all its not quite as dynamic as the web so it should be easier to index.

In truth its a mixed bag. Some things like like crawling are easy, others such as indexing are much harder then you would think. I thought I would write down why this is.

Getting more out of psql (The PostgreSQL CLI)

CRAIG KERSTIENS — Thu, 21 Feb 2013 22:55:56 GMT

After my last post I had a variety of readers reach out about many different tweaks they’d made to their workflows using with psql. One people Grégoire Hubert had a wondeful extensive list of items. Grégoire has been a freelance in web development and he has worked with Postgresql for some time now in addition to being the author of Pomm. Without further ado heres what he has to say on how he uses psql:

Get the most of psql

Psql, the CLI postgreSQL client, is a powerful tool. Sadly, lot of developers are not aware of the features and instead look for a GUI to provide what they need. Let’s fly over what can psql do for you.

Feel yourself at home

One of the most common misconception people have about CLI is «They are a poor user interface». C’mon, the CLI is the most efficient user interface ever. There is nothing to disturb you from what you are doing and you are by far fastest without switching to your mouse all the time. Let’s see how we can configure psql at our convenience.

First, you’ll have managed to choose a nice and fancy terminal font like monofur or inconsolata. Do not underestimate the power of the font

The nice line style shown above can be set with \pset linestyle unicode and \pset border 2. This is just an example of the many environment variables you can play with to get your preferred style of working out of psql.

For example, I found the character ¤ the most accurate to express nullity (instead of default NULL). Let’s just \pset null ¤ and here it is:

SELECT * FROM very_interesting_stat;
┌──────┬──────┬──────┬──────┬──────┐
│ a │ b │ c │ d │ e │
├──────┼──────┼──────┼──────┼──────┤
│ 9.06 │ ¤ │ ¤ │ ¤ │ ¤ │
│ 7.30 │ 3.55 │ 7.57 │ 3.31 │ ¤ │
│ 7.20 │ 5.08 │ ¤ │ 6.58 │ 5.90 │
...

Another hugely value to get environment variables is colors in the prompt. Colors in the prompt are important because it makes easier to spot where output starts and ends between two interactions at the console. The PROMPT1 environment variable will even let you set an indicator to notify you are inside a transaction or not, give this a try for a sweet surprise…

\set PROMPT1 '%[%033[33;1m%]%x%[%033[0m%]%[%033[1m%]%/%[%033[0m%]%R%# '

I also like to disable the pager by default \pset pager off and display the time every issued query takes \timing. If you are used to psql, you may notice in the picture above, some content is wrapped. This is \pset format wrapped option.

Of course, writing all that on every connection would be a pain, so just write them in a ~/.psqlrc file, it will be sourced every time psql is launched.

If you are familiar with bash or other recent unix shells, you might also declare aliases in your configuration file. You can do the same with psql. For example if you want to have a query for slow queries such as from this earlier post but not have to remember the query every time you can set it up as:

\set show_slow_queries
'SELECT
(total_time / 1000 / 60) as total_minutes,
(total_time/calls) as average_time, query
FROM pg_stat_statements
ORDER BY 1 DESC
LIMIT 100;'

Now, just entering :show_slow_queries in your psql client will launch this query and give you the results:

 total_time | avg_time | query
------------------+------------------+------------------------------------------------------------
295.761165833319 | 10.1374053278061 | SELECT id FROM users WHERE email LIKE ?
219.138564283326 | 80.24530822355305 | SELECT * FROM address WHERE user_id = ? AND current = True

Psql at your fingertips

Now you have got a fancy prompt, here is the real question you ask, what can psql do for me ? and \? has all of the answers. It has built-in queries to describe almost all database objects from tables to operators, indexes, triggers etc… with clever auto-completion. Not only completion on tables and columns – but also on aliases (sweet), SQL commands (w00t) and database objects.

Now we can enter some SQL commands. As usual, you need to check in the documentation how the heck to write this damn ALTER TABLE. Relax, psql proposes inline documentation. Just enter \h alter table (auto complete w00t) and you ll be ok.

Interacting with your editor

psql provides two very handy commands: \e and \i. This last command sources a sql file in the client’s current session. \e edits the last command using the editor defined in the EDITOR shell environment variables (aka vim). This grant you with real editor feature when it comes to writing long queries. What psql does, it saves the buffer in a temporary file and fires up the editor with that file. Once the editor is terminated, psql sources the file. Of course, you can use your editor to save queries in other places where they would be under version control, but the \e has a serious limitation: it spawns only the last query. Even if you sent several queries on the same line. (Note that \r clears psql’s last query buffer).

Note: \ef my_function opens stored function source code (With auto completion, I know, it’s awesome).

Vim users can here benefit from Vim’s server mode. If you launch a vim specifying a server name (let’ say “PSQL”) somewhere, and set the EDITOR variable as is export EDITOR="vim --servername PSQL --remote-tab-wait then psql will open a new tab on the running vim with the last query and run it as soon as you close this tab. Tmux or gnu/screen users will split their screen to have Vim and psql running on the same terminal window.

Call a friend

Vim power users know it is possible to pipe a buffer (or selection) directly in a program that can be … psql (Using the :w !psql syntax). Even from the shell, you might want to take advantage of the fantastic \copy feature that loads formated file in the database (I use it to load apache logs). But always having to specify connection parameters are a hassle. Let’s use shell environment instead. Psql is sensitive to the following variables:

PGDATABASE
PGHOST
PGPORT
PGUSER
PGCLUSTER (debian wrapper).

Set them once for all in you shell environment and call psql to connect to the database. In case you want to skip password prompt, you can store your pass in a 600 mode access file named .pgpass in your home (do not do that on shared or exposed computers). Although this is nice for development database servers, I do NOT recommend this for production servers since it should not be easy to mess with them.

Resource for additional information is … the man page and Postgres Docs. All PostgreSQL documentation is an example of what software reference documentation should be. Enjoy!

HtmlToWord

Tom Forbes — Mon, 18 Feb 2013 07:38:08 GMT

You can find the code here on github and the package here on PyPi I have written and continue to maintain a reporting system for a group of pentesters. During/after the tests the results and details are inputted into a web application using a WYSIWYG editor called Redactor (which is pretty awesome!)...

How I work with Postgres – psql, My PostgreSQL Admin

CRAIG KERSTIENS — Wed, 13 Feb 2013 22:55:56 GMT

On at least a weekly basis and not uncommonly multiple times in a single week I get this question:

@neilmiddleton I’ve been hunting for a nice PG interface that works within other things. PGAdmin kinda works, except the SQL editor is a piece of shit

Sometimes it leans more to, what is the Sequel Pro equivilant for Postgres. My default answer is I just use psql, though I do have to then go on to explain how I use it. For those just interested you can read more below or just get the highlights here:

Set your default EDITOR then use \e
On postgres 9.2 and up \x auto is your friend
Set history to unlimited
\d all the things

Before going into detail on why psql works perfectly fine as an interface I want to rant for a minute about what the problems with current editors are and where I expect them to go in the future. First this is not a knock on the work thats been done on previous ones, for their time PgAdmin, phpPgAdmin, and others were valuable tools, but we’re coming to a point where theres a broader set of users of databases than ever before and empowering them is becoming ever more important.

Empowering developers, DBA’s, product people, marketers and others to be comfortable with their database will lead to more people taking advantage of whats in their data. pg_stat_statements was a great start to this laying a great foundation for valuable information being captured. Even with all of the powerful stats being captured in the statistics of PostgreSQL so many are still terrified when they see something like:

 QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Hash Join (cost=4.25..8.62 rows=100 width=107) (actual time=0.126..0.230 rows=100 loops=1)
Hash Cond: (purchases.user_id = users.id)
-> Seq Scan on purchases (cost=0.00..3.00 rows=100 width=84) (actual time=0.012..0.035 rows=100 loops=1)
-> Hash (cost=3.00..3.00 rows=100 width=27) (actual time=0.097..0.097 rows=100 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 6kB
-> Seq Scan on users (cost=0.00..3.00 rows=100 width=27) (actual time=0.007..0.042 rows=100 loops=1)
Total runtime: 0.799 ms
(7 rows)

Empowering more developers by surfacing this information in a digestable form, such as building on top of pg_stat_statements tools such as datascope by @leinweber and getting this to be part of the default admin we will truly begin empowering a new set of user.

But enough of a detour, those tools aren’t available today. If you’re interested in helping build those to make the community better please reach out. For now I live in a work where I’m quite content with simple ole psql here’s how:

Editor

Ensuring you’ve exported your preferred editor to the environment variable EDITOR when you run \e it will allow you to view and edit your last run query in your editor of choice. This works for vim, emacs, or even sublime text.

export EDITOR=subl
psql
\e

Gives me:

Note you need to make sure you connect with psql and have your editor set, once you do that saving and exiting the file will then execute the query

\x auto

psql has long had a method of formatting output. You can toggle this on and off easily by just running the \x command. Running a basic query you get the output:

SELECT *
FROM users
LIMIt 1;
id | first_name | last_name | email | data | created_at | updated_at | last_login
----+------------+-----------+----------------------------+------------+---------------------+---------------------+---------------------
1 | Rosemary | Wassink | Rosemary.Wassink@yahoo.com | "sex"=>"F" | 2010-07-01 18:16:00 | 2011-05-14 11:47:00 | 2011-06-07 23:04:00

With toggling the output and re-running the same query we can see how its now formatted:

\x
Expanded display is on.
craig=# SELECT * from users limit 1;
-[ RECORD 1 ]--------------------------
id | 1
first_name | Rosemary
last_name | Wassink
email | Rosemary.Wassink@yahoo.com
data | "sex"=>"F"
created_at | 2010-07-01 18:16:00
updated_at | 2011-05-14 11:47:00
last_login | 2011-06-07 23:04:00

Using \x auto will automatically put this in what Postgres believes is the most intelligible format to read it in.

psql history

Hopefully this needs no justification… having an unlimited history of all your queries is incredibly handy. Ensuring you set the following environment variables will ensure you never lose that query you ran several months ago again:

export HISTFILESIZE=
export HISTSIZE=

\d

And while the last on the list one of the first things I do when connecting to any database is check out whats in it. I don’t do this by running a bunch of queries but rather checking out the schema and then poking at definitions of specific tables. \d and variations on it are incredibly handy for this. Here’s a few highlights below:

Listing all relations with simply \d:

\d
List of relations
Schema | Name | Type | Owner
--------+------------------+---------------+-------
public | products | table | craig
public | products_id_seq | sequence | craig
public | purchases | table | craig
public | purchases_id_seq | sequence | craig
public | redis_db0 | foreign table | craig
public | users | table | craig
public | users_id_seq | sequence | craig
(7 rows)

List only all tables with dt:

\dt
List of relations
Schema | Name | Type | Owner
--------+-----------+-------+-------
public | products | table | craig
public | purchases | table | craig
public | users | table | craig
(3 rows)

Describe a specific relation with \d RELATIONNAMEHERE:

\d users
Table "public.users"
Column | Type | Modifiers
------------+-----------------------------+----------------------------------------------------
id | integer | not null default nextval('users_id_seq'::regclass)
first_name | character varying(50) |
last_name | character varying(50) |
email | character varying(255) |
data | hstore |
created_at | timestamp without time zone |
updated_at | timestamp without time zone |
last_login | timestamp without time zone |

One more pro-tip if you’re running a transaction with many tables and forget which are involved in it you can run ‘\d *transaction*’ and it’ll display tables curently affected.

Have a tool you prefer, have something you use daily in psql that I missed, or interested in helping create a new admin experience please reach out and lets talk craig.kerstiens at gmail.com

World of Warcraft, Modern RPGs, and the Gear Grind

Ben Overmyer's Site — Sun, 10 Feb 2013 02:00:00 GMT

After a full year of not playing, yesterday I jumped back into World of Warcraft. I’d heard enough good things about the latest changes that I decided to give it another shot. It remains to be seen whether that’ll last or not. Something that did make me pause, though, is the obvious reminder that WoW is based on a gear grind. As soon as my character, a worgen warlock named Exsecratus, made it to the new continent, I almost immediately replaced my raid epic gear from the Cataclysm days with greens. Today’s junk is better than yesterday’s hard-won treasure. That theme carries across many modern MMORPGs, but in all honesty, it’s also present in tabletop RPGs these days. Dungeons & Dragons is one of the worst culprits, but there are plenty of others. That got me to thinking. Is the gear grind necessarily bad? The perpetual need to have better equipment may seem like a thin excuse to keep people playing a game, but taken as a component of a larger and more complex system, it’s a reliable foundation to build on. In a game like my own Ingenium, where if you have too many static modifiers it makes the randomization provided by the dice useless, a gear grind based on incremental bonus increases wouldn’t make much sense. However, if there were a game out there that had more complex calculation involved with point bonuses, then having a Sword +513 wouldn’t necessarily be any more unbalanced than a Sword +3. The trick is that to get to that level, you need to involve multiplication and division in the core mechanic, and that’s the real holdup for a tabletop RPG. Let’s face it - most tabletop gamers don’t want to sit and recalculate their stats every time they pick up a new item. In video games it’s possible because the game itself handles it automatically in the background without the player even being aware of the math. To introduce infinitely-scaling item stats into a tabletop RPG, you would have to do one of two things - either create a phone/tablet app to handle the math for you, or make the math so simplified that it would be possible to calculate quickly. Being a web developer by profession, I’m partial to the first one. These days it’s not unreasonable to expect gamers to have a smartphone. How would you handle bringing the gear grind into a tabletop game?

Introducing django-db-tools

CRAIG KERSTIENS — Fri, 08 Feb 2013 22:55:56 GMT

For any successful web application there is likely to come a time when you need to conduct some large migration on the backend. I dont mean simple add a column here or add an index there, but rather truly sizeable migrations… Going from MySQL to Postgres or migrating from an older version of Postgres such as a 32 bit instance to a newer 64 bit instance. In these cases the default approach is to just schedule downtime often throwing up a splash screen saying so.

For many sites this approach is simply wrong and lazy, with little effort you can improve the experience and there by ease the burden in conducting these types of migrations. By having the ability to turn your site into a read only mode which Simon Wilson talked about in his post on Lanyrd you can still continue to operate just in a limited capacity. Andrew Godwin further talks about some of this as well in regards to the Lanyrd move and even includes the script they used to migrate data from MySQL to Postgres. Though just in talking with Simon about this a week ago it occurred to me they had not released the code for their read-only mode.

Finally onto the announcing, today I’m releasing django-db-tools. This is currently a very lightweight utility that allows you to flip your site into two modes.

Anonymous Mode

For sites that offer a bulk of their data to unauthenticated users anonymous mode will be what you want. This ensures all users appear logged out and thus cannot interact with data. To enable anonymous mode you’d simple set the environment variable or config var on heroku as follows:

READ_ONLY_MODE = True

Restricting POSTs

The other bucket of sites is one that allows users to stay logged in but not insert data. Django did not appear to have a convenient means to know whether data was actually being inserted into the DB or not. As a good practice when inserting data it should be receiving a HTTP POST. The GET_ONLY_MODE mimmicks all POSTs as if they were sent via GETs thus hopefully eliminating inserting data into your application. To turn it on simply set the environment variabel or config var on heroku to:

GET_ONLY_MODE = True

Installing

The tool itself is largely middleware, to install:

Run pip install django-db-tools or add it to your requirements.txt
Add db_tools to your INSTALLED_APPS in your settings.py
Add 'dbtools.middleware.ReadOnlyMiddleware', to your MIDDLEWARE_CLASSES in settings.py

Contributing

As with all code this is largely a work in progress. There’s many items still to do such as providing default copy and error pages and potentially handling other edge cases. I’d welcome others to contribute and give feedback if they find it helpful or how it can be improved on Github.

Want to write a search engine? Have some links

Ben E. C. Boyter — Thu, 31 Jan 2013 01:29:36 GMT

A recent comment I left on Hacker News managed to get quite a lot of up-votes which surprised me since it was effectively just a collection of links about search engines. You can read the full thread at http://news.ycombinator.com/item?id=5129530

Anyway since it did do so well I thought I would flesh it out with some more information. Here are a collection of posts/blogs/discussions which go into the details of how to write a search engine.

Increasing the Encryption Noise Floor

Pete Keen — Sun, 27 Jan 2013 20:06:32 GMT

Inspired by Tim Bray's recent post about encrypting his website, I decided to enable and force HTTPS for bugsplat.info. The process was straightforward and, turns out, completely free. Read on to find out how and why.

Why?

Because I think the whole web should be encrypted and I figured I should practice what I preach. Bugsplat is primarily static html and is completely public but that doesn't mean I can't increase the encryption noise floor, so to speak. If only the websites with secret data are encrypted then just using them is suspicious. If every site is encrypted then there's nothing to be suspicious about.

How?

Bugsplat is deployed to my RamNode VPS (notice: affiliate link) with Capistrano::Buildpack, a Capistrano add-on that allows you to deploy applications using Heroku-style buildpacks. Recently I added support for simply configuring HTTPS with a few options in your Capfile. Here's the relevant config from bugsplat.info's Capfile:

set :use_ssl, true
set :force_ssl, true
set :ssl_cert_path, '/etc/nginx/certs/bugsplat.info.crt'
set :ssl_key_path, '/etc/nginx/certs/bugsplat.info.key'

:use_ssl enables listening on port 443 with SSL and the two path options just tell nginx where to find the keys on the server, which are deployed separately with Puppet. :force_ssl adds this small snippet to the exported nginx config file which redirects plain requests to SSL:

if ($ssl_protocol = "") {
   rewrite ^https://$server_name$request_uri? permanent;
}

As for the certificate, I ended up going with a free certificate from StartSSL. This certificate doesn't necessarily guarantee that I am who I say I am because I just had to validate an email address, but it does guarantee that the connection is encrypted which is really all I care about. At some point I plan on going through the verification steps needed to get Class 2 certificates from StartSSL, but that's for another day.

Crafting in Tabletop RPGs

Ben Overmyer's Site — Sun, 13 Jan 2013 02:00:00 GMT

One of the systems near and dear to my heart in MMORPGs is crafting. I love being able to make things from materials that I’ve collected myself, and in so doing improve my own capabilities without having to rely on an external source (e.g., loot, or vendors). Very few tabletop RPGs have ever attempted to include a crafting system, and those that have (Dungeons & Dragons, I’m looking at you) focus primarily on high-cost, high-powered magical items. Here, I explain how a crafting system for a tabletop RPG might work, taking several cues from MMORPGs. Crafting in MMORPGs has three primary component systems - gathering, refining, and building finished items. Gathering is the collection of raw resources from the world. This usually takes the form of withdrawing units of resources from a resource node that randomly spawns in the world. Refining takes those raw materials and works them into usable forms. For example, smelting iron ore into iron bars is one such process. This step is sometimes optional. Building finished items takes resources and puts them together into a final usable form. In MMORPGs, the process of actually crafting the item is abstracted away such that the player doesn’t need to worry about things like quenching steel in the proper temperature of liquid. Tabletop RPGs generally can’t follow MMORPG mechanics, since a lot of the latter is based on intensive calculation behind the scenes. With crafting, my perception is that it’s not popular because it’s not a story component - in a tabletop gaming session, anything that smacks of busywork is not going to go over well. To address that concern, there are two possibilities:

Make the whole crafting process fun and exciting, or
Make the crafting process something that happens between adventures.

Personally, I think a mix of the two is ideal. The gathering process should be divided into two parts - acquiring rare raw materials through normal adventuring, and acquiring mundane raw materials in a quick process that occurs between gaming sessions. The refining process should be purely between sessions, unless it’s something story-bound like imbuing a raw material with a specific magical aura that can only be done by a powerful hermit-mage that lives in the…. you get the picture. The crafting of the final item could go either way, though I’d tend to put it at the very beginning of a game session before the actual adventure starts to “prime” players’ minds. A big component of crafting, and one of its greatest variables, is time. In real life, it takes a long time to smith a sword. In World of Warcraft, it takes seconds. Crafting in tabletop RPGs, as a character activity, should take little time. Crafting is a secondary system, and it’s not epic to say that your character spent a month of in-game down time smithing a new longsword in a borrowed forge while the other members of the party twiddled their thumbs. This problem can be explained away in two ways:

Give the character NPC assistants to do the crafting for him, or
Make the crafting process itself something magical that takes little time.

I prefer the first option, since it involves more story hooks and feels less like a deus ex machina. If a player discovers he has the materials to make an upgraded suit of armor just as a session is about to begin, you can allow him to do that and retcon that he contracted the work out while he was adventuring in the last session. This can tend to make retconning a standard, accepted practice for crafting, but I think that’s of minor concern in this case. The other consideration when designing a crafting system for use in a tabletop RPG is the effect this can have on player equipment and expectations of loot. As such, it can’t be designed in a vacuum - it needs to be part of the system from the beginning, or you risk turning a game into a Monty Haul campaign inadvertently. The secret here is in the timing - you can gate the frequency of crafting more powerful items by introducing a crafting skill advancement system and making it as restrictive as you like, thereby keeping crafted items in line with looted items (or not, if that’s what you prefer). The Dungeons & Dragons system of spending experience to craft powerful items is one way to gate the acquisition of crafted items, but I think it’s fundamentally flawed in that it makes little sense from a story point of view. Sauron didn’t lose part of his essence to craft the One Ring - he just lost time and resources. To make something, a player has three resources available to him - Time, Money, and Effort. The ratio can vary between those three, but if you abstract out those concepts, you can arrive at a more sensible requirement ladder than Dungeons & Dragons’. In the end, crafting is a perfectly viable addition to a gaming system, as long as it’s well-integrated with the rest of the game. I’m working on one such system for inclusion in a game of mine, and I like to think the players will love it.

Code a Search Engine in PHP Part 5

Ben E. C. Boyter — Thu, 10 Jan 2013 07:34:18 GMT

[Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.][1]

This is part 5 of a 5 part series.

[Part 1][2] – [Part 2][3] – [Part 3][4] – [Part 4][5] – [Part 5][6] – [Downloads/Code][7]

So we need to convert the indexer to a method that wont consume as much memory. Looking at how it works now we can determine a few areas that could be improved before we implement out new method.``` public function index(array $documents) { $documenthash = array(); // so we can process multiple documents faster

Code a Search Engine in PHP Part 4

Ben E. C. Boyter — Thu, 10 Jan 2013 07:31:34 GMT

Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.

This is part 4 of a 5 part series.

Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code

So previously we had 4 main issues identified. Im going to go with low hanging fruit and add a simple porn filter.

Code a Search Engine in PHP Part 3

Ben E. C. Boyter — Thu, 10 Jan 2013 07:27:57 GMT

[Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.][1]

This is part 3 of a 5 part series.

[Part 1][2] – [Part 2][3] – [Part 3][4] – [Part 4][5] – [Part 5][6] – [Downloads/Code][7]

At this point I had a large chunk of the web sitting on my disk waiting to be indexed. The first thing we need to think about is what are we going to index. Since we are dealing with HTML files we have marked up content can can pick to a certain extent what we want to index on. For our search im going to index on the title and the meta description tags. Titles are still a part of search engine ranking algorithms as many people search for things like “Yahoo” and “Facebook” and the title/URL usually contains this information. The meta description had fallen out of favor with being something search engines use these days due to being gamed, but since its unlikely the top 1 million are going to be doing this I am going to use it as well. These two terms should work fairly well at our scale and ensure our index isn’t too massive. We can always add the rest of page content at a later point.

Code a Search Engine in PHP Part 2

Ben E. C. Boyter — Thu, 10 Jan 2013 05:03:54 GMT

Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.

This is part 2 of a 5 part series.

Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code

The first implementation has a few issues that are pretty apparent. Chiefly is performance. Secondly it stores the index in a single folder which is loosely related to the first. Finally it shows duplicates in the search results. Lets fix the above and test things again to see how far we can go.

Code a Search Engine in PHP Part 1

Ben E. C. Boyter — Thu, 10 Jan 2013 04:55:54 GMT

[Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.][1]

This is part 1 of a 5 part series.

[Part 1][2] – [Part 2][3] – [Part 3][4] – [Part 4][5] – [Part 5][6] – [Downloads/Code][7]

I imagine that if you have landed on this page you probably have an interest in search and search engines. Not only that, you have probably read the following [Why Writing Your Own Search Engine is Hard by Anna Patterson (of Cuil fame) http://queue.acm.org/detail.cfm?id=988407][8] If not, go read it and come back here. I have always had an interest in search and indexing I thought I would take the time to write a simple indexer as a way of testing if I had actually learnt anything.

Full Text Search with Whistlepig

Pete Keen — Wed, 09 Jan 2013 09:51:11 GMT

Yesterday I suddenly developed the intense need to add search to this site. Among the problems with this is that the site is kind of a weird hybrid between static and dynamic, and it has no database backend. If posts were stored in Postgres this would be a trivial matter, but they're just markdown files on disk. After flailing around for awhile I came across a library named Whistlepig which purported to do in-memory full text indexing with a full query language.

November 5, 2013: I've removed search because nobody used it and this way the site can be 100% static.

First pass: Regular expressions

To rewind a bit, my first horrible stab at search was to find all of the posts that matched a user-provided regular expression:

@results = @posts.find_all do |post|
  post.body.match /\b#{user_query}\b/
end

This is of course complete madness. Not only am I allowing the user to put whatever they want in a regex but it only matches whole words. Not to mention that, while I only have around sixty pages right now, that number's never going to go down.

Whistlepig to the Rescue

Whistlepig is a small text search index. Small as in not very many features and not much code, but the features that are there are perfect for my needs:

Full query language
In-memory, in-process
Arbitrary number of indexes for the same document

Here's a full example of how to index and query a document:

require 'rubygems'
require 'whistlepig'

document = "Hi there"

index = Whistlepig::Index.new "index"

entry = Whistlepig::Entry.new
entry.add_string "body", document

docid = index.add_entry entry

query = Query.new("body", "hi")
result = index.search(query)
assert_equal docid, result[0]

The indexing code in bugsplat's app is not much more complicated. Here's the interesting bit:

@pages_by_docid = {}

@pages.each do |page|
  entry = Whistlepig::Entry.new

  entry.add_string "body", page.render(@strip_renderer)
  entry.add_string "name", page.name
  entry.add_string "title", page.title.downcase
  entry.add_string "tags", page.tags.join(" ").downcase
  entry.add_string "page_id", page.page_id
  entry.add_string "blog_post", page.is_blog_post? ? "yes" : "no"
  docid = @index.add_entry(entry)

  @pages_by_docid[docid] = page
end

In bugsplat a Page encapsulates everything about an entry written in Markdown. I maintain six indexes on the pages, including body rendered with a Markdown-stripping, downcasing renderer, name which is the canonical name of the post, title, tags, page_id which is a short-code type of thing, and blog_post which is a simple boolean as to whether the post has a date or not.

"Why so many indexes?" you may find yourself asking. Because instead of just implementing search and being done with it, I went and refactored the guts of the blog to use it throughout. See, I had these terrible little things everywhere, all over the place:

@page = @pages.find_all { |p| p.has_tag? params[:tag] }

Doing linear searches across the list of in-memory pages isn't too terrible but man it bugged me to have to repeat that everywhere. Instead of that, I can do nice things like this:

@tagged_pages = @pages.search(params[:tag].downcase, "tags")

Each time I found myself iterating over all of the pages to get a subset I replaced it with a search query. The code is much nicer to read and faster, although almost all of it is cached as static HTML in production.

Try it out!

Go ahead and search for some stuff and let me know what you think! And next time you find yourself with a full text search problem, see if Whistlepig would help you out. It's not for everybody, but it's very good at what it does.

Conquering 2013: Goals for the Year

Ben Overmyer's Site — Sat, 05 Jan 2013 02:00:00 GMT

I’m joining the vast numbers of bloggers who are publicly declaring their goals for the new year right now. Scott Dinsmore of Live Your Legend fame has challenged people to, well, Live Your Legend in 2013, and offers a Goal Setting & Action Workbook to help achieve that. I took four hours this morning to fill it out and plan my goals for the year, and this blog post summarizes the results of that. Two of my six big goals for the year have to do with gaming, but the rest are personal goals. I’ll share all of them with you after the jump.

Goal #1) Release a Computer Game

I’ve worked, briefly, in the computer game industry. I’ve started numerous game projects. But I’ve never actually released a finished game. 2013 is the year to change that. While I won’t be releasing my dream game this year (the MMORPG tentatively called Starborn: Universe), I’m going to fire off a single-player action sandbox game called Starborn: Genesis. My release goal is by December 1, 2013. Future blog posts will detail this, my most difficult goal of 2013, as I work on it.

Goal #2) Join a Local Developer Group

This goal is much smaller than #1. The purpose, though, is no less important; it has been said that you are an average of the five people you spend the most time with, and all of the five people I hang out with most are at best only tangentially related to computer game development. I want to connect with people more, and I especially want to connect with others who are passionate about computer game development. Joining a local developer group is an important part of that.

Goal #3) Take a Vacation with Treasure to Hawaii

My wife Treasure and I have not yet taken a vacation outside of the Midwest. As both of us have traveled the world and love new places, this is a problem. This year, I’m committing to getting us both to Hawaii for a week in June, near or on our 4th anniversary. It will be an event to remember, I’m sure! Now I just need to figure out how to pay for it…

Goal #4) Lead a Yoga Class

I don’t know a thing about yoga. I’ve never taken a class and know only a few movements from Wii Fit. So why would I make a goal to lead a class? Because I’m interested in it, and no one ever achieved anything meaningful by aiming low. I’ll need to join a class and become proficient enough over the coming year to teach a community class by the end of December. I’ve already picked out the location and instructor to help me achieve that. It’s just a matter of getting it done.

Goal #5) Reach 1,000 Visitors Per Month On Dicejockey

Yep, my blog wasn’t going to slide by without a major goal this year. Dicejockey’s not about Top Ten lists or Five Steps to Success, but I like to think I say things worth talking about, and that’s worth pushing harder on. My life is my passion, and Dicejockey is a window to that passion. With a little luck and an extra dose of hard work, I hope to make Dicejockey a popular choice for gaming geeks.

Goal #6) Collaborate on Something With Tasha

My sister Tasha and I don’t talk very frequently. This isn’t because we’re not on speaking terms; it’s just because we don’t have much in common. This year I want to change that. We’re both very passionate about a wide variety of topics, and I’m sure we can find something to truly connect on. I’ve got a tentative plan on how to achieve this goal, and I really want to get closer to my family, so I know I’ll meet this goal by year’s end. So those are my goals for 2013. How about you? What are you looking to achieve this year?