Hacker News Personal Blogs 2013 | Top 100 Blogs

2013 Year in Review

Ben Overmyer's Site — Sun, 29 Dec 2013 02:00:00 GMT

A great deal happened in 2013 for me and my family. Sadly, though, I missed the mark on all six of my goals for the year. Despite that, I think 2013 can be counted a success in general.

My Goals for 2013

My first goal for 2013 was to release a computer game. For a variety of reasons, this never came close to completion. I overestimated my free time and, to be honest, my motivation. I ended up spending far more time developing tabletop games. I'll get into that in a second. My second goal was to join a local developer group. Technically, I succeeded at this, since I started going to MNPHP meetups for awhile. However, because I didn't sustain my attendance, I'm counting this one failed. There is promise here, though. My third goal was ambitious - take a vacation to Hawaii. Saving up the money to do so, especially amidst changing jobs, proved too difficult for me. My fourth goal, lead a yoga class, failed for more than just inability to do so. Most yoga schools these days seem to focus solely on the physical side of yoga, with no thought for the spiritual side. I had a moment of clarity when I realized that I couldn't get to the level of proficiency in just a year to be qualified to teach both the physical and spiritual sides. I may revisit this goal some year in the future, but not in the near tense. My fifth goal was well on its way to success when I sabotaged it by re-incorporating Dicejockey into my main site. Though I wanted to reach 1,000 visitors per month for at least one month in 2013, my desire to pull in everything to one destination representing me was greater than my desire to continue a blog dedicated solely to MMORPGs and tabletop RPGs. My sixth and final goal for 2013, to collaborate on a project with my sister Tasha, went fairly well for the first half of the year. Unfortunately, I think I was too ambitious in the project we chose to work on, because my contribution would have been a large number of artworks that my skill level and free time did not allow for. Still, this goal was inspiring, and I'll do something along these lines in 2014.

Major Happenings of 2013

I changed jobs twice this past year. The first time, I left Group 3 Marketing because I felt I was stagnating as a developer. I joined a startup called Sportsdigita and worked heavily on Symfony2 projects. While I certainly grew as a developer at Sportsdigita, the culture was damaging everything else about my life. In August, then, I left that company to work for the Minneapolis Star Tribune, the largest newspaper in Minnesota (and, possibly, the Midwest). The Star Tribune has proved to be a wonderful balance between interesting projects and good culture, and it doesn't hurt that it pays well, too. My contributions to Silver Gryphon Games grew this year. My efforts on the web side increased, and my interest in this side of the business is definitely heating up. I released the first web tool for Ingenium, and it's proved very useful - internally, anyway. I did more writing this year in previous years, releasing no less than three adventures for Ingenium. Two of those I did all the writing for, and the third I contributed a third of the final word count to. This also marked the first year where I managed a convention on my own, using the newly-minted “Booth 2.” All in all, 2013 was a successful year for me. I got back into tabletop gaming in a big way this year. I started what has become my longest-running D&D campaign ever. I started playing Warhammer 40k and Warmachine on a weekly basis with some good friends. I did a lot of miniature painting, and produced some works that I'm actually proud of, which hasn't happened before. I also began writing the rules for my own war game, and that could bode well for Silver Gryphon Games, based on early playtest feedback. I met several new friends and strengthened relationships with others. I've been more social-minded this year than the past couple years. I've definitely gone to more conventions this year than in past years, also.

Closing Thoughts

Many positive things happened this year, but I'm disappointed that I wasn't able to follow through with a single one of my goals. I grew as a developer, a gamer, and a person in general in 2013. Overall, despite the lack of goal completion, I think 2013 was a win for me. Here's to an even better 2014!

Using Beyond Compare 4 Mac Beta with Git

Donat Studios — Fri, 27 Dec 2013 01:29:40 GMT

A little over a year ago I wrote a post about using Beyond Compare on Mac via Wine. A native version is now in beta and open to everyone! If you haven't tried Beyond Compare, I suggest you do. If you have you already know how awesome it is.

I've been testing the Alpha for many months now and thought I'd throw together some instructions for getting it going with git.

You can download the beta here: http://www.scootersoftware.com/beta.php

After you have the App installed, the next step is to install the command line tools. All other steps require this.

From the App, go to the "Beyond Compare" drop down menu and choose "Install Command Line Tools...". It will then prompt you for your password. After this, you bcomp file1.txt file2.txt

Using with Git CLI

The next step is to set it up as a git diff tool and merge tool. Git already has native support for it built in thanks to the Linux client having existed for years now.

This can be achieved simply by running the following commands.

$ git config --global diff.tool bc3
$ git config --global difftool.prompt false
$ git config --global merge.tool bc3

This simple setup will leave git diff and git merge functioning as usual, as well as allowing you to use Beyond Compare for the task by doing git difftool and git mergetool.

I highly recommend giving Beyond Compare a go as a merge tool next time you need to do a merge as it makes it wonderfully simple (when you understand whats going on with the 4 sections anyway).

Using with Tower 1.x

Update: Tower 2.0.6+ include built in support for Beyond Compare which the following instructions will interfere with.

Quit Tower
Navigate to ~/Library/Application\ Support/Tower
Create a CompareScripts folder if one does not already exist.
Download bcomp.sh and place it in the new CompareScripts folder.
Make it executable via chmod +x bcomp.sh
Download CompareTools.plist and place it in the ~/Library/Application\ Support/Tower directory. Rename / move any old version that might exist.

Using with SourceTree

Open SourceTree
Open "Preferences" from the SourceTree menu
Along the top bar, choose the "Diff" tab.
Under External Diff / Merge
1. for Visual Diff Tool choose Other then in the Diff Command enter /usr/local/bin/bcomp and for Arguments enter $LOCAL $REMOTE
2. For Merge Tool choose Other and in Merge Command enter /usr/local/bin/bcomp and for Arguments enter $LOCAL $REMOTE $BASE $MERGED

On GitHub

Nemo's Home — Wed, 25 Dec 2013 02:00:00 GMT

I am an Internet addict. And the website I’m most addicted to is called GitHub. GitHub is a social code hosting website that is totally awesome. Not your general run-of-the-mill awesome, but rather ass-kicking best-thing-in-the-world awesome. If GitHub was a ninja, it would be Po, defeating the evil clutches of SourceForge.

When Linux was released a couple of decades ago, it was hosted on a university FTP server. Today, the days of FTP are way past, and unless your code is hosted on a code-sharing website like GitHub, it is as good as dead. The last decade has seen an explosive grown in software and the open source movement. Things like GitHub, Linux, Android, Facebook have been made possible due to the combine efforts of millions across the globe following the Unix philosophy of doing one thing well.

Ahh, I digress. The point I’m trying to make is that it is very hard to explain to a layman how important GitHub has been to the software community. It has been used as a collaboration platform for writers, law makers, governments, programmers, and even musicians. People have used its issue tracking feature to even plan weddings. And above all, people adore GitHub. It is one of the few startups that have been accorded God status in the community.

The question here is what makes GitHub tick? As the largest hosting site for code, it obviously makes a huge impact just by virtue of being there. But its tide of features and innovative progression had made it a darling of all. For instance, their 1 click to fork feature has allowed people to contribute to any project so much easier.

GitHub has an amazing User Experience, making sure it works perfectly on all devices large or small. Their amazing support makes sure all of its users are happy as possible. Their regular meetups makes sure that GitHub is invested in the software community themselves. And their GitHub Store lets people invest back in GitHub by romoting them via Tees, stickers and even laptop sleeves.

My personal experience with GitHub has been overwhelmingly positive. People have stood beside GitHub even as they faced major issues, and for me that is indicative of a trust in GitHub that no money can buy. The GitHub API (which I’ve used more than once) lets developers create their own apps on top of GitHub which others can use to create even more awesome things.

For me GitHub is more than just a coding website. It is a testament to creativity and the Hacker Way, reminding me every day that anything is possible.

Testing a NAS hard drive over FTP

Linux Engineer's random thoughts — Fri, 20 Dec 2013 01:00:00 GMT

So I have this NAS made by my ISP, that does a lot of things; but recently, I started having issues with its behavior. Recorded TV shows had lag/jitter while replaying, and the same happened with other types of videos I put on it. I narrowed it down to …

Digging in with Foreign Tables

CRAIG KERSTIENS — Thu, 19 Dec 2013 22:55:56 GMT

I wrote a couple months back about exploring FDWs. Its become quite clear to me, despite still having ample room for improvement, they’re not getting enough attention. Foreign data wrappers are perhaps better thought of as a foreign table, or even better yet as a view into some remote data source. They don’t take care of auto-updating or syncing data, thats all up to you, but it gives you a straight forward mapping to work with remote data easier.

The first step in working with FDWs is getting them setup. I wrote an earlier post on how to do this manually. And if you’re on Heroku theres an even easier solution if you want to setup a mapping entirely from one DB to another. The pg-extras CLI plugin has a command fdwsql which will generate the SQL to map all the tables for you. To run it simply specify the prefix app and database:

heroku pg:fdwsql yourprefix APP::DATABASE_URL

This will generate a lot of SQL. From here you’ll want to connect to the database where you want those foreign tables to be visible. Then run all the SQL. This will create all the foreign tables, this will mostly look just like another view or table to you in \d.

Tips on working with them

For the most part you can work with your foreign tables just like any other view or table. You can insert into them, read from them, join against them. Though currently foreign tables have some performance limitations, such as when joining it may return a lot more data than you expect then join. To make your performance a bit more ideal you can follow a few basic principles.

Lets look at some example tables to highlight this:

> \d
users
todos

In this case users is local and the todos are a foreign table. Looking at each of the schemas we have something like you might expect:

> \d users
Table "public.users"
Column | Type | Modifiers
-------------+-----------------------------+-----------
id | integer | not null
email | text |
created_at | timestamp without time zone |
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"users_created" btree (created_at)
> \d todo
Foreign Table "public.todo"
Column | Type | Modifiers
-------------+-----------------------------+-----------
id | integer | not null
user_id | integer |
desc | text |
created_at | timestamp without time zone |
status | boolean |
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"todo_created" btree (created_at)

Quora answer about writing a search engine

Ben E. C. Boyter — Fri, 06 Dec 2013 03:31:30 GMT

The following I posted on Quora in response to the question “I am planning to make a small scale search engine on my local system, but I don’t know from where to start?”. It’s a reasonable answer so like my Mahalo one I thought I would make a copy for myself.

I agree with Wolf Garbe and that you are better off in your case starting with existing technologies, have a look at http://yacy.net/ and SphinxSearch as well. However if you are doing this to learn and not just deliver a product I can provide a few links for you.

Go Binary Sizes Are Growing out of Control

Donat Studios — Tue, 03 Dec 2013 01:25:43 GMT

Update 2019-07-17: I have released a follow up post Go Binary Sizes Are Relatively Stable which speaks on what's happened in the time since this was published.

Note: This is not intended as a "Go Sucks" post. I love Go. I am not saying the developers are lazy or dumb or any of the things Reddit has implied. I am not implying I could build a better compiler. Rob Pike and the Go team are geniuses whom I look up to. The tone of this was intended to be a light "man I wish binaries were smaller" but that is not how Reddit seemed to take it. Also, the title is hyperbole and not meant to literally mean no one has control over it.

I started toying with Go about two years ago before the 1.0 release. I love the language, it meets me in a comfortable place between C and a scripting languages like PHP. I find it fun to write and an easy way to bust out performant code quickly. The binaries for a simple Hello, 世界 were large though, around 55KB. I figured this was something the Go team would work out later; it is a lot of space just to send a string to standard out.

With the release of Go 1.0 though, I noticed the binary size had grown slightly to 242KB. Discouraging but livable. I was quite hopeful with the 1.1 release, but it had actually nearly doubled in size coming out to a whopping 405KB. Then this morning I compiled it again in the newly release 1.2 and to my horror the ever simple program came out to 557KB.

What on earth is going on in that binary? I am no expert on compilers but I would imagine the vast majority of what is in there is completely unnecessary and could be optimized out. Changing the script to "Hello, World" rather than Go's flaunted unicode version has zero effect on binary size, as one would expect but I was hopeful.

For reference the complete source of my Hello, 世界 follows. Using fmt comes out even larger.

package main

func main() {
    println("Hello, 世界")
}

Here is a collection of binaries compiled in different versions of Go. Notice how imgavg grows from 1.9MB to 3.7MB. My applications are all small tools. I can only imagine the effect this has an already large application.

I really hope something is done about this. I would love to see tiny (<32KB) binaries from Go but right now that doesn't seem to be a priority for them. The fact that Hello, 世界 fills half a floppy means the language is not be useful for lighter or embedded environments.

As noted in the header there is an addendum to this:

Go Binary Sizes Are Relatively Stable

Making HackerTray

Nemo's Home — Thu, 28 Nov 2013 02:00:00 GMT

A few days back, I found the excellent HackerBarApp via Hacker News. Hacker News, for those of you who don’t know, is tech news website run by YCombinator. Hacker Bar was the simplest way of accessing HN stories that I’d ever seen. Unfortunately, it was only for Mac (made using rubymotion) and even though the source was available, it was of no use to me as a Linux User.

I decided to make a clone of Hacker Bar that would work on Linux. My first choice of the stack was node-webkit, an application framework that allows you to build cross-platform applications using HTML, CSS, JS, and modules from the node.js ecosystem. After reading a lot about node-webkit, I figured out that building this application in node-webkit (as it stands) would not be possible. Or rather, it would not work under Ubuntu and its derivatives because of lacking appindicator support in node-webkit. More details here.

The next obvious language and stack of choice was Python + Gtk. I’d already played a little bit with Gtk and Python some time back, so I knew the basics. But I’d never build a real application with PyGtk, just toys and small scripts. I found a basic skeleton app that was written for AppIndicator and modified it somewhat to form the base of HackerTray.

The next challenge I faced was keeping the check boxes always checked despite of any number of clicks after the first. That is, we don’t want any menu item to be “un-checked” at any moment. A basic idea is to do this (partial code):

def open(self, widget):
	if(widget.get_active() == False):
		widget.set_active()
	webbrowser.open(widget.url)

def addItem(self, item):
	#create a new CheckMenuItem (i)
	i.connect('activate', self.open)

However, this does not work as expected, because the widget.set_active() call also results in the activate event being fired, which ultimately calls open. This means on a click to an unchecked menuItem, the open function is called twice. This results in the browser opening the link twice.

As a workaround, I disabled the event handler in case it is a checked menuItem:

def open(self, widget):
	if(widget.set_active() == False)
		wiget.disconnect(widget.signal_id)
		widget.set_active(true)
		widget.signal_id = widget.connect('activate', self.open)
	webbrowser.open(widget.url)

def addItem(self, item):
	#create a new CheckMenuItem (i)
	i.connect('activate', self.open)
	i.signal_id = i.connect('activate', self.open)

The next thing I worked on was a persistent memory for the app. In a nutshell, I needed to make sure that the tick on an item remained there, even if the app was restarted. This meant writing a list of all the “viewed” items into a file. After looking at shelve for a bit, I just rolled my own implementation , based on storing the data into ~/.hackertray.json file.

After that I worked on packaging the app into a python package, so that it could be easily installed. The python packaging tutorial was an easy to use guide that let me create the package easily and push it to the Python Package Index. A few issues in the package were found, and were fixed quickly thanks to the pull request by @brunal.

After improving the README a bit, I posted about it on Hacker News, where it failed to get any traction. I re-tried with a link to the HackerTray website, and that fell flat as well. It was on the next day, when I posted it to HN for the third time, that it took off. After 50 or so upvotes, I found that my instance refused to run because it had hit the API Rate Limit on the excellent node-hnapi. I quickly pushed a fix that used a list of servers to hit as fallback in case it crossed the Rate Limits.

After a lot of feedback from HN, I started work on a node-webkit based clone of hackertray for Windows. I should be able to release it in a few more days, if nothing else crops up. Keep watching this space for info. If you have any queries, just file an issue on GitHub or contact me.

University Presentation

Tom Forbes — Wed, 27 Nov 2013 20:19:28 GMT

So I did a presentation on Information Security at University today. I think it went rather well, however I couldn’t show a couple of the demonstrations due to some SkyDrive files only being available online. That sucked because those were my best demonstrations, but overall I was happy. A few peopl...

PostgreSQL - It's kind've a nifty database

Brightball Articles — Wed, 27 Nov 2013 11:42:00 GMT

This is a presentation I recently gave to provide an overview of PostgreSQL and some of it's excellent features, including full-text search, multiple built in datatypes, data compression and extensions. Also, Morgan Freeman is narrating. You're welcome.

My Dimension in Rift

Ben Overmyer's Site — Wed, 27 Nov 2013 02:00:00 GMT

I started playing Rift again, after trying out Everquest 2 and Guild Wars 2 also. Until Everquest Next Landmark comes out, Rift may be my fantasy MMO of choice. We'll see.

Hack things, acquire clothing.

Tom Forbes — Fri, 22 Nov 2013 18:59:42 GMT

I discovered an XSS flaw in a website a month ago and reported it to the owners. As a thank you they sent me a hat, a rather large american sized t-shirt and a pair of “DeFeet” socks (guaranteed to stay cooler and drier than any other brand). I didn’t expect them to ship something like that overseas...

Books I read in 2013

Krishna's blog — Tue, 19 Nov 2013 02:00:00 GMT

A list of books read in 2013, including fantasy, historical fiction, and science fiction.

The best Postgres feature you're not using – CTEs aka WITH clauses

CRAIG KERSTIENS — Mon, 18 Nov 2013 22:55:56 GMT

SQL by default isn’t typically friendly to dive into, and especially so if you’re reading someone else’s already created queries. For some reason most people throw out principles we follow in other languages such as commenting and composability just for SQL. I was recently reminded of a key feature in Postgres that most don’t use by @timonk highlighting it in his AWS Re:Invent Redshift talk. The simple feature actually makes SQL both readable and composable, and even for my own queries capable of coming back to them months later and understanding them, where previously they would not be.

The feature itself is known as CTEs or common table expressions, you may also here it referred to as WITH clauses. The general idea is that it allows you to create something somewhat equivilant to a view that only exists during that transaction. You can create multiple of these which then allow for clear building blocks and make it simple to follow what you’re doing.

Lets take a look at a nice simple one:

WITH users_tasks AS (
SELECT
users.email,
array_agg(tasks.name) as task_list,
projects.title
FROM
users,
tasks,
project
WHERE
users.id = tasks.user_id
projects.title = tasks.project_id
GROUP BY
users.email,
projects.title
)

Using this I could now just append some basic other query on to the end that references this CTE users_tasks. Something akin to:

SELECT *
FROM users_tasks;

But where it becomes more interesting is chaining these together. So while I have all tasks assigned to each user here, perhaps I want to then find which users are responsible for more than 50% of the tasks on a given project, thus being the bottleneck. To oversimplify this we could do it a couple of ways, total up the tasks for each project, and then total up the tasks for each user per project:

total_tasks_per_project AS (
SELECT
project_id,
count(*) as task_count
FROM tasks
GROUP BY project_id
),
tasks_per_project_per_user AS (
SELECT
user_id,
project_id,
count(*) as task_count
FROM tasks
GROUP BY user_id, project_id
),

Then we would want to combine and find the users that are now over that 50%:

overloaded_users AS (
SELECT tasks_per_project_per_user.user_id,
FROM tasks_per_project_per_user,
total_tasks_per_project
WHERE tasks_per_project_per_user.task_count > (total_tasks_per_project / 2)
)

Now as a final goal I’d want to get a comma separated list of tasks of the overloaded users. So we’re simply giong to join against that overloaded_users and our initial list of users_tasks. Putting it all together it looks somewhat long, but becomes much more readable. And as a bonus I layered in some comments.

--- Created by Craig Kerstiens 11/18/2013
--- Query highlights users that have over 50% of tasks on a given project
--- Gives comma separated list of their tasks and the project
--- Initial query to grab project title and tasks per user
WITH users_tasks AS (
SELECT
users.id as user_id,
users.email,
array_agg(tasks.name) as task_list,
projects.title
FROM
users,
tasks,
project
WHERE
users.id = tasks.user_id
projects.title = tasks.project_id
GROUP BY
users.email,
projects.title
),
--- Calculates the total tasks per each project
total_tasks_per_project AS (
SELECT
project_id,
count(*) as task_count
FROM tasks
GROUP BY project_id
),
--- Calculates the projects per each user
tasks_per_project_per_user AS (
SELECT
user_id,
project_id,
count(*) as task_count
FROM tasks
GROUP BY user_id, project_id
),
--- Gets user ids that have over 50% of tasks assigned
overloaded_users AS (
SELECT tasks_per_project_per_user.user_id,
FROM tasks_per_project_per_user,
total_tasks_per_project
WHERE tasks_per_project_per_user.task_count > (total_tasks_per_project / 2)
)
SELECT
email,
task_list,
title
FROM
users_tasks,
overloaded_users
WHERE
users_tasks.user_id = overloaded_users.user_id

CTEs won’t always be quite as performant as optimizing your SQL to be as concise as possible. In most cases I have seen performance differences smaller than a 2X difference, this tradeoff for readability is a nobrainer as far as I’m concerned. And with time the Postgres optimizer should continue to get better about such performance.

As for the verbosity, yes I could have done this query in probably 10-15 lines of very concise SQL. Yet, most may not be able to understand it quickly if at all. Readability is huge when it comes to SQL to ensure its doing the right thing. SQL will almost always tell you an answer, it just may not be to the question you think you’re asking. Ensuring your queries can be reasoned about is critical to ensuring accuracy and CTEs are one great way of accomplishing that.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Restricting Thrift clients to specific IP addresses with Twisted

Tom Forbes — Mon, 11 Nov 2013 19:17:11 GMT

Apache Thrift is pretty awesome - you can build Twisted bindings for your Thrift interface file that work fantastically. There is one thing that took me a while to figure out: I want to restrict clients connecting to the service to a specific set of IP addresses stored in a database. There were thre...

Load a Github Gist with Composer

Donat Studios — Thu, 07 Nov 2013 04:36:45 GMT

Composer is amazing for pulling in packages, but what if you find a Gist that isn't Composer aware? Fear not, as Composer has the magical ability to pull in repositories not explicitly set up in Packagist by defining them in the "repositories" section.

But what if you wanted to load a single class from a git gist?

You can add the following repositories section to your composer.json file, adjusting name and url as necessary. The name should be a vendor/package (all lower) style name you make up for it, and url will be from the Clone this gist box on GitHub.

"repositories": [
    {
        "type":"package",
        "package": {
            "name": "turin86/wssoapclient",
            "version": "master",
            "source": {
                "url": "https://gist.github.com/5569152.git",
                "type": "git",
                "reference":"master"
            },
            "autoload": {
                "classmap": ["."]
            }
        }
    }
]

Then in your require section, add your selected name as follows.

"require": {
    "turin86/wssoapclient": "dev-master"
}

If the name of the class doesn't match the filename in a friendly autoload-able manner, you'll have to replace "classmap": ["."] with "files": ["ClassToAutoload.php"] replacing ClassToAutoload.php with the filename. More info on autoloading can be found here. Note that if you are experimenting with autoload settings you may need to remove your vendor directory and composer install, as updates to the autoload were not reflected for me after a composer update.

After this, just run composer update and voilà, you are loading a gist with Composer.

Embedded Linux Conference Europe 2013 notes

Linux Engineer's random thoughts — Wed, 30 Oct 2013 01:00:00 GMT

So I was in Edinburgh this year, and I took notes as I usually do. These are intended for personal consumption (do no expect LWN-style reports), but as more people were asking me to share them, I thought why not do it in public ?

Embedded Linux timeline

by Chris Simmonds …

Installing the PHP "memcached" Extension

Donat Studios — Wed, 23 Oct 2013 13:42:40 GMT

Please Note

This tutorial only covers the default PHP installation that ships with Mac OS X / macOS. If you have installed a new installation this does not cover you.

For those using brew, you can simply brew install the appropriate brew formula for the extension.

These directions are verified to work with Mac OS X 10.9 Mavericks through macOS 10.14 Mojave.

System Integrity Protection

If you are running OS X El Capitan or newer you will need to disable System Integrity Protection to modify system files and directories.

To disable System Integrity Protection, boot into recovery mode by restarting and then holding ⌘R as you hear the startup chime. Then start the Terminal from the Utilities menu.

Run the following command

$ csrutil disable

Then reboot. You are good to go.

If you wish to turn it back on, follow the instructions above but instead use:

$ csrutil enable

Click here for a full post on the topic.

The first step is to install the latest version of Xcode. On modern versions of MacOS this can be done from the App Store.

After this, install the Xcode developer tools.

$ xcode-select --install

Special note to Mojave users if the following steps fail, you may need to reinstall your Xcode header files. Instructions for that can be found here.

Next we will need to install the required dependencies before we can build the extension. If you are not already using Homebrew you should be.

$ brew install wget autoconf pkg-config libmemcached

You will also want to make sure you have PEAR installed; full instructions can be found here but can be summarized as follows:

$ wget http://pear.php.net/go-pear.phar
$ php go-pear.phar

Then in the /tmp directory we will create a folder we can work in.

$ cd /tmp
$ mkdir memcached-work
$ cd memcached-work

Next we will use pecl, part of pear, to fetch the current version of the extension.

The version of the memcached extension will vary so you will need to update the paths accordingly.

$ pecl download memcached
$ open memcached-{{version}}.tgz
$ cd memcached-{{version}}/memcached-{{version}}
$ phpize
$ ./configure
$ make
$ sudo make install

Finally you will need to add the following line to your php.ini

extension = memcached.so

You can verify your installation with the following:

$ php --info | grep memcached\\.

Depending on your setup now you may want to restart apache.

$ sudo apachectl restart

You should be all set to go!

How to: Disable macOS System Integrity Protection

Donat Studios — Wed, 23 Oct 2013 13:42:40 GMT

If you are running OS X El Capitan or newer you will need to disable System Integrity Protection to modify system files and directories.

To disable System Integrity Protection, boot into recovery mode by restarting and then holding ⌘R as you hear the startup chime. Then start the Terminal from the Utilities menu.

Run the following command

$ csrutil disable

Then reboot. You are good to go.

If you wish to turn it back on, follow the instructions above but instead use:

$ csrutil enable

My wishlist for a Star Wars Galaxies 2

Ben Overmyer's Site — Sun, 20 Oct 2013 03:00:00 GMT

Star Wars Galaxies 2 will most likely never happen, owing to the fact that SOE has publicly stated that it will never work on an MMO based on an external IP again. Whether that's true or not, if somehow this project were to be conceived, here are the changes from the original that I'd personally want to see in a sequel.

Modern Graphics Engine

This is the most obvious one. My personal choice for SWG2's engine would be Unreal Engine 4, owing to its extremely impressive technology. The “Infiltrator” demo video for UE4 shows that it would be perfect for the realistic art style of the Star Wars: Galaxies franchise, as compared to the more stylized art direction of Star Wars: The Old Republic.

A Return to Skill-Based Character Progression

Like most SWG veterans, I lamented the loss of the skill tree progression system when the New Game Experience took over. With SWG2, I'd want to see it return with a vengeance. Particularly, I'd like to see the depth and breadth of trees expand, allowing for greater freedom than even the original. Several aspects of the Star Wars universe were never fully explored in the first game, or were only given light treatment, such as slicing or smuggling.

More Player Races

The selection of available races in the original was good, allowing for most of the popular choices. I would like to see the Togruta, Chiss, and Devaronian races make a core appearance in SWG2. I'd also like to see droids as a player race, though that might be too complicated to balance against other races' gameplay.

Bigger Worlds

The original SWG planets were pretty big, but they frequently felt empty. The expansion worlds had stronger design, but felt smaller. I'd want to see planets in SWG2 be more expansive than the original SWG planets, but have more points of interest throughout each. For example, the planets in the first SWG felt a lot like the same planet given a different skin each time. While I don't want quests to provide a meta plot, I would like to see localized content that gives each world its own character.

Greater Freedom of Movement

Let's face it, movement was pretty horrible in the original SWG. You couldn't jump or fly over obstacles, and it was completely possible to walk straight up a sheer cliff face. While we don't necessarily need the full parkour aspects of EverQuest Next, I do want the ability to jump over things, be prevented from walking up impossible surfaces, and be able to climb, swim, and otherwise move as a person should be able to move.

Atmospheric Vehicles

I know these were added to the original SWG, but it came only in the last few months of the game's life. Given how important they are to the Star Wars universe in general, it would be a crime not to include these from the beginning in SWG2. I want to be able to bullseye womp rats in a T-16, over take a cruise over the highlands of Corellia in my own YT-1300. I also want shuttle transports to actually have real transit time, and not be a wait-and-teleport affair like they were in the original.

Player Factions

While it was great to have Imperial and Rebel affiliations in the original, to really ramp up the sandboxy nature of Star Wars Galaxies, the sequel would need to have player factions. This is particularly important if the game is set in a time period after the fall of the Empire, where a billion new factions would be vying for power. I see player factions being akin to massive alliances of Player Associations, rather than just an enormous PA. They'd also have to be able to influence planetary politics, like how Imperial and Rebel players were able to influence things towards the end of the original SWG.

A Player Exchange

Unlike the auction house format of current MMORPGs, I'd like to see something closer to the original vendor system in SWG. You'd be able to search available items on a local, planetary, or galactic level, as well as purchase them, but you'd need to physically travel to the vendor in order to pick up the item or items. This is important for two reasons: to encourage exploration and travel, and to support the possibility of player deliverymen - “FedEx” quests that actually matter. I want to have to travel to Tosche station to pick up some power converters.

Greater Visual Customization of Crafted Items

I liked that crafters could dye armor in SWG, but a sequel would need to take it to a new level. Item appearance should be further separated from item game mechanics than in the original. There is a vast variety of armor appearances in Star Wars, for example, and that should be reflected in SWG2. I don't want just a dozen different kinds of armor in different colors. I want several dozen, if not more. If World of Warcraft can do it, so can SWG2. Involving the Player Studio could help defray the cost of developing new models and textures for components of items.

Vehicular Combat

If we can fly in the atmosphere, we should be able to fight in it also. I'd want to see hover tanks finally make an appearance, among other things. If PlanetSide 2 can balance heavy armor against infantry, Star Wars Galaxies can do it too. This list could be even longer, but the above are my biggest wishes for a Star Wars Galaxies 2. How about yours? What would you want to see in a sequel?

Hello, World!

Linux Engineer's random thoughts — Sun, 20 Oct 2013 01:00:00 GMT

Update : I updated the information below with the 2020 tech.

So I finally did it. You're reading it right now. My personnal website/blog.

I should be posting here about things that cross my mind as well as various projects I've been working on. And maybe even new projets I …

First post

Krishna's blog — Sat, 19 Oct 2013 03:00:00 GMT

A personal blog about mundane observations on reading, programming, and gaming, with occasional updates and vain attempts to drive traffic to the blog.

Adding tail-call optimization to Python

Tom Forbes — Tue, 15 Oct 2013 22:04:05 GMT

Tail-call optimization is a trick many languages and compilers use to avoid creating excess stack frames when dealing with recursive code like this:...

Tooling for Simple but Informative Emails

CRAIG KERSTIENS — Sun, 13 Oct 2013 23:55:56 GMT

Emails are one of my favorite methods of communicating with users. Its works as a quick test for product validation. It works well at one->some->many-> all. Its still highly effective even as much noise as we receive in our inboxes. Over the years I’ve tried a lot of email tools from custom built solutions, to newer entrants that help around drip actions (intercom.io and customer.io), to more “enterprise” tools such as Marketo. While I have varying opinions on all of those, I still find myself coming back to a simple one off script setup to deliver clear concise emails.

Getting the Data

The first step of any email is deciding what you want to do, but hopefully you know that already. The part that is usually a bit more effort is actually getting the list to send it to and formatting it appropriately. I usually opt for SQL. While the specifics of the query of course always vary it common follows a general structure:

WITH initial_data AS (
SELECT
email,
app_name,
information_about_app
FROM
users,
apps
WHERE users.id = apps.user_id
AND some_filter_to_limit_data
),
candidates_for_email AS ... --- likely to have additional CTEs
--- Finally I build up the list
SELECT email,
array_to_string(array_agg(data_for_email), '
') --- an important note is to add a newline or not here depending on how you wish to format it
FROM candidates_for_email
GROUP BY email;

The query structure you’ll want is first column email, second column whatever data you want to include in your email.

From here I usually create a dataclip of it. This makes it easy to allow my data to change over time. If I’m testing an email for data over the last 7 days I just come back in 7 days and I have new data. It also lets me easily share and iterate on the data. The nice part is there’s an easy way to click a button and get the data as a CSV which is what you want for sending.

Once you download the CSV you’ll want to remove the header line as its not needed for the script.

Sending the Mail

To actually send the email you’ll need this script, which is largely credited to @leinweber:

require 'mail'
require 'csv'
FILE = ARGV[0]
Mail.defaults do
delivery_method :smtp, {
address: 'smtp address',
port: 587,
domain: 'gmail.com',
user_name: 'craig.kerstiens@gmail.com',
password: ENV.fetch('EMAIL_PASSWORD'),
authentication: :plain,
enable_starttls_auto: true
}
end
def send_email(address, app)
mail = Mail.new do
to address
from 'Craig Kerstiens <craig.kerstiens@gmail.com>'
subject "Your email subject in here"
body generate_body(app)
end
end
def generate_body(app)
%Q(
Hi,
Your list of apps:
#{app}
Various email content in here...
)
end
CSV.parse(File.read(FILE)).each do |line|
address = line[0]
app = line[1]
m = send_email(address, app)
puts m.to_s
p m.deliver!
puts
puts
end

You’ll want to make sure to export the PW of your email provider with EXPORT EMAIL_PASSWORD=pw_here

You can easily download this script from off of Github’s Gist. I’d recommend using an email service provider other than Gmail in sending your emails such as mailgun as they’re built to handle sending a large amount of emails. Finally send your emails:

ruby email.rb nameofyourfile.csv

My Uni's timetable system sucks, so I built a better one.

Tom Forbes — Sat, 12 Oct 2013 02:05:09 GMT

tl;dr The timetable system sucks, so I made one that works Getting your timetable sorted at Uni has never been fun. In years 1 and 2 of my study the department posted a timetable for each year showing all modules and students were expected to remove the classes they did not take, which while not the...

Objective-C

Dan Palmer — Sat, 12 Oct 2013 02:00:00 GMT

If I mentioned that I like C, C++ or Python to other students on my course, or colleagues, there would be no reaction. There are things you can criticise about each one, but they are all very safe bets. When I tell people that I enjoy writing Objective-C however, they are confused and often quite hostile towards the language.

I am by no means an Objective-C expert, but I’ve been thinking through the reasons why I like it, so this is a random collection of reasons why I enjoy using the language.

Purchasing a £30,000 numberplate for the price of a bus ticket

Tom Forbes — Mon, 07 Oct 2013 04:40:33 GMT

Regtransfers.co.uk is a website that allows you to purchase customized numberplates for your car or motorbike. They boast a large number of famous clients and short numberplates are often on sale for upwards of £20,000 (the plate ABC 4 is up for £30,000). While playing with their site I discovered a...

Breaking out of secured Python environments

Tom Forbes — Wed, 25 Sep 2013 23:02:33 GMT

A week or so ago I was browsing /r/Python and I saw a link to a website called rise4fun.com , which is a Microsoft Research project that contains a lot of cool demos and tools that you can run in your browser. The demo I was linked to was a restricted Python shell that could be used to experiment wi...

Aboard the Nautilus

Nemo's Home — Sat, 21 Sep 2013 03:00:00 GMT

I’d done a post on this a long time back (2009), detailing what all softwares I use on a daily basis. This is an update to that post.

Since the last post, I’ve moved on to using Linux, using Elementary OS as my primary OS. Over the time period this post was written, I’ve shifted from using Cinnamon to Openbox and finally to elementaryOS’s pantheon as my Desktop Manager. I’m thinking of switching to Arch Linux, just to get a faster experience. I use Synapse as my application launcher, because its much faster than anything else out there.

For most of my web browsing needs, I rely on Google Chrome Stable and a daily build of Chromium (v31 as of now) for most of my work. I switch between them all the time. I use Firefox (stable) only to test out my projects from time to time.

The current editor I use is Sublime Text. It is everything you need, and much more. I’m still to get started using its build system, and its plethora of packages; but its still an excellent choice for a daily use editor. On the command line, I use Vim, git (with SCM Breeze), fasd and tig, which is an excellent git interface on the command line.

I listen to music on my own browser-based music player, called Muzi, YouTube and GrooveShark.

For my terminal needs, I use Gnome-Terminal. I use Byobu to manage my session, and often connect to it from other computers as well. Its an excellent multiplexer that fits in my workflow really well.

I use Imo.im on both the Desktop and my tablet to chat. I occasionally use ReText for editing markdown files. I use RedShift on my laptop and f.lux on my iPad to help me sleep better. I recommend it to everyone who is suffering from eye-strain or wants to sleep better.

On the browser, my most visited sites would be Hacker News (via hckrnews.com), WorkFlowy for managing my to-do list and GitHub on a daily basis for most of my projects.

Hardware

I own a old Nokia X3-02, and will be upgrading to a Firefox OS Phone soon enough. I use a Dell Inspiron 1545 as my personal machine. I also use an iPad 2 (with 3G) on a daily basis (mostly for reading). I also own a Dayan Zhanchi 3x3 and a 5x5 shengshou speed cube.

iPad Apps

The must have iPad apps for me are Chrome, imo, and iBooks. I have installed Mailbox alongside GMail, and haven’t used GMail since I installed it. I sometimes write stuff using Plaintext, and sketch using Paper. I read my RSS feeds using Newsify and Feedly.

Extensions

Chrome Extensions that I use on a daily basis include Chime for wonderful notifications (highly recommended), Ghostery for getting a tracker-free internet , HTTPS Everywhere to keep me secure, LastPass to manage passwords and Stylish for customizing the looks of various websites.

Dream Setup

My dream setup would consist of a lightweight Ubuntu Laptop that I can carry around that still has lots of processing power and battery life. I’ll prety much be satisfied by any high-end Android phone as long as it has a decent battery life.

Inspecting .NET applications with ILSpy

Tom Forbes — Thu, 19 Sep 2013 15:40:40 GMT

Every once in a while I come across an application that is so comically insecure that I feel the urge to blog about it. The application in question is a .NET application to manage care homes and provide a Medical Administration Record for residents. Staff login to the app using a username and passwo...

Introducing SingleBugs the Bug Tracker for Single Developers

Ben E. C. Boyter — Mon, 16 Sep 2013 04:50:08 GMT

Introducing the first beta release of SingleBugs, the bug tracker aimed at single/solo developers. Are you a solo developer? Do you find setting up Mantis/Fogbugs/Bugs.net/et.al too complex and a waste of your time? Do you want a single solution that easily syncs and backs up between your machines? Try SingleBugs.

Guaranteed to save you time and money setting up a bug tracker. Guaranteed to be the fastest bug tracker you have ever used.

Disabling muting while typing in Google hangouts

CRAIG KERSTIENS — Thu, 12 Sep 2013 23:55:56 GMT

Google hangouts is awesome, its my preferred method for most audio/video calls these days. When running a group call I often dial into a separate phone if I have a better phone available for the group. It also got around the annoyance that when you are typing google automatically mutes you. This for most people is pretty subpar. While dialing in to the hangout can still be nice, you don’t have to do so to get rid of the annoying muting while typing. To fix such simply open up your terminal and run:

 defaults write com.google.googletalkplugind exps -string [\"-tm\"]

This clever hack discovered courtesy of @timtyrrell passed along to me by @mattmanning and @blakegentry

Diving into Postgres JSON operators and functions

CRAIG KERSTIENS — Wed, 11 Sep 2013 23:55:56 GMT

Just as PostgreSQL 9.3 was coming out I had a need to take advantage of the JSON datatype and some of the operators and functions within it. The use case was pretty simple, run a query across a variety of databases, then take the results and store them. We explored doing something more elaborate with the columns/values, but in the end just opted to save the entire result set as JSON then I could use the operators to explore it as desired.

Here’s the general idea in code (using sequel):

result = r.connection { |c| c.fetch(self.query).all }
mymodel.results = result.to_json

As the entire dataset was stored as some compressed JSON I needed to do a bit of manipulation to get it back into a form that was workable. Fortunately all the steps were fairly straightforward.

First you want to unnest each result from the json array, in my case this looked like:

SELECT json_array_elements(result)

The above will unnest all of the array elements so I have an individual result as JSON. A real world example would look something similar to:

SELECT json_array_elements(result)
FROM query_results
LIMIT 2;
json_array_elements

{“column_name”:“data_in_here”} {“column_name_2”:“other_data_in_here”} (2 rows)

From here based on the query I would want to get some specific value. In this case I’m going to search for the text key column_name_2:

SELECT json_array_elements(result)->'column_name_2'
FROM query_results
LIMIT 1;
json_array_elements
-----------------------
"other_data_in_here"
(1 rows)

One gotcha I encountered was when I wanted to search for some value or exclude some value… Expecting I could just compare the result of the above in a where statement I was sadly mistaken because the equals operator didn’t translate. My first attempt at fixing this was to cast in this form:

SELECT json_array_elements(result)->'column_name_2'::text

The sad part is because of the operator the cast doesn’t get applied as I’d expect. Instead you’ll want to do:

SELECT (json_array_elements(result)->'column_name_2')::text

Of course theres plenty more you can do with the JSON operators in the new Postgres 9.3. If you’ve already got JSON in your application give them a look today. And while slightly worse, if you’ve got JSON stored in a text field simply cast it with ::json to begin using the operators.

The Fizzbuzz Bug Tracker A Bug Tracker for Single Developers

Ben E. C. Boyter — Fri, 06 Sep 2013 08:22:26 GMT

This serves as the announcement of my new bug tracker product I am working on.

My needs are pretty specific and none of the existing bug trackers I have tried have meet the following goals.

Speed. Searching, adding projects/issues/comments should be instant. Any time waiting on the bug tracker is wasted time.
Outlook style view of projects/issues/comments. This should allow me to get an overview of how I am tracking.
Not require me to spin up a full web-server. I don’t want to have to install a full stack web-server just to track bugs.
Sync across all devices. I want it to just appear on every device I own without me having to worry about backups or keeping things in sync.
Be single user. I don’t want to have to login or perform user management.
Support multiple projects. I also want to be able to have multiple issues per project.

With that in mind let me introduce the Fizzbuzz Bug Tracker. Fizzbuzz is a single executable that works on Windows, Linux and OSX (once I have a machine to compile it on). A suggested workflow is to copy it onto your Dropbox folder (or equivalent) where it will be synced across all devices. Fizzbuzz has a strong emphasis on speed. Adding projects or issues, searching across projects and issues, drilling into projects or issues is near instant.

Collection of Letters for Neural Network OCR Training

Ben E. C. Boyter — Tue, 03 Sep 2013 08:59:42 GMT

I was looking for this on Google the other day and unable to find it. Essentially what I needed was a collection of images which are all the same size, but of different fonts so that I use them for training Neural Networks and test other OCR techniques. Since I couldn’t find any I thought I would upload my own collection.

I used the below images when working on my thesis. From memory over 20 different fonts and sizes were used to create about 200 examples of each letter. The full data set proved to be pretty accurate when it came to recognizing most examples of text I found on the web.

Saving Resources for Humans in PHP

Ben E. C. Boyter — Tue, 27 Aug 2013 03:23:43 GMT

One of the issues I have run into running searchcode.com is that a huge amount of time spent serving pages is serving them to bots (about 900,000 hits a day are from bots). I have been watching the load averages and they can spike to over 30.00 occasionally which is pretty bad for a 4 core/ht system. I don’t have any problems with bot’s really other then the fact that I cannot really control them. Sure you can specify crawl delays but its of limited use if the bot chooses to ignore it.

The Rule of Thirds - followup

CRAIG KERSTIENS — Tue, 13 Aug 2013 23:55:56 GMT

Several months back I wrote about how we do higher level, long term planning within the Heroku Postgres team. If you haven’t read the previous article please start there.

The exercise or rule of thirds is intended to be approximate prioritization and not a perfect science. Since that time I’m familiar with some teams both in and out of Heroku who have attempted this exercise with varying levels of success. We’ve now done this process 4 times within the team and after the most recent exercise attempted to take some time to internalize why its worked well, creating some more specifics about the process. Heres an attempt to provide even more clarity:

Gather data ahead of time

Its really common to have a list things to work on, but knowing the impact of those is commonly pure speculation. There may be some people that talk to customers, but even then its a subset of your actual customer base. Going into the exercise as much data you can have ahead of time on impact of features and specific problems helps. In our case we do this by:

Surveying current customers and users
Surveying attriters
Engaging with customer facing teams to hear trends
Input from external parties such as analysts on trends

Allow for casual discussion

We typically conduct our planning exercise at an offsite, this is a multi-day time of team bonding, planning, hacking. We intentionally schedule our planning excercise towards the end of the offsite. This allows us to have updates/presentations frmo the data we’ve gathered and from those that are customer facing. Presentations are meant to be short and direct, discussion can flow casually after. This gets a lot of people on the same page at a smaller level and reduces the problem of too many cooks in the kitchen come time for the actual exercise.

The rule of thirds

Creating the list

Coming to the exercise itself… We begin by everyone writing a list of their ideas individually, this is meant to be a list of the features we want to place on the grid. At this point theres no prioritizing of difficulty or impact. In addition each list while individually created does not have to contain items that only pertain to you, its more a comprehensive list of all the things you can think of that may be important to do.

Bucketing part 1

Once individual lists are created you can then collectively or designate one or two people to clean it up. We do this in two forms:

Removing duplicate items, which there should be several of.
Bucketing my a common/theme idea, this simply makes things more digestable

If you’re a big group of greater than 7 then it may be advisable to designate two people to do this exercise together. If a smaller group it can be manageable to coordinate collectively.

Bucketing part 2

Once you’ve removed dupes, identified themes, and removed excess items (depending on your team size you’ll find how many feels right - we aim an average of 5-6 per square for a team of 10) its then on to actually putting them on the grid. In the past we’ve done this a variety of ways but our most recent process seemed to be quiet efficient. We gave each item 60 seconds, at the end of that minute wherever the item was it was left there. This forced some quick discussion on impact and difficulty but in the end left us at a very good hit rate without taking multiple hours to complete the exercise.

Final pass

We intentionally design it so that low effort and high impact is on the top right corner. Finally once everything is on there we allocate names to the tasks, and put boxes around items we’re planning to do in the coming months. With boxes make it very clear of what we are doing as well as explicitly things we are not. The initials or names make it clear of how loaded down people are. If your name is on 3 tasks that are high difficulty, then you’re likely over allocated.

At this point things usually fall out pretty quickly and we emerge with some rough roadmap that in retrospect we’ve followed pretty accuately.

C# Vector Space Implementation

Ben E. C. Boyter — Tue, 13 Aug 2013 08:59:09 GMT

Since I am writing lots of Vector Space implementations in Go, Python etc… I thought I would add another one in C#. This one is a little more verbose then either the Python or Go implementations. The verbosity is mostly due to not using any of the nice C# LINQ functionality which would really reduce the size.

In any case here it is in case you are looking for a simple implementation of this useful class.

GoLang Vector Space Implementation

Ben E. C. Boyter — Mon, 12 Aug 2013 09:51:30 GMT

UPDATE – This is now actually available as a real Golang import with tests. Get it at https://github.com/boyter/golangvectorspace

I have mentioned this before somewhere but one of the first things I usually attempt to implement in any programming language I want to play with is a vector space. Its my own personal FizzBuzz implementation. It usually covers everything I need to know in a language (imports, functions, string manipulation, math functions, iteration, maps etc…) so I consider it a good thing to get started with.

The missing PostgreSQL documentation

CRAIG KERSTIENS — Wed, 07 Aug 2013 23:55:56 GMT

For a couple of years I’ve complained about the Postgres documentation and at the same time paraded it as one of the best sets of documentation I’ve encountered. In many ways the reason I veer towards Postgres as well as Python and Django is the quality of their documentation. If you need to find details about something its documented, and more importantly well and thoroughly documented.

In large part I came to Python by happenstance through Django, and Postgres through happenstance of an employer. Yet, Django was very little of an accident. The Django Tutorial got me a large part of what I needed to know and more excited about development than I had been in some time. Python has done some work at adding docs to make this even better, sadly its still very much needed for PostgreSQL.

Whats Missing in the Postgres Docs

Theres a huge variety of types of documentation, off the top of my head theres:

Reference docs (Postgres excels at this)
Onboarding (Postgres tutorial huh?)
Tailored guides (Postgres? I can haz? Nope… We don’t understand….)

Postgres is great if you know the name of what you’re looking for, but if you don’t you’re entirely left in the dark.

Understanding the power of Postgres

Postgres is good enough at performance, good enough at usability, and awesome at how powerful and flexible it can be. But all of this is entirely lost if you have to know the esoteric name of what you’re looking for.

What the hell is an hstore… In so many ways KVstore makes infintely more sense. In the same sense PLV8, I have to know not only what PL stands for but V8 as well, versus the JavaScript extension for Postgres.

I understand there are plenty of reasons why some of these things are the way they are, but its also limiting how great the broader perception is. Postgres externally is this hard to use DB, that well is just a database, versus giving developers a set of powerful and useful functions to make their lives better.

The Solution

Lets fix things, there are a ton of people that would love to know more about all things Postgres. This ranges from a good set of onboarding docs, to specific blog posts on topics that people are curious about. Just last week I got an email about improving the Postgres tutorial… Yes theres a tutorial hidden in the 2000 page set of documentation for Postgres. Its simply old, mostly uninteresting, and well just needs to be completely recreated. A great alternative would be a few tutorials/guides for:

Noobs to databases in general (Total 101 guide)
Building and architecting your application with Postgres (App Devs)
Administering and maintaining Postgres (DBAs)
SQL and reporting in Postgres (consumers of data, analysts, product people, marketing, etc.)

If jumping in and contributing to fixing the core tutorial isn’t your cup of tea because you don’t want to learn and write in SGML, send a pull request to postgresguide.com or do a [guest post on my blog](mailto:craig.kerstiens@gmail.com]. If thats too much effort please just let us know, what do you want to see - craig.kerstiens at gmail.com

A look at Foreign Data Wrappers

CRAIG KERSTIENS — Mon, 05 Aug 2013 23:55:56 GMT

There are two particular sets of features that continue to keep me very excited about the momentum of Postgres. And while PostgreSQL has had some great momentum in the past few years these features may give it an entirely new pace all together. One is extensions, which is really its own category. Dimitri Fontaine was talking about doing a full series just on extensions, so here’s hoping he does so I dont have to :)

One subset of extensions which I consider entirely separate is the other thing, which is foreign data wrappers or FDWs. FDWs allow you to connect to other data sources from within Postgres. From there you can query them with SQL, join across disparate data sets, or join across different systems. Recently I had a good excuse to give the postgres_fdw a try. And while I’ve blogged about the Redis FDW previously, the Postgres one is particularly exciting because with PostgreSQL 9.3 it will ship as a contrib module, which means all Postgres installers should have it… you just have to turn it on.

Let’s take a look at getting it setup and then dig into it a bit. First, because I don’t have Postgres 9.3 sitting around on my system I’m going to provision one from Heroku Postgres:

$ heroku addons:add heroku-postgresql:crane --version 9.3

Once it becomes available I’m going to connect to it then enable the extension:

$ heroku pg:psql BLACK -acraig
# CREATE EXTENSION postgres_fdw;

Now its there, so we can actually start using it. To use the FDW there’s four basic things you’ll want to do:

Create the remote server
Create a user mapping for the remote server
Create your foreign tables
Start querying some things

The setup

You’ll only need to do each of the following once, once you’re server, user and foreign table are all setup you can simply query away. This is a nice advantage over db_link which only exists for the set session. One downside I did find was that you can’t use a full Postgres connection string, which would make setting it up much simpler. So onto setting up our server:

# CREATE SERVER app_db
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (dbname 'dbnamehere', host 'hostname-here);

Next we’ll actually create our user mapping. In this case we’ll take the remote username and password and map it to our current user we’re already connected with.

# CREATE USER MAPPING for user_current
SERVER app_db
OPTIONS (user 'remote_user', password 'remote_password');

And finally we’re going to configure our tables. There were some additional pains here as there wasn’t a perfectly clean way to generate the CREATE TABLE. Sure you could pg_dump just that table, but overall it felt a bit cludgey.

# CREATE FOREIGN TABLE users
(
id integer,
email text,
created_at timestamp,
first_name text,
last_name text
)
SERVER app_db OPTIONS (table_name 'users')

Now we’ve got all of our local data, as well as remote data. For that report against two databases where you previously wrote a ruby or python script, ran a query, constructed another query, then executed it you can directly do in your database. We can simply query our new table - SELECT * FROM users LIMIT 5;

But the real power of foreign data wrappers goes well beyond just Postgres to Postgres. Having a defined contract in translating from one system to another, will really allow reinventing the way we work with data. This is especially true in large datasets where doing ETL on terrabytes of data takes longer than asking the questions of it.

While we’re waiting for more FDWs to be ready to use in production situations the Postgres FDW is a great start, though the Redis one is on its way. Even better is that it ships with standard installs of Postgres, meaning it will see more usage and help push them to advance further.

One final nicety, you’re not required to have ALL Postgres 9.3 DBs, just one that can then connect to the others, so go ahead and give it try :)

EverQuest Next Revealed

Ben Overmyer's Site — Sat, 03 Aug 2013 03:00:00 GMT

I'm writing this just a couple minutes after watching the EverQuest Next reveal video on Twitch. I couldn't watch it live yesterday, but it was totally worth the wait. For an hour and a half, I was on the edge of my chair. The introduction was a beautiful piece of sand performance art, which I otherwise won't talk about. The rest of the video was an introduction into what will probably be the single most revolutionary MMORPG to come out since 2003.

Dave Georgeson, director of development for EverQuest Next, started things off with a reveal of the art style. Let me tell you, this is a gorgeous game. The whole thing is very painterly, and is rendered using a modified version of the engine powering Planetside 2.

All of the races have been redone, and we were treated to concept art for several: the High Elf, the Dark Elf, the Human, the Dwarf, and the Kerran. The Dark Elf and the Kerran were probably the biggest departure from previous EQ designs. Kerrans are distinctly more lionesque, with pronounced lion's heads instead of the old cat faces and huge builds. Dark elves look like standard dark elves, except they have draconic eyes and short horns. There's a story behind that that I really want to hear.

The art style for characters is exaggerated, even slightly cartoony. I'm really OK with that, especially since they're building SOEmote into the game from the beginning. It's easy to see the expressions of the players themselves on their characters, and that's marvelous for a roleplayer, but also for immersion and socializing in general.

Movement is fantastic. Characters slide, leap, vault, grab, and otherwise interact with the terrain in a very natural way. With EQN's “active movement system,” players are free to really enjoy moving around in the world. There are items and spells that allow additional ways to move, like the Boots of the Zephyr in the video that allowed the players to float gently downward. Brilliant stuff.

Then, we got into the Four Grails, as Dave put it.

The Core Game

Grail 1 is the “Core Game.” This was the least impressive of the Grails, but it was still nice to see. Character evolution from a mechanical standpoint is based on collecting classes from around the world and mixing and matching abilities from them. It sounds like a hybrid of Guild Wars 2's system and Rift's system. Weapon choice matters, like it does in GW2.

Destructibility

Grail 2 is “Destructibility,” and this was the first mind-blowing revelation. The entire game world is built of voxels, despite looking natural and perfectly designed. It's fully destructible, even though it heals over time. It's like if SOE took a look at Minecraft and applied that on a grand scale. The way it works, too, is gorgeous. You can definitely see the aftermath of epic battles in the environment that it occurred in.

A Life of Consequence

Grail 3 is “A Life of Consequence.” Besides the standard “every action you take matters” spiel, we were introduced to the idea of emergent AI in an MMO. SOE will create mob behavioral patterns - likes, dislikes, etc. - and then release mobs of that type into the world. They will live, react, and die according to how they're programmed to behave. This means it's entirely possible to have roving bands of orcs that actually decide to set up camp in a particular area - choose their own spawn point, as it were. It's also possible to drive them out.

Permanent Change

Grail 4 is “Permanent Change.” For the most part, Dave talked about the Rallying Cry system, which is a little like a server-wide public quest that lasts 2-3 months each… and has permanent effect on the game world. These Rallying Cries are promised to be different on every server, and that along with the Life of Consequence and Destructibility Grails ensure that no two servers will have the same environment and history, let alone community.

EverQuest Next Landmark

As if all this wasn't enough, then they got into the EverQuest Next Landmark game that's coming out later in 2013. Landmark sounds basically like Minecraft on an epic scale, except that one continent in each Landmark server will have EverQuest Next art direction enforced… and creations from those areas will make it into EverQuest Next as real, lasting landmarks.

Player Studio

On top of that, SOE's Player Studio will find a very interesting real money economy implementation for people who enjoy building things in Landmark; you'll be able to sell not only finished works, but components. The example that Dave gave was of a player creating a nice crenelated tower. People could buy that tower, then make castles from it and sell those castles, and the player who created the tower would get royalties based on how many of that tower were used in the castle design for every castle sold. It's a fascinating system.

Basically, EverQuest Next sounds like the game I've been waiting for my entire gaming life. I really want to hear about how the crafting system works, and more about all the races, but… it sounds awe-inspiring. EverQuest Next takes player involvement in the game to a level never before seen.

EverQuest Next - Landmark Focuses on Construction and Exploration, Is Free-to-Play

EverQuest Next + Jeremy Soule = Geek Delight

EverQuest Next Redefines Next-Gen MMOs Enhanced by Zemanta

Postgres Dollar Quoting

CRAIG KERSTIENS — Fri, 02 Aug 2013 23:55:56 GMT

After my most recent post on documenting your database I had a colleague and friend chime in:

{% blockquote @danfarina https://twitter.com/danfarina/status/362007008079126528 %} @craigkerstiens You may want to mention for another post the generality of dollar quoting: it’s not just for CREATE FUNCTION. {% endblockquote %}

Luckily I was able to convince him to create the post. You can read a bit more on him below, but without further adieu here’s a bit on dollar quoting within Postgres:

Postgres supports two forms of entry of data literals into the system. One is the familiar single-quote:

=> SELECT 'hello';
?column?
----------
hello
(1 row)

This format is problematic when one might be using single quotes in the textual string.

Postgres also supports another way to enter data literals, most often seen in CREATE FUNCTION, but can be profitably used anywhere. This is called “dollar quoting,” and it looks like this:

=> SELECT $$hello's the name of the game$$;
?column?
------------------------------
hello's the name of the game
(1 row)

If one needs nested dollar quoting, one can specify a string, much like the ‘heredoc’ feature seen in some programming languages:

=> SELECT $goodbye$hello's the name of the $$ game$goodbye$;
?column?
---------------------------------
hello's the name of the $$ game
(1 row)

This can appear anywhere where single quotes would otherwise be, simplifying tasks like using contractions in database object comments, for example:

=> CREATE TABLE described(a int);
=> COMMENT ON TABLE described IS $$I'm describing this,
including newlines and an apostrophe in the contraction "I'm."$$;

Or, alternatively, entry of literals for types that may include apostrophes in their serialization, such as ’text’ or ‘json’:

=> CREATE TABLE json(data json);
=> INSERT INTO json(data) VALUES
($${"quotation": "'there is no time like the present'"}$$);

Security

Even though dollar quotes can be used to reduce the pain of many quoting problems, don’t be tempted to use them to avoid SQL injection: an adversary that knows one is using dollar quoting can still mount exactly the same kind of attacks as if one were using single quotes.

There is also no need, because any place a data literal can appear can also be used with parameter binding (e.g. $1, $2, $3…), which one’s Postgres driver should support. Nevertheless, for data or scripts one is working with by hand, dollar quoting can make things much easier to read.

About the Author

Daniel Farina is a long time colleague and friend, having worked together at 5 different companies. He’s part of the Heroku Postgres team as the resident tuple groomer, and the creator of WAL-E.

As is always the case if you have articles you’d like to see created or if you’re interested in doing a guest post please feel free to drop me a line craig.kerstiens at gmail.com. And if you have articles you feel are helpful to others in the Postgres world drop me a note as well for including them in Postgres Weekly.

Automatically inline Python function calls

Tom Forbes — Thu, 01 Aug 2013 18:54:18 GMT

Edit: Code is here on GitHub Calling functions in Python can be expensive. Consider this example: there are two statements that are being timed, the first one calls a function that returns an integer while the second one calls a function that returns the result of a second function call which return...

Clean Repository Data Access in C#

Ben E. C. Boyter — Thu, 01 Aug 2013 02:52:14 GMT

Mostly as a self reference here is an extremely clean data access pattern possible using C# and Entity Framework. It saves you the effort of mocking the database context as the code you end up writing is so simple it is all compile time checked.

Essentially you define a very simple class which provides a single method for getting data (although you may want a save data method too) and make sure you add an interface to make unit testing/mocking easier.

Documenting your PostgreSQL database

CRAIG KERSTIENS — Mon, 29 Jul 2013 23:55:56 GMT

Just a few days ago I was surprised by what someone was doing with their database, and not in the typical horrifying travesty against mankind. Rather, it was a feature that while familiar with I’d never seen anyone fully take proper advantage of - COMMENT or describing tables. Postgres has a nice facility for you to provide a description for just about anything:

Table
Column
Function
Schema
View
Index
Etc.

The specific use case was a database acting as a datamart pulling in data from multiple sources to be able to report against disparate data. Over the years I’ve seen this occur really one three ways, the first is that a limited set of people, typically one person, have knowledge over all the datasources and thus far the sole individual responsible for creating reports and answering questions of the data. The second, is wide open access to anyone that wishes for it. In this case you often have people asking questions of the data, and because they don’t understand the relationships coming up to entirely wrong conclusions. The final approach is to create some external documentation, entity relationship diagrams, data dictionaries, etc. This last one often works okay enough, but often suffers from lack of updates and being too heavyweight.

A better solution, and all around good process is simply documenting clearly within the database itself. Simply comment each table and column, just as you would outside of your DB then it can be quite clear when inside the database working interactivly:

COMMENT ON TABLE products IS 'Products catalog';
COMMENT ON COLUMN products.price is 'Current price of a single item purchased';

While an obvious example above naming even the most mundance columns can help create more accurate reports. Then of course when you want to inspect your DB its quite clear:

\d+ users
# \d+ users
Table "public.users"
Column | Type | ... | Description
------------+-----------------------------+-...-+-----------------------------------------
id | integer | ... | auto serial pk
first_name | character varying(50) | ... | required first name of user
last_name | character varying(50) | ... | required first name of user
email | character varying(255) | ... | email address of account
data | hstore | ... | mix of data, city, state, gender
created_at | timestamp without time zone | ... | when account was created, not confirmed
updated_at | timestamp without time zone | ... | time any details were last updated
Indexes:
"idx_user_created" btree (date_trunc('day'::text, created_at))
Has OIDs: no

But it doesn’t necessarily have to stop there. Which actually brings me to one other item, you should be commenting your SQL just the same. SQL comments can be done easily by just starting a line with --, or you can have it at the end of the line with further info. Here’s a nice example:

-- Query aggregates all project names that have open past due tasks grouped by email
SELECT
users.email,
array_to_string(array_agg(projects.name), ',')) as projects # Aggregate all projects and separate by comma
FROM
projects,
tasks,
users
-- A user has a project, which has tasks
WHERE projects.id = tasks.project_id
-- Check for tasks that are due before now and not done yet
AND tasks.due_at > tasks.completed_at
AND tasks.due_at < now()
AND users.id = projects.user_id
GROUP BY
users.email

You comment your code, why shouldn’t you comment your database?

Gigablast Aquired and Code Posted

Ben E. C. Boyter — Mon, 29 Jul 2013 08:58:30 GMT

Interestingly it seems that Matt Well’s search engine Gigablast has been acquired by Yippy.com [1]1 [2]2 [3]3 (demo here http://demo.yippy.com/). Gigablast has always been one of my favorite search engines simply because it is so interesting. Started by a single guy, with an interesting blog and being one of the last true new indexes of the web it was always worth a look. While its sad to see it go this way I am happy that Matt presumably has been able to cash out on his creation. Well done to him. I must admit Gigablast had been going downhill for a while and this might explain why ProCog appeared and then vanished so quickly.

Reawakening the Starborn

Ben Overmyer's Site — Sat, 27 Jul 2013 03:00:00 GMT

About a year and a half ago, I started a project to create a sci-fantasy sandbox MMORPG. It was called Starborn: Universe. I went through the trouble of getting a HeroEngine/HeroCloud environment and began learning the toolset. Design documents popped up in my Documents folder and on the dev blog I started for the project. Then, I got busy with other things, and Starborn: Universe went dark. Until now. I'm dusting off the design documents, but Starborn: Universe is as much of an experiment as it is a game. See, my theory is that online player-driven games can start with very little and grow over time. The idea of the “Lean Startup” can apply just as much to an MMORPG as it can to an iOS app. The trick is that MMORPGs are hard to produce. They have a lot of moving parts and require a lot of resources on the part of the developer and operator. So how does one or two people accomplish this monumental feat? By starting small, and iterating quickly. If you're curious as to what I have in mind, head over to the Starborn: Universe devlog. That's where I'll be writing about Starborn.

CoffeeScript's Scoping is Madness

Donat Studios — Fri, 26 Jul 2013 01:19:44 GMT

CoffeeScript is a programming language that compiles to JavaScript. Its mission is to "expose the good parts of JavaScript in a simple way". I've heard a lot of interest in it lately from various channels.

After a coworkers departure, I inherited maintenance on our deployment bot, and it was written in CoffeeScript. While trying to make some small modifications, I ended up getting unexpected issues I didn't understand. Things I did not touch were suddenly breaking. I did not understand until I investigated the generated JavaScript.

It ends up CoffeeScripts approach to scoping of variables is to not scope variables. There is no shadowing. There is no var, let nor const equivalent, there is simply no way to explicitly define scope.

Variables in CoffeeScript are scoped to their outermost definition, regardless of whether the name was already in use elsewhere. When you use a variable of a name, its scope now includes all inner scopes, trampling any existing definitions of the variable in those scopes.

Let's examine the following examples:

CoffeeScript

test = (x) ->
  y = 10
  x + y;

alert test 5

Generated JavaScript

var test;

test = function(x) {
  var y;
  y = 10;
  return x + y;
};

alert(test(5));

Now let's make one small change and define a variable y in the outer scope, outside the function.

CoffeeScript

y = 0;
test = (x) ->
  y = 10
  x + y;

alert test 5

Generated JavaScript

var test, y;

y = 0;

test = function(x) {
  y = 10;
  return x + y;
};

alert(test(5));

The Problem

Notice the highlighting indicating the scope of the y variable. By defining y in the global scope all subsequent y's are also now that global. You may no longer have a variable named y in your code, at any deeper scope. The outer y is the only y.

Creating a global of a name given overrides all subsequent definitions.

In JavaScript var, let, and const keywords define the scope of a variable. Leave it out and it scopes downward until it hits a var definition of that variable or the global level. This means that inner scopes inherits from outer scopes unless it defines its own scope of the variable. CoffeeScript on the other hand offers you no method to scope a variable. Scoping is entirely left to the lowest level use of a variable name.

This behavior is dangerous and makes code very difficult to maintain. The problem is twofold. Firstly, when writing a function, you need to be aware of all shallower scopes variable definitions, as not to trample them. Secondly, when altering code at any scope but the outermost, you need to be aware of all deeper scopes variables as to not trample any of them.

The only "safe" way to write CoffeeScript is to assume all variable names are global. In practice they are.

There have been proposed solutions, such as a Go-lang style := for inner scoping, but these have not been accepted as of this writing. The creator of CoffeeScript, Jeremy Ashkenas, when questioned about it by Armin Ronacher on Twitter replied:

Not gonna happen ;) Forbidding shadowing altogether is a huge win, and a huge conceptual simplification.

This is not true for applications of any scale larger than toy. It makes development more complex, having to check every inner scope before using a new variable name. This is not simpler. The entire behaviour of all inner scopes can change with a single bad variable defintion.

Finally

CoffeeScripts goal of "expos[ing] the good parts of JavaScript in a simple way" was missed. Instead of improving JavaScript's scoping which can be kind of daunting to learn, they actually made it worse. If you want to improve JavaScript make every variable local until explicitly defined global, as is the behavior in the vast majority of languages.

JavaScripts "global unless defined as local" behavior is weird, but manageable. CoffeeScript's "local until defined elsewhere globally" behavior on the other hand is simply difficult to predict and makes it easy to accidentally change the entire meaning of code even when you entirely understand the behavior.

It is a bad behavior, period.

Update

I've a fight on Reddit here.
I've irked Reginald Braithwaite, the author of CoffeeScript Ristretto, so badly he's giving away 100 copies of his book for free!
- The author has upgraded me to Not Wrong™!

SSDs are awesome, buy one.

Tom Forbes — Thu, 25 Jul 2013 22:54:58 GMT

I recently brought a Samsung 840 Series Pro 256GB 2.5 inch SATA Solid State Drive and its easily the best PC hardware purchase I have ever made. Before I purchased it I was thinking about replacing my laptop as it was getting pretty sluggish, despite having decent specs even 2 years after I purchase...

Why I still recommend Windows

Nemo's Home — Sun, 21 Jul 2013 03:00:00 GMT

Even though I am a long time Linux user, and a big time fan of the many Linux distros that I’ve tried out over time, I still go around recommending Windows to people who ask me for advice. The only exception I make is when the person in question is a developer, in which case I try to convert them to the Church of Linux.

The main reason I recommend windows to non-developers is because it is a far better operating system than most Linux distributions (for general public). Now, before you bring out your pitchforks, hear me out.

The first and foremost reason that I give is that Windows sports a far better integration across all its services. Nautilus/Nemo in the Linux world do not reach same level of integration that Windows Explorer does. For instance, just look at the way the “Send To” feature works in explorer. To add a folder to the send to entry, you just have to add a shortcut to that folder inside the special “Send To” folder. On nautilus, the equivalent would be going about installing an extension, and editing a configuration file by hand.

Or take a look at how the “Play All” feature in Explorer. Or the “Libraries” feature in Windows 7. Or the simple way that you handle file sharing in Windows. Even though Linux has (arguably better) Samba support for files sharing, you have to go about editing a handful of files to make it work. I personally find apache easier to configure to just share files one way.

Games are another reason. Even though Steam is available on Linux, all of the non-Valve triple-A titles are missing on Linux. Even though I continue to buy and play the Humble Bundles that offer Linux as a platform, I’m reminded of the stark reality every day when my friends ask me if I’ve played a recent title such as Call of Duty, Metro, NFS or even Swapper.

The next peeve that comes to my mind is the ridiculous driver support. It has been improving since a long time, but its still not there. Even Ubuntu needs to fetch proprietary drivers for my WiFi support on Broadcom, which needs an internet connection in the first place. This means I need to find out a LAN network connection to even start using Ubuntu. Similarly the pain I’d to go through to install drivers for Ralink network drivers on a friend’s laptop was immense. I can never use circular scrolling or touchpad zoom on my laptop in Linux because there are still no drivers available anywhere for it. And don’t get me started on UEFI boot issues. No matter what people believe or pretend, hardware support is just not good enough to be relied on in the Linux world. As a side note, I haven’t synced my iPad in Ubuntu since I shifted to iOS 5, and apple driver support on Linux will remain abysmal forever just because iTunes will never be released for Linux. The last version of iOS that had music sync support (via libimobiledevice) was iOS 4.0 (released 3 years ago in June 2010).

Next I want to point put out the upgrade pain that everyone has to go through. It’s like a constant rite of passage, which turns a Linux noob into an actual user. I am yet to do an Ubuntu upgrade which went smoothly and didn’t break a thing; and I’ve been upgrading my Ubuntu since 10.04 was released. The upgrade tag on askubuntu is chock full of horror stories.

Another thing that frustrates me to no end is that the Ubuntu Dash, and the GNOME Overview are both slow as hell. I’m currently using Cinnamon, which is faster than both of these, but still an order of magnitude slower than the Windows Start Menu. Synpase is better, but cannot be set as the deafult.

I was using Windows 7 on my cousin’s laptop these last few days and I remembered the favourite app that I used to no end: Everything. It is the quickest file search I’ve ever used. The alternatives, in the Linux world are synapse, zeitgeist, and plain old locate command. The only issue is that I’ve to manually run updatedb manually, while Everything was always up to date, using the NTFS File Journal. To this date, I am yet to find a good enough alternative to Everything.

It is true that Windows lacks many of the good things that Linux distros provide, such as the excellent package management support, POSIX compatibility, and the plethora of tools we get on the command line; but at the same time, it is also a better operating system for most of the masses. I’ll continue to recommend Windows to all my non-developer friends till “The year of Desktop Linux” arrives.

Returning to Skyrim

Ben Overmyer's Site — Sat, 20 Jul 2013 03:00:00 GMT

The last time I posted about Skyrim, it was shortly after release. You can read the first in the series here. At the time, I was playing on the Xbox 360 version. This time around, I've purchased the Legendary Edition for PC. I'm also playing a Destruction mage instead of the heavily armed and armored warrior I focused on in the 360 version.

Besides taking a completely different character route this time around, I'm also focusing much more on quests other than the main line. I own a small house that I've built but not yet furnished, gotten a fair ways through the College of Winterhold quest line, and continue to light up vampires that seem to crop up all the time in areas I'm traveling to.

I heard so many awesome things about mods for the game that I just had to get the PC version. That, and I couldn't take screenshots in the 360 version, which was infuriating for someone who blogs about games. Today, I read an article someone reblogged on Reddit about a mod called Falskaar. Falskaar is unique in that it was produced chiefly by a 19 year old who declined to go to university in order to produce the mod. It took him a full year, and he's aiming to get a job at Bethesda by proving his dedication.

By all accounts, it's a damned good mod. I look forward to installing it when I finish with vanilla Skyrim.

If you're interested in hearing further about my adventures in Skyrim, by all means, let me know in the comments. Otherwise, next week I'll be writing about a topic near and dear to my heart…. sci-fantasy sandbox MMORPGs.

List of useful CAPTCHA Decoding Articles

Ben E. C. Boyter — Fri, 19 Jul 2013 02:46:43 GMT

This website ranks quite high in most search engines for the search term “captcha decoding” or some permutation of it. As such here are a collection of useful links if you are looking into doing such a thing. If any more come up I will be sure to update this post.

http://www.boyter.org/decoding-captchas/

Shameless self promotion but this link is why this page ranks so highly. Its an article I wrote some time ago about how to go about decoding a simple CAPTCHA. There is full source code and the principles can be applied to 90% of CAPTCHA’s out there. For the record it only came about because a colleague bet me that I couldn’t decode his websites CAPTCHA which was the one used in the article. Of course I waited till he changed it before publishing.

Displaying a processes output on a web page with Websockets and Python

Tom Forbes — Mon, 15 Jul 2013 21:40:38 GMT

A few days ago a colleague of mine asked me how you would pipe the standard output of a process into a browser. I hacked around for a few hours and came up with a websockets based solution (using Twisted and Autobahn.ws ) that you can see below ( Your browser needs to support WebSockets, sorry IE9 a...

About

Ben E. C. Boyter — Thu, 11 Jul 2013 07:02:47 GMT

I’m a software developer by trade who after years of leaving articles and blog posts all over the place has finally gotten around to consolidating them all into this single site.

I am a professional software engineer who has been working in development with a focus on testing for over 15 years. In that time I have played an integral part in developing and implementing testing methodologies for all sorts of projects including,

Decoding CAPTCHA’s

Ben E. C. Boyter — Thu, 11 Jul 2013 06:00:02 GMT

Looking for a practical guide to CAPTCHA decoding? All About CAPTCHA’s. This eBook will teach you how to identify weaknesses and exploit CAPTCHA’s from beginning to end. Buy now using Leanpub

Most people don’t know this but my honors thesis was about using a computer program to read text out of web images. My theory was that if you could get a high level of successful extraction you could use it as another source of data which could be used to improve search engine results. I was even quite successful in doing it, but never really followed my experiments up. My honors adviser Dr Junbin Gao http://csusap.csu.edu.au/~jbgao/ had suggested the following writing my thesis I should write some form of article on what I had learnt. Well I finally got around to doing it. While what follows is not exactly what I was studying it is something I wish had existed when I started looking around.

hstore vs. JSON - Which to use in Postgres

CRAIG KERSTIENS — Wed, 03 Jul 2013 23:55:56 GMT

If you’re deciding what to put in Postgres and what not to, consider that Postgres can be a perfectly good schema-less database. Of course as soon as people realized this then the common comes a question, is hstore or JSON better. Which do I use and in what cases. Well first, if you’re not familiar check out some previous material on them:

If you’re already up to date with both of them, but still wondering which to use lets dig in.

hstore

hstore is a key value store directly within your database. Its been a common favorite of mine and has been for some time. hstore gives you flexibility when working with your schema, as you don’t have to define models ahead of time. Though its two big limitations are that 1. it only deals with text and 2. its not a full document store meaning you can’t nest objects.

Though major benefits of hstore include the ability to index on it, robust support for various operators, and of course the obvious of flexibility with your data. Some of the basic operators available include:

Return the value from columnfoo for key bar:

foo->'bar'

Does the specified column foo contain a key bar:

foo?'bar'

Does the specified column foo contain a value of baz for key bar:

foo@>'bar->baz'

Perhaps one of the best parts of hstore is that you can index on it. In particular Postgres gin and gist indexes allow you to index all keys and values within an hstore. A talk by Christophe Pettus of PgExperts actually highlights some performance details of hstore with indexes. To give away the big punchline in several cases hstore with gin/gist beats mongodb in performance.

json

JSON in contrast to hstore is a full document datatype. In addition to nesting objects you have support for more than just text (read numbers). As you insert JSON into Postgres it will automatically ensure its valid JSON and error if its well not. JSON gets a lot better come Postgres 9.3 as well with some built in operators. Though if you need more functionality in it today you should look at PLV8.

Which to Use

So which do you actually want to use in your application? If you’re already using JSON and simply want to store it in your database then the JSON datatype is often the correct pick. However, if you’re just looking for flexibility with your data model then hstore is likely the path you want to take. hstore will give you much of the flexibility you want as well as a good ability to query your data in a performant manner. Of course much of this starts to change in Postgres 9.3.

10 Years Ago, in an MMO far, far away

Ben Overmyer's Site — Sat, 29 Jun 2013 03:00:00 GMT

Ten years ago nearly to the day, Star Wars: Galaxies launched to the world. As a remembrance of the adventures I had there, I'm writing now about the game as it was in those days and the community that surrounded it. As you read this, you may wish to listen to the playlist on YouTube that starts with the above video. It's a wonderful collection of the music from the game, familiar audio glitches included.

The features page for the first SWG website

One of the things I remember reading first about Star Wars: Galaxies was an interview with lead world artist John Roy about procedural planet generation, and how he would discover features of the landscape that were new even to the designers. That got me really excited about exploring these worlds that I'd only read about or seen on the big screen. I'm not sure how often this technique is employed in modern MMORPGs. My impression is that the vast majority of current worlds are slowly crafted by hand by artists, which is why we have so many beautiful vistas but so few boring areas to contrast them with. If all we ever see is beauty, we start to tire of it and look for something better.

I miss the rolling hills of Corellia where my first SWG toon “grew up.” Another thing that I truly miss about Star Wars: Galaxies is the wait times at the starports. You could only travel to another planet if you bought a ticket for a starship and waited at a starport for one. They arrived in ten minute intervals, which means frequently large groups of players would congregate at the main starports while waiting to travel to another planet. It made for some great conversations, and new friends.

A starport in Star Wars: Galaxies

Something else I miss about Star Wars: Galaxies was the grind. As a crafter who played before the concept of “practice mode” was introduced, I would sit there and grind out components to level up my skill for hours. When I ran out of materials or empty space in my inventory, I'd make a backpack, shove the extra items in there, and head out to my personal harvesters for more materials. This was before anyone was able to build medium or heavy harvesters, so there were still vast fields of personal harvesters dotting the landscape.

A personal mineral extractor

The grind, at least for SWG crafting, meant long periods of sitting there doing the exact same motions over and over again. This sounds really boring, but frequently after about five minutes of doing this, I would hit a kind of altered state of mind - a vaguely pleasant feeling of just “being.” The only other time I've ever felt that is when practicing Zen meditation, and I sorely miss it.

Before vehicles were introduced, everyone had to walk everywhere. Shuttles and starports were jammed full of players on their way someplace. Player cities didn't exist yet, but there were always massive collections of player houses exactly 1 km outside of every major NPC city. It made those cities feel a lot like real cities - you would always be passing by other players on your way either into the city core for transport, into the suburbs for shopping or socializing, or out into the wilderness for adventuring and hunting. Every city had its own distinctive feel, too, on my home server of Chilastra.

Corellia had some beautiful vistas

Coronet on Corellia was where I first landed, and it's where I spent the first few months of the game. It felt very rugged, like a city on the fringe. Theed, on Naboo, was where the entertainers and image designers hung out. It was definitely a “pretty” city. Mos Eisley was… well, a desert city. Because of all the difficult, unbuildable terrain surrounding Mos Eisley, it wasn't nearly as dense as the others. I think the players more nostalgic for A New Hope landed there. The other cities, like Restuss on Rori and the Imperial Outpost on Talus, had much smaller and more focused populations. Later, when they allowed players to build on some of the more advanced planets like Dantooine and Lok, more interesting communities started springing up.

In the end, Star Wars: Galaxies really was the best MMORPG experience I've ever had. The early months, even with all the bugs, network issues, and missing content (capes!?), are permanently etched in my memory as a wonderful time. Rest in peace, SWG.

Pivoting in Postgres

CRAIG KERSTIENS — Thu, 27 Jun 2013 23:55:56 GMT

Earlier today on an internal Heroku group alias there was a dataclip shared. The dataclip listed off some data grouped by a category, there was a reply a few minutes later with a modification to the query that used the crosstab function to pivot directly in SQL. There were immediately several reactions on the list that went something like this:

While a mostly simple function in Postgres (there are a few rough edges), it really is all too handy. So here it is in action. Taking some data that looks like

row identifier, in this case date
category grouping, in this case OS
value

Given a really basic query that generates some sample data it may look something like this:

SELECT generate_series AS date,
b.desc AS TYPE,
(random() * 10000 + 1)::int AS val
FROM generate_series((now() - '100 days'::interval)::date, now()::date, '1 day'::interval),
(SELECT unnest(ARRAY['OSX', 'Windows', 'Linux']) AS DESC) b;

You get results that look like:

But of course this isn’t overly helpful in comparing day to day overall. You can do so on a OS by OS basis, but its annoying enough as is. The easy solution is to simply use a pivot table on your data. Most people at this point would pull it up into Excel or Google Docs, or you can do it directly in Postgres. To do so you’ll first enable the extension tablefunc:

CREATE EXTENSION tablefunc

Then you’ll use the crosstab function. The function looks something like:

SELECT *
FROM crosstab(
'SELECT row_name, category_grouping, value FROM foo',
'SELECT category_names FROM bar')
AS
ct_result (category_name text, category1 text, category2 text, etc.)

Lets see it an actual action. Given the same query we used to generate fake data we can actually pivot on it now directly in PostgreSQL:

SELECT *
FROM crosstab(
'SELECT
a date,
b.desc AS os,
(random() * 10000 + 1)::int AS value
FROM generate_series((now() - ''100 days''::interval)::date, now()::date, ''1 DAY''::interval) a,
(SELECT unnest(ARRAY[''OSX'', ''Windows'', ''Linux'']) AS DESC) b ORDER BY 1,2
','SELECT unnest(ARRAY[''OSX'', ''Windows'', ''Linux''])'
)
AS ct(date date, OSX int, Windows int, Linux int);

And see some results:

Have fun analyzing your data directly in your DB now. And as always if you have feedback/questions/requests please feel free to drop me a line craig.kerstiens@gmail.com

Javascript Functions for PostgreSQL

CRAIG KERSTIENS — Tue, 25 Jun 2013 23:55:56 GMT

Javascript in Postgres has gotten a good bit of love lately, part of that is from Heroku Postgres recently adding support for Javascript and part from a variety of people championing the power of it such as @leinweber (Embracing the web with JSON and PLV8) and @selenamarie (schema liberation with JSON and PLV8). In a recent conversation it was pointed out that it seems a bit of headache to have to create your own functions, or at least having an initial collection would make it that much more powerful. While many can look forward to PostgreSQL 9.3 which will have a bit more built in support for JSON a few functions can really help make it more useful today.

These are courtesy of Will Leinweber. For each of the following functions I’ll highlight an example of using it as well. To get an idea of the data its being run on:

select * from example;
data
--------------------------------------------
{"name":"Craig Kerstiens", +
"age":27, +
"siblings":1, +
"numbers":[ +
{"type":"work", +
"number":"123-456-7890"}, +
{"type":"home", +
"number":"456-123-7890"}]}
(1 row)

get_text

CREATE OR REPLACE FUNCTION
get_text(key text, data json)
RETURNS text AS $$
return data[key];
$$ LANGUAGE plv8 IMMUTABLE STRICT;

Then using the function:

select get_text('name', data) from example;
get_text
----------------
Craig Kerstiens
(1 row)

get_numeric

CREATE OR REPLACE FUNCTION
get_numeric(key text, data json)
RETURNS numeric AS $$
return data[key];
$$ LANGUAGE plv8 IMMUTABLE STRICT;

Then using the function:

select get_numeric('siblings', data) from example;
get_text
----------------
1
(1 row)

json_select

create or replace function
json_select(selector text, data json)
returns json as $$
exports = {};
(function(a){function z(a){return{sel:q(a)[1],match:function(a){return y(this.sel,a)},forEach:function(a,b){return x(this.sel,a,b)}}}function y(a,b){var c=[];x(a,b,function(a){c.push(a)});return c}function x(a,b,c,d,e,f){var g=a[0]===","?a.slice(1):[a],h=[],i=!1,j=0,k=0,l,m;for(j=0;j<g.length;j++){m=w(b,g[j],d,e,f),m[0]&&(i=!0);for(k=0;k<m[1].length;k++)h.push(m[1][k])}if(h.length&&typeof b=="object"){h.length>=1&&h.unshift(",");if(u(b))for(j=0;j<b.length;j++)x(h,b[j],c,undefined,j,b.length);else for(l in b)b.hasOwnProperty(l)&&x(h,b[l],c,l)}i&&c&&c(b)}function w(a,b,c,d,e){var f=[],g=b[0]===">"?b[1]:b[0],h=!0,i;g.type&&(h=h&&g.type===v(a)),g.id&&(h=h&&g.id===c),h&&g.pf&&(g.pf===":nth-last-child"?d=e-d:d++,g.a===0?h=g.b===d:(i=(d-g.b)%g.a,h=!i&&d*g.a+g.b>=0));if(h&&g.has){var j=function(){throw 42};for(var k=0;k<g.has.length;k++){try{x(g.has[k],a,j)}catch(l){if(l===42)continue}h=!1;break}}h&&g.expr&&(h=p(g.expr,a)),b[0]!==">"&&b[0].pc!==":root"&&f.push(b),h&&(b[0]===">"?b.length>2&&(h=!1,f.push(b.slice(2))):b.length>1&&(h=!1,f.push(b.slice(1))));return[h,f]}function v(a){if(a===null)return"null";var b=typeof a;b==="object"&&u(a)&&(b="array");return b}function u(a){return Array.isArray?Array.isArray(a):b.call(a)==="[object Array]"}function t(a,b,c){var d=b,g={},j=i(a,b);j&&j[1]===" "&&(d=b=j[0],j=i(a,b)),j&&j[1]===f.typ?(g.type=j[2],j=i(a,b=j[0])):j&&j[1]==="*"&&(j=i(a,b=j[0]));for(;;){if(j===undefined)break;if(j[1]===f.ide)g.id&&e("nmi",j[1]),g.id=j[2];else if(j[1]===f.psc)(g.pc||g.pf)&&e("mpc",j[1]),j[2]===":first-child"?(g.pf=":nth-child",g.a=0,g.b=1):j[2]===":last-child"?(g.pf=":nth-last-child",g.a=0,g.b=1):g.pc=j[2];else{if(j[1]!==f.psf)break;if(j[2]===":val"||j[2]===":contains")g.expr=[undefined,j[2]===":val"?"=":"*=",undefined],j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0])),(!j||j[1]!=="(")&&e("pex",a),j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0])),(!j||j[1]!==f.str)&&e("sex",a),g.expr[2]=j[2],j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0])),(!j||j[1]!==")")&&e("epex",a);else if(j[2]===":has"){j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0])),(!j||j[1]!=="(")&&e("pex",a);var k=q(a,j[0],!0);j[0]=k[0],g.has||(g.has=[]),g.has.push(k[1])}else if(j[2]===":expr"){g.expr&&e("mexp",a);var l=o(a,j[0]);j[0]=l[0],g.expr=l[1]}else{(g.pc||g.pf)&&e("mpc",a),g.pf=j[2];var m=h.exec(a.substr(j[0]));m||e("mepf",a),m[5]?(g.a=2,g.b=m[5]==="odd"?1:0):m[6]?(g.a=0,g.b=parseInt(m[6],10)):(g.a=parseInt((m[1]?m[1]:"+")+(m[2]?m[2]:"1"),10),g.b=m[3]?parseInt(m[3]+m[4],10):0),j[0]+=m[0].length}}j=i(a,b=j[0])}d===b&&e("se",a);return[b,g]}function s(a){if(a[0]===","){var b=[","];for(var c=c;c<a.length;c++){var d=r(d[c]);b=b.concat(d[0]===","?d.slice(1):d)}return b}return r(a)}function r(a){var b=[],c;for(var d=0;d<a.length;d++)if(a[d]==="~"){if(d<2||a[d-2]!=">")c=a.slice(0,d-1),c=c.concat([{has:[[{pc:":root"},">",a[d-1]]]},">"]),c=c.concat(a.slice(d+1)),b.push(c);if(d>1){var e=a[d-2]===">"?d-3:d-2;c=a.slice(0,e);var f={};for(var g in a[e])a[e].hasOwnProperty(g)&&(f[g]=a[e][g]);f.has||(f.has=[]),f.has.push([{pc:":root"},">",a[d-1]]),c=c.concat(f,">",a.slice(d+1)),b.push(c)}break}if(d==a.length)return a;return b.length>1?[","].concat(b):b[0]}function q(a,b,c,d){c||(d={});var f=[],g,h;b||(b=0);for(;;){var j=t(a,b,d);f.push(j[1]),j=i(a,b=j[0]),j&&j[1]===" "&&(j=i(a,b=j[0]));if(!j)break;if(j[1]===">"||j[1]==="~")j[1]==="~"&&(d.usesSiblingOp=!0),f.push(j[1]),b=j[0];else if(j[1]===",")g===undefined?g=[",",f]:g.push(f),f=[],b=j[0];else if(j[1]===")"){c||e("ucp",j[1]),h=1,b=j[0];break}}c&&!h&&e("mcp",a),g&&g.push(f);var k;!c&&d.usesSiblingOp?k=s(g?g:f):k=g?g:f;return[b,k]}function p(a,b){if(a===undefined)return b;if(a===null||typeof a!="object")return a;var c=p(a[0],b),d=p(a[2],b);return l[a[1]][1](c,d)}function o(a,b){function c(a){return typeof a!="object"||a===null?a:a[0]==="("?c(a[1]):[c(a[0]),a[1],c(a[2])]}var d=n(a,b?b:0);return[d[0],c(d[1])]}function n(a,b){b||(b=0);var c=m(a,b),d;if(c&&c[1]==="("){d=n(a,c[0]);var f=m(a,d[0]);(!f||f[1]!==")")&&e("epex",a),b=f[0],d=["(",d[1]]}else!c||c[1]&&c[1]!="x"?e("ee",a+" - "+(c[1]&&c[1])):(d=c[1]==="x"?undefined:c[2],b=c[0]);var g=m(a,b);if(!g||g[1]==")")return[b,d];(g[1]=="x"||!g[1])&&e("bop",a+" - "+(g[1]&&g[1]));var h=n(a,g[0]);b=h[0],h=h[1];var i;if(typeof h!="object"||h[0]==="("||l[g[1]][0]<l[h[1]][0])i=[d,g[1],h];else{i=h;while(typeof h[0]=="object"&&h[0][0]!="("&&l[g[1]][0]>=l[h[0][1]][0])h=h[0];h[0]=[d,g[1],h[0]]}return[b,i]}function m(a,b){var d,e=j.exec(a.substr(b));if(e){b+=e[0].length,d=e[1]||e[2]||e[3]||e[5]||e[6];if(e[1]||e[2]||e[3])return[b,0,c(d)];if(e[4])return[b,0,undefined];return[b,d]}}function k(a,b){return typeof a===b}function i(a,b){b||(b=0);var d=g.exec(a.substr(b));if(!d)return undefined;b+=d[0].length;var h;d[1]?h=[b," "]:d[2]?h=[b,d[0]]:d[3]?h=[b,f.typ,d[0]]:d[4]?h=[b,f.psc,d[0]]:d[5]?h=[b,f.psf,d[0]]:d[6]?e("upc",a):d[8]?h=[b,d[7]?f.ide:f.str,c(d[8])]:d[9]?e("ujs",a):d[10]&&(h=[b,f.ide,d[10].replace(/\\([^\r\n\f0-9a-fA-F])/g,"$1")]);return h}function e(a,b){throw new Error(d[a]+(b&&" in '"+b+"'"))}function c(a){try{if(JSON&&JSON.parse)return JSON.parse(a);return(new Function("return "+a))()}catch(b){e("ijs",b.message)}}var b=Object.prototype.toString,d={bop:"binary operator expected",ee:"expression expected",epex:"closing paren expected ')'",ijs:"invalid json string",mcp:"missing closing paren",mepf:"malformed expression in pseudo-function",mexp:"multiple expressions not allowed",mpc:"multiple pseudo classes (:xxx) not allowed",nmi:"multiple ids not allowed",pex:"opening paren expected '('",se:"selector expected",sex:"string expected",sra:"string required after '.'",uc:"unrecognized char",ucp:"unexpected closing paren",ujs:"unclosed json string",upc:"unrecognized pseudo class"},f={psc:1,psf:2,typ:3,str:4,ide:5},g=new RegExp('^(?:([\\r\\n\\t\\ ]+)|([~*,>\\)\\(])|(string|boolean|null|array|object|number)|(:(?:root|first-child|last-child|only-child))|(:(?:nth-child|nth-last-child|has|expr|val|contains))|(:\\w+)|(?:(\\.)?(\\"(?:[^\\\\\\"]|\\\\[^\\"])*\\"))|(\\")|\\.((?:[_a-zA-Z]|[^\\0-\\0177]|\\\\[^\\r\\n\\f0-9a-fA-F])(?:[_a-zA-Z0-9\\-]|[^\\u0000-\\u0177]|(?:\\\\[^\\r\\n\\f0-9a-fA-F]))*))'),h=/^\s*\(\s*(?:([+\-]?)([0-9]*)n\s*(?:([+\-])\s*([0-9]))?|(odd|even)|([+\-]?[0-9]+))\s*\)/,j=new RegExp('^\\s*(?:(true|false|null)|(-?\\d+(?:\\.\\d*)?(?:[eE][+\\-]?\\d+)?)|("(?:[^\\]|\\[^"])*")|(x)|(&&|\\|\\||[\\$\\^<>!\\*]=|[=+\\-*/%<>])|([\\(\\)]))'),l={"*":[9,function(a,b){return a*b}],"/":[9,function(a,b){return a/b}],"%":[9,function(a,b){return a%b}],"+":[7,function(a,b){return a+b}],"-":[7,function(a,b){return a-b}],"<=":[5,function(a,b){return k(a,"number")&&k(b,"number")&&a<=b}],">=":[5,function(a,b){return k(a,"number")&&k(b,"number")&&a>=b}],"$=":[5,function(a,b){return k(a,"string")&&k(b,"string")&&a.lastIndexOf(b)===a.length-b.length}],"^=":[5,function(a,b){return k(a,"string")&&k(b,"string")&&a.indexOf(b)===0}],"*=":[5,function(a,b){return k(a,"string")&&k(b,"string")&&a.indexOf(b)!==-1}],">":[5,function(a,b){return k(a,"number")&&k(b,"number")&&a>b}],"<":[5,function(a,b){return k(a,"number")&&k(b,"number")&&a<b}],"=":[3,function(a,b){return a===b}],"!=":[3,function(a,b){return a!==b}],"&&":[2,function(a,b){return a&&b}],"||":[1,function(a,b){return a||b}]};a._lex=i,a._parse=q,a.match=function(a,b){return z(a).match(b)},a.forEach=function(a,b,c){return z(a).forEach(b,c)},a.compile=z})(typeof exports=="undefined"?window.JSONSelect={}:exports)
return JSON.stringify(
exports.match(selector,
data));
$$ LANGUAGE plv8 IMMUTABLE STRICT

Then using the function:

select json_select('.name nth-child(1)', data) as name, json_select('.numbers', data) as phone
from example;
name | phone
--------------------+------------------------------------------------------------------------------------------
["Craig Kerstiens"] | [[{"type":"work","number":"456-123-7890"},{"type":"home","number":"123-456-7890"}]]
(1 row)

javascript injection attack

create or replace function
js(src text) returns text as $$
return eval(
"(function() { " + src + "})"
)();
$$ LANGUAGE plv8;

Have any others you feel are essential when starting to work with JSON? Let me know craig.kerstiens@gmail.com. Beyond that give JSON and JavaScript a try inside your database.

The Joys of UNIX Keyboards

Donat Studios — Wed, 19 Jun 2013 02:10:51 GMT

I fell in love with a dead keyboard layout.

A decade or so ago while helping a friends father clean out an old building, we came across an ancient Sun Microsystems server. We found it curious. Everything about it was different from what we were used to. The command line was black text on a white background, the connectors strange and foreign, and the keyboard layout was bizarre.

We never did much with it; turning it on made all the lights in his home dim, and our joint knowledge of UNIX was nonexistent. It sat in his bedroom for years supporting his television at the foot of his bed.

I never forgot that keyboard though. The thought that there was this alternative layout out there seemed intriguing to me.

I am ruined for all other keyboards…

Mac is the main platform at my new job. I found myself unhappy with Apple keyboards; they are flat, bland, and completely unsatisfying to type on. I dug out an old AppleDesign keyboard, and used that for several months but I still needed something more.

I read about Happy Hacking Keyboard, and after some deliberation decided to give it a try. I thought I could learn to accept most of the layout changes, but the lack of arrow keys on the "Professional" model made me squirm a little. Luckily there is a "Lite" version that includes arrow keys.

I purchased the Happy Hacking Keyboard Lite2.

The Happy Hacking Keyboard much like the Sun Microsystem keyboard I found years earlier features a UNIX layout. They designed it with typing at a terminal in mind.

A Better Layout

For those unfamiliar with UNIX keyboards, there are a few distinctions.

The Control key moved to the Caps Lock keys usual position. This makes it much easier to hit common commands than its usual location, often without ever leaving home row. I can't imagine why anyone thought Caps Lock should have had this prominent of placement.
The Meta (◆ / ⌘ / Windows) key is in its rightful place next to the space bar, as on a Mac keyboard.
The Backspace key moved down a row to be above the Enter key. This change is my favorite change. Having a small reach, normally my hands have to leave home row to hit backspace. but on a UNIX board I have no issue hitting it with my pinky with the rest of my fingers remaining firmly on home row.
Escape is also in reach of home row, placed where you would usually find ~. I could have done without this change, but it is fine.
The ~ and \ have moved to the usual location of the Backspace. Working exclusively on UNIX systems now, I don't mind the \ being harder to reach. I could understand a Windows user finding this irritating.

The HHKB is a minimal version of the UNIX keyboard. To this end the F Keys are mapped to Fn + 1, 2, 3, etc. The HHKB encourages you to stay on home row; you can hit any key on the keyboard without leaving it. I find my typing accuracy and speed to have improved thanks to its use.

The HHKB features one more change I don't care for. The Backspace key by default performs Forward Delete. Luckily changing this is easy by flipping a toggle switch on the back of the board.

Closing

I recommend this keyboard to anyone who spends a lot of time at the terminal. You are able to do more of what you need without ever leaving home row, every important key is within reach.

Elsewhere

Discussion on Hacker News
Mentioned on BSD Now Podcast: Episode 333

Explaining your PostgreSQL data

CRAIG KERSTIENS — Thu, 13 Jun 2013 23:55:56 GMT

I’ve written a bit before about understanding the output from EXPLAIN and EXPLAIN ANALYZE in PostgreSQL. Though understandably getting a grasp on execution plans could probably use some more guidance. Yet, this time around I’m taking a bit of a cop out and highlighting a few tools instead of documenting myself, which I’ve done in a talk I’ve frequently given Postgres Demystified.

Getting at the Data

The first small thing you can do is actually retrieve the data in JSON form. By adding in (format json) right after your EXPLAIN or EXPLAIN ANALYZE command it’ll as you’d expect return it in JSON. To give an example:

# EXPLAIN SELECT * FROM users LIMIT 1;
QUERY PLAN
--------------------------------------------------------------
Limit (cost=0.00..0.03 rows=1 width=812)
-> Seq Scan on users (cost=0.00..1.50 rows=50 width=812)
(2 rows)

Then in JSON format:

EXPLAIN (format json) SELECT * FROM users LIMIT 1;
QUERY PLAN
------------------------------------------------
[ +
{ +
"Plan": { +
"Node Type": "Limit", +
"Startup Cost": 0.00, +
"Total Cost": 0.03, +
"Plan Rows": 1, +
"Plan Width": 812, +
"Plans": [ +
{ +
"Node Type": "Seq Scan", +
"Parent Relationship": "Outer",+
"Relation Name": "users", +
"Alias": "users", +
"Startup Cost": 0.00, +
"Total Cost": 1.50, +
"Plan Rows": 50, +
"Plan Width": 812 +
} +
] +
} +
} +
]
(1 row)

While its on my list to build some interesting apps by pulling in the JSON input, others may be equally as interested in taking advantage of this data in its JSON form. If you take a shot at building something with this output, as always I’d love to hear about it - craig.kerstiens@gmail.com

Despez

Of course if you’re itch isn’t in better tools for Postgres, you may just want to have a solution that works today. While its not perfect, one of the best ones out there is Dezpez’s explain tool. You can take any execution plan and paste it in and get some better visual representation of the result. You can also share them as well.

Build a Better Warrior in Rift

Ben Overmyer's Site — Sat, 08 Jun 2013 03:00:00 GMT

I've been playing Rift a lot lately, and the more I play, the better I understand the ins and outs of being a Warrior. This post has a few of my thoughts on the subject of tuning warrior builds. The class is really versatile, but focuses heavily on two aspects - tanking and dealing damage. That much is obvious. What is less obvious is that there are several ways to do each of these. My personal preference is for dealing respectable amounts of damage to multiple enemies while being able to take a lot of punishment before going down. My favorite ability for staying in the fight is No Permission to Die. It's in the fifth tier of the Warlord soul, and it can be a godsend. If you die within ten seconds of triggering it, you are resurrected at 50% of your maximum health. Properly timed, it can really save your bacon. Also, it's not on global cooldown, so it's really hard to NOT time it right if you have a good sense of your combat situation. My next favorite ability is Cornered Beast, which is the 32 point unlock for the Champion soul. It hits up to 6 enemies for a ton of damage over time AND generates Attack Points over time, making it a powerful threat-building ability. Awesome for tanks, and for area effect damage. Along with Warlord and Champion, picking the Reaver soul allows for a ridiculous amount of survivability. It has a number of abilities that boost armor, endurance, and resistances. A balanced mix of the three souls, like the build linked here (Champion (32) / Warlord (23) / Reaver (21)), will give you a decent AoE tanking playstyle. You won't win any DPS races, but it will keep you in the fight for a good long time.

Postgres Indexing - A collection of indexing tips

CRAIG KERSTIENS — Thu, 30 May 2013 23:55:56 GMT

Even from intial reviews of my previous post on expression based indexes I received a lot of questions and feedback around many different parts of indexing in Postgres. Here’s a mixed collection of valuable tips and guides around much of that.

Unused Indexes

In an earlier tweet I joked about some SQL that would generate the SQL to add an index to every column:

# SELECT 'CREATE INDEX idx_'
|| table_name || '_'
|| column_name || ' ON '
|| table_name || ' ("'
|| column_name || '");'
FROM information_schema.columns;
?column?
---------------------------------------------------------------------
CREATE INDEX idx_pg_proc_proname ON pg_proc ("proname");
CREATE INDEX idx_pg_proc_pronamespace ON pg_proc ("pronamespace");
CREATE INDEX idx_pg_proc_proowner ON pg_proc ("proowner");

The reasoning behind this is guessing whether an index will be helpful can be a bit hard within Postgres. So the easy solution is to add indexes to everything, then just observe if they’re being used. Of course you want to add it to all tables/columns because you never know if core of Postgres may be missing some needed ones

As included with the pg-extras plugin for Heroku you can run a query to show you all unused indexes. On Heroku simply install the plugin the run heroku pg:unused_indexes to show the size and number of times an index scan has been used. On a non Heroku Postgres database you can run:

# SELECT
schemaname || '.' || relname AS table,
indexrelname AS index,
pg_size_pretty(pg_relation_size(i.indexrelid)) AS index_size,
idx_scan as index_scans
FROM pg_stat_user_indexes ui
JOIN pg_index i ON ui.indexrelid = i.indexrelid
WHERE NOT indisunique AND idx_scan < 50 AND pg_relation_size(relid) > 5 * 8192
ORDER BY pg_relation_size(i.indexrelid) / nullif(idx_scan, 0) DESC NULLS FIRST,
pg_relation_size(i.indexrelid) DESC;
table | index | index_size | index_scans
---------------------+--------------------------------------------+------------+-------------
public.grade_levels | index_placement_attempts_on_grade_level_id | 97 MB | 0
public.observations | observations_attrs_grade_resources | 33 MB | 0
public.messages | user_resource_id_idx | 12 MB | 0
(3 rows)

Costs of Indexing

There are really a couple of primary costs when it comes to indexing your data. The first is the overall size of the index. Indexes take size on disk, fortunately in most cases disk is pretty cheap. If you’re limited on disk size and not on your current performance then its pretty clear the trade-off you want to take. If you do need to get the size of your index you can do that by running:

# SELECT pg_size_pretty(pg_total_relation_size('idx_name'));

The harder trade off to look at is the cost in terms of throughput. As your data comes in there’s a cost for maintaining that index as the data within it has to be computed. If you’re doing crazy regex’s in your index then you can expect this to have an impact on your throughput.

Composite Indexes vs. Multiple Indiviual Indexes

A composite index is an index that includes multiple columns. Given an example table of purchases:

# \d purchases
Table "public.purchases"
Column | Type | Modifiers
-------------+-----------------------------+-----------
id | integer | not null
item | integer |
quantity | integer |
color | integer |

You might want to add an index on item and quantity together. You can do this with:

CREATE INDEX idx_purchases_item_quantity_color ON purchases (item, quantity, color)

From now on if you included item and quantity in a query its likely it would use this index just as it would if you used item, quantity and color. If you have a large varied set of data within each of these such an index can prove very useful. The caveat is that if you’re querying against only quantity and color then this index is useless, it must include the item column.

In contrast if you have three individual indexes Postgres may combine these or simply use one that would be the most efficient out of the three.

CREATE INDEX idx_purchases_item ON purchases (item);
CREATE INDEX idx_purchases_quantity ON purchases (quantity);
CREATE INDEX idx_purchases_color ON purchases (color);

Of course in this case if you query any individual column it would use the index if appropriate.

What Else

What else do you want to know about Postgres Indexing? Drop me a line craig.kerstiens at gmail.com or hop over to Postgres Guide and read a little there or even contribute some articles of your own.

Postgres Indexes – Expression/Functional Indexing

CRAIG KERSTIENS — Wed, 29 May 2013 23:55:56 GMT

Postgres is rich with options for indexing. First you’ve got a variety of types, and beyond that you can do a variety of things with each of these such as create unique indexes, use conditions to index only a portion of your data, or create indexes based on complex expressions or functions. In cases where you commonly use various PostgreSQL functions in your application or reporting you can get some great gains from this.

Let’s take a look at a really simple case. Given a basic user table:

# \dt users
Table "public.users"
Column | Type | Modifiers
------------+-----------------------------+-----------
id | integer | not null
email | character varying(255) |
created_at | timestamp without time zone |

You may commonly want to run a report against it showing your signups by date. Let’s say you do this by running the query:

SELECT
count(*),
date_trunc('day', created_at)
FROM
users
GROUP BY
2;

If you’re commonly using date_trunc('day', created_at) for grouping, filtering, or projecting it out you can get some great gains by creating an index on this:

# CREATE INDEX idx_user_created ON users(date_trunc('day', created_at));

Of course you can go beyond the built in functions of Postgres and use more complicated functions you create yourself. For example if you have JSON stored within PostgreSQL, have PLV8 enabled, and want to create a Javascript function to parse and return the text for a given key:

# CREATE OR REPLACE FUNCTION
get_text(key text, data json)
RETURNS text $$
return data[key];
$$ LANGUAGE plv8 IMMUTABLE STRICT;

Of note in the above function is IMMUTABLE and STRICT. Immutable specifies that the function given the same inputs will return the same result. Strict means that if you send in NULL values you’ll get a null result.

Given some example data inside your JSON field:

{
"name": "Craig Kerstiens",
"location": "San Francisco",
"numbers": [
{
"type": "work",
"number": "123.456.7890"
},
{
"type": "home",
"number": "987.654.3210"
}
]
}

If you wanted to return just the name you could index on:

# CREATE INDEX idx_name ON users(get_text('name', json_data));

Or even combine with built ins for a case-insensitive version:

# CREATE INDEX idx_name ON users(lower(get_text('name', json_data)));

Indexes like all of the above can be useful when you’re filtering on something that postgres can take advantage of. In most cases any conditions with the exception of a LIKE beginning with a % work for this. With Postgres 9.2 even a count(*) in certain cases can take advantage of the index because of index only scans.

Whether you’re looking to take advantage of all the power of Javascript with JSON or another procedural langauge – or simply speed up a basic report using built in functions expression indexes can give you some great benefits.

My SQL Bad Habits

CRAIG KERSTIENS — Sun, 26 May 2013 23:55:56 GMT

I’m reasonably proficient at SQL – a coworker when pseudocoding some logic for him pointed out that my pseudocode is what he thought was executable SQL. I’m fully capable of writing clear and readable SQL – which most SQL is not. Despite that I still have several bad habits when it comes to SQL. Without further adieu heres some of my dirty laundry so hopefully others can not make the same mistakes.

Order/Group by Column Numbers

When quickly iterating on a query its a lot less typing to put the column number as the thing you want to order by. Here’s a quick lightweight example:

SELECT
email,
created_at
FROM
users
ORDER BY 2 DESC
LIMIT 5;

This gives me my last 5 users that have signed up for my site. Of course as soon as I have this I may want to add some data to it, like their first name so I can send them a welcome email. I quickly alter the query to:

SELECT
email,
first_name,
created_at
FROM
users
ORDER BY 2 DESC
LIMIT 5;

And now I have 5 users that have signed up ordered by their first name. Sure its obvious when you have 1 column you’re ordering by, but when you have GROUP BY 1, 2, 3, 4, 5, 6 which is actually open in one of my tabs currently its a bit more confusing….

Though if you really want to have some fun, share a query with someone that looks something like this:

SELECT
email as "3",
first_name "2",
created_at "1"
FROM
users
ORDER BY "1", "3" DESC
LIMIT 5;

Implicit Joins

I seldom use the syntax INNER JOIN. Instead I simply put the two tables in my where clause and ensure I have a where condition. The problem with ensuring I have a where condition is sometimes I don’t, especially when you’re dealing with 3 tables.

SELECT
email,
product.name,
product.price
FROM
users,
orders,
items
WHERE users.id = orders.user_id
AND orders.id = items.order_id

Is less clear (especially when dealing with 5-6 tables) than the alternative:

SELECT
email,
product.name,
product.price
FROM users
INNER JOIN orders on users.id = orders.user_id
INNER JOIN items on orders.id = items.order_id

Lack of comments

I comment my SQL far less than I comment my code, yet it can be done just as easily. For example I have this in one of my queries:

SELECT convert_from(CAST(E'\\x' || array_to_string(ARRAY(
SELECT
CASE
WHEN length(r.m[1]) = 1
THEN encode(convert_to(r.m[1], 'SQL_ASCII'), 'hex')
ELSE substring(r.m[1] from 2 for 2)
END
FROM regexp_matches(url_here, '%[0-9a-f][0-9a-f]|.', 'gi') AS r(m)
), '') AS bytea), 'UTF8');

While this has its own issues theres no documentation around what this actually does, in contrast:

--- DECODES url ---
SELECT convert_from(CAST(E'\\x' || array_to_string(ARRAY(
SELECT
CASE
WHEN length(r.m[1]) = 1
THEN encode(convert_to(r.m[1], 'SQL_ASCII'), 'hex')
ELSE substring(r.m[1] from 2 for 2)
END
FROM regexp_matches(url_here, '%[0-9a-f][0-9a-f]|.', 'gi') AS r(m)
), '') AS bytea), 'UTF8');

Comments also work well inline at the end of a line.

Large Manually Generated Lists

A lot of times in working with some specific data set I’ll manually or automatically generate a list that I want to filter. A common example is filtering out staging/dev environments. I’ll often manually search and prune the list, then save that result for the queries I’m going to build going forward. This is a bit of effort but still feels reasonable the downside is it results in something like:

SELECT
foo
FROM
bar
WHERE
bar.id NOT IN (34723, 42735, 32321, 47205, 20375, 30261, 26194, 109371, 9313, 6351, 20184, 50273, 34735, 39854, 23954, 25323, 23405, 30528, 50182, 29340, 47659, ... and the list goes on)

SQL is meant to be reasonable for containing some level of logic. Data changes, hard coding keys is going to bite you at some point, spend the extra effort and re-use something thats clear.

What else

I’m sure theres plenty more; I suspect within a few minutes of sitting down with someone they could point out some other bad habits. While I know mine at least some of mine I still often know the trade-off. What are yours? I’d love to hear to document them for others so hopefully they can prevent developing the same bad habits. Let me know; craig.kerstiens@gmail.com

Learning Ruby on Rails: Why? How? Pitfalls?

Brightball Articles — Thu, 09 May 2013 19:00:00 GMT

I've always been a proponent of the "right tool for the job" approach to programming. Different languages are well suited for different situations. Over the past 2 years I've spent a great deal of time with Ruby on Rails after coming from a background of PHP, Java and Perl. Here's how I got started and some of the lessons I learned along the way.

CX – Conference Experience (Facilitating Communication)

CRAIG KERSTIENS — Mon, 29 Apr 2013 23:55:56 GMT

Following up on my earlier post about CX or Conference Experience – I’m going to dig in a bit on how you get good conversation to happen. In the past two years I’ve been to nearly 20 conferences, I’ve been to conferences with great talks, with great parties, with great swag, and hands down my favorite conferences have always been a result of great conversation. With the number of talks that are recorded and immediately available online after, what can I say I’m a hallway track guy.

I’ve seen a number of conferences intentionally design around this concept, in some ways the unconference is purely a hallway track conference. I’ve also seen conferences that weren’t clearly planned for this and have pulled off some of the best situations where people turn their phones off and engage in real conversation.

Breaks

Breaks are always a hard part to balance. Every conference organizer I’ve talked to felt they had the perfect mix for breaks. If you’re designing for convesation then you need to allow time for conversation to actually happen. Whether a multi-track or single track conference 15 minutes for breaks doesn’t give you the time to actually engage in conversation. If multitrack its time enough to grab a drink then head to the next room. If single track its time to have the entire room leave and then come back.

Even 30 minute breaks fall into this category. At a multitrack conference I’m usually saying hi to enough people during a 30 minute break that I don’t have the chance to really get deep into a conversation. If I do it usually results in skipping the next talk.

Staggered Talks

Locations for Conversations

Having places for people to get away an talk is critical, in a venue that only has 1 main room and no convenient places for a quick coffee or beer it becomes very hard for conversations to happen. If its on the agenda to facilitate such things thats great, but going a step further and allowing for it to be facilitated naturally is even better. On more than one occasion I’ve been mid conversation wtih someone, we cut it off agreeing to pick it back up at the next break only to never locate them again for the rest of the conference. I’d have just as much preferred to continue the conversation there, but sadly there was in the middle of a talk room with no where else feasible to go.

Water

This one caught me by surprise… The event was PyCodeConf in Miami and it was one of the evening activities – a pool party with food, drinks, and mariachi band. Early into the evening, inevitably someone was thrown into the pool (by friends in a general good nature), but thus ruining a lovely iPhone. As a result I’m pretty most people, just as I did, went to their rooms but their cell phone away, or set it near their stuff. Much of the rest of the evening was an entirely internet and twitter free evening. Causing lots of uninterrupted conversations to happen and resulted in a pool full of geeks.

Welcoming noobs

The big difference between just assuming this will happen and actively working to facilitate it is who gets involved in the conversation. For a first time presenter there’s a wealth of nerves about talking. For a first time attendee there’s a wealth of knowledge to soak up. First time attendees may not feel as comfortable interjecting them into a conversation, approaching presenters, or talking to someone they’ve been following for years – yet they have just as much to add as anyone else.

Creating places where more conversation is happening helps ease this and build a better community.

CX – Conference Experience (People)

CRAIG KERSTIENS — Mon, 29 Apr 2013 23:55:56 GMT

Following up on my earlier post about CX or Conference Experience – I’m going to dig in a bit on getting the right people there. There’s a lot of different ways to approach this from having a good ratio of:

Designers to Developers
Females to Males
Noobs to well known community members
Overall lack of suits

While this is by no means a how we can solve all of the above problems, theres some bas

Talks

Quality talks are obviously important. A conference should at whatever cost ensure that the talks are good. This can happen a variety of ways, but regardless of the method you should ensure talks are worthwhile; since this is largely what most people are paying to attend for.

Invite-only

Perhaps the easiest way to do this is via invite only to speakers you know will do a good job. The downside ot this of course is that you must already know enough people to fill out a good agenda, and also you limit the ability for others to contribute.

Coaching/Practicing

This is not an either/or option with any of the other pieces for getting a good agenda. Having presenters do a trial run can ensure a minimum level of quality, and working with them to coach them can help make the talk even more effective. Its likely that the organizers know the audience as well as anyone, so no one better than them to help with this. Of course this is a time sink for both parties, but can give good returns. Of course this could be done independent of a specific conference such as through speakup.

At a very least getting a quick run through, outline, or something of that nature can ensure that a presenter doesn’t fill a 45 minute talk slot with only 5 minutes of content.

Open CFP

Likely the most common approach to getting speakers is having an entirely open CFP. Its typical then that either the organizers or a speaker selection commitee then discusses and makes selections. This can usually work to have a nice balance of experienced and known to be good speakers and newer less experienced speakers that can have great potential as well.

Open Open CFP

Going even further is an open CFP where all talks are published after the CFP then voted on. While this does a great deal to ensure transparency, it doesn’t necessarily improve ensuring theres a great line up of speakers. Being able to write an interesting talk proposal is an entirely separate process from delivering an interesting talk.

There’s a common question lately for both the open CFP and open open ones one whether to do blind review of the talks. The logic here is an attempt to be entirely fair, versus having some bias. This is an understandable goal, but can come at the expense of quality. Truth be told; I’m not entirely sure of a way to balance this.

Choosing

There’s definitely not a one size fits all. If you’re goal is to provide a good list of talks then you should keep that in mind in how you decide your talks. If you’re goal is to pull others in then it should be shaped differently. Zach Holman recently talked a bit about this and had some interesting ideas to minimize risk for new speakers.

The Drawback to Web Frameworks

Brightball Articles — Mon, 29 Apr 2013 20:37:00 GMT

Web frameworks are great, don't get me wrong here. They provide a structure and consistency across projects that will transcend developers over the life of a system while dramatically simplifying the code base amongst other wonderful side effects. But what's the downside?

"Make it faster."

Brightball Articles — Sun, 28 Apr 2013 05:22:00 GMT

I'm obsessed with performance tuning. It's an itch that can never fully be scratched. A sickness that can never be cured. Here's the story of how I caught the bug.

CX – Conference Experience (Talks)

CRAIG KERSTIENS — Sat, 27 Apr 2013 23:55:56 GMT

A couple of weekends ago I had the great opportunity to attend lessconf. It was an all around great conference, and as a result of the greatness I ended up having a conversation with a few people around conference experience. I must give much of the credit to Swift, as he mentioned he’d already been thinking alot about this since Waza. In general it feels like there’s a few key themes that any conference should focus on, then a lot of small things that can really push it over the top. Here’s a few:

In general the key areas for any great conference are:

Talks
Great people
Ensuring communication happens
Bonus points

Digging in deeper on the first area…

Talks

Invite-only

Coaching/Practicing

At a very least getting a quick run through, outline, or something of that nature can ensure that a presenter doesn’t fill a 45 minute talk slot with only 5 minutes of content.

Open CFP

Open Open CFP

Choosing

The Impossible Assignment

Brightball Articles — Thu, 25 Apr 2013 20:12:00 GMT

I had the opportunity to visit the class of one of my legendary former professors yesterday and got to share a classic story about him...the time he gave us an impossible assignment.

Create Better Villains for your Adventure

Ben Overmyer's Site — Sat, 20 Apr 2013 03:00:00 GMT

Villains are the mainstay of any good adventure. With very few exceptions, there is always some form of evil mind plotting dark things in the background. Whether this is a mindless mob of creatures that just happens to be devouring the locals, or a masterfully devious court magician manipulating a king to sinister ends, you can't get away from the guys.

This blog post aims to assist you in building memorable villains that are more than just numbers on a sheet of paper. If you're a player and not a game master, the character building advice in here might help you too, but this is specifically geared towards the folks weaving the story behind the encounters.

The Nature of a Villain

While it's fairly obvious that a villain exists to ruin the lives of good people, a truly memorable villain has much more than that going on. He has his own motives and desires. He may be acting in accordance with his own morals, or he might not care at all.

Most frequently, a villain's goals shouldn't be an end, but a way to reach some other starting point. The kobold king isn't sending raiding parties out to nearby villages because he feels like it; he's weakening the nearby human presence so his people can expand outward from their all-too-small cave system.

The depth of a villain's forward thinking depends on how intelligent you want him to be. Don't restrict a villain's planning just because he's a minor NPC in your adventure. Players will inevitably do something you don't expect, and if they just happen to latch onto your minor villain, you'll then have a direction to take that in.

Journey of the Hero

The player characters will only grow stronger as a result of having a characterful villain as their foil. You can consider your adventure a success when not only does it give your PCs the magical loot of awesomeness they've been waiting for, but it also encourages them to reconsider their characters' actions and motives.

Joseph Campbell's Hero with a Thousand Faces is a fantastic guide to setting up an epic story. His writings on the “road of trials” in particular are relevant to creating worthwhile villains for the players.

A player character that's just starting out usually has weak stats, basic equipment, and almost zero personality. She might have a single personality quirk or cliché that her player jots down to guide her role-playing, but rarely will players take the time to build a full story for their characters. After all, that's what they're hoping you will do.

As the PC encounters and interacts with your adventure's many challenges, locations, and characters, she will grow in both power and personality. If her player is into role-playing, this will happen sooner rather than later, and with more depth. She will need a good villain, though, to balance that development against.

Brainstorming

Creating a villain should start with looking at what you want to achieve with the adventure and with the campaign at large. An adventure isn't a set narrative that you can just write and expect to stay the way you wrote it. The players will do things you don't expect and interact with the environment in ways that force you to adapt the adventure to fit their actions. As such, your villain should support your goal with the adventure, rather than just play to the script.

Jot down a few notes about your adventure's premise. Is it a classic battle of good versus evil? Is it a day in the life of an adventurer? Is it an exploration of the unknown? Think about your favorite movies or books that fit the theme you're going for. What twists in those stories really got your blood pumping?

Potential for Loss

Think about what would really hurt the player characters. I don't mean in terms of hit points lost or character sheets thrown away – I mean tragedies that specifically target each PC's hopes, dreams, and beliefs. Whether the players are into role-playing or not only determines whether you have more or less fodder for this; it's still possible to make a tactical gamer sit back in his chair with a thud when his character loses something dear to him. It usually just changes the context.

Build something of that potential loss into each villain of the adventure. You don't need to include every player character in every adventure, but you shouldn't focus on only one player character either. Also, choose a handful of NPCs and target their potential for loss also. It helps build a living world when you include characters other than the players' in your adventures.

Potential for Gain

Saruman was originally a hero, but gave in to temptation.

Now think about what the villain can offer the player characters. Temptation plays on base emotions; use it. For tactical gamers, this could be a particular piece of loot that they want desperately. For role-players, it might be a bit of information about a relative they've been searching for. For every player, write down at least one thing that the villain might have the ability to provide.

As with the potential for loss, also make sure that some of your NPCs stand to gain something by working with the villain. Even if that incorruptible priest of all that is good can't possibly be turned to the dark side, the players might not know that, and that creates tension. Tension is what every great story (and adventure) is built on.

Villains are Story Generators

Other than trying to provoke emotional tension from players through the potential for loss or gain, a good villain should also act as a way to move the story along. The best stories, whether they're explosion-ridden Hollywood blockbusters or 1500-page literary masterpieces, are about the heroes changing in some way over the course of the story. Villains are almost always the catalyst for that change.

If your players are craving action, send a wave of minor baddies against them. Let them know that the villain made this happen, whether directly or indirectly. Weave a spider's web with your villain at the center. Even if he is not of the evil genius persuasion, he's still fully capable of planning ahead.

Summary / TL;DR Version

Keep your villains close to your players' hearts. Tension is a great story device, so use it. Tempt the PCs. Hurt the PCs. Make a list of the villain's goals and aspirations.

And, above all else, remember: villains are not just stat blocks, they're people too.

Heroku's Acquisition 2 Years Later

CRAIG KERSTIENS — Thu, 18 Apr 2013 23:55:56 GMT

Just over two years ago, Heroku was acquired. I was around and peripheral to this just before the acquisition and came on board only barely after. While there is a large group of people that have been there longer than I have (several for 4+ years) I’m still commonly asked how things have changed, how things work, and other questions of that nature. I wrote about some of these processes over a year ago in the months after joining Heroku around our hiring, our teams, and how we work. Many of these things haven’t changed, and yet almost always at a conference I’m asked how are things different since being acquired.

Here’s my personal take (and while I don’t typically include this – to be safe, this is not an official Heroku view of what’s changed).

Are things different since the acquisition?

Heroku in many ways operates like a wholly owned subsidiary or as an independent business within Salesforce. We still have our own office space, our own IT (if you can even call it that), and in general entirely own workflow. I have a salesforce email account that automatically forwards to my Heroku google apps account, and that’s about as much as I know about it. In fact, we’re currently preparing for our new office and its planned to be our home for somewhere between the next 5-10 years.

At the same time some things do flow through Salesforce, some of those things are logistical some strategic. First the perhaps most unfortunate part – hiring. This has not really changed how we hire, but rather once you are coming on board theres more paperwork. In general this is the most painful part that instead of 1/2 pieces of paper theres a few to sign. At the same time some of these come with some greater gains such as benefit, etc. To be honest I dont fully recall what all of the paper work is, but either way there is a bit more of it. The other area we really see this effect is expense accounts. All expenses flow through concur, which in my opinion is a pretty good solution. I’ve used many worse expense solutions and seen few if any better.

Do they influence the product?

Not really

In general we at Heroku aim to have some broader alignment around what we’re trying to accomplish. Salesforce believes in helping their customers become a customer company. We believe that developers are worth of great experiences. Salesforce believes in ensuring its customers are successful. Heroku believes that developers should focus on adding value of their customers not just keeping the lights on. In all of these things its important to have alignment. Not for the company we’re building for the next six weeks or six months, but for the next six years. Given both Salesforce’s and Heroku’s worldview I believe what we’re trying to accomplish is in good alignment.

But does Salesforce influence our product roadmap? No, we aim to listen to our customers problems and build to solve those problems and through that improve the way software is delivered. At the time Salesforce acquired us, the pieces of Cedar were already in motion; including Procfile support, logplex, and other pieces. Later was fully productized Cedar and which was a goal we’d long had – making other languages such as Node, Java, and Python available. This wasn’t driven because Salesforce wanted Java, but because we saw value in delivering the same value of the platform to other communities.

Have we changed?

Sure, we’ve changed, but I’d surmise very little if at all due to Salesforce. We’ve grown from a 20 person company to now over 100. We were a primarily local team and now have people all over. Company offsites get a bit harder to coordinate with 100 people, and finding a date that absoluely everyone can make it is nearly impossible. Many of our changes are more strictly a change of growth than they are because of Salesforce.

I dont get it, why acquire you then?

Because we’re aligned in a much bigger vision, we can both work together long term to accomplish our goal.

The internet is changing the world; software is everywhere.

I believe Salesforce truly understands this. Heroku does as well (though, at the time of the acquisition I fully suspect many Herokai held their breath to see how it would all go). Six weeks went by without trying to ‘change us’, then six months, and here we are over two years later. In many ways so many of us have started to better understand that Salesforce truly understands the above.

If you understand the above, then you know that delivering software and improving that process gives any business a competitive advantage. This is at the core of the value we aim to provide. E.g. If you ranked all companies in terms of how strong they aligned with Heroku, Salesforce might be at the very top.

Profiling Django templates with Django-Debug-Toolbar

Tom Forbes — Thu, 18 Apr 2013 23:46:58 GMT

My last post about the speed of Django’s templating language caused a bit of a stir and it was clear that people didn’t really have a clue how long the templates were taking to render in their applications. Enter Template-timings Template-timings is a panel for Django-debug-toolbar (which everyone s...

Using array_agg in Postgres – powerful and flexible

CRAIG KERSTIENS — Wed, 17 Apr 2013 23:55:56 GMT

In almost any application it’s common to want to aggregate some set of values together, commonly in a comma separated form. Most developers do this by running a query to get much of the raw data, looping over the data and pushing it into a set, appending each new value to the appropriate key. Hopefully, it’s not a surprise that there’s a much better way to do this with PostgreSQL.

Postgres has a flexible and robust array datatype that comes with a variety of functions. Even without taking advantage of the array datatype in your application, you can still take advantage of some of the functions to get the functionality you need. Lets take a look at an example schema and use case.

An example

Given a project management application, you may have users who have projects that have tasks. An example piece of functionality might be to send an email with a list of all projects that have tasks that are past their due dates of completion. Your schema might look something like this:

 # \d users
Table "public.users"
Column | Type | Modifiers
------------+-----------------------------+-----------
id | integer | not null
email | character varying(255) |
...
# \d projects
Table "public.projects"
Column | Type | Modifiers
------------+-----------------------------+-----------
id | integer | not null
user_id | integer | not null
name | character varying(255) | not null
...
# \d tasks
Table "public.tasks"
Column | Type | Modifiers
--------------+-----------------------------+-----------
id | integer | not null
project_id | integer | not null
completed_at | timestamp without time zone |
due_at | timestamp without time zone |
...

To get a list of all projects that have tasks that haven’t been completed, you would start with something like:

SELECT
projects.name
FROM
projects,
tasks
WHERE projects.id = tasks.project_id
AND tasks.due_at > tasks.completed_at
AND tasks.due_at > now()

This would give you a list of projects which you could then easily join this with users:

SELECT
users.email
projects.name
FROM
projects,
tasks,
users
WHERE projects.id = tasks.project_id
AND tasks.due_at > tasks.completed_at
AND tasks.due_at > now()
AND users.id = projects.user_id

At this point you’ve got everything you need to pull this up into Ruby, Python, or other language of your choice and then build the full set. However if this is thousands or even hundreds of results you’ll be spending more time than necessary, grouping this data for a sensible email. With 3 other small changes you can have this already formatted for you to immediately send of in an email. The first is using a handy function called array_agg which will aggregate items and then you can format them how you wish. The second is just ensuring you’re grouping correctly. Finally you’ll want to unnest the array so it formats the data in a clean way for you.

Looking at it all put together:

SELECT
users.email,
array_to_string(array_agg(projects.name), ',')) as projects
FROM
projects,
tasks,
users
WHERE projects.id = tasks.project_id
AND tasks.due_at > tasks.completed_at
AND tasks.due_at > now()
AND users.id = projects.user_id
GROUP BY
users.email

This would give you a nice clean result of projects that have overdue tasks that you could then send to the user in an email:

 email | projects
---------------------------+-------------------
craig.kerstiens@gmail.com | blog, timetracker
craig@heroku.com | foo, bar, baz

Building a Search Result Extract Generator in PHP

Ben E. C. Boyter — Wed, 17 Apr 2013 02:23:56 GMT

During some contracting I was doing recently there was a requirement to implement some search logic using only PHP. There are no issues with that but it turns out I couldn’t find a decent extract generator handy as usually one would just plug into the search engines provided version to do this.

Off the top of my head I could only think of one example I was aware of which lives in Sphider (for the record it lives in searchfuncs.php from line 529 to 566). Sadly it has a few issues. Firstly the code is rather difficult to understand, and more importantly it usually has accuracy issues. A quick search turned up this link http://stackoverflow.com/questions/1436582/how-to-generate-excerpt-with-most-searched-words-in-php on StackOverflow. The second answer looked promising but its even more difficult to understand and a bit of profiling showed some performance issues will all of the regex going on in there.

Scaling Evangelism – Creating Advocates

CRAIG KERSTIENS — Tue, 16 Apr 2013 23:55:56 GMT

The first area I tend to think about when it comes to developer marketing is around advocates. A simple definition of an advocate is someone that speaks/writes in favor of some thing.

Creating advocates is a means of creating more of myself to go out and talk loudly and ideally effectively. However, they do take time to cultivate and it is a hard item to track. But by cultivating advocates I’m able to scale what would otherwise be a bottleneck of my own personal time.

Mentorship

The people in your group of 3 you’ll have nearly daily communications with, even if quick. These are the first people you think to work and collaborate with. It’s the case with all, but especially so here that there may be some strong mutual benefit. With both the groups of 3 and 12 you’ll be looking to:

Co-author content with
Collaborate on projects
Help amplify their content

Helping them Succeed

For a few years I’ve followed a similar process to this. Essentially creating advocates is done through first making them successful, then empowering to talk about their success. I’ve often done this by giving people a private channel to me – through IM, SMS, Phone, Skype, or even in person. Sure its easy enough to get my work email address, its a pretty easy one to guess, and even at that address I do aim to respond to every request. These channels in reality don’t make me more available; but they do allow a different type of communication to happen.

My personal approach has found it more out of place to ask how someones weekend was over email – IM or Skype make this easy. Broader questions of how someones actually doing in life generally are much easier to do over a coffee or a beer. This allows for a deeper relationship; at the same time I can ensure everyone I interact with is successful. It’s easier to ensure someones successful if they’re willing to talk to you, actually connecting with people enables this.

Communicate

Once someones been successful theres usually some value in their story. This doesn’t need to come in a marketing-ese approach though. Instead contribute back the value of what you did to the broader community with clear steps how they can get the same benefit. For many developers they feel talking about their own success is a bit of imposter syndrome. While this isn’t always the case; if it is you can help to work with someone to highlight the value in what they’ve done. This could be as simple as some encouragement to actually helping review drafts of blog posts.

Of course once someones started talking about what they were able to accomplish help them celebrate the success. Yelling from the rooftops only helps to further their efforts and give them the confidence to do it all over again.

Conclusion

To create advocates you don’t have to write a list of people you wish to shoot for, but realizing there are limiitations on your time can help you be more intentional. Knowing that where you spend your time and who you spend it with has a cost can allow you to be more explicit about ensuring you’re providing value to them, from there it should naturally run its course. Want to discuss this more? Email me at craig.kerstiens at gmail.com.

Doing Marketing (for developers) Differently

CRAIG KERSTIENS — Fri, 12 Apr 2013 23:55:56 GMT

As developers we can tend to be a fickle bunch; especially in certain open source communities. We like to see intelligence and systems applied to things, hope for a better world through making things open, and appreciate when others relate to our world as we often attempt to relate to theirs. Marketing is often a loaded word when it comes to developers – leaving mixed feelings about webinars, email campaigns, and the like.

A quick clarification in terms of marketing I’m referring to product marketing, but of a technical product. This is based on the idea that developers are the new kingmakers and when marketing a product to them it needs to be done differently.

While many of the above steps can add immense value, they often miss when it comes to developers. By changing these processes only slightly there can be much more efficiency in connecting with and actually delivering the value you hope for to developers.

Email Marketing

This is an area that there are more and more companies already starting to solve the right problems. Companies like intercom.io and customer.io are allowing for better tracking of what users are doing and notifying them at the right time instead of via massive impersonal campaigns. Developers build systems that take into account similar factores every day, you should be doing the same.

Emails that are sent at the wrong time or miss on what someone is trying to do can often do more damage towards building a rapport than good. Pushing content to solve a recent issue or problem at a relevant time should be table stakes for any email thats sent.

Community Engagement

Whether at conferences or in online collaborations developers are more social than so many give them credit for. We appreciate when things are done to improve the greater good particularly for developers, but often for the world as well. From supporting individual developers or projects financially such as through gittip or certain kickstarter projects all the way to employing developers to work full time on open source.

Being Better Than Webinars

After doing a webinar for a particular audience I had a Heroku co-worker come up to me. This individual has contributed many ideas that have become core functionality such as the Cedar stack and is generally open minded and listens very well. He came up to me and genuinely asked, “so what is a webinar?” I explained its like a live webcast with an opportunity for questions at the end. He was quiet for minute and responded, “Why can’t we just say that?”

There are some words that simply send off the wrong signals. Webinar rather clearly is one of those. As a community we’ve shown that we have a desire for this kind of content. The conference community has continued to grow steadily, and while they are many parts to every conference a pretty common one is the talk track itself. A talk at a conference is essentially a live in person webinar, of course the experience is often a bit richer.

Instead of the term webinar what is wrong with online office hours or webcasts with an opportunity for questions? Even doing the same thing with a different name is underdelivering on what we’re capable of.

I welcome hearing from others directly at craig.kerstiens@gmail.com on what they’re doing to engage with developers thats working. Not only working in the sense of eyeballs or dollars, but in adding value, in improving communities, in make the world better for developers. If there’s enough interest would be happy to post more detail around things I’ve seen work and what I hear works for others.

More holes than swiss cheese

Tom Forbes — Thu, 11 Apr 2013 23:24:30 GMT

tl;dr Never trust user input. Note: Before I published this I contacted the owner of the site and he has since replaced it with a new and more secure version When I browse the net I often see the hallmarks of a security issue with websites that I visit, and little alarm bells go off in my head. I wa...

Profession Talents for Ingenium

Ben Overmyer's Site — Sat, 06 Apr 2013 03:00:00 GMT

This post presents a prototype profession mechanic we're working on for Ingenium 2nd Edition. We're looking at giving each profession its own Profession Talent to add a little more variety to the game.

The Priest profession from first edition was pretty…. plain, and there was nothing priestly about her, other than role-playing ques. So with this new mechanic, she gets a Talent that reinforces her role as a messenger of the gods.

The priest profession from first edition was presented like this:

Priest

Starting Talent Pool: Mystic

Attributes: +1 Magical Aptitude

Speed: 3

Hit Points: 1d8

Advancement: Healing Arts

Starting Gold: 1d4 x 5

Priests are those divine focal points who channel their gods' will into physical manifestations. Their magic is not based on arcane rules and laws; rather, they are blessed as Chosen and receive gifts of power for as long as they serve their gods.

The flavor text stands in Second Edition, but there are a number of changes. Here's the new Priest block:

Starting Talent Pool: White Magic

Attribute Bonus: +1 Magical Aptitude

Hit Die: d6

Talent: Exorcism

Exorcism

The Priest has the divine gift and knowledge to pull a demon out of a possession victim. She makes a Willpower roll against the demon's Challenge Level, and if the total is higher, the demon vacates the possessed body as a cloud of roiling black smoke. The victim then returns to his original stats, and if this causes any damage he has sustained to exceed his normal maximum Hit Points, he dies instantly. If his Willpower was at least half of the demon's, he remembers everything he did while possessed. A failed exorcism increases the Challenge Level of the target demon by 1.

About Profession Talents

Every profession gets a unique Talent with this mechanic, and one that scales with the character's power. She will never out-level her profession Talent.

Also, by adding “Profession Talents,” we add another option for expansion down the road.

Unlocking Hillsborough

Dan Palmer — Sat, 06 Apr 2013 02:00:00 GMT

I’m writing this on the train home from Rewired State’s latest event: National Hack the Government Day 2013 (event summary page). It was another great event with the same friendly atmosphere that goes along with so many (especially Rewired State’s) developer events. My friend Elliot and I won in one of the categories, and so this post is mostly about what we did, how we did it, and why we think it’s important.

Why I Blog

CRAIG KERSTIENS — Sun, 31 Mar 2013 23:55:56 GMT

I blog because I’m lazy. There’s more too it though: In any given day I may explain something to someone, the first time I do this I make a bit of a mental note. The second time I do this, especially within a short time frame I make a physical note of this in the form of the title of a blog post. Once I’m already to a second time of doing this its almost inevitable I’ll continue repeating myself – and its valuable to others.

Becoming replaceable

My goal in doing this is actually to become heavily replaceable. Keeping information locked away means I’m the only one capable of doing it. Making myself highly replaceable means I can continue to work on new and interesting things.

Lots can be shared

I don’t recall exactly where I read it, I believe it may have been James Governor, but any email that doesn’t contain proprietary info or trade secrets should be blogged. I fully expect the sentiment around this is something to the effect that anything that contains good ideas/learning/processes can also be helpful to others.

Distilling information

For every time I explain something I often linger about one piece too long, hit something that doesn’t need to be discussed, or miss something entirely. Putting an idea or guide together in written form you re-read it, which is hard to do when simply verbally explaining. If you’ve video taped yourself before for public speaking practice you know this is an awkward but valuable experience. Its much easier to practice as an exercise in writing.

Economies of scale

While in some ways I am absolutely lazy when I blog I get economies of scale I couldn’t otherwise have. On a given day on a high day I may have 15 one on one conversations. Every post has the opportunity to become a one on one conversation.

One on one conversation is my preferred method instead of in a large group. In a related area I’ve been having an ongoing conversation wth Ryan Daigle on whether blog posts should have comments. I opt for having you reach out to me directly via email, he believes comments should be available. Putting this in words crystalizes that to me its just relative to how you prefer to interact.

Summary

As with any other post, this is one conversation I’ve now had a few times over. Hopefully this gives some basis to why I feel others should contribute content as well. Whether you want make your self dispensible, refine your thoughts, or reach new economies of scale I believe its a worthwhile exercise for many. If you have other reasons you feel its valuable as always please reach out.

Being Defiant: Finishing Out Rift's Free Tier

Ben Overmyer's Site — Sun, 31 Mar 2013 02:00:00 GMT

Today is the last day of March and the last day I'll be writing about the MMORPG Rift as it applies to tabletop RPGs. If you've missed any of these posts, you might want to check them out. Go ahead, I'll wait.

This post is a retrospective and has little to do with tabletop RPGs. If you're just looking for new ways to experience your favorite RPG, come back next week – I've got something very interesting planned!

Generally, it takes me about one month to get bored of an MMORPG. I'll steep myself in the lore, powerlevel up to about 75-90% of endgame, get involved in crafting, and then… burn out. The novelty factor wears off, and I stop playing.

Most of the time, it's not a conscious decision to stop playing. I'll just go a day without playing the game, since I find something else I want to do more that one day. Then two days go by. Then three. Before I realize it, it's been over a week since I last logged in, and I have no intention of going back.

A rare handful of MMORPGs keep my interest for longer than that. The list so far has been Star Wars: Galaxies and World of Warcraft. Now, I'm adding Rift to the list.

I originally reinstalled the game because it was going to be free and I'd heard good things about Dimensions. So, I did, and started this blog series to tie the game to tabletop gaming. It just clicked. The novelty factor has worn off, and I'm still playing. That bodes well for its longevity, and I'm going to resubscribe.

The Dimensions are awesome. The fishing is weirdly addictive. And playing an MMO with a great and friendly community is refreshing.

If you're interested in trying it out, jump on the Faeblight server and look me up. I'll be playing either my Eth cleric Amun or my Kelari mage Sidious.

Being Defiant: Currencies Beyond Gold and Silver

Ben Overmyer's Site — Sun, 24 Mar 2013 02:00:00 GMT

In Rift, there are five ways to acquire new armor and weapons. You can take it from the corpses of monsters you kill. You can run through dungeons, collecting new pieces from the tougher monsters and bosses. You can run from rift to rift, defeating the planar invaders and getting new gear from the rift rewards. You can craft it or purchase it from other players who craft gear. You can also use special currencies earned through various means to purchase new gear from special NPC merchants.

In existing tabletop RPGs, there are really only three ways to get gear – buying it from merchants, looting bodies, and finding it lying around in dungeons.

Many MMORPG loot mechanics won't work in a tabletop RPG because the gear just appears in the player's inventory when certain conditions are met. In a tabletop game, gear rarely just appears. I've touched on crafting in tabletop RPGs before, so this time around I'll focus on special currencies.

Rift's Currency for Planar Invasions

In games like Rift and World of Warcraft, you receive units of these currencies for achieving feats of excellence. For example, in Rift, you earn “planarite” and “sourcestone” for defeating planar invaders. Sourcestone is much rarer than planarite, each unit of its most common version being worth 50 planarite. It's only awarded for helping defeat special rift events like full invasions.

While there's only one version of planarite, there are multiple types of sourcestone, each being dropped by progressively more difficult planar invaders. In order to get the higher sourcestones, you must defeat higher-level invaders.

All of these can be traded in for impressive, rare gear. The best items cost a combination of sourcestone and planarite.

How to Deal with Special Currencies in Tabletop Games

In tabletop RPGs, by comparison, you're looking at a single currency – gold, credits, silver, whatever. In some games there are multiple types of currency but only one “track.” For example, in Dungeons & Dragons, you have copper, silver, electrum, gold, and platinum, but they're all just higher denominations of each other. This currency is accepted everywhere from all merchants.

To make sense, a unique currency needs to be widespread in use and creation, and to make sense in a setting, the powers-that-be need to allow that currency to exist alongside the official currency of the realm. That last part is historically difficult to come by, but in fictional worlds, we can gloss over that fact.

The best reason to include a currency like this is to gate powerful gear behind difficult tasks. It also provides a way to uniquely theme gear and work in factional alliances and reputation.

You could make up a large order dedicated to service to a particular god, for example. Instead of just being all priests or all paladins, this order could include a variety of skilled craftsmen who produce special gear. The craftsmen may only part with the gear in exchange for a special currency that only the order uses. Higher-ranking members of this order may be giving the PCs quests, and in gratitude for their help, rewards them with some of the order's currency.

This also offers role-playing opportunities, oddly enough. Possession of particular currencies could trigger interesting reactions from NPCs. Maybe they're associated with the producer of the currency. Maybe they're afraid of the producer. Maybe it's more complicated than that.

If it makes sense for your setting, try adding a special currency and unique rewards for that currency to your next adventure. It just might take your campaign in a very interesting direction.

Bird Watching

Beetle Space — Fri, 15 Mar 2013 09:00:00 GMT

A friend of mine recently took his bird watching hobby to another level. Armed with a 400mm lens, a scope and a sturdy tripod, he and his wife scour the local area for birds. The absence of leaves makes it easier to find them. [1]

I met up with them …

Prioritizing and Planning within Heroku Postgres

CRAIG KERSTIENS — Wed, 13 Mar 2013 22:55:56 GMT

Over a year ago I blogged about Heroku’s approach to Teams and Tools. Since that time Heroku has grown from around 25 people to over 100, we’ve continued to iterate and find new tools that work for how we do things. For many of the product management and software engineering books I’ve read I’ve yet to find something that helps a team priorize in a fashion I that feels right.

One process emerged nearly a year ago from within the Heroku Postgres team and is now followed by many others. Within a team this process is now commonly conducted each 6 months. Lets take a look at how this process looks

It Starts with Ideas

Hopefully having ideas of things to work on isn’t a problem, if it is just go spend some time with customers – listen to their problems, see how they use the product, then come back and write down the ideas. For most teams this is simply an excercise of thinking back and writing it down. Some teams at Heroku have resorted to keeping running backlogs of things they’d like to do this. We do this by keeping a Trello board which columns for:

New ideas
Ponies
Stallions

Ponies and Stallions are things that would be great to do, however a sizeable amount of work must be done on them and we’re not currently tackling them. Ponies are less sizeable and likely to get done not in coming weeks but perhaps in coming months up to a year. Stallions are great but large effort and may or may not get done but in the category of things we would like to be able to do.

Once you’ve got your ideas whether in your head on a backlog we begin by writing them out typically on sticky notes or index cards.

Laying it all out

From here we create a simple grid:

The grid has two axis. One is for impact the other for difficulty. At this point we aim to lay out every idea that we’ve already written down into a quadrant. Commonly this is done at team offsites where the team is free of distractions and able to devote appropriate time to it. Being able to accomplish this in one sitting with the team is important to having cohesion around the result.

A plan

At this point hopefully its quite obvious what you want to tackle. If there’s anything in the top right it should be an easy win for something for you to focus on. From we often transcribe this into a powerpoint/keynote document and highlight things that we will definitely aim to accomplish in the next 6 months as well as things we’re intentionally not working on. This leaves us with an artifact of both things we will work on and explicit things we wont work on.

What works for us

In general we try to have more work than we can tackle to ensure we’re constrained in a good form and not wasting idle time of people. Ensuring we’re selective about the things we’re working on and that we’re working on the right things works for us. We’ve found this simple exercise valuable for many teams to plan and ensure we’re working on those right things. Of course this may not work for everyone but for our goals and culture aligns well for us.

If you’ve got simple but unique techniques that work for your team as always would love to hear about them – craig.kerstiens@gmail.com

Just how slow are Django templates?

Tom Forbes — Wed, 13 Mar 2013 20:17:38 GMT

Edit 2: I made a Django debug toolbar panel that profiles your Django templates and all their components. You can find it here: https://pypi.python.org/pypi/django-debug-toolbar-template-timings Edit: It appears that Django does have some form of template caching. The graphs have been updated to inc...

Fixing Database Connections in Django

CRAIG KERSTIENS — Thu, 07 Mar 2013 22:55:56 GMT

If you’re looking to get better performance from your Django apps you can check out Pro Django, PostgreSQL High Performance, or read some my earlier posts on Postgres Performance. All of these are of course good things to do – you can also start by correcting an incredibly common but also painful performance issue, that until 1.6 is unaddressed in Django.

Django’s current default behavior is to establish a connection for each request within a Django application. In many cases any particularly in distributed cloud environments this is a large time sink of your response time. An example application running on Heroku shows a typical connection time of 70ms. A large part of this time is the SSL negotiation that occurs in connecting to your database, which is a good practice to ensure security of your data. Regardless, this is a long time in simply establishing a connection. As a point of comparisson its commonly encourage that most queries to your database are under 10ms.

An example that highlights this in a small lightweight application shows the bulk of a request time being within a connection displayed by New Relic:

One option to remedy this is by running a connection pooler on your Database side such as Pgpool or PgBouncer. In fact Ask the Pony already highlighted these potential gains. While running an external DB they’re essentially testing the benefits of conncetion pooling. This is an obvious gain and can be in a much more lightweight format.

Connection Pooling in Django

As Django establishes a connection on each request it has an opportunity to both pool connections and persist connections. There are two major options for pooling, each works quite well with Django and provides some dramatic improvements. While the first request may take the 70ms of connection time, subsequent requests show absolutely no connection time since the connection already exists. This is highlighed by these two comparissons of before and after in actually the times it grabs a connection:

Clearly theres plenty of value to having a persistent connection or a pool within Django itself. As of today theres a few options for that:

Django-PostgresPool

The first Django-PostgresPool is created by kennethreitz. As in general I’d encourage the use of dj_database_url you can easily begin using his package (once installed) with:

import dj_database_url
DATABASE = { 'default': dj_database_url.config() }
DATABASES['default']['ENGINE'] = 'django_postgrespool'

An important thing to note is if you’re using South you’ll also want to setup the adapter for it:

SOUTH_DATABASE_ADAPTERS = {
'default': 'south.db.postgresql_psycopg2'
}

djorm-ext-pool

The second option djorm-ext-pool is created by niwibe. Once you’ve installed djorm-ext-pool you then add it to your INSTALLED_APPS within your settings.py. From here then you can setup your pool:

DJORM_POOL_OPTIONS = {
"pool_size": 20,
"max_overflow": 0
}

django-db-pool

The third and final option is django-db-pool. You can set it up with:

DATABASES = {'default': dj_database_url.config()}
DATABASES['default']['ENGINE'] = 'dbpool.db.backends.postgresql_psycopg2'
DATABASES['default']['OPTIONS'] = {
'MAX_CONNS': 10
}

Gotchas

Each of these does work with recent versions of Django, though in some cases there are gotchas. If using a prodution worthy python web server such as Gunicorn or uwsgi and running with gevent or eventlet some edge cases can present themselves. Regardless of potential gotchas it is worth attempting this and of course providing feedback to maintainers and the community as you find those.

The future

Django more recently has directly started to address these issues of large costs of establishing a connection. The first major step here is this patch from Aymeric. You can find more dicussion around this particular patch here. Essentially with this patch which will hit in Django 1.6 developers then get a persistent connection which will help reduce the time. If you’re interested in trying the 1.6 master you can do this by adding it to your requirements.txt as:

https://github.com/django/django/archive/master.zip

At this point it does not introduce pooling which could allow even more gains, though I’m sure if there’s enough need it’ll be on a roadmap at some point. Though, as it stands today before 1.6 your best bet is one of the above options.

Simple database read scaling without sharding in rails

CRAIG KERSTIENS — Wed, 06 Mar 2013 22:55:56 GMT

In an earlier post I provided a high level overview of sharding. Sharding while a very solid approach to scaling capacity versus simply only relying on vertical scaling can also be a time intensive one. Additionally in some cases certain sites may only need extra capacity for a short lived period of time. Fortunately theres a nice middle ground alternative for scaling capacity that works well in quite a few cases. It even has a benefit that can potentially in place of sharding.

This method results in scaling your reads to replica databases, you can do this on Heroku by taking advantage of followers. A follower is a read only database on Heroku Postgres that receives asynchronous updates of your data usually only lagging a very few commits behind. This means you can write all of your data to the leader (main) database, and then read from another.

While you can arbitrarily do this there’s some major benefits to doing it based on the models. This is because Postgres maintains a cache on each instance its running on. Though you may have the same dataset, Postgres maintains frequently accessed or queried data in the cache giving you better performance. For more on this you can read earlier posts on PostgreSQL Performance.

Setting it up with Rails

With a follower database created you can begin adding support for this to your application. The first thing is to add the gem to your Gemfile:

gem 'ar-octopus', :require => "octopus"

Then of course to install it with bundle install. Now we can actually begin to add the code needed to have specific models access the follower.

octopus:
shards:
shard_sqlite:
adapter: sqlite3
database: db/db_one.sqlite3
pool: 5
timeout: 5000
shard_pgsql:
adapter: postgresql
username: postgres
password:
database: db_two
encoding: unicode

class Project < ActiveRecord::Base
octopus_establish_connection(:adapter => "sqlite3", :database => "db_one")
end

Finding fried chicken with C-Sharp, Mono and ServiceStack

Tom Forbes — Sat, 02 Mar 2013 23:57:06 GMT

I think I just solved a very first world problem - Where to find the nearest takeaway that sells fried chicken. When you are stumbling out of Spiders at 1am in the morning and craving some SFC goodness the last thing you want to do is wander the backstreets of hull in the cold looking for an open ta...

Why Code Search is Difficult

Ben E. C. Boyter — Fri, 01 Mar 2013 01:26:36 GMT

I was chatting with a colleague the other day and he was asking me why code search is a difficult problem. After all its not quite as dynamic as the web so it should be easier to index.

In truth its a mixed bag. Some things like like crawling are easy, others such as indexing are much harder then you would think. I thought I would write down why this is.

Getting more out of psql (The PostgreSQL CLI)

CRAIG KERSTIENS — Thu, 21 Feb 2013 22:55:56 GMT

After my last post I had a variety of readers reach out about many different tweaks they’d made to their workflows using with psql. One people Grégoire Hubert had a wondeful extensive list of items. Grégoire has been a freelance in web development and he has worked with Postgresql for some time now in addition to being the author of Pomm. Without further ado heres what he has to say on how he uses psql:

Get the most of psql

Psql, the CLI postgreSQL client, is a powerful tool. Sadly, lot of developers are not aware of the features and instead look for a GUI to provide what they need. Let’s fly over what can psql do for you.

Feel yourself at home

One of the most common misconception people have about CLI is «They are a poor user interface». C’mon, the CLI is the most efficient user interface ever. There is nothing to disturb you from what you are doing and you are by far fastest without switching to your mouse all the time. Let’s see how we can configure psql at our convenience.

First, you’ll have managed to choose a nice and fancy terminal font like monofur or inconsolata. Do not underestimate the power of the font

The nice line style shown above can be set with \pset linestyle unicode and \pset border 2. This is just an example of the many environment variables you can play with to get your preferred style of working out of psql.

For example, I found the character ¤ the most accurate to express nullity (instead of default NULL). Let’s just \pset null ¤ and here it is:

SELECT * FROM very_interesting_stat;
┌──────┬──────┬──────┬──────┬──────┐
│ a │ b │ c │ d │ e │
├──────┼──────┼──────┼──────┼──────┤
│ 9.06 │ ¤ │ ¤ │ ¤ │ ¤ │
│ 7.30 │ 3.55 │ 7.57 │ 3.31 │ ¤ │
│ 7.20 │ 5.08 │ ¤ │ 6.58 │ 5.90 │
...

Another hugely value to get environment variables is colors in the prompt. Colors in the prompt are important because it makes easier to spot where output starts and ends between two interactions at the console. The PROMPT1 environment variable will even let you set an indicator to notify you are inside a transaction or not, give this a try for a sweet surprise…

\set PROMPT1 '%[%033[33;1m%]%x%[%033[0m%]%[%033[1m%]%/%[%033[0m%]%R%# '

I also like to disable the pager by default \pset pager off and display the time every issued query takes \timing. If you are used to psql, you may notice in the picture above, some content is wrapped. This is \pset format wrapped option.

Of course, writing all that on every connection would be a pain, so just write them in a ~/.psqlrc file, it will be sourced every time psql is launched.

If you are familiar with bash or other recent unix shells, you might also declare aliases in your configuration file. You can do the same with psql. For example if you want to have a query for slow queries such as from this earlier post but not have to remember the query every time you can set it up as:

\set show_slow_queries
'SELECT
(total_time / 1000 / 60) as total_minutes,
(total_time/calls) as average_time, query
FROM pg_stat_statements
ORDER BY 1 DESC
LIMIT 100;'

Now, just entering :show_slow_queries in your psql client will launch this query and give you the results:

 total_time | avg_time | query
------------------+------------------+------------------------------------------------------------
295.761165833319 | 10.1374053278061 | SELECT id FROM users WHERE email LIKE ?
219.138564283326 | 80.24530822355305 | SELECT * FROM address WHERE user_id = ? AND current = True

Psql at your fingertips

Now you have got a fancy prompt, here is the real question you ask, what can psql do for me ? and \? has all of the answers. It has built-in queries to describe almost all database objects from tables to operators, indexes, triggers etc… with clever auto-completion. Not only completion on tables and columns – but also on aliases (sweet), SQL commands (w00t) and database objects.

Now we can enter some SQL commands. As usual, you need to check in the documentation how the heck to write this damn ALTER TABLE. Relax, psql proposes inline documentation. Just enter \h alter table (auto complete w00t) and you ll be ok.

Interacting with your editor

psql provides two very handy commands: \e and \i. This last command sources a sql file in the client’s current session. \e edits the last command using the editor defined in the EDITOR shell environment variables (aka vim). This grant you with real editor feature when it comes to writing long queries. What psql does, it saves the buffer in a temporary file and fires up the editor with that file. Once the editor is terminated, psql sources the file. Of course, you can use your editor to save queries in other places where they would be under version control, but the \e has a serious limitation: it spawns only the last query. Even if you sent several queries on the same line. (Note that \r clears psql’s last query buffer).

Note: \ef my_function opens stored function source code (With auto completion, I know, it’s awesome).

Vim users can here benefit from Vim’s server mode. If you launch a vim specifying a server name (let’ say “PSQL”) somewhere, and set the EDITOR variable as is export EDITOR="vim --servername PSQL --remote-tab-wait then psql will open a new tab on the running vim with the last query and run it as soon as you close this tab. Tmux or gnu/screen users will split their screen to have Vim and psql running on the same terminal window.

Call a friend

Vim power users know it is possible to pipe a buffer (or selection) directly in a program that can be … psql (Using the :w !psql syntax). Even from the shell, you might want to take advantage of the fantastic \copy feature that loads formated file in the database (I use it to load apache logs). But always having to specify connection parameters are a hassle. Let’s use shell environment instead. Psql is sensitive to the following variables:

PGDATABASE
PGHOST
PGPORT
PGUSER
PGCLUSTER (debian wrapper).

Set them once for all in you shell environment and call psql to connect to the database. In case you want to skip password prompt, you can store your pass in a 600 mode access file named .pgpass in your home (do not do that on shared or exposed computers). Although this is nice for development database servers, I do NOT recommend this for production servers since it should not be easy to mess with them.

Resource for additional information is … the man page and Postgres Docs. All PostgreSQL documentation is an example of what software reference documentation should be. Enjoy!

HtmlToWord

Tom Forbes — Mon, 18 Feb 2013 07:38:08 GMT

You can find the code here on github and the package here on PyPi I have written and continue to maintain a reporting system for a group of pentesters. During/after the tests the results and details are inputted into a web application using a WYSIWYG editor called Redactor (which is pretty awesome!)...

How I work with Postgres – psql, My PostgreSQL Admin

CRAIG KERSTIENS — Wed, 13 Feb 2013 22:55:56 GMT

On at least a weekly basis and not uncommonly multiple times in a single week I get this question:

@neilmiddleton I’ve been hunting for a nice PG interface that works within other things. PGAdmin kinda works, except the SQL editor is a piece of shit

Sometimes it leans more to, what is the Sequel Pro equivilant for Postgres. My default answer is I just use psql, though I do have to then go on to explain how I use it. For those just interested you can read more below or just get the highlights here:

Set your default EDITOR then use \e
On postgres 9.2 and up \x auto is your friend
Set history to unlimited
\d all the things

Before going into detail on why psql works perfectly fine as an interface I want to rant for a minute about what the problems with current editors are and where I expect them to go in the future. First this is not a knock on the work thats been done on previous ones, for their time PgAdmin, phpPgAdmin, and others were valuable tools, but we’re coming to a point where theres a broader set of users of databases than ever before and empowering them is becoming ever more important.

Empowering developers, DBA’s, product people, marketers and others to be comfortable with their database will lead to more people taking advantage of whats in their data. pg_stat_statements was a great start to this laying a great foundation for valuable information being captured. Even with all of the powerful stats being captured in the statistics of PostgreSQL so many are still terrified when they see something like:

 QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Hash Join (cost=4.25..8.62 rows=100 width=107) (actual time=0.126..0.230 rows=100 loops=1)
Hash Cond: (purchases.user_id = users.id)
-> Seq Scan on purchases (cost=0.00..3.00 rows=100 width=84) (actual time=0.012..0.035 rows=100 loops=1)
-> Hash (cost=3.00..3.00 rows=100 width=27) (actual time=0.097..0.097 rows=100 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 6kB
-> Seq Scan on users (cost=0.00..3.00 rows=100 width=27) (actual time=0.007..0.042 rows=100 loops=1)
Total runtime: 0.799 ms
(7 rows)

Empowering more developers by surfacing this information in a digestable form, such as building on top of pg_stat_statements tools such as datascope by @leinweber and getting this to be part of the default admin we will truly begin empowering a new set of user.

But enough of a detour, those tools aren’t available today. If you’re interested in helping build those to make the community better please reach out. For now I live in a work where I’m quite content with simple ole psql here’s how:

Editor

Ensuring you’ve exported your preferred editor to the environment variable EDITOR when you run \e it will allow you to view and edit your last run query in your editor of choice. This works for vim, emacs, or even sublime text.

export EDITOR=subl
psql
\e

Gives me:

Note you need to make sure you connect with psql and have your editor set, once you do that saving and exiting the file will then execute the query

\x auto

psql has long had a method of formatting output. You can toggle this on and off easily by just running the \x command. Running a basic query you get the output:

SELECT *
FROM users
LIMIt 1;
id | first_name | last_name | email | data | created_at | updated_at | last_login
----+------------+-----------+----------------------------+------------+---------------------+---------------------+---------------------
1 | Rosemary | Wassink | Rosemary.Wassink@yahoo.com | "sex"=>"F" | 2010-07-01 18:16:00 | 2011-05-14 11:47:00 | 2011-06-07 23:04:00

With toggling the output and re-running the same query we can see how its now formatted:

\x
Expanded display is on.
craig=# SELECT * from users limit 1;
-[ RECORD 1 ]--------------------------
id | 1
first_name | Rosemary
last_name | Wassink
email | Rosemary.Wassink@yahoo.com
data | "sex"=>"F"
created_at | 2010-07-01 18:16:00
updated_at | 2011-05-14 11:47:00
last_login | 2011-06-07 23:04:00

Using \x auto will automatically put this in what Postgres believes is the most intelligible format to read it in.

psql history

Hopefully this needs no justification… having an unlimited history of all your queries is incredibly handy. Ensuring you set the following environment variables will ensure you never lose that query you ran several months ago again:

export HISTFILESIZE=
export HISTSIZE=

\d

And while the last on the list one of the first things I do when connecting to any database is check out whats in it. I don’t do this by running a bunch of queries but rather checking out the schema and then poking at definitions of specific tables. \d and variations on it are incredibly handy for this. Here’s a few highlights below:

Listing all relations with simply \d:

\d
List of relations
Schema | Name | Type | Owner
--------+------------------+---------------+-------
public | products | table | craig
public | products_id_seq | sequence | craig
public | purchases | table | craig
public | purchases_id_seq | sequence | craig
public | redis_db0 | foreign table | craig
public | users | table | craig
public | users_id_seq | sequence | craig
(7 rows)

List only all tables with dt:

\dt
List of relations
Schema | Name | Type | Owner
--------+-----------+-------+-------
public | products | table | craig
public | purchases | table | craig
public | users | table | craig
(3 rows)

Describe a specific relation with \d RELATIONNAMEHERE:

\d users
Table "public.users"
Column | Type | Modifiers
------------+-----------------------------+----------------------------------------------------
id | integer | not null default nextval('users_id_seq'::regclass)
first_name | character varying(50) |
last_name | character varying(50) |
email | character varying(255) |
data | hstore |
created_at | timestamp without time zone |
updated_at | timestamp without time zone |
last_login | timestamp without time zone |

One more pro-tip if you’re running a transaction with many tables and forget which are involved in it you can run ‘\d *transaction*’ and it’ll display tables curently affected.

Have a tool you prefer, have something you use daily in psql that I missed, or interested in helping create a new admin experience please reach out and lets talk craig.kerstiens at gmail.com

World of Warcraft, Modern RPGs, and the Gear Grind

Ben Overmyer's Site — Sun, 10 Feb 2013 02:00:00 GMT

After a full year of not playing, yesterday I jumped back into World of Warcraft. I'd heard enough good things about the latest changes that I decided to give it another shot. It remains to be seen whether that'll last or not. Something that did make me pause, though, is the obvious reminder that WoW is based on a gear grind. As soon as my character, a worgen warlock named Exsecratus, made it to the new continent, I almost immediately replaced my raid epic gear from the Cataclysm days with greens. Today's junk is better than yesterday's hard-won treasure. That theme carries across many modern MMORPGs, but in all honesty, it's also present in tabletop RPGs these days. Dungeons & Dragons is one of the worst culprits, but there are plenty of others. That got me to thinking. Is the gear grind necessarily bad? The perpetual need to have better equipment may seem like a thin excuse to keep people playing a game, but taken as a component of a larger and more complex system, it's a reliable foundation to build on. In a game like my own Ingenium, where if you have too many static modifiers it makes the randomization provided by the dice useless, a gear grind based on incremental bonus increases wouldn't make much sense. However, if there were a game out there that had more complex calculation involved with point bonuses, then having a Sword +513 wouldn't necessarily be any more unbalanced than a Sword +3. The trick is that to get to that level, you need to involve multiplication and division in the core mechanic, and that's the real holdup for a tabletop RPG. Let's face it - most tabletop gamers don't want to sit and recalculate their stats every time they pick up a new item. In video games it's possible because the game itself handles it automatically in the background without the player even being aware of the math. To introduce infinitely-scaling item stats into a tabletop RPG, you would have to do one of two things - either create a phone/tablet app to handle the math for you, or make the math so simplified that it would be possible to calculate quickly. Being a web developer by profession, I'm partial to the first one. These days it's not unreasonable to expect gamers to have a smartphone. How would you handle bringing the gear grind into a tabletop game?

Introducing django-db-tools

CRAIG KERSTIENS — Fri, 08 Feb 2013 22:55:56 GMT

For any successful web application there is likely to come a time when you need to conduct some large migration on the backend. I dont mean simple add a column here or add an index there, but rather truly sizeable migrations… Going from MySQL to Postgres or migrating from an older version of Postgres such as a 32 bit instance to a newer 64 bit instance. In these cases the default approach is to just schedule downtime often throwing up a splash screen saying so.

For many sites this approach is simply wrong and lazy, with little effort you can improve the experience and there by ease the burden in conducting these types of migrations. By having the ability to turn your site into a read only mode which Simon Wilson talked about in his post on Lanyrd you can still continue to operate just in a limited capacity. Andrew Godwin further talks about some of this as well in regards to the Lanyrd move and even includes the script they used to migrate data from MySQL to Postgres. Though just in talking with Simon about this a week ago it occurred to me they had not released the code for their read-only mode.

Finally onto the announcing, today I’m releasing django-db-tools. This is currently a very lightweight utility that allows you to flip your site into two modes.

Anonymous Mode

For sites that offer a bulk of their data to unauthenticated users anonymous mode will be what you want. This ensures all users appear logged out and thus cannot interact with data. To enable anonymous mode you’d simple set the environment variable or config var on heroku as follows:

READ_ONLY_MODE = True

Restricting POSTs

The other bucket of sites is one that allows users to stay logged in but not insert data. Django did not appear to have a convenient means to know whether data was actually being inserted into the DB or not. As a good practice when inserting data it should be receiving a HTTP POST. The GET_ONLY_MODE mimmicks all POSTs as if they were sent via GETs thus hopefully eliminating inserting data into your application. To turn it on simply set the environment variabel or config var on heroku to:

GET_ONLY_MODE = True

Installing

The tool itself is largely middleware, to install:

Run pip install django-db-tools or add it to your requirements.txt
Add db_tools to your INSTALLED_APPS in your settings.py
Add 'dbtools.middleware.ReadOnlyMiddleware', to your MIDDLEWARE_CLASSES in settings.py

Contributing

As with all code this is largely a work in progress. There’s many items still to do such as providing default copy and error pages and potentially handling other edge cases. I’d welcome others to contribute and give feedback if they find it helpful or how it can be improved on Github.

Want to write a search engine? Have some links

Ben E. C. Boyter — Thu, 31 Jan 2013 01:29:36 GMT

A recent comment I left on Hacker News managed to get quite a lot of up-votes which surprised me since it was effectively just a collection of links about search engines. You can read the full thread at http://news.ycombinator.com/item?id=5129530

Anyway since it did do so well I thought I would flesh it out with some more information. Here are a collection of posts/blogs/discussions which go into the details of how to write a search engine.

Crafting in Tabletop RPGs

Ben Overmyer's Site — Sun, 13 Jan 2013 02:00:00 GMT

One of the systems near and dear to my heart in MMORPGs is crafting. I love being able to make things from materials that I've collected myself, and in so doing improve my own capabilities without having to rely on an external source (e.g., loot, or vendors). Very few tabletop RPGs have ever attempted to include a crafting system, and those that have (Dungeons & Dragons, I'm looking at you) focus primarily on high-cost, high-powered magical items. Here, I explain how a crafting system for a tabletop RPG might work, taking several cues from MMORPGs. Crafting in MMORPGs has three primary component systems - gathering, refining, and building finished items. Gathering is the collection of raw resources from the world. This usually takes the form of withdrawing units of resources from a resource node that randomly spawns in the world. Refining takes those raw materials and works them into usable forms. For example, smelting iron ore into iron bars is one such process. This step is sometimes optional. Building finished items takes resources and puts them together into a final usable form. In MMORPGs, the process of actually crafting the item is abstracted away such that the player doesn't need to worry about things like quenching steel in the proper temperature of liquid. Tabletop RPGs generally can't follow MMORPG mechanics, since a lot of the latter is based on intensive calculation behind the scenes. With crafting, my perception is that it's not popular because it's not a story component - in a tabletop gaming session, anything that smacks of busywork is not going to go over well. To address that concern, there are two possibilities:

Make the whole crafting process fun and exciting, or
Make the crafting process something that happens between adventures.

Personally, I think a mix of the two is ideal. The gathering process should be divided into two parts - acquiring rare raw materials through normal adventuring, and acquiring mundane raw materials in a quick process that occurs between gaming sessions. The refining process should be purely between sessions, unless it's something story-bound like imbuing a raw material with a specific magical aura that can only be done by a powerful hermit-mage that lives in the…. you get the picture. The crafting of the final item could go either way, though I'd tend to put it at the very beginning of a game session before the actual adventure starts to “prime” players' minds. A big component of crafting, and one of its greatest variables, is time. In real life, it takes a long time to smith a sword. In World of Warcraft, it takes seconds. Crafting in tabletop RPGs, as a character activity, should take little time. Crafting is a secondary system, and it's not epic to say that your character spent a month of in-game down time smithing a new longsword in a borrowed forge while the other members of the party twiddled their thumbs. This problem can be explained away in two ways:

Give the character NPC assistants to do the crafting for him, or
Make the crafting process itself something magical that takes little time.

I prefer the first option, since it involves more story hooks and feels less like a deus ex machina. If a player discovers he has the materials to make an upgraded suit of armor just as a session is about to begin, you can allow him to do that and retcon that he contracted the work out while he was adventuring in the last session. This can tend to make retconning a standard, accepted practice for crafting, but I think that's of minor concern in this case. The other consideration when designing a crafting system for use in a tabletop RPG is the effect this can have on player equipment and expectations of loot. As such, it can't be designed in a vacuum - it needs to be part of the system from the beginning, or you risk turning a game into a Monty Haul campaign inadvertently. The secret here is in the timing - you can gate the frequency of crafting more powerful items by introducing a crafting skill advancement system and making it as restrictive as you like, thereby keeping crafted items in line with looted items (or not, if that's what you prefer). The Dungeons & Dragons system of spending experience to craft powerful items is one way to gate the acquisition of crafted items, but I think it's fundamentally flawed in that it makes little sense from a story point of view. Sauron didn't lose part of his essence to craft the One Ring - he just lost time and resources. To make something, a player has three resources available to him - Time, Money, and Effort. The ratio can vary between those three, but if you abstract out those concepts, you can arrive at a more sensible requirement ladder than Dungeons & Dragons'. In the end, crafting is a perfectly viable addition to a gaming system, as long as it's well-integrated with the rest of the game. I'm working on one such system for inclusion in a game of mine, and I like to think the players will love it.

Code a Search Engine in PHP Part 5

Ben E. C. Boyter — Thu, 10 Jan 2013 07:34:18 GMT

[Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.][1]

This is part 5 of a 5 part series.

[Part 1][2] – [Part 2][3] – [Part 3][4] – [Part 4][5] – [Part 5][6] – [Downloads/Code][7]

So we need to convert the indexer to a method that wont consume as much memory. Looking at how it works now we can determine a few areas that could be improved before we implement out new method.``` public function index(array $documents) { $documenthash = array(); // so we can process multiple documents faster

Code a Search Engine in PHP Part 4

Ben E. C. Boyter — Thu, 10 Jan 2013 07:31:34 GMT

Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.

This is part 4 of a 5 part series.

Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code

So previously we had 4 main issues identified. Im going to go with low hanging fruit and add a simple porn filter.

Code a Search Engine in PHP Part 3

Ben E. C. Boyter — Thu, 10 Jan 2013 07:27:57 GMT

[Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.][1]

This is part 3 of a 5 part series.

[Part 1][2] – [Part 2][3] – [Part 3][4] – [Part 4][5] – [Part 5][6] – [Downloads/Code][7]

At this point I had a large chunk of the web sitting on my disk waiting to be indexed. The first thing we need to think about is what are we going to index. Since we are dealing with HTML files we have marked up content can can pick to a certain extent what we want to index on. For our search im going to index on the title and the meta description tags. Titles are still a part of search engine ranking algorithms as many people search for things like “Yahoo” and “Facebook” and the title/URL usually contains this information. The meta description had fallen out of favor with being something search engines use these days due to being gamed, but since its unlikely the top 1 million are going to be doing this I am going to use it as well. These two terms should work fairly well at our scale and ensure our index isn’t too massive. We can always add the rest of page content at a later point.

Code a Search Engine in PHP Part 2

Ben E. C. Boyter — Thu, 10 Jan 2013 05:03:54 GMT

Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.

This is part 2 of a 5 part series.

Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code

The first implementation has a few issues that are pretty apparent. Chiefly is performance. Secondly it stores the index in a single folder which is loosely related to the first. Finally it shows duplicates in the search results. Lets fix the above and test things again to see how far we can go.

Code a Search Engine in PHP Part 1

Ben E. C. Boyter — Thu, 10 Jan 2013 04:55:54 GMT

[Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so.][1]

This is part 1 of a 5 part series.

[Part 1][2] – [Part 2][3] – [Part 3][4] – [Part 4][5] – [Part 5][6] – [Downloads/Code][7]

I imagine that if you have landed on this page you probably have an interest in search and search engines. Not only that, you have probably read the following [Why Writing Your Own Search Engine is Hard by Anna Patterson (of Cuil fame) http://queue.acm.org/detail.cfm?id=988407][8] If not, go read it and come back here. I have always had an interest in search and indexing I thought I would take the time to write a simple indexer as a way of testing if I had actually learnt anything.

Conquering 2013: Goals for the Year

Ben Overmyer's Site — Sat, 05 Jan 2013 02:00:00 GMT

I'm joining the vast numbers of bloggers who are publicly declaring their goals for the new year right now. Scott Dinsmore of Live Your Legend fame has challenged people to, well, Live Your Legend in 2013, and offers a Goal Setting & Action Workbook to help achieve that. I took four hours this morning to fill it out and plan my goals for the year, and this blog post summarizes the results of that. Two of my six big goals for the year have to do with gaming, but the rest are personal goals. I'll share all of them with you after the jump.

Goal #1) Release a Computer Game

I've worked, briefly, in the computer game industry. I've started numerous game projects. But I've never actually released a finished game. 2013 is the year to change that. While I won't be releasing my dream game this year (the MMORPG tentatively called Starborn: Universe), I'm going to fire off a single-player action sandbox game called Starborn: Genesis. My release goal is by December 1, 2013. Future blog posts will detail this, my most difficult goal of 2013, as I work on it.

Goal #2) Join a Local Developer Group

This goal is much smaller than #1. The purpose, though, is no less important; it has been said that you are an average of the five people you spend the most time with, and all of the five people I hang out with most are at best only tangentially related to computer game development. I want to connect with people more, and I especially want to connect with others who are passionate about computer game development. Joining a local developer group is an important part of that.

Goal #3) Take a Vacation with Treasure to Hawaii

My wife Treasure and I have not yet taken a vacation outside of the Midwest. As both of us have traveled the world and love new places, this is a problem. This year, I'm committing to getting us both to Hawaii for a week in June, near or on our 4th anniversary. It will be an event to remember, I'm sure! Now I just need to figure out how to pay for it…

Goal #4) Lead a Yoga Class

I don't know a thing about yoga. I've never taken a class and know only a few movements from Wii Fit. So why would I make a goal to lead a class? Because I'm interested in it, and no one ever achieved anything meaningful by aiming low. I'll need to join a class and become proficient enough over the coming year to teach a community class by the end of December. I've already picked out the location and instructor to help me achieve that. It's just a matter of getting it done.

Goal #5) Reach 1,000 Visitors Per Month On Dicejockey

Yep, my blog wasn't going to slide by without a major goal this year. Dicejockey's not about Top Ten lists or Five Steps to Success, but I like to think I say things worth talking about, and that's worth pushing harder on. My life is my passion, and Dicejockey is a window to that passion. With a little luck and an extra dose of hard work, I hope to make Dicejockey a popular choice for gaming geeks.

Goal #6) Collaborate on Something With Tasha

My sister Tasha and I don't talk very frequently. This isn't because we're not on speaking terms; it's just because we don't have much in common. This year I want to change that. We're both very passionate about a wide variety of topics, and I'm sure we can find something to truly connect on. I've got a tentative plan on how to achieve this goal, and I really want to get closer to my family, so I know I'll meet this goal by year's end. So those are my goals for 2013. How about you? What are you looking to achieve this year?