Hacker News Personal Blogs 2011 | Top 100 Blogs

searchcode now supports regex code search

Ben E. C. Boyter — Sat, 17 Dec 2011 06:52:21 GMT

searchcode updates now supports regex code search

Similar to the soon to be defunt Google Code Search you can now search over 1 billion lines of code from github, bitbucket, sourceforge, google code and codeplex on searchcode. All you need do it search using a regex such as /[cb]at/ or /a{1,5}/ etc… You can filter by file extension as well using ext:[FILETYPE] EG, [/print_r/ ext:php][3] The only catch is due to constraints you can only do prefix regex not infix. Essentially this means the following search /lisp*/ will work as expected but /*lisp/ may not. The reason for this is that I am a single founder without funding and resources are finite. Assuming there is a demand for these features I can ramp up pretty quickly however (just a few config changes). Some searches such as [/[a-z]/][4] will take a while to return initially as the system warms up. You can see a lot more examples on the [features][5] page.

Google, Fix your Google+ API

Nemo's Home — Tue, 06 Dec 2011 02:00:00 GMT

Dear Google,

Please release the Google+ write API as well. In case you’ve forgotten, its been 6 months since you launched. I’ve got stuff I wanna post, stuff I wanna develop. Please keep to your promises, and don’t make me quote Steve Yegge.

Also, while you are at it, please fix this issue and make the list of +1d url’s available.

Thanks, A frustrated developer and Google+ user.

Someone jailbreak my iPad

Nemo's Home — Mon, 05 Dec 2011 02:00:00 GMT

This is a rant, mainly targeted at Apple. If you are a Apple fanboi, just go elsewhere

Please, and quickly. I can’t take another minute of this iTunes. It refuses to properly sync songs, does not update my songs tag info, unless I play them individually, and to top it all, my iPad gives out incorrect usage information about my apps. I tried adding them up, and it seems the total is way off. And since everything in thr Apple land lives happily inside the propertiery Apple File System, I cannot even check it realiabily. And don’t even get me started on the the application transfers. The file sharing system is so bad in iOS that almost evey other app comes built in with a wi-fi or ftp server. When alomst mevery app that deals with files starts ro do that, there is something definately wrong with your file system approach.

And why doesn’t my itunes recognize any video files at all. I was given a warning about needing QuickTime, and I installed it, but it still refuses to play them in iTunes, and doesn’t even give me an error string to hold on to.

Seriously, iTunes go back to the bloatware land that you came from. I’ll try my luck with something else from now on. And only if someone could give me the specs for the arcane plist format, I might write a PC version of the iBooks app. And calibre needs to fix up its pdf to epub approach. Inline styles are not used anymore, doesn’t it know already.

A few days later…

After spending a few more days with my iPad, I have developed a love/hate relationship wit it. I hate the ugly sync process, which still refuses to sync videos, telling me I have to delete everything to start sync. I love it for its ease of use, awesome touch, and aesthetics. I hate it for its lack of support, oepn sync protocols, and most of all Apple’s close mindedness regarding unsigned binaries.

Seriously Apple, if tablets are the computer of the future, treat them as auch. Let me treat it as a fully functional comouter, which lets me run whatever I want, with whatever privileges I want.

If Microsoft had tried to create such a walled garden of apps in Windows, would it have been so successful? Granted, people would love running apps from Windows 8 Marketplace, if only because it would keep their software updated. But Windows at least has a choice,to allow me to download a tiny binary from a small Windows Developer who does not want to pay MS just to let people use his/her apps.

An appstore is an excellent idea, executed brilliantly by Apple, but please follow MS and give us an official way to do whatever the hell I want to do with my iPad.

A few weeks later…

I really hate my iPad now, especially once I realise that my cheap Nokia cell phone has tether support, while my iPad does not. To do any task that may require you access to a normal file system, such as cloning a repository from github, or editing an document anywhere is impossible in the device. I love playing games on it, though, but it seems like a really costly device to play games and read books on. Get the Fire at less than half the price, and root it to install cyanogen mod (which will soon be available for the Fire).

Sometimes, I wonder if Indian carriers would have installed Carrier IQ on an iPad (which comes without a SIM card). I’ll find out as soon as I can root it.

One of my friends gave a very serious comment on the iPad : “It seems as if you have rented the device from Apple, rather than bought it”. And I’ve come to realise that its a reflection of the sad state that the electronics manufacturing industry is in today.

A few more weeks later :

I jailbroke my iPad :)

Further links :

How Heroku Works - Hiring

CRAIG KERSTIENS — Fri, 02 Dec 2011 22:55:56 GMT

I alluded in earlier posts of How Heroku Works that we have talented engineers. In fact I would venture to say that there is not a weak link when it comes to our engineers at Heroku. Ensuring we have talented engineers makes it easier for us to find other talented engineers and maintains a level of quality in our product. This means we must be very careful about not diluting our pool of engineering talent, which is where our hiring process becomes especially key. By the time we hire a new employee, we know without a doubt they’re a fit within our organization.

Our goal in hiring is seldom to fill a role, but more commonly to find more talented people share our goal (changing the world for developers).

So what’s our hiring process look like….

Review Resume/Github Profile
Initial Screen
Second Screen
Starter Project

While there’s definitely a process that we follow that’s not what’s interesting. We way too often get worried about the steps 1, 2, 3… Instead you should focus on what’s important: are they a fit? Can they get shit done? Who cares about how many phone screens someone goes through?! Five phone screens instead of two doesn’t make them a better fit for your company. The short of it is they go through enough screens that you feel comfortable and you progress them through the process. For us at any point in the process if someone is determined to not be a fit the process ends there. If the process does end the hiring manager will relay this in the appropriate form, though always in writing via email as well.

The hiring manager could debatably be the biggest difference between our process and others. When a candidate applies to a position it goes to the hiring manager (not an HR person). The hiring manager will be your manager once at Heroku it’s one in the same and this ensures from the start of the process the candidate and the manager mesh well. Yes, having the manager of a group review github profiles and resumes is extra effort, but who better to judge from a quick glance than engineers. In general as a manager you’re evaluated on the success of those you manage, as such you should be invested heavily in those you hire. In addition to this we find a big difference in the on boarding process and how quickly someone can succeed. We have used many approaches, but the success of someone at Heroku based on having their hiring manager and manager be the same individual is best highlighted below:

While every step in the hiring process is valuable starter projects may be the most valuable to ensuring quality. The final step with nearly everyone we hire is to invite them to come hack with us. Instead of parading someone around for a day long interview we get down to business and write some code. It could be something internal to Heroku, it could be an open source project we use, it could be something interesting that the candidate feels would add value to Heroku. Starter Projects vary slightly between each hiring manager.

Several of our managers prefer to lay out several potential interesting projects, talk through them with the candidate, and then let the candidate decide what they’d like to work on. Sometimes there’s a pressing need that the candidate can jump right in and add some value. It’s always important that the starter project is achievable, if it’s too broad of difficult for a 1-2 day period then the manager has failed in the hiring process. Regardless of the project it’s far more than an exercise on a white board, it’s actually what life is like at Heroku. We have lunch at the same table that we eat at every other day, we interact just as we normally would, and after work there may or may not be drinks just like any other day.

As a slight aside, we even conduct starter projects when current Herokai move from one team to another

Starter projects usually last anywhere from a day to several days. At the end of a starter project the candidate presents what they did, in a similar fashion to weekly demos that occur at workshop (more on that some other time). In earlier days it was nearly the entire company that would sit in, ask questions and give feedback. Now it’s a bit harder for all us to fit into one conference room, though there’s an open invite and anyone that wishes can sit in (often 10-20 Herokai). At the end of the starter project there’s no question that the candidate fits or doesn’t, often from both sides. Of course if it’s a fit we make an offer and welcome them into the family.

Wildcards

Ben E. C. Boyter — Mon, 28 Nov 2011 06:49:22 GMT

Just a quick update to let you know that wildcard searches have been updated to support prefix and postfix.

Essentially this means the following searches now all work,

*print

*clo*

clo*

In addition code characters such as { } [ ] < > are now being indexed. Finally searchco.de has been on a bit of a diet and should be a lot faster to load and search.

Skyrim: A Nord's Tale, Part 4

Ben Overmyer's Site — Fri, 25 Nov 2011 02:00:00 GMT

I formally joined the Companions, and learned that the Circle, that group that is the central leadership of the Companions, all are werewolves. I heard through the grapevine that players who join the Circle can also become werewolves. Since I've committed to the Companion track, I think I'll gun for that. I also took down two more dragons and learned a little more about that storyline. It looks like there's a big master dragon that's resurrecting other dragons. I forget his name, but he mocked (pitied?) me when he resurrected another dragon in front of me. Curious. I need to explore that more.

In light of the limitation on how many skills you can master, I've stopped exploring alchemy and magic in favor of my skill focuses - Heavy Armor, Two-Handed Weapons, Smithing, and to a lesser degree, Enchanting. I may drop Enchanting, depending on how easily I can acquire magical items in the future. I like the idea of smithing magical jewelry, and I'm starting to amass quite a collection of components for that. My most recent perk acquisition is Dwarven Smithing. Dwarven armor seems to be just a little better than steel armor, so I'll be upgrading my armor as soon as I can to that standard.

My Experience with the Deloitte Cyber Collegiate Threat Competition (2011)

Nemo's Home — Sun, 20 Nov 2011 02:00:00 GMT

I recently was part of a team at IIT-Roorkee that won the Deloitte Cyber Collegiate Threat Competition. It was a competition modeled after The Deloitte sponsored CCDC in the US. The event will be organized in the subsequent years as well, and hence this blog post will summarize my experience so as to help any future participants. Moreover, the organizing team has guaranteed us that the competition will be altered significantly in the coming years. This was the first year that this event was organized, after all.

I’ll go into the contest round by round, so beginning with Round 1.

Round 1

Deloitte came to our campus, with little promotion about the event. A presentation was given on the current scenario of Cyber Threat, particularly with respect to India. Free swag was awarded to people who asked some good questions, or answered some as well. After the presentation, a quiz was goven out, consisting mainly of questions about Web Application Security. A few of the questions asked to write down code to circumvent a particular issue (like SQL Injection). But it was mostly about stuff that every security conscious Web Developer would know about.

After the quiz, they selected the top 9 candidates, and asked us to form teams. Make sure that you attend the quiz with your friends, as we definitely had an edge by knowing everyone in our team before the event. The number of teams varied from campus to campus. But if you perform decent enough, you will be selected.

Round 2

Each of the teams were given a VM Image, and we were asked to hack into it. We were not allowed to exploit vulnerabilities in the guest OS, or things like VMWare, or try to boot into the image with another OS, but other than that everything went.

The VM had a library application, with several vulnerabilities. A challenge sheet was mailed to us, and we were expected to finish as many challenges as we could. Any further vulnerabilities not mentioned in the challenge could also be mentioned, but theywere only to be used in the case of a tie with another team.

The time duration for Round 2 was 15 days and we were supposed to submit our reports by then. We were able to complete most of the challenges after we found a blind sql injection vulnerablity. Further, we were able to get a copy of the obsfucated PHP code, which we converted to simpler versions easily enough. We had no way to make use of the code, but it did help us in identifying possible files and entry routes for vulnerabilities.

To get a good score in round 2, try to attack every point in the application. In our case, some of them were too stupid to be used in a real case scenario. For instance, we had password hashes appearing in images. Pour through the javascript code, and search like hell. Stuff like w3af might help you, but since its a limited application only, it is often easier to just track the application flow. We did try kernel level exploites, but the VM was fully patched and up to date.

Round 3

Round 3 was organized at Hyderabad and was a head on hack everything contest. We were handed 3 virtual machines, with lots of vulnerable services. We had to keep those services running, which were periodically pinged by a scorebot. Scores were awarded in three categories : attack, defense, and flags.

Attack points were earned upon getting a shell on any of the other team’s servers.

Flag points of awardedon the basis of getting access to secrets stored inside the other team’s servers.

Defense points were earned on the basis of status of your own service.

The network architecture was 3 tiered. A single central router routed requests to a team’s router, which was then connected to an individual team switch. A switch was connected to the host VM, and the attack machines. Two different subnets were created for attack and defense in each team’s router. All uVMs were present in the attack subnets.

Day 1

Day 1 consisted mostly of us learning about the network and trying to gain access into the other systems. All the services were highly vulnerable, and as a result, we had to patch that vulnerability in our own servers before we attacked anyone with it. DoS attacks started late in theday, but were ever present.

The VMs handed t us included a Windows Server 2003, a debian, and an Ubuntu. Only open source/freeware tools were allowed, and we used lots of stuff including :

Backtrack for almost everything since attack laptops given to us had Windows installed
LOIC for DoS attacks
Wireshark for packet analysis.
Snort for intrusion detection.
NMap for scanning services
Metasploit for trying out exploits
Cain and Abel for miscellaneous stuff

Day 2

Day 2 involves lots of pwning,and a surprise twist. All VMs for day 1 had been reset to their current state, and we had to do patch them all over again in first fifteen minutes of the session. Other than that, the increase in traffic was exponential. All our machines were scanned tonell. DoS attacks became normal, and the epic moment of the day was during the last session when we had our router pwned.

Pics from the Event are at Facebook.

Conclusion

Kudos to the Deloitte Team for organizing such a brilliant contest. We had lots of fun. They have assured us that next year it will be even bigger and better. And that the format will be entirely Different. Ext hear so this blog post might not be as helping as you may have thought.

Skyrim: A Nord's Tale, Part 3

Ben Overmyer's Site — Fri, 18 Nov 2011 02:00:00 GMT

So I finally solved the Sapphire Claw dungeon... something Shroud Barrow. Ended up being a Draugr corpse blocking my ability to depress a button to lower the drawbridge. I had to use my Unrelenting Force Shout to move it aside so I could lower the bridge.

Once that happened, though, things got awesome. I fought more Draugr, took out a new Draugr Overlord, and learned a new Shout… which I haven't yet tested out. On the way back, I found a bandit hideout with an AMAZING view of Whiterun. I may run that dungeon again just to see the view. It's simply awesome.

I went out looking for iron ore deposits, because I was excited to try out mining. I found a few, but the highlight of that particular adventure was when I was wandering along a path and, all of a sudden, a dragon dropped out of the sky right on top of me. Scared the crap out of me! I dodged out of the way and let Lydia shoot it with her bow while I hammered away at the thing with my Whiterun Axe. Eventually it went the way of its predecessor (Dragon Kill Count: 2), but that certainly was not something I was expecting. The long run to Whiterun hauling massive amounts of dragonbone was a very anticlimactic aftermath.

I'm getting closer to my goal of owning a house in Whiterun. 3,000 gold down, 2,000 to go. I'm trying to level my smithing and enchanting skills too, but those are expensive, since I don't always have the ingredients I need to craft things. Enchanting in particular is pricey.

The Big Story wants me to head northwest to a far-away town. Eventually, I will go that way, but for now I'm concentrating on exploring the area around Whiterun.

Skyrim: A Nord's Tale, Part 2

Ben Overmyer's Site — Tue, 15 Nov 2011 02:00:00 GMT

Alchemy in Skyrim is vaguely unpleasant at first, since in order to discover the first alchemical property of something, you must taste it. This is all fine and good, but some of these ingredients are less than appetizing - skreevers' tails, for example. On the other hand, blacksmithing is a real gem. I've learned how to create steel items now, though I don't yet have steel ingots. I discovered something interesting - iron ore deposits. I didn't have a pickaxe with me at the time, and apparently that's required to mine ore. I'm excited about this development, though - it means I don't have to buy ingots or ore! I became a Thane of Whiterun by helping defeat a dragon, and the Jarl assigned Lydia to me as my own housecarl. She's come in very handy, though her pathing AI has been very irritating at times. She was probably the only reason I survived an encounter with an ice troll, though she herself didn't remain conscious for the entire event.

Trolls are brutal, especially if left to heal for a few seconds. After ascending the Throat of the World to meet with the Greybeards, they gave me additional Words of Power to give me one new Shout and beef up my existing one. I like the Norse feel of the draconic language; very different from the usual Elvish/Latin draconic languages. Returning from Hrothgar, I solved the mystery of a ghost in a nearby barrow, only to discover a very complex dungeon hidden beneath it. As of this writing, I'm stuck on a puzzle in its depths. Supposedly, I should be able to rotate four stone pillars to lower a wooden bridge, but despite doing what I thought was the correct pattern based on clues in an adjacent room, the bridge remains stubbornly vertical. No idea how I'm going to fix this one.

Getting Started with Django

CRAIG KERSTIENS — Sun, 13 Nov 2011 05:55:56 GMT

For those completely new to web development, Django is a web framework that makes it easier to build web applications with Python. For those that have some knowledge of other web frameworks and Django you may be able to fly through much of the following. Django is a slight modification on the MVC construct which views itself as a MVT Model, View, Template. Django views a website as a project and within it smaller apps are contained.

Earlier we installed Django into your virtual environment. If your environment is loaded we can get started with a Django project. First lets create the project:

django-admin.py startproject myproject

This should have created a directory for your project called myproject. Within the myproject folder you’ll find some core files to every Django project.

$ ls
myproject venv
$ ls myproject
manage.py myproject
$ ls myproject/myproject
__init__.py settings.py urls.py

Let’s examine a little of each of the files and what they’re used for:

manage.py - A management utility for interacting with your Django project. In addition to the default commands available which you see by running python manage.py, you can create custom commands (we’ll get into that much later). settings.py - This is the settings file for your application. Here you’ll put various configuration and load things such as your database connection. urls.py - This is the place to setup how your urls will work. You’ll define a path for the url and then which code is to be executed when you visit that url.

Skyrim: A Nord's Tale, Part 1

Ben Overmyer's Site — Sat, 12 Nov 2011 02:00:00 GMT

Last night at midnight, I purchased Elder Scrolls V: Skyrim for Xbox 360. This morning I got up at my usual 5:30 and went straight down to the living room to play.

The next few blog posts will be brief updates on my adventures in Skyrim. They'll be out of character, more a log than a story.

After a dragon's attack conveniently freed me from captivity and nuked my captors, I was spirited away to a Nord village by a fellow survivor. As a Nord myself, it was interesting to see the culture surrounding the village, and how the villagers reacted to my chosen race as a familiar stranger.

Of the massive arsenal I acquired in the hour or so of play this morning, the weapon I've used the most has been a two-handed battle axe. I loved that I could, straightaway, put the axe on the grindstone and use an iron ingot to turn it into a [Fine] Battle Axe. Crafting systems ROCK!

I'm a little overwhelmed by all there is to do. With so many choices and quest givers practically throwing themselves at me, I just shut down and ignored the quests, opting instead to do some exploring. I plan on delving into the inventory/item system next, checking out what I can do in that arena.

I want to see if there's something to these spider eggs I picked up on my flight from the dragon...

Simple PHP ASCII Art Generator

Donat Studios — Thu, 10 Nov 2011 21:18:43 GMT

So yesterday I was bored and asked my friend what I should do. She replied "I don't know, draw ASCII art or something!" and sitting there with the terminal open I kind of wondered what it would take to write one of those image to ASCII scripts.

20 minutes later I had the following PHP Shell Script! It works by simply sampling the saturation of every Xth pixel and mapping that saturation to a character from an array. There is no nearest-neighboring or anything like that, just straight up sampling.

Huzzah!

And there you have it! A whopping 35 lines of code. There is a ton of tweaking you can do like changing the array of characters, or adjusting the $scale variable which is simply how wide of a sample it makes. Setting it to 1 for instance should sample every pixel on the Y and every other pixel on the X (so it scales properly). To the right is a sample output of the script.

Try it for yourself!

The script accepts local or remote image paths as the first argument. The following is a simple usage example you can use to get you started.

./ascii.php "https://donatstudios.com/images/site/JesseDonat.jpg"

It was a fun little project and I look forward to seeing if/what the internet comes up with for it. Fork me on Github, or post links in the comments! Happy hacking!

Postgres... The death of NoSQL

CRAIG KERSTIENS — Tue, 08 Nov 2011 22:55:56 GMT

NoSQL has long been a trend that many have talked about. While there’s a place for various key-value stores and tools such as memcache and redis, this will address most specifically how NoSQL is attempting to replace a traditional database. I’ve long been a fan of postgres and in general traditional relational databases. In a broad sense traditional databases offer multiple things.

RDMS

Data guarantees

The current major SQL databases (SQL Server, Postgres, MySQL, Oracle) offer guarantees around your data that doesn’t always exist with other systems. At a very high level this means when they say they have the data there’s not a chance they’ll loose it. When using something as a primary datastore this is always my first requirement. Data is a valuable commodity so keeping it around is obviously important. There are cases where exceptions exist (reporting applications are common here). The specific thing I always look for is that a system upholds the ACID properties. For a quick breakdown of these:

A is for atomic. In short it means no transaction can be partially completed, its all or nothing.
C is for consistent. This means you go from one consistent state to another. Meaning things like cascades and constraints are upheld and can’t be ignored for a period of time.
I is for isolation. This means transactions don’t get to interfere with each other.
D is for durability. This means once the transactions there its not going anywhere.

These basic principles make me feel pretty content with my data being safe. This doesn’t include things like backups and replication, but rather is a baseline for me feeling safe with a system.

Here’s a hint, many NoSQL solutions don’t enforce these which is where they get speed from

Consistent means for accessing data (SQL)

Many people complain about SQL and while its not a perfect language it is a common standard for accessing data. There are idioms that exist in Oracle that do not in Postgres and ones that exist in MySQL that do not in SQL Server, but on the whole ANSI-SQL is a common standard. This means if you learn one in large part you learn another (from an application developers perspective). This means you have a broader people to pull from when you consider moving from one to another, and that skills are more portable. While Mongo may be growing, its in no way guaranteed to be around in 5 years, nor is CouchDB. In fact there have been many NoSQL databases that have come and gone:

Tokyo Cabinent

Postgres

So there’s some shared things that make databases great, but in particular Postgres aims to be the single database capable of ushering in a death to NoSQL. While each item could easily be its own blog post hopefully the following calls out they key values and allows people to dive in deeper.

HStore

I strongly debated saving the best for last, but really just couldn’t wait. If there’s a single feature in Postgres that will kill NoSQL its HStore. HStore is a data-type that allows you to store a dictionary within postgres.

Custom datatypes

PostGIS

Location is all the buzz these days and more and more applications have some tough of location involved in them.

How Heroku Works - Maker's Day

CRAIG KERSTIENS — Mon, 07 Nov 2011 22:55:56 GMT

In my earlier post on Teams and Tools at Heroku, I mentioned how we value engineers’ time; their work has enabled us to build a great platform. As a result of what we’ve built, we’ve had great growth both of our platform and of our teams internally. With that growth inevitably comes different distractions on engineers’ time. Despite how a manager may plan things, engineering work needs long periods of uninterrupted time. To ensure that no matter what, an engineer has plenty of opportunity to do the work he or she was hired to do, Heroku has Maker’s Day.

Maker’s Day ensures that engineers get a full day of uninterrupted time to focus on making things.

The more consistent interruptions are throughout an engineer’s day, the more time will be lost due to context switching in addition to the time spent on those other activities. These interruptions may include a quick question from a manager, a question on a code problem someone else is working through, or an email or IM from a coworker. Regardless of the type of interruption, it causes an engineer to lose focus. According to Peopleware: Productive Projects and Teams, in a study regarding productivity among engineers, the top performers when surveyed said they were interrupted regularly 38% of the time versus the bottom performers, who were interrupted 76%. Context switching should be counted as fully wasted time for an engineer, and all too often as the number of meetings increases, the time involved with context switching is increased similarly to the following:

For more on how interruptions or context switching decreases productivity, Jeff Atwood has a great post about The Multi-Tasking Myth, which demonstrates …

Most people understand that context switching is bad, but another team may still have valid demands on your time. Pushing back against another team or manager isn’t always feasible; after all, we do work together, and each team at times may need something from another team. This is where Maker’s Day starts to come in. Every Thursday at Heroku is Maker’s Day.

Maker’s day is meant for making shit. Meetings don’t happen on Maker’s Day. If someone asks if that time on your calendar works for a meeting, the simple response is no–it’s Maker’s Day. Because Maker’s Day has been ingrained into our culture, engineers have no problem giving that response when there’s a request on their time on Maker’s Day. If someone in marketing, sales, or another non-engineering role wants to book meetings, they’re welcome to do so, but they’re going to be without engineers. However, even for non-engineers, Maker’s Day is equally invaluable; uninterrupted hours of focus at a time are amazing for productivity in any role.

Maker’s Day varies in how it is executed from person to person. Often the office is slightly less busy due to some engineers working from home or coffee shops to maximize their productivity. To an outsider, the office may appear business as usual: engineers sit at their desks, working. At lunch, everyone is sitting around the lunch table eating together. To the unobservant eye it may appear to be just any other day, but the engineers notice the difference. There will be significantly less interruptions by someone walking over to your desk, you won’t be pulled into meetings that distract you from features, and you know it’s an opportunity to accomplish a bulk of work laid out from your weekly planning meeting.

As Heroku has grown, meetings have increased, and the value of Maker Day’s has increased exponentially.

Whether you’re in the early stages of bootstrapping a company or at a large company of thousands of engineers, one of the best practices anyone can put into place is dedicated quality time for engineers to produce code. Maker’s Day is a fantastic way to ensure this happens on a weekly basis.

How Heroku Works - Teams and Tools

CRAIG KERSTIENS — Wed, 02 Nov 2011 22:55:56 GMT

Heroku is a largely agile company, we work in primarily small teams that talk via api and data contracts. Its also a company comprised primarily of engineers, even product managers often write code. Heroku as a platform drives many of the features not from top down, but from bottom up based on engineers desires or skunkworks projects. There’s many valuable insights into how Heroku runs efficiently for engineering.

I’ll be diving into many various practices that enable Heroku to put quality engineering above all else, but first let me highlight the team structure and tools that enable this.

Heroku is comprised of many small teams internally, each team operates much like an individual entity. The team chooses its own tools and best method for communication, though as a whole some form of Scrum is run throughout teams. Think of the unix philosophy of small sharp tools as in The Art of Unix Programming applied to teams and people.

For most teams this involves a weekly planning meeting earlier in the week. In such a meeting teams may conduct a retrospective, opportunities to improve the process the coming week, and of course plan tasks for the coming week. Its very important to note that planning tasks for the week doesn’t necessarily involve planning the deadline for them, but rather simply laying out what people are working on (more on this in a future post). Each team will record and track this in a tool of their own choosing. Several use pivotal tracker, one uses scrumy, some use email to distribute and track against personal to do lists. The method for tracking issues is again entirely up to the individual team. A one person team may choose to use a simple to do list, larger teams commonly use github issues and pull requests. Given the team is the one responsible for their own productivity the team is the one to choose what tools they use.

Meeting loads vary from person to person depending on what is the demands are on their time, though everyone at Heroku participates in some form of standup. Most teams do these daily as quick status stand-ups of what was worked on the day before and whats to be worked on the next day. In addition to the planning meeting and stand-ups, there is often collaborative engineering, and company wide gatherings.

Collaborative engineering once again varies depending on which engineers are working together. Engineers may get in front of a white board or in front of machines and simple collaborate. For engineers together in the office this is often the most productive way. These practices work the same for remote employees, though slightly modified for the high touch interaction. For remote employees this often works as pair programming via Skype. Skype is indispensable for allowing remote workers to feel far less remote. Skype alongside typewith.me and you have an unbelievable easy to collaborate not just with 1 other, but with multiple parties to work through a document on a given topic. For smaller activities of communication asynchronous is key. This ranges from campfire most commonly during common working hours when someone is likely to be at a machine, to email when the return on a request may take slightly longer.

Finally there is the all common company wide meeting, which occurs weekly. The structure of this varies from status updates to broader ongoings. Its often the perfect time for engineers to hear about what sales is doing or get updates on teams you don’t commonly interact with. Along with common status updates there will be broader company updates.

Consistently across all teams you’ll find these principles which allow us to ensure the quality of engineering as we continue to grow:

Small teams that talk across defined API’s and data contracts
Teams using the tool that they believe is best for the job
Frequent asynchronous communication
Collaboration (including for remote employees)

The key in Heroku running efficiently is primarily allowing each team to run as it chooses. Heroku works because we have talented engineers, the best thing we can do for those engineers is allow them to work productively. Often only they know the best way to accomplish this, so who better to let them accomplish it than themselves.

Installing Python Packages

CRAIG KERSTIENS — Tue, 01 Nov 2011 22:55:56 GMT

Now that you have you system and project environment all setup you probably want to start developing. But you likely don’t want to start writing an entire project fully from scratch, as you dive in you’ll quickly realize theres many tools helping you build projects and sites faster. For example making a request to a website there’s Requests, for handling processing images there’s Python Imaging Library, or for a full framework to help you in building a site there’s Django. With all of these there’s one simple and common way to install them. But first a little more on how it all works.

All major Python packages are hosted on PyPi (Pronounced Pi-P or Cheeseshop). When you use a common python installer it will:

Search for the package you specify
If you specify a version will use it, otherwise will use the latest
Will download the source for that package
Install it into your Python environment

Now for actually installing… Lets get started with installing the three packages below. At this point you should at least have a fresh Python environment, however you don’t have an immediate way to install packages. The defacto Python package installer is pip.

Earlier we setup virtualenv to help isolate our python packages we were working with. First lets go ahead and create a folder for our project then setup a new environment for the project we’ll work on:

$ mkdir myapp
$ cd myapp
$ virtualenv --no-site-packages venv

If we list the contents of the directory you’ll now see a folder venv. Within this folder you’ll find all the parts of the environment that virtualenv just created:

$ ls
venv
$ ls venv
bin include lib

Now you’ve got a sandboxed environment that exists but you haven’t loaded it. You can now activate and deactivate this any time you like. Once you do this it customizes your path to use the packages you’ve installed for this environment. To load your environment when in the myapp directory:

$ source venv/bin/activate

To deactivate this simple:

$ deactivate

Now that we’ve got your environment loaded installing your packages should be pretty simple. Ensure that you have your virtualenv loaded and then run:

$ pip install requests
$ pip install PIL
$ pip install Django

Now that you’ve installed your packages you want to be able to share this with others to make it easy to get setup. You could provide a list of everything your application needs to run manually, or because its Python you can expect it to make it easy for you. Pip has a wonderful command freeze that will show all of your packages and their versions that are installed. Simply run:

$ pip freeze

However, this only outputs the information. Along with this pip has a canonical form for listing requirements and installing them from a file. The filename is commonly a requirements.txt. To create this we simply pipe the results of pip freeze to this file.

$ pip freeze > requirements.txt

Next we’ll talk about a few more advanced items in dependency management, then finally we’ll get started on building an application.

Getting Setup with Python

CRAIG KERSTIENS — Fri, 28 Oct 2011 06:55:56 GMT

This is the first of a multipart series to getting started with Python. Throughout this guide we’ll walk you through a full setup. For starters if you’re a mac or linux user you already have Python on your system. You should be able to confirm you have python my opening up a terminal window and running:

$ python --version
Python 2.7.2

As long as you see a Python version 2.5.x-2.7.x you should be fine to continue. From here we’re going to work through setting up your Python project environment. For this we’re going to use virtualenv. For those of you not familiar virtualenv is a self-contained python environment. It holds its own copy of python and any libraries you install. This allows you to work on multiple projects with different versions of libraries.

While we’re installing virtualenv we’re also going to go ahead and setup PostgreSQL as we’ll be using it later. If you’re on a mac you’ll first need to setup homebrew. Homebrew is used for installing various system packages. If you’re on linux, in particular Ubuntu you can skip down to the steps for setting up your environment.

First for Mac users lets setup homebrew which will allow us to install various system packages:

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

$ curl -O http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg | sh setuptools-0.6c11-py2.7.egg
$ easy_install virtualenv
$ brew install postgresql

$ sudo apt-get install virtualenv
$ sudo apt-get install postgresql

Now you have Python, virtualenv, and postgresql all installed. We can now focus on setting up the initial start to a project.

In the next part we can start with installing some Python packages.

Hello World Collection

Ben E. C. Boyter — Tue, 18 Oct 2011 07:48:09 GMT

I was looking through the logs of what people are searching for and noticed there is a large amount of “Hello World” searches such as “hello world java” which actually came up with no results. I couldn’t have that continue so I quickly integrated the Hello World Collection

You can now search for hello world in 400+ languages (although curiously no in Clojure). Just type hello world and the language you are searching for EG,

Underscore.js joins the party

Ben E. C. Boyter — Mon, 17 Oct 2011 07:45:28 GMT

Had a spare moment and added underscore.js documentation in.

underscore.js template

_ pluck

Still working on C++ Ruby and C#. I will be adding backbone.js soon however as I am using that on my own projects.

Github +1 URLs

Nemo's Home — Mon, 17 Oct 2011 03:00:00 GMT

I was working on the Google +1 Listing API (undocumented). So here’s a list of my current +1 urls on github.com. Most of the projects pertain to web-designing. I’ll update this list automatically every week or so, provided I remember to/set a cron job.

jayferd/color.js	color.js - The missing color library
typicaljoe/taffydb	taffydb - TaffyDB - an open source JavaScript Database for your browser
antirez/lamernews	lamernews - Lamer News -- an HN style social news site written in Ruby/Sinatra/Redis/JQuery
felixge/node-mysql	node-mysql - A pure node.js JavaScript Client implementing the MySql protocol.
daneden/animate.css	animate.css - A big ol' goody bag filled with CSS animations for WebKit, Firefox and beyond.
javve/list	Do you want a 7 KB cross-browser native JavaScript that makes your plain HTML lists super flexible, searchable, sortable and filterable? Yea
azer/jekyll-social-activities	jekyll-social-activities - a jekyll project template to list social network activities
e1ven/Robohash	Robohash - RoboHash.org
donpark/node-robohash	node-robohash - node.js implementation of Robohash. It's neither complete nor render general SVG.
lg/marshmallow	marshmallow - An open source Campfire server
tmcw/big	big - presentations for busy messy hackers
All of the Hooks	Service Hooks are available for more events (issues, pull requests, forks, etc). Update them through the API!
chromakode/karmabot	karmabot - A highly extensible IRC karma+information bot written in Python.
moserware/PHPSkills	PHPSkills - An implementation of the TrueSkill algorithm in PHP
nodejitsu/docs	docs - Community powered rocket fuel for node.js
spin.js	An animated CSS activity indicator with VML fallback.
Alice.js Demos	Alice.js Demos. Alice.js (A Lightweight Independent CSS Engine) is a micro JavaScript library focused on using hardware-accelerated capabili
tcorral/Design-Patterns-in-Javascript	Design-Patterns-in-Javascript - Based in examples on Head First Design Patterns
atduskgreg/srender	srender - John Resig's Simple Javascript Templating turned into a jQuery Plugin
mrdavidlaing/functional-javascript	functional-javascript - A fun set of koans to teach you functional programming techniques in Javascript
mrdavidlaing/javascript-koans	javascript-koans - Koans to learn Javascript
robrighter/current	current - Node.js app for visualizing http requests on a lan
bcoe/endtable	endtable - A ridiculously simple Object Mapper for Node running on top of CouchDB.
sproutcore/sproutcore	sproutcore - JavaScript Application Framework - JS library only
tpope's Profile	tpope (Tim Pope). You're not logged in! Login; Pricing & Signup. Name: Tim Pope. Website/Blog: http://tpo.pe/. Company: Waiting on t
tpope/vim-fugitive	vim-fugitive - fugitive.vim: a Git wrapper so awesome, it should be illegal
github/gitignore	A collection of useful .gitignore templates
Chosen - a JavaScript plugin for jQuery and Prototype - makes select boxes better	Standard Select.
harvesthq/chosen	chosen - Chosen is a library for making long, unwieldy select boxes more friendly.
rthauby/Paige	Paige - Super simple project page generation
docco.coffee	Docco is a quick-and-dirty, hundred-line-long, literate-programming-style documentation generator. It produces HTML that displays your comme
LeaVerou/prefixfree	prefixfree - Break free from prefix hell!
twitter/scala_school	scala_school - Lessons in the Fundamentals of Scala
arcturo/library	A library of free eBooks we're working on
tcr/selection.js	selection.js - A tiny JavaScript DOM selection library for modern browsers and IE5-8.
First Annual Octocat Dodgeball Invitational	Why? We were brainstorming in the office and decided we should throw balls at our enemies. But why stop at destroying our enemies with foam
nide - Beautiful IDE for Node.JS	nide. Beautiful IDE for Node.JS. nide is a web-based IDE for Node.js, designed with simplicity and ease-of-use in mind. The current version
mikeal/request	Simplified HTTP request client.
unconed/TermKit	TermKit - Experimental Terminal platform built on WebKit + node.js. Currently only for Mac and Windows, though the prototype works 90% in an
If Dropbox Used GitHub’s Pricing Plan	If Dropbox Used GitHub's Pricing Plan. What if Dropbox used GitHub's pricing model? Folders? Yes, folders. I have a lot of folders.

Google Killing off Code Search

Ben E. C. Boyter — Sat, 15 Oct 2011 07:43:32 GMT

Google has just annouced that they are killing off two new products, Buzz and Google Code search (read here). Buzz is a no brainer (although I am certain someone out there is using it) but code search is a little sad. Its also both annoying and an opportunity for http://searchco.de/

Its annoying because I have branch in my source control which integrates with Google code search. I was about to push it out too. This kills a few hours integration work which is annoying. I can understand Googles point of view though. I imagine that the Grep code search was rather CPU heavy and that storing all that source code (including versions) would have really eaten up some disk space. I would be curious to see their stats though as im certain there is enough traffic there to make something viable.

Must Use Web Applications

Nemo's Home — Sat, 15 Oct 2011 03:00:00 GMT

Here are a few of the applications that I would heavily recommend.

Workflowy

Here’s how Workflow describes itself.

WorkFlowy is a simple, but powerful way to manage all the information in your life.

Here’s their introductory video:

If that does not hook you, I don’t know what will.

Clipboard

Clipboard is a content archiver tool that makes it quick, snappy, easy, and cool. It has got tons of features and is still in private beta. However, since Michael Arrington blogged about it, it has begun accepting larger number of invites. I’ve only started to use it, but it has been quite awesome till now.

My favorite feature is embed. Everything I’ve embedded on this page is via clipboard, as a demo. You can clip tweets, pics, videos and what not and embed it on your blog easily.

Ge.tt

Ge.tt is one of the many file sharing sites that seem to have cropped up in Web 2.0. Its USP is its simplicty, however don’t be fooled by it. It has got lots of features as well:

Share URLs while your stuff is uploading
Share without even logging in
Drag and drop upload
Versioning for file Uploads
Share multiple files under a single upload
Limited Analytics (See number of Downloads)

So yes, its not as powerful as many others, but has got quite enough features to keep me busy.

Minus]

Minus is a simple image sharing service. It was amongst the first to offer Drag-And-Drop upload back before it was cool. Right now, it is trying to become the next flickr, allowing people to subscribe to each other. If you are someone who posts cool pics regularly, check this out.

My primary browser is Chromium, and here are some Chrome Applications that I use regularly:

Offline GMail

Was a Chrome extension that used Google Gears for letting you use GMail completely offline.

Pros:

Looks cool
Allows multiple accounts
Drafs facility
Labels

Cons

Not all the functionality of Online GMail

Collaborative Editors

I like collaborative editing, working together with people in real-time. Unfortunately, the biggest entrant fizzed out. However, there are still quite a lot of competitors left in the field.

Google Docs

I tend to avoid Google Docs usually, as it is too much of a bloat for me. The integration with Google Chat is good, but sometimes all you need is a plain text editor. There is also an Offline Google Docs application, although it does not allow one to edit documents. Also no presentations.

Etherpad

Etherpad is quite good, as a plain text collaborative editor. I tend it to use it frequently, and it has some excellent features. It makes a thousand revisions of each of my posts, and allows me to play through them, and see who made wat change in real-time. All this, for free. Plus you get chat, and basic formatting (bold, italics, underline).

The previously mentioned workflowy keeps your lists in sync over time, so it is collaborative, though not in real time.

Also, mention goes out to pastehtml, which does an excellent job. Its neither collaborative nor has sharing, but it had me hooked at editable markdown. You can type in markdown, publish in html, and come back and edit your documents, as per your your heart’s wish.

Grooveshark

I rarely listen to music online, but when I do, its either on Youtube or Grooveshark.

Looking back at this document, it seems that there are not many web-apps that I use.

Some other applications that I regularly use, in no particular order:

FreedCamp - A free alternative to Basecamp
IssueBurner - Simple Issue Tracking via Email
Postary - A simple blogging platform. “Write.Share.”
LastPass - Password Manager

I’ll probably add some Chrome Extensions later as well.

Coverage

Ben E. C. Boyter — Mon, 10 Oct 2011 07:41:06 GMT

It was a long weekend here in Australia and as such I decided to take it with my girlfriend and spend time totally off the grid. As I came back from my time off I had a look at my general stats and was very surprised to see some massive spikes. Looked into it a bit more and it seems searchco.de has gotten some coverage from a few sites. See the below,

Blogging with Jekyll

Nemo's Home — Mon, 19 Sep 2011 03:00:00 GMT

For the past few years, there has been a revolution in the blogging scene. People have moved towards better hosting providers, better blogging tools, with automated, and delayed blogging becoming the norm. Posts are written months in advance, and proof-read dozens of times, before making it to the general public. Wordpress, Blogspot, Tumblr, Posterous, Texpaterrn are pretty much everything that most of the bloggers use. However, there have been some silent niche entries in this market. Static Site Generators.

Static Site Generators

Static Site Generators are tools which you use to generate a static version of your site. Instead of using a dynamic scripting language (such as php), your tool takes in your markup & combines it with your blog posts to generate an html only version of your site. This version is then uploaded(for eg, via ftp) and is then visible as your blog. Most such tools are written in languages such as ruby, python, node.js, and erlang. The most commonly known are Jekyll, Hyde, nanoc, and webby. An excellent list is available here.

Benifits of using Static Site Generators

Markup

For me, the most important part of using Jekyll is that it allows me to use markdown as my writing syntax. Markdown is a markup language that is compiled to HTML. It is supposed to be a highly readable version of html. For instance it uses backticks(`) to write code.

`this` becomes this

Also, you can use * to emphasize text(strong or emphasis). Link creation is not the horrible <a href=> that you remember, but the sleek looking [Link text](Link URL). Similarly it offers lots of other features. You can even specify alternative markup languages, such as textile in jekyll configuration.

Ease Of Blogging

You can write blog entries very easily. I’ve added markdown syntax bindings to vim, my favorite editor, and geany has a markdown plugin as well. I am of the opinion that you should write text entries in a text-editor, not a textarea in a browser window. I used Windows Live Writer for quite some time and still believe that it is far ahead of anything else in the market. But the wpost format that it uses is propertiery, and as such stopped me from importing blog posts anywhere.

Revision Control

Revision Control, such as git, works best with text-files. Since your blog entries are now just plain text files, you can easily store them under version control, easily reverting blotched commits, making branches, and merging errors. And in case you do not know it yet, git is awesome!

Layout Tools

Jekyll allows you to define your layout using Liquid Templating. {{content}} translates to the variable content. Similarly you can iterate over blog posts, by using Liquid tags for foreach. The best part is that this is all done before publishing your website, meaning that the final result is always just pure html. You can easily create static portions of your site (such as headers, footers, sidebars).

Getting Started With Jekyll

I’ll start with jekyll, since it is the most used one out there, and runs on github. It even powers this very blog. First, you must install jekyll. On an Ubuntu machine, sudo apt-get install ruby rubygems && gem install jekyll should work. If you are working on development using Ruby, I’d recommend you to rvm, instead of plain vanilla distro ruby installs. For windows folks, install Ruby, and the Devkit using RubyInstaller.org. After that run gem install jekyll.

Instead of preparing a site from scratch, we will instead be forking an existing site running on jekyll, and using it to model our own. This is partly to shield this tutorial from html/css/js which is irrelevant in this case. In this particular case, I will be using my very own website. Download the source code for my website from github and extract it somewhere. Next, you will have to delete all the content inside the _posts, projects, _drafts, contact, & data directories as I do not give permission to you do use those in your site.

Next, create a file called _posts/2011-month-date.md with the following format :


---
title: Title Of Post
layout: post
---
The blog entry follows here in markdown. You can use *italic*, **bold**, [Link](http://google.com), #header1, ##header2 etc.

For more markdown details, visit its official page at http://daringfireball.net/projects/markdown/syntax#autolink

Now, open up the _layouts/ and edit default.html. You’d need basic html skills to replace my photo with your own, and change links to various places. After you are done, just cd into the root directory of the site, and run jekyll. Some output should confirm that the server ran successfully. Open up http://localhost:4000 in your browser, and you will see your web-site running.

Now for the hosting part. All of your current site is hosted in _site folder, so you can upload it anytime you want. Or, if you want, you can use github as your hosting provider. Just follow the instructions on pages.github.com and you should be be up and running in no time.

Six Months of Ubuntu

Nemo's Home — Tue, 13 Sep 2011 03:00:00 GMT

I installed Ubuntu as my primary OS back sometime in February. Not that I’d not tried it earlier. In fact, I’d used a copy of Ubuntu 3 back in day. But this time around, Windows (from my dual boot) just gave up and died. The partition with Windows got heavily corrupted, lost lots of data, and ultimately I had to format it. And Ubuntu dragged through all that. And here I’m today, a veteran of 3 Ubuntu versions, starting with 10.10, and right now on the 11.10 beta.

What have I learned ? That its better than Windows, for one. But several other things as well. I’m writing this post from the general linux-distro scene, and not just Ubuntu in specific. For the period before Feb, I’d been using Linux Mint as my primary OS for quite some time. But Ubuntu Natty Alpha brought all that Unity love (which I doted on once), and I had to move to 11.04

Reasons to switch to Linux

Its free!
Its free! No more paying up for Windows. People who had Windows ship with their computers would be delighted to know that there is something called a Windows Refund, which allows you to be compensated by the cost of Windows, if you decide not to use it.
Better file system. If you’ve ever lost yourself in the mirth of files in “Program Files”, “AppData”, “Application Data”, “sytem32” and the like, you’d be delighted to know that there is a very well balanced binary management system in Linux. All binaries are in PATH, unlike Windows, where some softwares do that, and most don’t. So you can actually run php, git, ruby from any damn place that you want. People who have tried to compile Java programs using javac on Windows might remembring updating PATH in Windows. No more of that in Linux.
Free softwares! The majority of good software on Windows is paid (and even more with Mac). But in the Linux world, (almost) everything is free. What is paid for then? Some games, high level commercial softwares and the like. But most of the stuff is free.
Package Management Apparently Windows 8 will have some sort of app-store with it. Which is a long time coming in Windows. But guess what, almost all Linux distros have some flavor of package manager built in. Debian (and Ubuntu) has apt-get, Arch has pacman, Fedora has yum. And installing softwares is as easy as it could be. Need mysql? Just type sudo apt-get install mysql. Done. Boom. Just like that.
Customization Level Even though I don’t like the messing up of Gnome by Ubuntu, there are tons of alternatives available. I’m right now using Gnome-Shell, and have plans to move to xmonad, which is another Window Manager. Almost every feature on the Linux desktop can be customized.
Terminal I’d always had planned several learn bash scripting kind of to-dos but never got to it until I started using Linux. Even if you don’t script, the actual power of your machine is opened by the terminal. Hacking away in vim, and browsing sites using elinks, and ordering Pizzas in Command Line is as geeky as it gets. Gnome is designed in such a way to allow a normal user to use his computer fully without touching the terminal, but if you use it, it gets better & better.
Applications There was a time I used to be a Windows fanatic, using WMP, Zune, Everything and what not. But now, I’ve got a bigger arsenal of softwares. Ever used Audacious? That’s my default music player, and its awesome.
Software Development You use Windows to develop stuff for Windows. I’m nowhere near to writing applications in C#. I’ll probably be hacking away scripts in ruby, node, python, bash, and building stuff using xul, gtk, webgtk, and qt. All my web applications are ultimately deployed on Linux machines, so it makes sense to write them on Linux. And only Linux has the ease of language package managers, like rubygems, npm, and pip.
Open Source I haven’t yet checked out the Linux Kernel source code, but I’m thinking of getting my hands dirty real soon. Ever since I’ve joined github, I’ve been introduced to several awesome coders, projects, and organizations. And guess what? Its all open source! Meaning I actually spend less time writing parsers for xml, and more time working on applications.

What I’ve Learned

Be utterly fearless Back when I was in Windows, a simple partition deletion used to scare me to death. Now? I’m ready for anything. If you, like me, go play with the alpha of all your favorite stuff, then things will break. And it will be fun to solve all that stuff. You will learn a thousand new things in the process, gain lots of rep on askubuntu, and become a Ubuntu Jedi Master. OK, maybe not the last bit, but you shall become utterly fearless of all danger. I had to use my computer without network accesibility for three days. And it didn’t even give me a GUI. So I just drudged along for 3 days straight on the console. :)
Community Sometimes, I feel people do not get my helpful comments, and offerings of help on Facebook. Its not their fault. They’ve never been introduced to a proper online, helpful community before. The Linux community is helpful, worldwide, readily available, and has probably handled the problem that you are facing now.
Server Administration I’m still lacking on the cloud front, as I don’t have servers powerful enough to host virtual machines. But otherwise, I’ve handled lots of stuff. ssh is my thing, and I use git to deploy applications like a pro. I’ve moved on from apache to nginx to cherokee and what not.
Take Backups Remember all those “Take Backups before installing Windows/Ubuntu” things that popup while installing your OS. I never paid attention to them, until recently. Now, I’ve got backups scheduled on Dropbox, SpiderOak, and a custom SparkleShare server. I make sure to host my code on github, or my personal git repositories and backup images to Picasa. Everyone knows hard discs are unreliable, so why not make your data redundant and take backups.

The only counter argument would be that Windows has much more stream-lined view of things, with almost everything offering a GUI based application to amange stuff. That’s just not the Linux way. Still, I’m not one of those crazy fanatics who go around preaching Linux (maybe I am!). I feel that if you are using your computer to just open Chrome, Firefox, VLC, and Word, then anything would do. You could probably install android on your computer and do everything you are doing right now. If you are a heavy gamer, then better stay with Windows. And if you are a programmer, switch to Linux.

Windows 8

I installed a developer preview copy of Windows 8 recently, and although I was really awed by its designs concepts, I feel that it is still lacking in certain fronts. For instance, it seems to me that Microsoft is trying to get 8 to the tablets much more readily than on the desktops. Why? Because, there is nothing that can undermine Windows superior market share in Desktops for the near future. However Android & iOS have a very strong presence. Ergo, Windows 8 goes to tables.

I installed all my favorite programs on Windows 8, but had to switch back after a few days. As a developer, it just does not offer me the same freedom, and easy workflow that I am accustomed to. For instance :

Installing Ruby with build support for gems took a lot of time and patience.
Cygwin does not compare anywhere near to a bash shell. Its like this tiny humble brother of shell who turns away when you say PATH. You have to type all those monstrous C:/\ kind of paths.
No App Store. Windows 8 dev preview does not have an app store yet, which was one of the reasons I had tried Win8. I’m used to frequently browsing the Ubuntu Software Center just to find something that fits.
I found that my productivity reduced to half when I was on Windows. I spent too much time looking for stuff which could be done by a single line bash on Linux

Tags: operating systems, linux, ubuntu, windows 8, weblog ifest 2011

Programming in Node.JS

Nemo's Home — Mon, 12 Sep 2011 03:00:00 GMT

After my attempts at python, Ruby on Rails, this was the time for node.js. You ask me what is node.js ? Remember when Google Chrome came out and went blazing past the rest of the browsers in Javascript benchmarks. That was because of its internal Javascript Engine, called V8.

Soon, V8 was developed as a backend to an evented IO library, that is now known as node.js. Initially, it was named just node, but to prevent confusion, and explain its javascript inheritance, it was renamed node.js. This is the water-cooler moment of the language. If you know node, you’re the cool guy on the block. So what is so special in node ?

Evented IO

Basically node allows you bring home the same good anonymous functions from JQuery on your server. In advanced terms, node allows evented input/output, meaning all the IO calls are non-blocking and evented, or scheduled in parallel.

Traditional I/O:

data = file.read("/foo/bar");
  // wait...
  doSomething(data);

Evented I/O:

file.read("/foo/bar", function(data) {
  // called after data is read
  doSomething(data);
});
otherThing(); // execute immediately;

How To Install

The official installation guide is present at https://github.com/joyent/node/wiki/Installation. I’d present a slightly different version, using nvm, i.e Node Version Manager.

Unix/Linux

Dependencies

There are some dependencies on installing node.

sudo apt-get install git-core build-essentials libssl-dev

Run the corresponding command for your distro (yum etc).

Installing nvm

Clone nvm: git clone https://github.com/creationix/nvm.git ~/.nvm
Include nvm: . ~/.nvm/nvm.sh //The dot is important
Source it to your bash file. Basically copy the above command(2) inside your ~/.bashrc.

Fetching node

Node is under heavy development at the moment. Development of node is carried across a stable & a testing branch. The stable branch is the one that you should prefer. As of now, stable is v0.4.11 and testing has reached v0.5.6.

Now run the following commands: nvm install v0.4.11 to install v0.4.11 nvm use v0.4.11 nvm alias default v.04.11 to make it default

You can type which node to see the actual node binaries being used.

Package Methods

I would strongly advice against using your distro versions of node, unless you are on a rolling release distro, such as Arch. Please do not run sudo apt-get install node to install node. This would only cause much anguish and pain later on.

As of now, even the beta of Ubuntu 11.10 holds v0.4.9 and is likely to do so for the next 6 months.

Windows

As I’ve stopped using Windows for quite some time, here are instructions from the official node installation guide.

Windows Build (Node v0.5.5): http://nodejs.org/dist/v0.5.5/node.exe I would again recommend to install http://nodejs.org/dist/v0.4.11/node.exe for stability reasons.

Self-contained binaries are available at http://node-js.prcn.co.cc

Node Package Manager

All cool programming languages come with their own package managers. Ruby has rubygems, Python has pip, PHP has PECL, perl has CPAN, and node has npm.

npm holds a large collection of packages that are the extra batteries that don’t come included in node. If you need to parse documents, or do some other fancy stuff in node, don’t look farther than npm. If you need it, chances are, it already has a package in npm. See list of packages on the npm site.

Unix

A simple one line install is available for npm

curl http://npmjs.org/install.sh | sh

After that, you can install any package by npm install package. For instance, install jade by npm install jade

Windows folks can clone the npm repository and run the included nmp.bat file, and hope that it works.

Simple Servers in Node

node.js comes with “batteries included”, and part of that battery is node’s ability to instantly create web servers. Yes, right inside your program, you can easily create web servers, which will hold full compliance to HTTP.

This is a very simple HTTP server, written using the http module(included) in node :

var http=require('http');
http.createServer(function(request, response) {
  var headers = { "Content-Type": "text/plain" };
  response.sendHeader(200, headers);
  response.sendBody("Hello, World!\n");
  response.finish();
}).listen(8000);

As you can see, the createServer function takes a callback function as its argument. The callback function is called for each of the requests. All events are handled easily and instead of a server handling threads, memory etc, node just handles requests. In essence, a request generates an event, which is then handled by the callback function provided.

This is quite similar to the way we program event loops in javascript on the browser.

The good stuff

There are loads of interesting projects using node. Visit the modules section on the node wiki for a list of interesting node modules available. These include node clients for various libraries such as Databases(mysql, postgre, sqlite, Cassandra etc), Microframeworks (lik Sinatra), Frameworks, wikis, CMS, parsers and what not.

I’d recommend starting out with Connect which is a middleware for node, and allows you to wrap your application easily around it. For databases, you can either go with the standard Relational ones(like mysql) or be brave, and take a spin with the noSQL ones like CouchDB, Cassandra, or MongoDB. All of them have native bindings available for node.js.

References

Node On GitHub, including the wiki, documentation & code
Nodejs.org

Blogs & Other resources

NodeJitsu, a company working completely on node.
Getting Started with Node.JS, Express and CouchDB
6 Must Have Node.js Modules
Some cool node projects
HowToNode
The node Beginner book
NodeCasts

StackOverflow Questions on node

Michael Hart Has Passed Away

Beetle Space — Thu, 08 Sep 2011 10:00:00 GMT

I just learned that Michael Hart, founder of Project Gutenberg has passed away.

I met him once in April of 2006. He gave an overview talk on Project Gutenberg in the building I worked in on campus. I assumed they had flown him over for the talk and was surprised …

Slow

Ben E. C. Boyter — Tue, 06 Sep 2011 07:38:11 GMT

Well it seems searchco.de has been very slow to respond over the last few days. Turns out I had some rogue processes running for another domain chunews.com which were eating a lot of the CPU. I just killed them and switched off the processes for maybe good as that website really needs an overhaul anyway.

Hard Numbers

Ben E. C. Boyter — Thu, 04 Aug 2011 07:29:46 GMT

Thought I would quickly post some hard numbers showing the number of searches since searchco.de launched. The results surprised me somewhat actually. 2011-7 13636

2011-6 8154
2011-5 6183
2011-4 8532
2011-3 10886
2011-2 6476
2011-1 948

I could dig into the results more closely since a lot of those could be the ajax firing off events like “a” and the like but I do like the upwardish trend.

Learning Ruby on Rails

Nemo's Home — Mon, 01 Aug 2011 03:00:00 GMT

Continuing my quest on Web Designing, I’ve started learning Ruby On Rails, which is like the most-hyped web framefork of the moment. After all, Github runs on Ruby On Rails, Redmine uses Rails, and so do Basecamp,Hulu,Scribd, and even Twitter. Even though RoR has Ruby in its name, its just a namesake.

Learning Ruby and learning Rails are entirely two different routes, and learning one only gives you a slight advantage in the other. I’m learning Rails by the excellent book, Ruby On Rails 3 Tutorial, by Michael Hartl. It covers Rails 3, which is one reason I picked it as Rails 3 is quite different from Rails 2 in comparision.

Rails in a few words would be described as a Web framework that makes writing web applications really, really easy. And I really mean that. I’ve been programming in Rails for ~2 days, and I can confortably say that it is better than any other PHP framework (viz CakePHP, CodeIgniter, Kohana) simple because it is powered by Ruby.

And the beauty of Ruby is not in its implementation, but in its elegance. Reading ruby code is lie reading seeing a visual presentation. While PHP is the paragraphed, prose version of the same stuff. Simply put, PHP allows you to do the same things, but essentialy it was not readable enough to match Ruby’s elegance.

Rails follows the MVC pattern (Model-View-Controller) for development, and uses it strictly. It has got its own conventions, but as I found out, the concept of convention over Configuration makes much more sense in Rails than it ever did in PHP. All that time I spent in the CakePHP console was nothing compared to the interactivity of the Rails Console(rails c). Starting development server(rails s) is as easy as running the production server(rails s --environment production).

Instead of writing down another beginner tutorial on Rails, I’d rather direct you to some of the excellent Rails resources :

Ruby on Rails 3 Tutorial, the book I’m reading for learning Rails. Highly recommended
RailsBrains, offline API for Rails
Rails Getting Started, Official Rails Getting Started guide
Rails@Stackoverflow

Photo Booth Plist Rebuild/Repair Script

Donat Studios — Wed, 27 Jul 2011 13:13:20 GMT

Update 2023-10-17 The script has been verified to work in macOS Sonoma.

When I switched from an older MacBook Pro to a MacBook Air the backup from my old MacBook Pro was corrupted and unmountable. However, I had preserved a backup of my Photo Booth Photos and wanted the Photo Booth application to recognize them. This is just a matter of creating a Recents.plist, an ordered list of the image files.

While adding these photos to the Recents.plist file Photo Booth uses seems straightforward enough, arranging hundreds of photos from different naming schemes across various OS versions manually is pretty tedious.

To fix this, I wrote a script that automatically creates a new plist file with the photos in the correct order. As a safety measure, the script backs up the existing Recents.plist as Recents.plist.bk.[time], allowing easy restoration if needed.

Usage

Please Note - NEW

This script is compatible with macOS versions up to macOS 14 (Sonoma). However, it relies on PHP. While PHP was bundled with macOS until macOS 12 (Monterey), subsequent versions require a separate PHP installation.

I recommend using Homebrew. You can find Homebrew installation instructions on their official website.

After setting up Homebrew, install PHP by entering the following command:

brew install php

To Begin With

Ensure all photos and movies you wanted loaded into Photo Booth are located in your Pictures/Photo Booth Library/Pictures/ folder.
Ensure the Photo Booth Application is closed to avoid overwriting our changes.

Simple Method

Added September 10, 2012
Open Terminal and run the following command.

curl -Ls https://gist.githubusercontent.com/donatj/1108691/raw/Photobooth_rebuild.sh.php | php

Git Method

Simply executing the following should both download it and set it executable.

git clone https://gist.github.com/1108691.git PhotoBoothRepair

Non-Git Method

Save the script somewhere accessible to you via Terminal.
Navigate in Terminal to the directory where you saved the shell script
Set the file executable by running
```
chmod +x Photobooth_rebuild.sh.php
```
Execute the shell script
```
./Photobooth_rebuild.sh.php
```
All Done - Fire up Photo Booth to see our results!

Known Limitations

All Leopard format names will come first regardless of date simply for lack of anything to go on. They're just numbered and contain no date information.

Quick update on Eiridia

Ben Overmyer's Site — Sat, 09 Jul 2011 03:00:00 GMT

I was laid off two Wednesdays ago and then went to a science fiction convention, which is why there was no blog post last week. As you read this, I’ll be at a Society for Creative Anachronism event for the weekend. This is going to be a short post just detailing a bit of where Silver Gryphon Games is at with the Eiridia Campaign Setting for Ingenium.

The Campaign Setting book includes a host of new races, Talents, professions, and spells, in addition to giving a good overview to the game world. The writing is currently being done by two people – myself, and a freelancer we’ve hired to do some of the mechanical portions. We have roughly a fourth of the manuscript done.

In the next few weeks, I’ll be putting out a call to find a cover artist. Once the cover is complete, the manuscript should be finished enough that we’ll begin playtesting in earnest. If you’d like to get involved with that, there are some very cool incentives to being an active “alpha” player. Email me and we’ll talk about it.

Announcing Planet IITR

Nemo's Home — Sat, 09 Jul 2011 03:00:00 GMT

Planet IIT-R is a blog collection similar to the Planet Ubuntu, Wordpress, and the likes. The basic idea is to create a single address where all blogs of IIT-R are aggregated. This way, people can easily follow happenings and blogs at IIT-Roorkee without individually following various blogs. I’ve added a few blogs to it already, including those of Divye Kapoor, Sanath Rath, Wona, Arasu and the likes.

A complete list of blogs in the planet is available at http://www.planetaki.com/iitr/subscriptions/

Update: There is also a feature in planetaki to suggest a website for the planet. Unfortunately, you need to be logged in to planetaki (ie create a planet) to suggest a site. In case you are, you can use the suggestions feature to suggest websites.

To suggest a blog to add to the list, please use the form below. You can also see the current spreadsheet here :

You can also email me for any further queries.
Further Links :

Nusphere PhpED Dark Theme

Donat Studios — Fri, 08 Jul 2011 06:34:57 GMT

It would seem there is a fair deal of call for an updated version of my PhpED dark theme from colleagues as well as random folks on the internet who’ve seen screenshots of my PhpED installation. I previously had my Dark Theme for 5 linked on my Review of PhpED, but I figured I could have a page dedicated to it.

Screenshots

Download and Installation

The first step is to download the proper version.
Click here to download the PhpED 6 Version or here to download the PhpED 5 Version.

After you have downloaded the file you will need to replace your hl.cfg.

In Windows 7 or Vista this should be located in or around:

C:\Users\YOUR_USER_NAME\AppData\[Roaming / Local]\NuSphere\PhpED\config

Likewise in XP it should be located in something like:

C:\Documents and Settings\YOUR_USER_NAME\Application Data\NuSphere\PhpED\config

The final step is to simply change your background color to a dark color. The background color is oddly saved in a different configuration file than the highlighting. This can be found in Tools ⇒ Settings ⇒ Editor ⇒ Appearance

While you're in there you may want to tweak the settings some to your taste, but that is all there is to it.

What if Linus Torvalds designed Google+

Nemo's Home — Thu, 07 Jul 2011 03:00:00 GMT

Google’s announced its next big thing, Google+ to take on Facebook left people wondering if the next version would be called Google++. Inspite of all the great work that Vic Gundotra has put into Google+, it still lacks something. The creator of linux, Linus Torvalds. Google+ is as-of-now the social network for tech-geeks who are part of a field trial experiment(monkeys!) on the site. Google+ is still halfway geek, and in my imagination it would have been the ultimate geek power tool if it had been designed by Linus. Unfortunately, he is busy developing the linux kernel (10 million lines, 2% being his), and we would have to suffice with these thoughts:

Open Source : Google+ would be open sourced. Anyone could run Google+ on their own servers, and use it to create their own Social Network
Command Line : The default Google+ client would run natively on linux/unix but would be ported later to Windows using cygwin.
Difficulty-of-use : Gone are the click and point days. You’d be required to have absolute mastery over at least 5 different commands before you can even post a single item to your feed.
Full Control : You will have full control over whatever you post. You can make 4 changes, stage them, take them back, commit 2 of them, edit a post, and recommit before pushing it to the server. However all of this will be stored.
Git Backend : Both the client and the server will use git as its backend to store history, revisions, links, and circles.
Circles : would be called trees. And you can tag your trees to take a snapshot of your friends lists at a time.
Branching : would allow you to create multiple versions of your profile. The default version will be called master, while you can continue your secret development in alpha, staging, beta branches. Circles will automatically be associated with branches and auto-post items.
Merging : will allow for collaborative posts.
After 12 years of development, Google+ will reach version 2.1
No deletions : Everything in history has an importance. You are allowed to use rebase to rewrite history. Beware: use cautiously. Incorrect usage may lead to painfull scenarios.
A mascot(direwolf ?) would be found for Google+ (probably after it bit Linus in an aquarium)
Facebook would spend $421m fighting Google+
The manual of Google+ would be a labryinth of switches amd command line arguments for all the features that it came with. A user would be expected to read through the entire manual, or at least the first one-third, before being able to do something of use with the service.
To block a user, you must enter his id in an ignore file
3 years hence, the next big thing would be ghub offering a farm-like service to create, host, and customize your own google+ servers.

Note: Based on the history of linux, and the usability of git.

Announcement

The famous announcement of Linux Kernel on the comp.os.minix mailing list is well-known. Slightly re-written, this is how Linus might have announced Google+.

>From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)  
>Newsgroups: web.social.facebook  
>Subject: What would you like to see most in facebook  
>Summary: small poll for my new social network  
>Message-ID: <2011Jun28.205708.9541@klaava.Helsinki.FI>  
>Date: 28 Jun 11 20:57:08 GMT  
>Organization: Google  

>Hello everybody out there using facebook -

>I’m doing a (free) social network (just a hobby, won’t be big and professional like facebook). This has been brewing since dec, and is starting to get ready. I’d like any feedback on things people like/dislike in facebook, as my site resembles it somewhat (same layout of the news feed and comments(due to practical reasons) among other things).

>I’ve currently ported feed(1.08), comments(2.1), photos(3.14), and likes(1.40), and things seem to work. This implies that I’ll get something practical within a few months, and I’d like to know what features most people would want. Any suggestions are welcome, but I won’t promise I’ll implement them :-)

>Linus (torvalds@kruuna.helsinki.fi)

>PS. Yes – it’s free of any facebook code, and it has pipelined js. It is NOT protable (uses google accounts), and it probably never will support anything other than Google App Engine, as that’s all I have :-(.

Tags: WeBlog iFest 2011, humour, linux, linus

Vector Space Search Model Explained

Ben E. C. Boyter — Wed, 29 Jun 2011 02:19:48 GMT

A mate of mine was looking at a previous article I wrote about Decoding CAPTCHA’s where I pointed people to the following article (PDF) http://la2600.org/talks/files/20040102/Vector_Space_Search_Engine_Theory.pdf

He was having some difficulty understanding it so I thought I would write up a very simple explanation of what’s actually happening in the vector space.

The vector space isn’t actually that complicated, but getting your head around how it works takes a few steps. Let’s imagine for the moment that we are going to search over a collection of documents which only contain one word “boffin”. Each document has a different count of our word but only contain those words. We can represent this visually using a simple graph.

New RPG in the works!

Ben Overmyer's Site — Tue, 28 Jun 2011 03:00:00 GMT

Generally I post on Saturdays, but for this I thought I’d make an exception. Sunday I sat down in a coffee shop called Latté Da in Vermillion, SD and wrote another RPG system. Took me a couple hours, and it hasn’t been playtested or even simulated yet.

The premise is that each character has six attributes. Each Attribute is a die type, and any dice roll is comprised of three Attributes and (in the case of PCs and special NPCs) a Hero Die. The Hero Die functions like a normal die, unlike the Wild Die in Savage Worlds.

There are no levels, classes, or skill trees. While the system is fantasy-based, it could be used for any genre with very little tweaking. Borrowing from a certain RPG from Atarashi Games, players can throw out Challenges for each other when they think someone’s having too easy a time of it, just as the GM can. That includes NPCs.

The rules are about six pages long. You can download the prototype from here:

http://ge.tt/9wBgyU5

Thoughts? Comments? Concerns?

Eiridia, My Fantasy Setting

Ben Overmyer's Site — Sat, 25 Jun 2011 03:00:00 GMT

Just over eleven years ago, I began the setting that will see publication next year. It has evolved and gone through dozens of sweeping changes. The result is something that generates excitement when I talk about it with others, and that makes me really happy. There is no greater joy for a writer than to see his works enjoyed by others.

My own reading enjoyment comes from a very epic slant. I like stories where the hero wins, but only by the narrowest of margins. I like stories where the entire world is at stake. I like stories where someone grows from simple origins into a legendary hero. These themes came through in Eiridia’s history.

The system it’s built for owes equal parts of its style to original D&D, Diablo, World of Warcraft, and 4E. This mix works oddly well, considering the fervor with which some crowds denounce each of those games in turn. It helps that I enjoy all of them, I suppose. With Ingenium, I sought to create a game that didn’t get in the way of the epic stories I wove for my players. It was designed to be simple to learn and simple to play, but with enough complexities to interest the more tactical or creative players.

Eiridia mixes a number of different real-world and fictional analogues together to form a cohesive whole. I took inspiration from Robert E. Howard’s Hyborea. Jur’gaan is a feudal land of warlords and dark magic. The Oriental influence is strongest there. Tantris is a desert populated by bandits. Arabia played the biggest part there. Nardora is a dystopia, its sprawling pristine cities ruled absolutely by a cult and its Queen. The Mongol Hordes provided the original inspiration for Nardora. Tandarus is a city-state populated almost entirely by the winged Tandari, making it a city of angels. Mercedes Lackey’s The White Gryphon inspired that race and their city. Rekellia is a weakened domain, a remnant of a once-great empire that spanned the entire continent. It was born of stories of Rome. Redami is a kingdom that, through the shortsighted malice of its populace, unwittingly created its own worst enemy. It had no one inspiration that influenced it most.

The other continent in Eiridia, Khathana, was initially written for the stillborn fantasy setting for Æther, my business partner’s brainchild. Not wanting to see the concept wasted, he handed it over to me to see if it could be incorporated into Eiridia. Given the epic, sweeping nature of Khathana, it was a natural fit. I’m in the process of pulling together the histories now.

I’m planning on shifting the focus of this blog to writing about Eiridia. If you like the new focus, let me know about it in the comments. If you hate the new focus, let me know, too. I’ll still continue with my plans, but it’s nice to know what the audience I’m writing for thinks.

Why Indian Government Sucks at Technology

Nemo's Home — Wed, 22 Jun 2011 03:00:00 GMT

Note: All opinions are mine alone. Please keep your opinions limited to comments.

This is in wake of the not-so-anonymous attacks on the nic servers. Apparently a few people had decided to brand their own version of AnonymousIndia and hack into the nic servers. This had been viewed as almost normal in the Indian Media. This is after all something that happens every other day in India. Its just the Indian Army website, nothing that we care about.

Even though there are excellent tech-security companies in India, we have never developed the right attitude to it. People still think that hacking is fun, and its something that could never happen to them. I’ve never seen people reading the fine print on the thousand social web sites they join now a days. Piracy is rampant in India without any checks and the Indian Government is silent.

Why? Because we are used to it. It has always happened this way. Information leaks have been a major part of the Indian history and would remain so unless we realize that the more we embrace technology; the more we become dependent on it, it is coming one step closer to the edge.

A recent tweet by @divyekapoor reminded me of the AADHAR project, which aims to give out a unique identification number to each Indian citizen.

Hacks like these ensure that I will resist giving my Biometrics to the UID project till they've suffered atleast 4 security breaches.
— Divye Kapoor (@divyekapoor) June 18, 2011

I read out the UIDAI docs and it tells us that it would “prescribe protocols to ensure the confidentiality, privacy and security of data”, and “follow the confidentiality, privacy and security protocols prescribed by the UIDAI”. A search on the uidai.gov.in website gives out a little detail :

The UID database will be guarded both physically and electronically by a few select individuals with high clearance. It will not be available even for many members of the UID staff and will be secured with the best encryption, and in a highly secure data vault. All access details will be properly logged.

This is the most that they have to say on the subject. If that makes you feel safe, remember the earlier fiasco involving the security of the EVM machines. Ultimately I’m with the government on that one, however, since a user had to have physical access to actually do any harm to the machine. But it was possible and Election Council kept denying it. As it happened, it seems that all our govt. agencies are probably trying more to hide their technical requirements, rather than make them open.

So why is this approach taken in India? Beauracracy? probably yes. But I feel that unless the technocrats rise up and actually enter the Indian Govt. technical agencies (such as NIC), nothing would change. For instance I find that almost everything that Indian government does is tailored specifically for Windows. There is no reason to plaster “Works best in Internet Explorer 6 at 1024x768” in all your web sites, we already know how much the Indian government. So much so that they get it pre-installed everywhere, even in government schools, and technical institutions.

Yet there is a small faction that is working tirelessly in the other direction as well. For instance the Sakshat project has been under the news recently as well. It would ship with a version of android (which one?) with wi-fi, bluetooth and other frills. However the project has been known for shadowing and changing its details at each conference it is unvealed, so beware. It may suddenly change from an android to a Windows phone in the next one. Or maybe a blade server (speaking of which, the uid project ordered 68 of them.

I’ve worked a little bit on online Geo-Mapping tools earlier (mostly using Google Maps API). However, I wanted some accurate data for one of my projects (such as geographical boundaries). Other sources for this data are not as reliable, and I found the bhuvan online tool to be extremely accurate in this respect. If you’ve forgotten Bhuvan, its the Indian version of Google Maps. It was supposed to be the tool for mapping things. Unfortunately, as things have planned out, most of its claims have been rubbished (like 10m resolution power), or made null due to the extremely slow servers it uses(and it was supposedly optimized for low bandwidths). If you’re plannig to fight google, you’ve to step up your game. Try checking out the horrible design of the Bhuvan website. Leaving aside its horrible interface, Windows only support (.net), installation of additional plugins to just run it, and , it had brilliant geographical data(collected via various government agencies). However as it turns out, this data is not public. Why? It seems that ISRO plans to sell this data to people interested in using it. Wow! So a public funded agency decides to make money from the development done using our money. It’s been close to 3 years since its launch. And I’d be highly interested in knowing where it managed to sell this data.

In all fairness, though they’ve said that they would only sell the high resolution data, while making the general data freely available to users. But I don’t see fair unless its own my system, damn it. If by free you mean I’d have to open your website each time I need to find a village, it seems we have different thoughts about the word’s usage. Oh, and there’s a link on the website’s home page to a section called APIs which redirects to the Bhuvan Software download page, where guess what, there are no APIs at all. I managed to dig a few links and download two versions of their APIs, which seem to be downloaded copies of the documentation of the open layer javascript protocol they are using for mapping in the browser. So the data is available, it seems. But they’re forgetting to mention where.

Seriously, people wake up. This is the 21st century, and the most hyped buzz word today is open-source(well, after cloud). And if you really want to work on things, make them open source. Not just the technology, but the data as well. Becauase open data is essential to growth and planning of a nation, as Hans Rosling keeps on reminding us. Meanwhile, DRDO decides to go ahead and develop its own operating system. Why? Because it will be closed-source and will be much more secure than any of the variants of linux. Or so they think.

The Informatics center still runs thousands of its websites in ASP (not even ASP.net), and this fact alone is enough to scare me off. With major corporations suffering from data leakage (Sony, Gawker) where user access was compromised, it is high time that the Indian Govt. someone realizes that you cannot secure your systems by locking them in a vault. These companies did the same, and look at the result.

Even the American Government has come up with an open-source initiative. Their website data.gov is a collection of applications, apis and raw data collected by various government agencies. And to top it all, the US government invited the top application developers from their platform at data.gov to the White House, hailing them as unsung heroes of the new age. If you’re interested in reading more and taking a stand for the open data democracy, take a look at this whitepaper by the Netherlands Organisation for Applied Scientific Research for a keen review of what are the major barriers to a government from sharing its data. Also go ahead and donate some money to wikileaks, while you’re at it.

And where is the Indian government at this? Not very far behind, but lagging nonetheless. Let us take the prime example of the decennial festival that is the Census. Apparently all census data is free (as it should be). But there is a minor caveat. The entire site is made in asp (which doesn’t really matter, I just don’t like it), and all the data (tables, figures, maps) are in pdf format. So, you can access the data personally, on a single page. Page by page, it might be thousands of documents, figures, charts, and what not. But since it is all in pdf format, it is locked down. You could parse it by some means, but the data is supposed to be free, in the best format possible so that everyone can use it easily. And the geographical boundary data(which I mentioned earlier) is also available in the census results, but only via a java applet, which does not allow access to the raw data, that an application developer would need. Am I expected to file an RTI application, just to know the exact boundaries of my state. Or perhaps, I should just pay ISRO and be done with it.

As an ~~additional benifit~~ easter egg, the census website states the following:

The Census of India or any data or content providers shall not be liable for any errors in the content, or for any actions taken in reliance thereon.

All efforts have been made to ensure the accuracy and currency of the content on this website. However, users are requested to verify/check any information with us to obtain appropriate professional advice before acting on the information provided in the website. In no event will the Government or office of the Registrar General India be liable for any expense, loss or damage including, without limitation, indirect or consequential loss or damage, or any expense, loss or damage whatsoever arising from use, or loss of use, of data, arising out of or in connection with the use of this website.

And lastly, as an analogue to the excellent Right To Information Act, there must be an analogous Right To Technology act. It should empower each and every person in the country to know what is happening behind the scenes. We demand total transparency in technological decisions. Not just the passing of tenders, but the decisions which involve actual technological development. For. eg at the CDAC website, the government offers a trial version of various forensic tools they’ve developed. Why aren’t they open sourced? Why were they writen using a particular language? If RTI fought off the beauracracy, this could help us eliminate the old technocrat thinking from the India Govt. If we are able to get the right data, in the right format, thousands of application developers across the world are willing to create great ways to access that data. Data by itself is not enough, however. It must be met with an equal resolve from people to make it accesible, and usable.

This could be a turning point in the Indian Govt. Either they could continue what they’ve been doing and meet their doom in a major state sponsored hack crippling the entire nation. Or they could take a step back, and do things the right way.

And what is the right way?

Update :According to an article in the Economic Times, there is a working draft for bringing Open-Source in e-governence systems under work.

Setting Up Sparkleshare Server using Gitolite and Ubuntu

Nemo's Home — Mon, 20 Jun 2011 03:00:00 GMT

Introduction

Everybody seems to be all about open-source cloud-backup and sync solutions now-a-days. The hype is all around the cloud, they say. However cloud is just a stupid concept for sales people, that I prefer to avoid. However people are coming up with all kinds of crazy ideas to create their own dropbox clones. A few similar services include SpiderOak, Ubuntu One, Sugar Sync, and Wuala. However, not all of them are compatible with Linux (unlike Dropbox, which is).

Comparision

So here’s a minor comparision of some famous clients :

Dropbox : Cuurent Leader, offers everything from sync, collaboration, sharing, public links, upgradable storage and is the de-facto client for synchonization tasks. However there have been a few issues regarding its privacy issues recently.
Ubuntu One : Ubuntu One is Ubuntu’s fighting offering to Dropbox. Its excellent, with an open source api, that allows one to create applications for the Ubuntu One platform very easily. However, the server-side of Ubuntu-One is still closed source, which means you cannot setup it on your own servers (similar to dropbox). Canonical has hinted that it might be made open source in the future.
Wuala : is a file-backup network where you trade your own hard disk space for extra storage. This allows wuala to offer higher space at a much lower offering rate.
SpiderOak : I’m using this currently along with SparkleShare and Dropbox. It has proven to be very robust, allowing me to backup almost anything to its servers. I’ve got a 5GB account which is more than enough for me till now. Its very powerful interface allows one to control each and every aspect of your backup/sync/share process. Also it boasts of a true-privacy feature, meaning that all your documents are encrypted before being sent to dropbox. It also means that you can only reset your password from your own computer.

Take a look at http://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software for a better comparision of several other services as well.

Installing SparkleShare using gitolite

This is a simple tutorial on running your own sparkleshare server as a hosting server. Note, that this implementation should ideally be built as a separate module for sparkleshare-admin, which is still under works as of writing. Sparkleshare’s basic concept is to use git repositories as storage places. In case you don’t know what git is, I’d recommend this guide for more details. In short it is an awesome revisioning system for use by anyone managing code(or content for that matter as well). It allows you to keep track of what is happening with your directory, and revert back to earlier versions (among several other things).

Sparkleshare asks you to setup a git-server somewhere and use it as a remote storage system. It offers out of the box support for git hosting providers github and gitorious. It also allows you to add your own custom servers as well. Enough description, lets get down to some work :

Setup Gitolite

Assumptions :

You are running a stable Linux OS (Fedore/Debian/Ubuntu etc)
user@host1 is your own computer
user2@host2 is the primary computer where you intend to start the server
The gitolite username is sparkle

#On your host machine (which will be remote admin to the git share)
ssh-copy-id user2@host2:/tmp/user.tmp
#should not ask for password:
ssh user2@host2
sudo apt-get install gitolite 
sudo dpkg-reconfigure sparkleshare
#Configuration Options may vary, but remember the gitolite user name that you specified
logout #Come back to your own computer
git clone sparkle@host2:gitolite-admin 
#Should work, or else you did something wrong. Go read the [gitolite docs](http://sitaramc.github.com/gitolite/doc/)
cd gitolite-admin
nano conf/gitolite.conf

Setup WildRepos

Edit the file and add the following lines at the bottom :

repo	share/[a-z0-9]{6}
	C	=   @all
	RW+	=   CREATOR

Now we need a method to allow anyone to create git repositories on the server. This is accomplished via Gitolite’s very powerful Wildcard Repositories feature.

#Login back to server
ssh user2@host2
#Since we are using package install method, sparkle's password needs to be set
sudo passwd sparkle
su - sparkle
nano .gitolite.rc
#search for $GL_WILDREPOS and set it to 1
logout
#Now we push our admin repo to add the wildrepo settings
#you're still inside the gitolite-admin directory, right
git push

Setup Client

Now, your server is all setup, but there is still stuff to be done :

#On user1@host1
mkdir -p ~/.ssh
sudo add-apt-repository ppa:warp10/sparkleshare
sudo apt-get update
sudo apt-get install sparkleshare libwebkit1.1-cil git-core
sparkleshare start &
sparkleshare stop

When you run sparkleshare for the first time, it asks you for a few things, including your email-id. Fill in those details, but do not setup your repository yet. You need to first allow your sparkleshare account access to gitolite.

cd ~/.config/sparkleshare
ls #Should reveal files called sparkleshare.email.key and sparkleshare.email.key.pub
cp sparkleshare.email.key.pub /path/to/gitolite-admin/keydir/
cd /path/to/gitolite-admin/
git commit -am "Added sparkleshare client1"
git push

Now if all goes well, you’d have allowed acess to gitolite for this user. We now need to re-run the sparkeshare setup again. Find it in your Applications. Now when it asks you to fill a repository path, type in the following details :

Server: sparkle@host2
Path: /share/fh73ah

Please take care of the slashes, otherwise sparkleshare fails to recognize it as a valid ssh address. Instead of fh73ah, you can type any alphanumeric string of 6 characters. You can change this in your gitolite-admin conf.

After your first sync is complete (in which it tries to clone your existing repo, and gitolite creates it for you), you can find a folder called Sparkleshare in your home directory. This contains all your personal sparkleshares, including your first one. Put in any content inside the fh73ah and it would be automatically synchronized.

Conclusion

The best thing about sparkleshare is that you can use your own server under your own rules. I’ve synced 143GB via Sparkleshare so far, and it has been working excellent so far. It takes a complete history, takes care of moves (git) and allows you to keeo huge backups easily. Just drag and drop, and forget. If you want to sync already existing folders, just drag them , and alt+drop them inside the shared folder. This way a sym-link gets created, which refers to the original directory. The sparkleshare folder on my computer takes up hardly a few kbs, but syncs worth 150gb.

This method is only useful if you need to manage multiple accounts on the same host. Otherwise, you can refer to this excellent post on webupd8 for instructions to install it to a single user system (which does not involve the complication of gitolite). I’ve been looking for some gitolite management scripts (I’ve written a few as well) which would allow one to easily add their own ssh keys. This way anyone can easily setup accounts on the system. However, as of now, this is just a dream.

Two Systems, One Game

Ben Overmyer's Site — Sat, 18 Jun 2011 03:00:00 GMT

Recently I read a post over on the RPG Corner about using one system for the Dungeon Master and one separate one for the players. This sounded like a fascinating idea to me. Many of the commentators chimed in with good thoughts, like “how do you reconcile the appearance of disparate magic systems?” That last question makes the assumption that your game setting has distinct spells that players need to be able to recognize.

What if the setting assumed magic could produce wondrous effects, and that its practitioners had mysterious powers that no one but them could understand?

The discussion on the RPG Corner post, however, limited itself pretty drastically by sticking only to Dungeons & Dragons or derivatives thereof for its rules material. What if we brought other, completely different, systems into this mix? Say… Savage Worlds? Or Ingenium? Or even jump the setting gap altogether, and use Mutants & Masterminds?

Before you start yelling about this not being in keeping with the Old School philosophy, allow me to explain. This is completely in keeping with the Old School Revival – not because it adheres to rules sets with decades under their belts, but because it relies heavily on a distinctly OSR concept: that the purpose of the game is to explore the unknown, conquer danger, outwit enemies, and develop a story that keeps all of its players and the GM engaged. Newer games like 4E could be played like this, and are, but for the most part they tend towards being tactical games rather than role-playing games.

So here’s my proposal.

At the start of a campaign, before the first session, let the players choose a rules set to play by. They can choose anything, as long as the GM is at least passingly familiar with it, and all of the players must adhere to it.

The GM then chooses a system to use as the behind-the-scenes crunch. The players don’t need to be aware of the system.

The part that you need to think about here is bridging the two in terms of common situations. The players need a solid turn order, and damage and ability usage need to be abstracted enough that they can be converted between the two systems on the fly. This is easier than it sounds, as long as you set this up beforehand. This is why the GM needs to be familiar with the players’ system.

As an example, let’s say the players are using Savage Worlds and the GM is using Labyrinth Lord. Turn order in Savage Worlds is handled via poker cards as initiative, with Jokers being wild. To reconcile this with Labyrinth Lord, where initiative is determined via a roll, the GM might use 3d20 as his initiative roll instead of the usual. Granted, this changes the probability of the NPCs going first rather drastically in their favor, but no one ever said the game had to be fair – just fun, for everyone involved.

Savage Worlds has a concept of “Edges” and “Hindrances” similar to other games’ Advantages and Disadvantages. Labyrinth Lord has no analogue to this. However, the vast majority of these are things that the players themselves have to work out, and this has no bearing on Labyrinth Lord. For example, one Edge is “Quick,” and lets the player redraw initiative cards until he has a 6 or higher.

Combat is the toughest to bridge between Savage and LL. Savage Worlds uses wound levels and has exploding dice – that is, if you roll the highest number on a die, you get to roll it again and add the two together (potentially infinitely). However, there is a way to get the two systems to play nice together. NPCs still roll their usual damage dice, but set a specific amount of damage as equivalent to a Shake or Wound result that makes sense for the PCs’ power level. Status effects like poison or daze have equivalent rules in both systems that are relatively simple.

Whenever the two systems you’re using collide in a way that seems unbridgeable, use whatever resolution is the most fun for everyone.

So what do you think? If you’ve ever combined two systems in this way, how did it work out for you?

New Website [ CaptNemo.in]

Nemo's Home — Mon, 13 Jun 2011 03:00:00 GMT

Just if someone's still following this blog around(don't, its already dead). I've moved over to my new website (http://www.captnemo.in). Its running from github and will be a perfectly static website where the power of my awesome magical skills shall finally be revealed. Just joking. Its hosted on the awesome servers of github and you might want to check it out..

Once, in a forest beyond time…

Ben Overmyer's Site — Sat, 11 Jun 2011 03:00:00 GMT

This mighty forest spreads for miles upon miles. Many hundreds of different kinds of trees grow here, from the pines of the west to the oaks of the deep forest. Though the furthest edges are occasionally visited by men, the vast majority remains unseen by human eyes. This is an ancient forest, one that has stood for countless millennia. The first ages of the world when gods walked the earth and men were but a whisper in the minds of their creators saw this forest come to being from a single tree. That tree, the Eversource, still stands in the heart of the forest. Its trunk is so wide that fifty men could not encircle it. It is so tall that its canopy dwarfs even the mighty ironoaks, whose branches reach a hundred heights of man into the air. No disease or infestation has ever dared to take hold in the Eversource, and through its power the trees around it have flourished and spread.

The forest holds many wonders, and many horrors. There are beasts within that would shake the bravest hearts. Even the trees in some places have grown wild and angry, and do harm to any animal large enough to make its notice. The rains that come frequently nourish the beasts as well as the trees, and there are things in the deepest parts of the wood that have grown to sizes equal to the titanic ironoaks.

What follows here are some inspirations your games can take from such a forest as this one. I present a handful of creatures, new materials, and adventure ideas to make your Saturday a little more entertaining. The creatures are presented using the Labyrinth Lord format.

Creatures

Fearwillow

Stat	Value
No.	1 (1d2)
Align	Chaotic
Move	N/A
AC	6
HD	6
Att	3 (branches)
Dmg	1d8/1d8/1d8
Save	F3
Morale	8
Hoard	None
XP	320

These trees occur when the seed of a willow tree takes its sustenance and incubation from the corpse of a demon. They are the same size and shape as a regular willow tree, but their trunks, branches, and leaves are all white. Red spots run the length of the trunk. Whenever a living thing passes within a few yards of a fearwillow, the tree’s branches will shake and shiver as if in anticipation. If a living thing passes under the willow, then the wispy tree’s branches will caress it gently. Every touch of a fearwillow causes a sense of growing dread. If something spends too long in the grasp of the tree, the fearwillow will suddenly seize it and draw it towards the trunk, where an unholy maw will open. Unless the creature breaks free, the abomination will devour it whole, feeding on the terror it has been cultivating. It takes two full days for a fearwillow to consume its prey. When finished, the entire body will have been incorporated into the tree, and it will grow in size in proportion to the terror of the victim.

The bark of a fearwillow can be dried, ground up, and steeped to make a tea. This tea grants the ability to sense fear and feed on it, letting the imbiber recover 1 HP per round spent in the immediate vicinity of something that is afraid. This ability lasts for one month for every pot of fearwillow tea consumed.

Hunter Beetle

Stat	Value
No.	1d4 (1d4)
Align	Neutral
Move	120’ (40’)
AC	3
HD	4
Att	1
Dmg	1d6
Save	F2
Morale	7
Hoard	None
XP	80

These insects grow to be a hand’s length in size at adulthood and live in dark parts of the wood that have little light beneath the canopy. It has three horns coming from its head and is a ghostly grey in color. When it flies, its wings create a buzzing tone that sounds like the hum of a gargantuan tuning fork. Its preferred method of attack is to eject a stream of foul-smelling liquid onto its prey. This bile has a paralyzing effect, causing total paralysis within 8 rounds. The hunter beetle is content to wait for its prey to stop moving before it advances and begins to feed. It will attack anything up to an elephant in size, as anything bigger than that is unaffected by the bile. Creatures the size of a squirrel or smaller are generally ignored.

A hunter beetle’s bile can be harvested from its body after it dies and retains its potency for a full year. Only an ounce is necessary to paralyze a man. The paralysis wears off after an hour.

Adventure Ideas

Long-dead civilizations are not the only source of treasure to be found in this world. Clever and knowledgeable adventurers can discover many things within the depths of an ancient forest. Here I present some single-line adventure ideas to get your mind working.

An alchemist hires the party to retrieve an ingredient that can only be found in the heart of the forest.
One of the party is attacked by a poisonous creature of the forest, and they must find the antidote before it’s too late.
The party finds an old hermit in the forest who offers to teach them a strange new spell if they assist him in finding his “daughter.”
A talking tree asks the party for help against a company of orcs that are felling trees to make a war camp.
A royal chef is tasked with preparing a delicacy for a tyrant, and needs the party’s help in order to procure the ingredients.

Grep Match a Tab

Ben E. C. Boyter — Wed, 08 Jun 2011 02:19:24 GMT

Ever wanted to match a tab while using grep for some reason? The trick (under bash anyway) is to Ctrl+V and then press the tab key so you get whatever you are looking for.

$ cat file_to_grep.txt | grep "^log    "

I was trying to match a file for the exact match of log and then a tab. Without the tab I ended up getting back a bunch of junk results like “logger” “logging” “login” etc…

Clojure 1.3 Now Avaliable

Ben E. C. Boyter — Tue, 07 Jun 2011 02:19:04 GMT

Added Clojure 1.3 documentation today. The Clojure language itself is pretty tight, but its nice to have something to browse through when looking for some elusive method.

You can view it by searching for clojure

One thing I have discovered while doing this is that if I have a slight amount of familiarity with the language it really makes parsing it easier. This stands to reason as I can pick up errors more easily, but somewhat surprising. Another thought is that I think there should be a standard for programming language documentation. Every language has its own which means each one needs a specialised parser.

A Craftsman’s Magic

Ben Overmyer's Site — Sat, 04 Jun 2011 03:00:00 GMT

Not all magical items are weapons or armor. The list below is of a few items that might prove useful to a clever adventurer. Some may make excellent flavor items for particularly interesting NPCs.

Forgemaster’s Hammer – This hammer is ensorcelled by an ancient dwarven forgemaster. Its weighty iron head is engraved with seven runes of power, and its hilt is wrapped in dragonhide leather carved from the corpse of an adult red dragon. Generally such hammers are not sold or traded. A forgemaster will pass this hammer down to his first-born son when he dies or becomes too old to smith any longer. The hammer grants the wielder the ability to shape any metal as if it were the most malleable copper. If used as a weapon against a foe in nonmagical metal armor, any successful hit will permanently reduce (or increase, if using descending AC) the AC of that armor by 1. If this penalty would negate the armor’s AC bonus, the armor is destroyed. The hammer’s true purpose, though, is in the forging of masterworks of metalcraft. Legendary swords have been forged with this hammer, and its value for historical reasons may outweigh its value as a magical item.

Saw of Severing – One of many like it, this hand saw was enchanted to allow it to cut through anything without dulling. It has two magical properties; one, it can cut through any nonmagical solid inanimate material at a rate of 1 foot in thickness every round. Two, the blade of this saw will never dull.

Box of Endless Nails – This box is invaluable to any carpenter. It appears as a simple oak box one foot long, one foot wide, and six inches deep. A latch made of iron holds the box shut. On the underside of the box is a symbol of three crossed hammers burnt into one corner. If a person opens the box, he will find a dozen nails of the exact size, thickness, and material that he needs for the job at hand. No matter how many he removes from the box, if he closes and latches the box, then opens it again, there will be twelve new nails exactly as before.

Spade of Well-Digging – This small iron spade has a mahogany handle. Carved into the handle is a shallow impression of a farmer digging a well. If a person spends five rounds digging a hole with this spade, the hole will expand to a circle four feet in diameter and ten feet deep. The hole will instantly fill with clean, drinkable water. If the ground that the hole was initially dug in is not large enough to accommodate the well (say, if the wielder was digging on the lip of a cliff), the hole will enlarge but it will not fill with water. After one day, the well vanishes, leaving the terrain as it was before the ground was disturbed by the spade.

Link Love

Ben E. C. Boyter — Thu, 02 Jun 2011 02:18:41 GMT

With the fall from grace of the TWiT podcast (less Dvorak and no Calacanis makes for boring shows) I went looking for new podcast’s to keep me entertained over the last couple of months. Here are a few that I highly recommend.

Tech Podcast TechZing Live

The boys from tech-zing are full of energy, always come up with new stuff and usually manage to do one thing technical each show that makes me want to scream with frustration. All in all good stuff. They also interviewed both Calacanis and Dvorak which were probably 2 of the better shows they did.

IIT-JEE 2011 Results

Nemo's Home — Wed, 01 Jun 2011 03:00:00 GMT

I worked out the entire IIT-JEE 2011 results and the end result is available here. I’ve intentionally removed the Application Form Number (partially) in the results, so that it may not be misused. I’m thinking of trying a full-scale birthday permutation attempt on the JEE site for the Application Form. What do you think? Will it be worth it?

Nginx HttpCoreModule Documentation

Ben E. C. Boyter — Sat, 21 May 2011 02:18:17 GMT

Added the Nginx HttpCoreModule Documentation to the index. Hopefully helps someone out there!

You can view it here Nginx HttpCoreModule Documentation

Environment Structure for Django Apps

CRAIG KERSTIENS — Tue, 17 May 2011 06:55:56 GMT

I’ve been writing applications off and on for nearly 4 years now, since before Django 1.0 was even released. I must say the framework could not be better described than by its own tagline “The Web framework for perfectionists with deadlines”. Among the things I love about it are:

However, after using the framework for nearly 4 years I’m just now discovering my preferred way of managing environments. I know there’s still a bit of back and forth on development environment/IDE, but as far as configuring actual project environment I’ve become very comfortable with what I’ve now been using for many months. It also allows for someone else bootstrapping their environment incredible quickly as well. Below is a quick cookbook of how to do this on OSX and Ubuntu.

$ sudo port install python27
$ sudo port install py27-virtualenv
$ sudo port install postgresql90

Now for setting up your project:

$ mkdir example
$ cd example
$ virtualenv-2.7 --no-site-packages .
$ source bin/activate

$ sudo brew install python
$ sudo brew install virtualenv
$ sudo brew install postgresql

Now for setting up your project:

$ mkdir example
$ cd example
$ virtualenv --no-site-packages .
$ source bin/activate

$ sudo aptitude install python
sudo aptitude install virtualenv
sudo aptitude install postgresql

Now for setting up your project:

$ mkdir example
$ cd example
$ virtualenv --no-site-packages .
$ source bin/activate

For those of you not familiar virtualenv is a self-contained python environment. It holds its own copy of python and any libraries you install. Now that you’ve setup your virtualenv we can go through the process of installing django and setting up your repository. This is the same across all of the above platforms:

Add to a .gitignore file:

bin
build
include
lib
.Python
*.pyc

Add to a requirements.txt file:

Django==1.3.1
psycopg2==2.4.1

Then run:

$ bin/pip install -r requirements.txt

This should fully installed any of your required apps and makes it easy for others to do the same to begin contributing to a larger app. Finally if you like you can create your git repo from this an make your first commit:

$ git init
$ git add .
$ git commit -m 'my first django virtualenv'

Learning Python, PyGTK

Nemo's Home — Mon, 16 May 2011 03:00:00 GMT

I had been meaning to learn either Python or Ruby for a long time but had been unable to decide. I had a basic understanding of both of these, but I never had the chance to build an entire application in either. And I’m not talking about using Django or Ruby on Rails (which are both brilliant), but building a desktop application using GTK. The application I was first aiming for was a P2P sharing client which would be completely decentralized and offer several special features, such as :

NAT Transversal using UDP Hole Punching
HTTP based file sharing as well (so that people can install their clients, and still use IDM for downloading stuff)
Client discovery & routing behind firewalls along the lines of Skype (using supernodes)

However my dreams were shattered by the first point itself, being the difficulty of NAT Transversals. Further ahead was the question of networking as well, which could have potentially driven me nuts. I worked a bit on it, using STUN, twisted, but gave up on it soon as being unfeasable as a learning project.

The next idea came to me when I got tired of downloading stuff using axel from the command line and started itching for something similar to Internet Download Manger for Ubuntu. The closest thing to a download manager on Ubuntu is FatRat and is something I really don’t like. It is based on Qt and prouds on working as a front-end for several file-sharing websites as well. What I needed was in fact a fast download manager, which keeps tracks of what I download, and does not require keeping a terminal open all the time.

I found GwGet, which occassionaly looks much better in the source version than in the one from the Ubuntu Repositories. I really liked this one, except for the fact that it was made using C++, and used single threaded downloads (like wget). As a result it was quite slow, and not upto my needs.

That was when I thought of the idea of creating a download manger using [GTK][gtk] + Python/Ruby. I looked around for axel ports in Python/Ruby and found PyAxel, which beat axel in some of my benchmarks (after this patch). For the past two days, I have been working on PyGTK, Glade, Anjuta and several other IDEs, none of them to my liking. I really prefer Vim :)

So far, the work on PGet has been minimal. I’ve worked out threading, and little parts of GUI which were stripped from GwGet. As of now, it is still under works, but I am hoping for a release real soon. After all, it is not for nothing that they call Python a dynamic langugae

For more details, please go the PGet project page on Github. I will be posting further updates over there. In case someone is following this blog, you can view the source for this website at github and maybe even fork it!

Puzzles, Life & Other Things

Nemo's Home — Sat, 30 Apr 2011 03:00:00 GMT

Since I’ve already decided to make this my newer blog, why not just continue in the same spirit and write a little of the events of my highly boring, lazy life. For one, I was part of the SDSLabs 1st yearly trip to Robber’s Caves which was highly enthralling. We enjoyed a lot, and as a bonus I learned to play Mafia. Somehow the concept of not-knowing and yet trying to deduce out a solution in Mafia seems quite interesting for a Party game to me. Among other things, my sister got me a new Rubik Cube (which should be my 6th or 7th I guess), and I’ve been practising quite a lot (to the dismay of my friends and teachers), My timings have not been upto the mark as they were last year, but I’ve been improving and I average around 80 seconds per solve. I’ve been focusing on learning the entire Fridrich, and try to learn 2-3 moves per day.

I also spent some time reading a brilliant new fantasy series called “The Kingkiller Chronicles” by Patrick Rothfuss. It is a brilliant new debut series in fantasy fiction, and already has 2 books out from its planned trilogy : The name of the Wind, and The Wise Man’s Fear. The series is highly praised, and you were to believe me, one of the best pieces of Fantasy ever written. But it is not the fantasy about this book that makes it so great. Its the general themes of love, tragedy, enemity, and knowledge that make it brilliant. Kvothe, the protagonist of the series is a charming character who is telling the story of his life to the Chronicler. Enough on the book, just go ahead and read it!

Oh, and I’ve got a RSS feed for blog using Jekyll already. Its available [here] (/atom.xml)

Game Of Thrones

Nemo's Home — Tue, 19 Apr 2011 03:00:00 GMT

For the uninitiated, Game Of Thrones is a high budget fantasy TV series currently being screened on HBO. It is based on George R.R Martin’s epic fantasy, A Song Of Fire & Ice, book one of the Game Of Thrones series. I had started reading the book a few weeks ago (not knowing then about the TV series), but never quite got the time. However after the highly promoted Trailer, and the absolutely brilliant episode 1, I’ve picked up the book once again.

I’ve decided to keep myself aloof from all the spoilers, and read the book along with the series. This means that I will be finishing the book in a very long time, but also that I will be able to get a much better insight into what is going on the series. For eg there are very many characters in the TV series, which are not yet introduced, and it was interesting to see them come upfront in the book.

Is anyone else reading the book for the first time along with the TV series ?

Below is the ground from the book that each episode contains. I will continue to update this as the series goes ahead :

Episode :

Chapters 1-8
Chapters 9-17
Chapters 18-24*
Chapters 25-29
Chapters 30-35

* Only half of the chapter is present in the respective episode.

Let me know what you think of Game Of Thrones in the comments…

jQuery Hello World Plugin

Ben E. C. Boyter — Sat, 16 Apr 2011 02:17:36 GMT

I was doing a simple job test the other day and one of the questions involved creating a simple jQuery plugin. Having never created one myself I had to look into how to do it. I couldn’t find a dead simple hello world plugin example so I thought I would create a simple example here for people to look at.

(function( $ ){
  $.fn.HelloWorld = function() {
    $(this).html('Hello World!');
  };
})( jQuery );

The above essentially just attaches a new function called HelloWorld to the basic jQuery object. You can then call it using the below,

Convert MPO to Anaglyph or Stereo Images

Donat Studios — Thu, 14 Apr 2011 07:41:33 GMT

Convert MPO to JPEG

This tool converts MPO files from devices like the Nintendo 3DS, Fujifilm FinePix Real 3D, or other 3D cameras that support MPO files, into more usable images. It's useful for viewing and sharing 3D photos in a more commonly used format.

Unlike many other online tools that simply strip one of the eyes from the result, presenting you with a single image, this tool incorporates both the left and right photos, preserving the depth and detail of the original 3D images.

Additionally, it can convert MPO files into an anaglyph format, allowing the images to be viewed with standard bi-color 3D glasses.

Gallery

jQuery Documention Now Avaliable

Ben E. C. Boyter — Thu, 14 Apr 2011 02:16:56 GMT

It has been a long time coming, but jQuery is now in the index and can be searched on.

You can by searching for jquery

EDIT And a list of all functions to go with it, List of all jQuery Functions

Introduction

Nemo's Home — Sun, 10 Apr 2011 03:00:00 GMT

Welcome to my github pages. This will be my personal code blog site, or something of that sort. I will soon be transitioning this site to Jekyll for easier publishing and maybe move away from wordpress. (Done)

I am a proficient coder in PHP, working on various internal projects at SDSLabs.

Here is my current setup :

Dell Inspiron 1545
T-6400 Intel Core 2 Duo
Ubuntu Oneric Ocelot 11.10 (Default Primary OS)
~~Elementary OS (Under Testing)~~ Will be moving to Gnome Shell soon
~~Samurai WTF, Backtrack 5~~ for pen-tests
Moved from Windows after remaining a staunch supporter for 5 years

Simple WPF VB.Net JumpList Example

Donat Studios — Thu, 07 Apr 2011 13:00:31 GMT

Today I decided to spruce up a VB.Net application I wrote in college and still use almost daily. In the process of converting it from .Net 2 to .Net 4 I wanted to add a splash of Windows 7 goodness; there were some very simple operations I wanted to add to JumpLists. The problem I ran into with all the examples I could find is that they either called entirely different applications (the Microsoft examples all called Notepad) or the example was in C#. The application is in VB.Net and I intend to keep it that way, as it is my .Net language of choice.

The interesting problem with JumpLists is they are just shortcuts. They call an executable with the specified arguments. Microsoft’s reasoning for this is that they are still available while the program is no longer running. The initial execution of an application placed in the taskbar registers the JumpLists, after which point they are cached by the system and continue to be available even when the application is no longer running. This potentially limits the usefulness of JumpLists as they do start a new instance of your application. If anyone knows how to catch the arguments on a single instance application, I would be very interested to hear it. The best I’ve come up with is simply passing messages to my current instance from a new instance and then terminating the second instance, but this is awkward to my eye.

XAML can be used to create JumpLists in the Application.xaml file, which many other examples will demonstrate – but the issue here is there is no way to get the path to the current executable via XAML, so you can only call other executables whom you know the full path to.

The answer then is to create our JumpLists dynamically. The best place to do this is in your Application.xaml.vb file. The New method will be used to instantiate and register the JumpLists, whereas the Startup event will be used to handle arguments passed to the new instance of our application.

Below is a very simple example of JumpLists not too far off from what I ended up going with in my own application. Feel free to leave me comments as to any improvements or changes you would make, or go ahead and fork the gist.

How to run apt-add-repository behind firewalls (#iitr)

Nemo's Home — Fri, 01 Apr 2011 03:00:00 GMT

Copied from OMGUbuntu

Press Alt-F2 and type “gksu gedit /usr/lib/python2.6/dist-packages/softwareproperties/ppa.py”
Find line 88, change “keyserver.ubuntu.com” to “hkp://keyserver.ubuntu.com:80”
Save and close

Note that this is the default setting in Ubuntu Natty Narwhal (11.04), and was only applicable for Maverick or older versions.

How I store Enumerations in the Database

Ben E. C. Boyter — Thu, 31 Mar 2011 02:16:26 GMT

One of the things I come across in databases now and then is a collection of single tables with a name like “MessageType”. You have a look in them and it turns out to have 6 or so rows with no foreign key relationships. Every single time it turns out to be someone had the idea to store an Enumeration (Enum) type in the database. Not a bad idea as it turns out since you can add sort options, soft deletes and the like, but the implementation of a single table for each one is flawed.

PHP User Agent Parser

Donat Studios — Wed, 30 Mar 2011 06:51:09 GMT

Demo

Requirements

PHP >= 5.4.0

Download

The script is available over on GitHub or via Composer:

$ composer require 'donatj/phpuseragentparser'

Usage

The original procedural use is as simple as:

<?php

// v0 style global function - @deprecated
$uaInfo = parse_user_agent();
// or
// modern namespaced function
$uaInfo = donatj\UserAgent\parse_user_agent();

echo $uaInfo[donatj\UserAgent\PLATFORM] . PHP_EOL;
echo $uaInfo[donatj\UserAgent\BROWSER] . PHP_EOL;
echo $uaInfo[donatj\UserAgent\BROWSER_VERSION] . PHP_EOL;

The new object oriented wrapper form:

<?php

use donatj\UserAgent\UserAgentParser;

$parser = new UserAgentParser();

// object-oriented call
$ua = $parser->parse();
// or
// command style invocation
$ua = $parser();

echo $ua->platform() . PHP_EOL;
echo $ua->browser() . PHP_EOL;
echo $ua->browserVersion() . PHP_EOL;

Currently Detected Platforms

see: List of Detected Browsers

Lesson Learnt – Save Assets

Ben E. C. Boyter — Tue, 29 Mar 2011 02:15:49 GMT

I really should learn from my past mistakes, and usually I do, but this one comes to bite me again and again. I never save assets.

This results in me having to find fonts, recreate images and the like over and over. If you do a hard refresh of this page (CTRL+F5) you will notice the logo looks a little different. This is because I couldn’t find the font I used before and had to find something similar.

Coming in April–WGR 2!

Ben Overmyer's Site — Thu, 24 Mar 2011 02:00:00 GMT

The next version of WoW Guild Retrieve is coming, and this will be a doozy of an update.

Slated for an April 30th release, WGR 2 brings a host of new features.

Caching – The plugin will only retrieve data from the Armory every 4/12/24 hours, depending on what you set the frequency to. This means high traffic sites will no longer experience temporary blacklisting.
New data source – Instead of pulling from the old XML service, v2.0 will parse the new Armory. This means new information will be available for display, including (but not limited to) Achievement Points.
More flexible limit options – The new version lets you set a specific level limit instead of just the option of 85-only. You will also be able to limit the players displayed by rank.
New styling options – Visual control is back! Instead of just “dark” and “light” themes, you will be able to control the colors of each of the fields through the plugin settings page.
Column display toggling – Now you will be able to toggle individual columns on or off. Everything but the name column will now be optional.

In addition, there will be some behind-the-scenes changes to make the plugin play nicer with other plugins and generally be easier to use.

iOS Documention Now Available

Ben E. C. Boyter — Mon, 21 Mar 2011 01:15:12 GMT

Added iOS documentation to join the OSX documentation.

You can by searching for appleios

Attribution 101

CRAIG KERSTIENS — Sat, 19 Mar 2011 05:55:56 GMT

Continuing with the recent posts on metrics and marketing. I want to give a quick primer on attribution. To any marketing or analytics people out there, simply skip this it would aim to be a primer recap at best for you.

The very general meaning behind attribution is to give credit. When it comes to web products this can be giving credit for lots of things:

Its really wonderful when there’s a direct mapping in correlation. Take for example the case where:

Disclaimer: I have little to no formal training in stats or analytics, have simply learned through launching products so take this for what its worth, someone that has been there and done it.

Lorem Ipsum Word Generation

Ben E. C. Boyter — Fri, 18 Mar 2011 01:14:28 GMT

Something I tend to need from time to time is some generated lorem ipsum text as a placeholder. I would normally search for lorem ipsum, click through to lipsum.com click paragraph etc…

I thought this was too much work, so I quickly added a lipsum generator.

To use it just type the following,

lorem:100

Where you can replace 100 with any number of words up to 1000. Works for the following search texts as well,

Wona Oct-Dec '10 Review

Nemo's Home — Thu, 17 Mar 2011 02:00:00 GMT

Download the issue on the WONA archive website.

Note about Archival: This post used to live on the (now-dead) piratecoders.co.cc website. I’ve moved it here for archival’s sake.

The latest issue of WONA turned up 3 months late at my doorstep. Other than the fact that it was missing an apology letter for this lateness, it was a good step in the forward direction by WONA in general. Once you take aside the pleasantries, and the sarcasm, I felt that WONA was, on a general scale, surpassing what it had been doing till now, and moving towards a better (and hopefully a quicker) issue. However, this one was definately not the one to be labelled perfect. So here comes the review :

Cover : Let’s start with the cover. Not much to write about it, other than the fact that it took me quite some time to figure out what was Maradone doing on it. Nice work in putting up the WONA logo (which I’m genuinely fond of). Nice choice of color scheme, and over-all good work.

Editorial : The magazine starts at a good note with the editorial promising us what’s inside in a nutshell. This was the best write-up of issue for me, and did its job well. I was enticed into reading further, and got a gist of what was about to come.

Almost Famous : Almost Famous has quite a history of its own, and is one column that every person who lays his hands on WONA definately reads. This time around, you held out Jan Flaming for me. As for the interview itself, it was clearly written, had some nice questions, some imaginary answers, and overall does quite well for the reader. On a second take, this was probably the reason it ends up on the first page this time.

Murphy’s Strip : Alas if Murphy had been here, he might have taken a shot or two at you. How about : “Any comic that can go wrong will go wrong.” Or maybe “If Vela could draw stupid sketches, he would.” Still, nice concept, bad execution. Could have been better, definately, especially with regards to sketching. Still 42 times better than the other strip.

NewsNotes : were 4 pages of news, which arrived 3 months too late. Nobody remembers any of these incidences happening, let alone being part of them. PAN-IIT was way bigger than your coverage, and deserved a bit more. Rest all the newsnotes were in a bit, exactly what they were supposed to be : News, without any sarcasm or humour. However I’m heavily against the use of one and half pages to glorify pages. I know how costly it is to push each page into the magazine, but please don’t fill it with something that no one bothers to read. Medal winners, please don’t mind but the two tables were a waste of space, if not something else. Or else the nine people from WONA’s news section could not find enough news. Anyone could have gotten these lists from the Sports Council a day after the event ceremonies. Give me something better.

Face-Off : was usual stuff. Nice choice of topic, nicely edited, with some actual points being thrown around, this turned out fine.

Big Story (Devil Wears Prada) : In spite of my initial skeptism about the story (I knew about it before the issue came out), I really liked it. This article goes ahead and proves that what you write on is unimportant (or at least less so) as long as you write it well. In a world of geeks, bringing out a cover story on fashion is really a bold move, and I must applaud you for having the guts to do so. The article was well written, and a joy to read. However I’d come to hear, from several sources, that people quoted in the article did not in fact give one to WONA. Please take care not to make up stuff next time. You’re a news mag, stick to the status quo, please.

Verbatim : WONA gets its hands on the most respected professor on on our campus, and does quite a good job of it as well. Nice questions, interesting answers make up a good read. In fact, the only negative point of the article was its placement.

WonaLeaks : This was the article that forced me to write this review. If it was an attempt at humour, it failed badly. As an attempt to mix news and wonaspeek, it fails even badly. Mixing kangaroos, indian cricket team, koala bears and a state prosecutor is definately not the recipe for sucess. It might work in movies (Spaceman + Potato Head + Zombie Dolls + Cowboy + Alien with 3 eyes = Oscar), but definately not on paper. Nobody remembers the event the article talks about. Its relation to wikileaks is not enough to demand a WonaLeaks icon. I shall forever remember this as the worst of writing, and imagination that ever came out of wona.

Tech-ila Shots : The in house tech article arrives a few months late. (Google just released Cr-48). I would have personally liked to see something else here (iPhone vs Android, iPhone 5, Ubuntu vs Windows etc). But as it was, the article was well thought of, and did exactly what it planned to. It could have done with a bit of pruning though, and the author might have liked to tell us a bit more about the OS itself (it only mentions the fact that its web based, comes as pre-installed, will not run intensive applications).

Random Ed : I’m not sure if it really is an editorial, but I’ll stick with the title. Nicely written in short. However the purpose and intent of such an article is lost in the true random nature of the article itself. The article fails to reach a resolution, and delievers nothing at all. Indeed most of the thoughts mentioned in the article, must have sprung to every person’s brain at some time or another. Then what’s the need of this article? Is it philosophy that Wona’s trying to dive into? Or perhaps its just a conspiracy to get all students in the campus to think more randomly, leading to a decrease in entropy of the thoughts of the profs, and as a consequence, simpler question papers. (For the skeptics, something similar has already happened, and our thought patterns are involed in the entropy of a system.)

Ethics : was actually the cover story (which one comes to know only after re-reading the cover). The author is unclear of the intent of the article, and it steers in various directions. I’m still unsure as to where it ends, and whether Arasu’s mumblings (Another Brick in the Wall) are part of it. One might raise question as to the relevance of the last section itself, but this article had its moments as well. In short, nice concept, nice writing, but could have done with a bit of restructuring.

Canine Strip : Another page wasted. The time spent by the sketcher could so easily have been devoted on bettering the other one. Needless to say, wona seems to be lacking in people who actually write, and the it results in a page filled with a stupid comic about dogs taking over the campus. Seriously, get some creativity. Even zombies would have done better.

WORC : Good decision in continuing this column. I was afraid it might get scrapped along with Agony Aunt(which was a very good decision). I can just hope that you actually asked some persons to create those pie charts. Dr. Sinhval seems to have taken his time in answering the questions, and his reply is full of facts, explanations, and ideas. Nice work by the ed team here, definitely.

Other than the articles there are a few more areas of interest I would like to point out.

Design: Seriously, have you people ever heard of vector graphics. Your designers really need a course on making scalable graphics. The need for Darth Vader to illustrate an article is fine, as long as you have a big enough image to fit. Trying to scale a 400×300 jpg into a page will not do. Even the cafe de norma ad was pixellated. The girl wearing Prada proves that it was possible to pring clearer graphics . However on the very next page, there’s an overdose of black. Similarly pixellated were most of the other pics on the mag. The 3 monkeys illustrating the second last page could have been something better. The essence of the German flag was lost in black & white(Almost Famous) . And just so you know that everyone noticed , dark colored pics in the background make text unreadable (Chrome OS).

Ads : I don’t know the reasons behind it, but from a reader’s point of view, an issue with only 3 ads is really awesome. Especially once you compare it with Kshitiz’s latest issue. I know it must have been really hard to cover the costs, and manage the finances and all that. But folks over here are smiling for your hard work. You finally published it (albeit 3 months late), and that’s what matters.

Placement : could have been better. Some articles should have gotton more coverage than given. Perhaps you should think of making Tech-ila bigger. I would’nt have minded the least if Verbatim had gotten a bit larger. The ethics article was divided into portions, I couldn’t understand, and might have done with fewer sections.

Cover : A little side note as to why the article change from the cover to the article itself. Isn’t the cover supposed to hold titles as well. On second thoughts, I might be wrong to suggest your current layout, but can you just put up page numbers! They would surely help.

And finally a note from my side : Wona is an excellent magazine. Despite it having a status quo of its own on its supposed unreadability without an excellent understanding of wona’s inner sanctum, I believe that the magazine is an essential part of life at iitr. It is one of the few sections in our campus, whose work is actually a part of our daily life. Thanks to the entire team for working so hard on this issue.

This review is my way of letting you know that there are people who care about what you write. There are people who wait for the issue, and who are determined to do so till they pass out. Consider it a friendly nudge and a little feedback from my side. Its up to you decide what your mag is after all. With hopes that the next issue is even better than this one

Update: Friends over at wona inform me that the delay in the mag was due to an issue on the administration’s side rather than wona’s side. If that was the case, the blame’s partially on your staff advisors (spelled incorrectly in the mag’s first page) : Dr. M.J. Nigam, and Dr. B.R.Gurjar. I’m still not sympathatic to the excuse given entirely, and believe that it might have come out a bit more sooner with some more effort on wona’s side.

List of Most Commonly Used PHP Functions

Ben E. C. Boyter — Wed, 16 Mar 2011 01:13:43 GMT

One of the things about ranking in Search is that you need to consider all sorts of methods of working out what is relevent. Google broke new ground (although the idea had already existed) with its PageRank algorithm which supplied better search results then all the other search engines. For what I am doing however I need to consider what programmers are looking for. One thing that I considered some time ago was working out which are the most common functions in a language and adding this as an additional signal to ranking.

List of MySQL, SQL Server 2008 and Apache Directives

Ben E. C. Boyter — Tue, 15 Mar 2011 01:13:47 GMT

Added some more lists now that the feature is working correctly.

List of MySQL Functions
List of SQL Server 2008 Functions
List of Apache 2 Directives

List of all GNU/Linux and Windows Commands

Ben E. C. Boyter — Tue, 15 Mar 2011 01:11:51 GMT

Today I can announce a small but possibly useful feature which is the addition of lists. They are here as way of making it easier to find what you are looking for if the main search lets you down and you want to try a browser inline search. They can also be useful if you just want to scroll through everything in a particular subject. I have the following lists already created,

Startup/Bootstrapped Marketing Recap

CRAIG KERSTIENS — Tue, 08 Mar 2011 05:55:56 GMT

If you have an hour to spare its well worth it to look back and look back at my series on startup/bootstrapped marketing. But if you’re short on time and want the high level summary here’s the quick recap:

Part 1 - Focus on SEO

BATF – Big Arse Text File

Ben E. C. Boyter — Tue, 08 Mar 2011 01:10:23 GMT

Ever needed the ability to track bugs and features without using a full featured bug/feature tracker? What about storing all your random notes such as server details, blog ideas, books to read, urls etc, without using a full featured CMS or the like. Want to have everything searchable and in the most platform independent format possible?

Enter the BATF. I have always been a fan of the big arse text file (BATF) for keeping track of the above. The catch being I wanted it centralised so I could get at it from any machine I was on (assuming internet access). I also wanted it to provide a simplistic version system. Tags would be useful too.

Reading Metrics to Evaluate Marketing

CRAIG KERSTIENS — Thu, 03 Mar 2011 05:55:56 GMT

A short while backed I talked about tactically measuring metrics for your site/company. Recently I talked a bit about methods of marketing. A large key to getting the most out of your time and money is to properly report against the intersection of these two items. First I’m going to make the assumption you’ve read those posts, if you haven’t go back and do that. Next this is heavily on the assumption that you’re using Google Analytics as your primary tool for measuring metrics and have setup goals appropriately.

Within measuring you’re metrics you’ll have abandonment at each level. You may have some visitors that never register, and many that register but never purchase anything. It’s wonderful if you’re able to immediately have full insight of the best means of marketing to drive revenue, however realistically it occurs in a more phased approach. The first step is to drive visitors and almost immediate second is to convert those users as registered.

Acquisition -> Activation

This at a high level makes sense, as does at a high level knowing you should be targeting users in your target market. However, even slightly drilling into this you realize that all traffic is not equal. It’s often known that CPC and CPA advertising does not convert well for users, though can drive traffic. This may not always be the case. For Registry Stop to make this analysis easier we’ve created a custom report in google.

To create your own custom report simply click Manage Custom Reports under the Custom Reports area then “Create new custom report”. The custom report ability in google gives you much more ability to drill into the data that you already have at a higher level. To track effectiveness of converting visitors to registered users and which sources are effective at this you’d create something that looks like:

Here while very simple we’re able to see some very key information quickly. Here’s an example of how it would appear over a few day period:

As you progress in your stage of bootstrapping or growing your startup you’ll want to grow custom reports that allow you to report against as many metrics as possible. The first and last part of my day is spent pouring over these reports. Having this data readily available allows us to drive our business based on data. Perhaps the hardest part of all of this is admitting when the data is counter to what we expect and following its advice.

Batch RewriteRule Generator

Donat Studios — Tue, 01 Mar 2011 09:58:44 GMT

Find this useful?Buy me a coffee!

PayPal:

GitHub:

Ko-Fi:

I find myself often needing to set up a large number of 301 redirects from an excel file of old to new url’s. Until this point, writing these had been a fairly exhausting process as you need to be certain to escape every character that could be picked up by the regular expression engine, or risk unintended consequences. On top of that, writing these rewrites to catch GET parameters in any order is complex to say the least. While working through a large list of 301s for a very picky server where Redirect 301 style redirects were not an option, I began setting up RewriteRule's by hand and decided there had to be a better way. Voila! My 301 RewriteRule Builder was born.

You can simply enter a list of tab or space delimited urls (copied from Excel works great) and it will spit out the generated RewriteRule’s for you. GET parameters on the old urls are broken up and built into multiple RewriteCond’s set up to work in any passed order.

I hope someone finds this useful. Any comments are welcome below, of feel free to fork me on github!

Please make sure you have RewriteEngine On somewhere near the top of your .htaccess, before you attempt any rewrites.

Need more help? The tool can only do so much, and sometimes you need human help. I offer .htaccess help as a paid service and have 10+ years of Apache experience. Send me a note on my contact page and we can discuss your options.

Fork my source on Github!

Setting up Goals and Funnels - Google Analytics

CRAIG KERSTIENS — Tue, 01 Mar 2011 05:55:56 GMT

I had a recent request on how to setup a funnel in Google Analytics. If you’ve missed by first post on some tips for Google Analytics first check that out. With most websites today there is some portion of the site that is event and not page based, meaning you have some workflow on the page based on Javascript. If this is the case you’ll want to fake a page view instead of an event in order to entirely use it in funnels and goals.

A personal recommendation is actually to use both, goals and funnels. The key a funnel is that you need to have successive steps that occur in some order. With regards to metrics tracking this is absolutely needed, but typically you may have 1-2 total funnels with many steps in your site versus goals where you could have 10-15 single goals. For Registry Stop we’ve structured our site so that our earlier stage goals become the same as steps in later stage funnels. For us in almost all cases the first part of the funnel is the visit, the second is registering for an account. We do have independent goals for visits and registrations as well, but we do not have funnels on those goals.

A key to getting the most use out of your funnels is to know that there is a workflow to follow to getting to that end goal. To highlight this slightly more visually let me walk through an example:

The parts of your funnel will be:

Once here it becomes a bit more intuitive. You can begin by simply adding a goal. Your goal types should seem mostly intuitive, as mentioned in an earlier post you cannot use an Event in a goal. For this reason you can fake a pageview as if it actually occurred and then create your goal against that non-existent page view. If you want a few more details of how to do this check out the previous post . So for an example, we have a goal on Registry Stop that detects when a page view occurs as a result of a registry being synced. Because this workflow is heavily javascript and flow based we fake the page view and track it as if the page was actually visited. The goal itself looks like:

For setting up your funnel as we mentioned above its generally a set of page views. A very key item is the check box next to the first item in your funnel. If you check this is means any other steps in the funnel are not counted unless you the first step is completed. If you have a very structured 1, 2, 3 workflow this makes sense. However, if there are various ways for the goal to complete then be very careful about selecting this.

For this same goal above we have a corresponding funnel to track in detail how our conversion flows. The funnel itself looks like this once setup:

This results in a funnel report that looks like:

Using wget and xargs

Ben E. C. Boyter — Mon, 28 Feb 2011 07:46:55 GMT

The joy of the linux/unix command line is how versatile the commands are. I recently had 50,000 URL’s I needed to download in a text file. I was thinking about writing a crawler in Python to do it but ended up just doing the following,

cat urllist | xargs -P16 wget -i

A 16 thread (process really) webcrawler in a single command. Joy.

Evaluating Paying for a Blog Post

CRAIG KERSTIENS — Sat, 26 Feb 2011 05:55:56 GMT

At a recent meetup I talked a bit about how I’d been using blog posts on other blogs, both free and paid for as a primary user acquisition tool. I was very shocked, when several were surprised and curious on the method for this. In tech startups coverage is common, but its usually just that press, not paid for press. I must say I love how the tech community doesn’t force people to pay to get the word out, but it is very much a competition; that might be just as much work as paying.

In contrast the wedding industry is very much a pay to play space. If you give some money you can get some attention.

First things first, contact the blog you’re interested in being written up in and ask for their media kit. If they welcome sponsored posts, then it is likely called out in their media kit. However, this isn’t always the case, if you’ve noticed posts on their blog that have been sponsored posts but pricing isn’t called out in their blog then email them explicitly and inquire.

Once you’ve got their media kit its time to do some digging. My process has first been to validate their numbers. Most blogs include their unique visitors and page views in their media kits. I immediately jump over to compete to check if their numbers are even in the ballpark. To clarify in the ballpark can be somewhere around 1.5x of compete. There are a surprising number of blogs that may be 20x off in the numbers they are stating. This could mean you’re outright lying on your stats, or that you’re not running a solid enough business that you know how to effectively track your numbers. It could be, because your blog exists on 5 different domains, while compete I’m checking only the primary. Regardless, if you’re numbers aren’t close, it often means you’re not as together as we’d like.

If they pass the first smoke screen of the stats being in the ballpark, then we can move on to evaluating sponsoring a post. Traffic’s a big factor, unique visitors are important as well as page views. Next we’ll typically look for how active your users are. Do users actively engage in comments, where there are active commenters there’s usually opportunity to get a bit more out of your post.

Next would be, how frequent are posts. Are you looking at 1-2 posts that go up per day, or 10. If 10, it simply means your content will be pushed to the bottom of the page pretty quickly, in this case if page views are exceptionally high, it may mean that users only view 1-2 posts per day and miss the others. I’ve historically done this on a subjective basis, but it could easily be a number that is calculated and factored in.

So you start with this basic methodology for one blog, then do it for a few more. Its pretty simple to compare 3-4 blogs on their potential value, but when you really start expanding this you could be looking at 100 blogs. If that’s the case it does help to have some structured method. We typically weight their unique visitors, page views, a factor of how accurate they are against compete, their commenter level, and finally their post frequency. We multiply that weight against the cost of a sponsored post and there you have your priority in terms of which blogs to begin advertising on. Its usually best to try 2-3 blogs to determine your return. But even one can give you an idea of results.

A quick recap of the basic formula:

JQuery and Django Autocomplete

CRAIG KERSTIENS — Sat, 26 Feb 2011 05:55:56 GMT

In a couple of various places I’ve seen light requests of how to put autocomplete in for a Django web application. Here’s a really light weight version with a view and autocomplete functionality using:

from django.utils import simplejson
def autocompleteModel(request):
 search_qs = ModelName.objects.filter(name__startswith=request.REQUEST['search'])
 results = []
 for r in search_qs:
 results.append(r.name)
 resp = request.REQUEST['callback'] + '(' + simplejson.dumps(result) + ');'
 return HttpResponse(resp, content_type='application/json')

For the jQuery autocomplete and call:

Bootstrapped/Startup Marketing Part 4

CRAIG KERSTIENS — Wed, 23 Feb 2011 05:55:56 GMT

We’ve talked some about SEO, media/blog posts, adwords, no one of these is a magic bullet. Some work better for different reasons. As I mentioned in the first post, if you haven’t checked out the post on tactically measuring metrics then please do. If you have followed those steps and explore each of these options, then you should have an idea of which one works well for you and which doesn’t. The final piece of marketing may be a bit harder to measure, but is going to do great things towards growing your brand to users and visitors.

Retargetting Retargetting is the idea of showing an ad to a user that has already visited your site. It’s pretty basic, someone comes to your site and you have a pixel that loads telling your ad network they’ve visited. From then on they may see your ad when randomly browsing the web. With my most recent venture into the online registry space I was browsing Chiacgo Tribune and Slashdot and came across our ads. There’s absolutely no contextual relevance there, but I checked and it was due to our retargetting. This will make it appear as if you are everywhere to your users. If you do want to heavily monitor what retargetting is doing for you, the best place to do it is around your retention metric.

How do you do retargetting? Sounds like a complicated process slightly… Well it’s simple you don’t, let one of the major ad networks do it for you. The first step for it, is to use image ads. Text based ads may work great for google and facebook, but on most of the websites your retargetting will run on you want to have images that the blogs typically serve. The 3 key form factors you’ll want are:

If you want your shop to seem like a mom and pop site that has 100 visitors a month, then retargetting will hold absolutely no value for you. But even if you do only have 100 visitors a month, with retargetting they’ll get some confidence that you’re a real online brand with a presence.

Bootstrapped/Startup Marketing Part 3

CRAIG KERSTIENS — Sat, 19 Feb 2011 05:55:56 GMT

For this third part on the series I’m going to dive into what people perhaps most traditionally think of with marketing startups, online advertising. Online advertising can work, but its definitely not cheap and it does take a good about of pounding at it to know what works. I’m going to break up the three key types of advertising, based on the way I’ve utilized them and evaluated them recently.

Contextual The first is search advertising, or contextual. The biggest usage of this has of course most recently been with google AdWords. For the complete novices out there, this is where you’re ad would appear as text based on certain search keywords. You’re the one that is able to determine the keywords, though google does rank you for relevance and how much you are willing to pay respective of the other person bidding on the same keywords.

For a startup I’d very strongly discourage advertising for keywords you already appear as the first result. A part of this goes back to part one of the series, SEO doesn’t cost you anything other than time, so invest in it. If you’re already at the top of the results, why potentially pay $2 for a click, when you’re already there. With limited budget you want to be very selective about which keywords you target. In our experiences the best targetting can be with regards to your competitors or similar products. Also take advantage of google’s tools here. There are many tools within adwords that will suggest new keywords, rank your relevance, and show traffic to certain keywords.

Always remember, what you really want is conversion, so make sure that’s what you’re tracking against performance. The key to doing this is linking your Google AdWords account to your Google Analytics. Google mentions in several places you should do this, but is very light on the instructions of how. So for a really quick how to:

We followed a rather structured process to identify what worked. If there’s enough interest in this specifically I’ll do a full follow up post later but the high level steps:

Vertical Most people term this display advertising, display is typically a banner/image ad going up on some website. I prefer viewing it as vertical, because if you target your display advertising correctly on sites you want to be on, its definitely more targetted than a standard billboard. Google does have some options for display advertising, but I’ve found their image approval process very painful.

With display or vertical advertising there’s a few key’s I’ve found valuable:

Image approval process matters as you dont have multiple weeks to spare
The ability to target/select your sites is an obvious must
Filtering out MFA (Made for Adsense) sites in the case you do open it up slightly by site type

On the third item theres two ways of doing this, you can manually check daily in google, and exclude sites. What you’re looking for here is sites with < 100 impressions and abnormally high click through rates. The other option for managing and tweaking these vertical campaigns is to let someone else do much of the optimization. My recent favorite for this is AdRoll. There’s definitely a slight premium over going directly to google, but they do let you easily get your campaign running and very much help optimize.

Bootstrapped/Startup Marketing Part 2

CRAIG KERSTIENS — Thu, 17 Feb 2011 05:55:56 GMT

For the second part of the series we’re going talk a bit about finding the influencers in certain industries. We’ll get to more traditional means that people think of later, and if you’ve missed our first post that dealt mostly with SEO make sure you check it out first. In most online ventures there’s a key set of influencers, often times these are blogs or podcasts. Blogs can receive a huge readership, which are often very loyal.

The first step to taking advantage of this is obviously finding the correct blogs. As a byproduct of being in the valley I spend plenty of time reading Techcrunch among many other blogs. While a post on Techcrunch might result in massive traffic spike, or some moderate feedback, its definitely not in our target demographic. If our goal was investors then Techcrunch might be viable, but here we’re talking about marketing and marketing to core users at that. The blogs for your core demographic should be pretty straight forward and you can find them through google or other basic means.

The hard part might be once you’ve found them, how do you get on them. The first thing to do is work on networking with them if at all possible. If you’re in the same area as bloggers try seeing if they typically attend certain meetups or go one step further and simply organize a meetup or tweetup yourself. Or simply network, find others in your space that can make in intro. Hands down a first person intro will work better than any other method.

If for whatever reason networking and getting directly in touch doesnt work (which can often be the case). You can always resort to cold emailing them. This can work. There’s a few keys to this approach though:

The next step will bleed a bit more into the next post in this series, but here’s a small preview: Advertise to your influencers, not your users. If you have a limited ad budget, don’t advertise to your target user. Advertise to the influencer of your target user. We’ll get into more detail on this in the next post but a quick example:

If you’re building a sports site, why advertise on facebook for people who like sports when you could advertise to someone that works on espon.com. If the latter notices you and see’s interest they could potentially write you up or refer you to people that might be of interest.

Bootstrapped/Startup Marketing Part 1

CRAIG KERSTIENS — Tue, 15 Feb 2011 05:55:56 GMT

This is the first of a 4 part series on marketing for startups/bootstrapped companies. Much of the learnings from this are a result of experiences with Registry Stop. The key to each of these is going to be measuring and reacting to your efforts. If you need help on this, check out previous post around metrics for startups.

So without further adieu, on this initial post of the series we’re going to talk a bit about the biggest free way to get traction and traffic for your startup. The best way to aquire free traffic to your site, is to ensure your site is optimized for search engines or more commonly SEO. Sure you can pay $3 for your ad to show up on certain keywords, but why spend the $3 per click if you can simply ensure you’re the first search result. There are slightly different methods for this for each search engine, but we’ll cover a broad set of items to pay attention to.

Sitemap Most sites have a sitemap.xml at their root level. This could perhaps be one of the biggest pieces of getting indexed that you can pay attention to. This xml tells search engines what pages they should index, how frequently they are updated, and the priority of the page. If you have dynamic pages, you should have this sitemap.xml generated so that it captures all pages.

If you need a little more reading on creating your sitemap take a look at: http://www.google.com/support/webmasters/bin/answer.py?answer=183668

Meta Tags Many search engines pay little attention to these tags, but that doesn’t mean that all don’t. You do want these tags to be as unique as possible per page and relate as much as possible to the content. The really key meta tags you want to have would be your description and your keywords. For a little more information you can check out: http://searchenginewatch.com/2167931

Other Tags While meta tags make pretty straight forward sense, other tags are immensly powerful in how a search engine indexes your site. The first is the title tag, you absolutely want it in and want it to be unique per page. You also want to correctly use html markup throughout your page. Just because your page looks like you want, doesn’t mean a search engine will understand it the same way. It’s important to use proper headings, including h1, h2, h3, to show just that, your headings and importance. You also want to be careful about the use of tables when not representing tabular data.

Once your page is live, you’ll want to submit it to google and other search engines. Do Not use a tool to do this, submit to the major search engines manually. Once you’ve done this you’ll start to appear in search results. The key from here is to begin monitoring what users are searching for, from within google analytics, and when they visit your site. A few key links to submit and manage your sites, as well as evaluate how you’re doing include:

MySQL Export to CSV

Ben E. C. Boyter — Tue, 15 Feb 2011 01:33:51 GMT

Ever needed to export data from MySQL into a CSV file? Its actually fairly simple,

SELECT * INTO OUTFILE '/tmp/name.csv'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\'
LINES TERMINATED BY '\n'
FROM [tablename]

Certainly easier then writing a quick Python/Perl/PHP script to do the job.

Requirements Gathering for Consumer Startups

CRAIG KERSTIENS — Tue, 08 Feb 2011 16:49:20 GMT

Most all development projects start with a hunch at a problem. Seldom do you have the opportunity of enough resources prior beginning building to fully vet all assumptions and define all requirements. Or at the very least if you do, you’re not in startup mode. For this reason the very first thing you build is often not the perfect solution. If you’re lucky its a start at a solution, and even if its not, if you’re close users will tell you what they want.

What this leaves you with is a couple of key items. First is get to the minimum product you can to vet your idea. Most commonly known as Minimally Viable Product. This should be the minimum product you need to vet your idea, and add some form of value for users. Once you’ve created this, don’t refine, don’t keep iterating, launch. More time won’t let you perfectly solve the problem, getting it in front of users will help you solve things perfect.

Second is make feedback simple. Often times support can be difficult, if its a form that requires 5 fields on your website forget it. If at all possible, provide multiple ways for users to give feedback. ALL ways should be simple for users. Email is great, because every user already has it. If it can be built on your website, great, but you should make registering as light as possible. Personally, I’m a big fan of Get Satisfaction. They give a great embeddable widget and provide a service that just simply works.

Third, and perhaps most important: Listen to your users! When users start to give you feedback, that’s the best requirements you can receive. There’s a couple of pieces to this step. When users give you feedback, you want to acknowledge them. Give some recognition for giving feedback; respond as positively and supportively as possible. If at all possible to act on requests do it, it will set you apart from the typical services they use of never getting a response and never seeing a change.

In short three key steps make for requirements gathering that trumps all else:

Tactical Steps for Startup Metrics

CRAIG KERSTIENS — Fri, 04 Feb 2011 05:39:54 GMT

Metrics are obviously a very valuable area for start-ups, if you don’t believe in metrics and think you’re idea wins just because its great then you better start searching for your next day job. Dave McClure has done a great talk on start-ups several times over, you can check out a video and corresponding slide show at:

http://www.ustream.tv/recorded/5336115 http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version

And besides, it’s a pirates acronym, so it’s got to be great. But translating these from concept to actual technology metrics is also something that needs discussion. You can’t exactly say we’re going to use Google Analytics and let it magically tell you everything, furthermore all of the web developers out there talking about tweaks on Google Analytics do little to actually tell you what you need to know.

To start with I’m going to lay out a few of the key features of Google Analytics, assuming it’s the key backbone for web analytics, and because it really is an amazing free tool.If you’re not familiar with google analytics go and explore before reading this, we’re going to skip right over the basics of what are visitors and traffic sources directly to what you need to use and customise to get valuable metrics for a start-up. Diving right in a few key items:

Goals - These are custom items that you want to setup. What is a goal? Well that can be a bit abstract but it can be a specific page that was visited, time spent on the site, or a certain number of pages per visit.

Events - Here you can start track events to any type of your liking. In most cases this can be some in page action, but the key here is to each event you can track a value. This starts to become useful if you want to track most shared content, most liked products, or other items. Typically those kinds of items might exist within your in-house database, but exposing it to Google Analytics gives you an extra level of reporting integration

Custom Reporting - Google is great out of the box, if you feel like it just grazes the surface then you haven’t explored it. But in some cases it’s easier to create a custom report of data that you can’t get exactly to, or to get very quick insight into similar data.

So this is a really high level, but how do you use each of the above to tie to actually implementing the metrics you care about? We’ll go through an example of how you track each of Dave McClure’s 5 key metrics based. For those of you not familiar with his metrics, please go to the above links and listen to his presentation first.

Events with Google Analytics and Tricking Pageviews

CRAIG KERSTIENS — Thu, 03 Feb 2011 02:28:56 GMT

Google analytics is great out of the box, the basic tracking tag on every page will do a lot for you. Unfortunately most people never get beyond this. There are two key items with tracking that you can do that will let you get a bit further. There’s also plenty more on the reporting side, but we’ll get to some of that later. On the tracking side the first item is event tracking. This is perhaps most commonly used for tracking various Javascript events that occur during a visit, however it can also be a bit more flexible towards tracking values. A very simple example might be:

Or a real life example of this, might be on a FAQ screen, clicking the link to an anchored section of the page:

But events by their sheer nature give a bit more flexibility with that value field. In the case of a user sending a message

you might be able to track how many recipients it has, or any other numeric value you want to track.

Events overall are great, but you’re limited to the set Google Analytics report to know whats happening with them. So much of Google Analytics is based around page views, fortunately Google makes it easy to entirely fake a page view. If you’re wondering why you’d care whether it’s a page view versus events, we’ll get to that in a later post. For now what’s important to know is that you can fake any page view with: _gaq.push([’_trackPageview’, ‘/somepagenamehere]);

While seemingly small tweaks and extra additions these two items will create massive value for what you can actually do with Google Analytics. Stayed tuned later for how you use these with the base Google Analytics to actually get the value.

Converting Bookmarklet to Chrome Extension

CRAIG KERSTIENS — Wed, 02 Feb 2011 10:34:49 GMT

Google’s documentation is pretty good when it comes to how to create an extension that opens a full page and has large functionality. But if you’re more interested in transforming an existing bookmarklet into an extension there’s not great quality on it. The steps themselves are really quite simple. The big key that’s not heavily documented is creating a background html that creates an event listener. After the jump is a full sample that would then call your javascript to activate the bookmarklet:

manifest.json

background.html

chrome.browserAction.onClicked.addListener(function(tab) { chrome.tabs.sendRequest(tab.id, {fun: “callBM”}) }); </script> bm.js

MySQL Command Line Import UTF-8

Ben E. C. Boyter — Thu, 27 Jan 2011 23:24:36 GMT

Ever wanted to command line import some data into MySQL and keep the encoding type? Turns out its not that difficult. Just a simple command line option. That said I have to look it up all the time.

mysql -u USERNAME  -pPASSWORD --default_character_set utf8  DATABASE > file.sql

That will import things across with the correct encoding type. I think personally that the face that MySQL fails to throw an error or even raise a warning when it encounters these sort of issues is wrong but where you have no choice the above fixes problems.