Hacker News Personal Blogs 2018 | All Blog Posts

Empathy and Failures of Democracies

Essays by Matthew Eric Bassett — Mon, 31 Dec 2018 18:20:00 GMT

This is a meandering discussion about empathy's role in public policy debates and popular culture's role in affecting empathy. So can popular culture help to nurture or destroy empathy in politics? I try to set up a framework for exploring this, but raise more questions than I answer.

2018: Reflections on trying to start an internet business

Max Rozen — Mon, 31 Dec 2018 07:52:00 GMT

A review post on what I've done this year in terms of trying to start an Internet business

The Post

🦉 olshansky 🦁 — Mon, 31 Dec 2018 04:23:09 GMT

Olshansky's review of The Post

2018 in review

D13V — Mon, 31 Dec 2018 02:44:11 GMT

Looking back at 2018, I’m stunned how much it had to offer. I spent most of it on a sabbatical, and working on startup projects. Here are some highlights.

This might seem like bragging. I realize it’s rare to have it so good, and for many people it’s not the case. I’d like to look at everything I got and experienced this year through the lens of gratefulness.

I’ve worked a lot in the preceding years to be able to afford this, yet, I am incredibly lucky. So many things could have gone wrong and prevented me from enjoying one of the best years of my life. I’m grateful they didn’t.

First highlight, I learned a lot.

Some of the technical skills I learned are hands-on deep learning, web development with React/Django/Graphql, serverless computing with Lambda and Typescript.

I got better in some “soft skills” - public speaking, nonverbal communication, negotiation, persuasion, active listening.

I learned that the most important and most effective soft skills are to care about the other person, and to be open an honest. To take the long term view. Win-win!

I learned a few things just for fun. I dove into various aspects of blockchains and mechanism design such as Arrow’s impossibility theorem. I fulfilled a long time desire to learn about quantum computing and mechanics and to understand Shor’s factoring algorithm.

I baked bread and experimented a lot. Some of it turned out so amazing, I was told it was Tartine-level quality. That made me feel proud.

Second, I worked on my personal development.

I published 19 blog post and wrote one short fiction story I haven’t published. Writing is a way for me to process life. I’m looking forward to do more writing every year.

I quit caffeine on March 14 - it was one of the hardest things I accomplished and I felt a significant boost of energy after that. The withdrawal was a bitch.

I made exercising an almost daily habit. I was active before, but not daily. I started slow, just ten push ups a day, and every month I added more - squats, stretches, bends and twists. Later on I got back to lifting weights.

I kept training Brazilian Jiu Jitsu. Martial arts have given me so much. The more I train, the more interesting and fun it gets.

I took a nearly two month break from alcohol. I have done one month break for the previous ten years, but this year I increased the duration. I’m proud of my current relationship with alcohol.

I meditated. I clocked more than 2000 minutes on Headspace, and more untracked.

I did a 30 day no complaint fast. I couldn’t complain about something without suggesting how it would be fixed. It’s amazing how this rewired my brain.

I’m currently working towards a 30 day no excuse - where I have to say what I would change to succeed. This has been even more powerful rewiring.

I started FINALLY breathing through the nose. It’s surprisingly simple to achieve. I used SomniFix mouth tape at night, and after about a month of that I’m breathing through the nose nearly 100% of the time.

Third, I got to travel plenty and do fun things :D

I went to Rwanda, Tanzania, Morocco again, Northern Spain and the Pyrinees, Colorado, Bulgaria, and I got to go around California a bit - from Mount Shasta to LA.

I also got to race a Ferrari and ride a Jetski. Can’t complain.

I went home and got to hug my parents and brothers. ❤️

Fourth, I got to spend time with close friends and make new ones

I went to my friend’s wedding. I hosted friends for Fourth of July, Friendsgiving, and Christmas, and enjoyed cooking for them.

I met, brainstormed, collaborated and worked with amazing and inspiring people. If you were one of them - thank you ❤️.

I got to meet in person many of my closest friends, and with others I did video calls. That makes me happy looking back.

I also got to interact with several dogs and occasionally take care of them, and they brought me a lot of joy. ❤️

Fifth, I read books and listened to audio books.

I counted ~16 books I read and 6 audio books. I didn’t count how many podcasts I listened to, but many of them discussed books as well. I read a partial draft of a book that I enjoyed.

To highlight my favorite books - “Never split the difference” and “Without you, there is no us”. I was also moved by “Flowers for Algernon” and “Essays after Eighty”.

Sixth, I worked on a variety of projects.

I started a company, called AskAround, which would unlock the world’s information and failed. I was naive and impatient and that caused me and my cofounder to split.

I pivoted to building a crowdsourced research assistant, renamed to ImmerseMe and build a working service. I eventually pivoted out of it again.

I consulted one company in the NLP and voice space for a month, but ultimately found another cofounder and started working on a new startup.

Meanwhile my wife started build an airplane so I also started helping with that from time to time.

My workout habit turned out contagious, and several friends started joining me in lifting weights three times a week. I enjoy leading the group, designing the program, and it keeps me accountable in my own training. Gainz!

Last but not least, I got to spend a lot of quality time with my wife.

She’s my closest person and I cherish how our relationship has grown through the years, and how we share our path in life. ❤️ ❤️ ❤️ ❤️

It was not all butterflies and rainbows in 2018

I struggled with a bunch of things. Among them addiction to social media and youtube, procrastination, and building a sense of personal motivation.

Without external structure to impose deadlines and incentives on me, felt lost and often devolved into hours of binge watching Youtube. One of my main 2018 takeaways is the intrinsic motivation and discipline that I’ve been slowly building.

Still a long way to go. I didn’t know what to do with my life. I still don’t. At least I have a direction. And I feel a strong desire to live and see everything life has to offer. I’m thankful for 2018.

Favorite Books Read in 2018

dominik.net — Mon, 31 Dec 2018 02:00:00 GMT

My favorite books out of the 103 books (~35,000 pages) that I read in 2018:

Sci-Fi, Fantasy, and Speculative Fiction

The Baroque Cycle (Quicksilver, The Confusion, System of the World) — Rich, resplendent, and towering, this series of three (or eight, depending on how you count) books soars from the Glorious Revolution to the Court of the Sun King to rambunctious adventures around the globe. Filled with scientific references, historical tidbits, and cameos (as well as larger parts played) by historical figures, this series is a delight.

Norse Mythology — Ancient tales, lovingly retold in clear, crisp prose.

History

Rubicon — The events a hundred years before, and immediately after, Caesar crossed the Rubicon. Compellingly told and woven into a fine tapestry depicting the gradual and then sudden dissolution of the Roman Republic.

The Square and the Tower — Erudite and thoughtful consideration of the interplay between networks (the eponymous square of the title) and hierarchies (the tower) across history.

On Grand Strategy — A series of vignettes selected from history, highlighting themes of grand strategy across time.

Biography

Grant — A human portrayal of a simple man from Southwestern Ohio; of his struggles, his character, his flaws, and his virtues.

Elon Musk — Written in 2012, this book offers insight into Elon Musk‘s early life, initial success in business, and current motivation and drive.

A Mind at Play — I first heard the name Claude Shannon in undergrad, in a little experimental course called 6.095: Information and Entropy. I remember being fascinated at the simplicity and depth of the theory that Shannon laid out. This book surveys his life and the development of that theory, as well as Shannon‘s own incredible combination of intellectual breadth and depth.

The Dream Machine — A tour de force history of the early days of interactive computing. A window into a near-forgotten era, and of the work of J.C.R. Licklider, who did much to bring the dreams of that era into the reality of instantaneous, planetary-wide, communication and computing that we take almost completely for granted today. Highly recommended.

Economics

Information Rules — Though written at the turn of the century, this book‘s dated references do not distract from the underlying soundness of the theory and insight offered about the network economy.

Prediction Machines — A brief survey of the business implications of the deep learning revolution, with a look at what sort of things are ripe for disruption and how that might happen. Accessibly written, even for a non-technical reader, this book does a great job of illustrating what sort of applications we’re likely to see in the future and why.

Business History

Deal of the Century — A fascinating portrayal of the dissolution of the telephone monopoly in the early 80s. Memorable characters and a stirring story well told make this a surprisingly hard-to-put-down read.

The Box — Shipping containers. Seemingly so obvious and yet, as this book reveals, not.

Bad Blood — In the end, if the science doesn‘t work, the science doesn‘t work. This book illustrates just how long hype and spin and bluster and fraud can hold off a collision with reality.

Shoe Dog — Part autobiography, part business history by the founder of Nike, this book focuses on the early years. Heartfelt and honest; warmly written. Perhaps most surprising, to a modern reader, are the challenges of financing an aggressive growth strategy in a world before venture capital.

Leadership, Culture, and Self-Development

Make Time & Deep Work — This pair of books, best read together, share strategies and tactics on how to make time for deep work. In a world filled with shiny distractions (digital and otherwise), finding time to focus on what‘s important (as well as determining, what, in fact, is actually important) is itself of vital importance.

Thinking in Bets — Entertainingly written by a former champion poker player, this book applies lessons learned around the card table to life and business. Insightful and a fun read.

Good Strategy, Bad Strategy — I‘m surprised this book was not on any of the reading lists of strategy courses I took in business school. Perhaps the clearest articulation of how to formulate strategy, as distinct from sloganeering, a common doppelgänger thereof.

Powerful — From one of the authors of Netflix’s famous culture deck comes this amazing book. Honest, direct, filled with excellent, actionable guidance on how to create a company culture and what sort of company culture one should strive to create.

The Culture Map — A practical, insightful look at international business cultural differences, told with humor and heart. Useful to anyone who works with folks from anywhere else.

Politics

The Revolt of the Public — Originally self-published in 2014 and republished, in an updated form, this year by Stripe Press, this book, by a former CIA analyst, looks at technological change as the catalyst that has caused the public to revolt against leaders and provoke a crisis of authority. Weaving together such seemingly disparate events as the Arab Spring, Occupy Wall Street, Brexit, and Trump, the book makes a compelling case that all these movements, at their heart, are enabled by the distributed, unmediated communication enabled by the Internet. The author argues that these movements “from the outside” often lack a positive agenda but are instead able to unify various factions through a common desire to oppose the status quo, without agreement on what or how to replace it. Highly recommended.

After Europe — This short book presents the case that migration, particularly that precipitated by Libya and Syria, was the European establishment’s kryptonite. Europe’s ideals of human rights and equality put European leaders in an awkward position of either (in the eyes of their critics) ceding national sovereignty or heartlessly turning away from people in need. Europe’s own refugee laws were built for a different era and unequipped to meet the necessary scale. This disruption-by-migration has not yet been settled in Europe and it is unclear how it will be settled, as evidenced by the rise of a variety of populist movements, on both right and left, and no consensus on how to resolve inherent contradictions in the foundation of Europe. A thought-provoking read.

The Jungle Grows Back — This book argues that the postwar order as we know it did not just happen. It was created — deliberately — by the United States and the Allies in an attempt to prevent the horrors of the 20th century from ever recurring. The book briefly surveys the creation of the status quo before turning to look at the beginning of its dissolution, as America turns inward after ill-advised adventures abroad. The implications of what the world could look like, were the postwar order overturned, are harrowing. A dire, but well-written brief.

Philosophy

Skin in the Game — A delightful read, replete with wisdom. This book can be a bit curmudgeonly at times, but overall it’s a enjoyable discourse on the necessity of accounting for downside risks / disincentives — i.e. making sure not only that the people who make a decision can benefit if it goes well, but that they have some downside if goes poorly. And lots of illustrations on what happens if that isn’t the case.

Science

Behave — An incredible read, providing a deep look into human biology and neuroscience as we know it now, viewed through the lens of what makes us behave the way we do when we do. Insightful, humorous … deeply thought-provoking. Be sure to read the delightful footnotes.

Stumbling on Happiness — An entertaining survey of the state of practical cognitive psychology and how we, as humans, routinely mispredict what will actually make us happy, as well as deceive ourselves about ourselves. Recommended.

Year in Review 2018

Ben Overmyer's Site — Mon, 31 Dec 2018 02:00:00 GMT

As 2018 comes to a close, I'm looking back on the year and reflecting.

It was a big year for me. On January 1, I proposed to my fiancee, and she accepted. We got married in October. In February, for the first time since 2005, I left the country. My fiancee, a couple friends of hers, and I all went to Punta Cana for a 7-day vacation. In May, I quit my job working for Spok and took a consultant position with Solution Design Group.

With encouragement and support from Sarah, I paid off most of my credit cards over the course of 2018. I also paid off one of my loans. I saved up enough to pay cash for our honeymoon to Thailand next year.

For the first time ever, I bought a current events book. I haven't been able to read much of it, though, since I normally read at night before bed and this particular book is the opposite of relaxing.

I also spent a great deal of time programming generator tools for tabletop role-playing games. This year marked the emergence of my website for such tools, Iron Arachne.

Finally, I started to really put normal effort into Silver Gryphon Games again. I've been blogging there regularly and working on multiple new books for the company.

Books I Read in 2018

ntietz.com blog - technically a blog — Mon, 31 Dec 2018 02:00:00 GMT

Every year, GoodReads has a Reading Challenge, where you set how many books you want to read and record them as you go. This year, I got serious about it, and it was a wonderful motivational device. I set a goal of two books per month, and I just eked it out over the finish line, finishing my 24th book this morning.

Here are some of the best.

Rework and Remote

These are two modern classics by DHH and Jason Fried. These fit so well into my thoughts about what a workplace should be and the culture we should cultivate that they weren't mind blowing - rather, they were an incredible distillation of things I've wanted to say, in an incredibly clear manner. These two alone let me point to them and say: this is the kind of company I want to build.

Don't worry, the third in this series, It Doesn't Have To Be Crazy at Work, is one of my first books to read in 2019.

Salt, Fat, Acid, Heat

This book has changed the way I cook. It teaches you the fundamentals - mentioned in the title - and how to understand and apply them to any dish you are cooking. I was a good cook before this, but now I'm a vastly more capable home chef.

For anyone looking to step up their cooking game, to really understand what they are doing and break their reliance on recipes, this is a fundamental you deserve to have on your shelf.

Built: The Hidden Stories Behind our Structures

Have you ever wondered how subway tunnels are dug under rivers? How skyscrapers are built to withstand disasters? How structures stay standing for so many years? Well, this is the book for you. This is an incredible peek into what actually goes into creating and maintaining the structures we rely on every day to live in, drive through, and work from. Hands down one of my favorite books I've read all year.

The Three-Body Problem and The Dark Forest

Such an excellent series. It is shockingly well written, and the credit is due to both the author and the translator, who is himself an award-winning sci-fi author. This series is one of the best I've read in a while. It is both an interesting universe and a believable one, with good characters and interesting plot.

The Monk of Mokha

This is a fascinating true story behind a young man's attempts to bring Yemen's coffee to the American market. Whether or not you are interested in coffee, this is a fascinating story which shows you the human side of the production of this little brown bean.

And the rest...

The rest of the books were good, but you can only have so many favorites. These are presented in reverse chronological order of my reading:

A Hologram for the King: a fantastic novel by the great Dave Eggers.
Never Split the Difference: an interesting perspective on negotiation with many intensely interesting anecdotes. I'm not sure how much of this I really can apply.
The Checklist Manifesto: this book makes me want to go make checklists for everything. I want to apply what I've learned here to software engineering, but that's going to be a slow process.
High Output Management: this book isn't just for managers. He defines "middle manager" in an interesting way which includes many (if not all) knowledge workers and it is very transferable into my daily work. Even just knowing that "dual reporting" is a thing - and how to manage it - is very helpful.
Code Girls: let's just say, this is an incredible peek into the work of these incredible women who helped the Allies win WWII. This should be part of the history curriculum.
Nino and Me: a fascinating story of the friendship of two legal scholars. This isn't the first Garner book I bought (Black's Law Dictionary was), nor is it the last (Garner's Modern English Usage), but it is certainly the most page-turning.
Start Small, Stay Small: music to my ears. This is an excellent resource on how to build a bootstrapped company, and something I intend to revisit.
Coders at Work: interesting interviews with some of the legends of software engineering and computer science. You must take it with a handful of salt. The biggest thing I took away was that all these successful people work in incredibly different ways, so there really is no single best way of working.
Factfulness: imagine a TED talk in book form, and this is what you get. Pretty interesting and eye opening.
Skin in the Game: this book got me to think about things from a perspective I have really never had before. I loved being challenged to think so differently. I'll read more Taleb. I won't buy everything he says, but it's worth reading for the new perspective.
The Signal and the Noise: such a good read by Nate Silver. Highly recommended.
Hit Refresh: this book made me realize how much Microsoft had changed under Satya Nadella's leadership. To an extent it feels like (and is) a marketing piece for Microsoft, but it boosts my confidence that some of their acquisitions (like GitHub) will live on and keep being wonderful.
The Big Short: Inside the Doomsday Machine: a great look at what led to the housing bubble collapse and market crash of 2007-2008. Not really an uplifting topic, but a great read.
A Wrinkle In Time: this is a classic, and I'm glad to have finally read it.
Do What You Love and Other Lies About Success and Happiness: I love the title but did not enjoy the book. It was written in an incredibly dry, overly academic style.
Fire and Fury: Exactly what you'd expect from a book about the current administration written this early.
Ready Player One: this was a great look at how bad our future could be if we blindly lean into technology and corporatism. Let's not, okay?

2019 is going to be great, and I have a massive list of books I want to read (and a smaller list of ones I actually will read, as always).

Design is compromise

Steph Ango — Mon, 31 Dec 2018 02:00:00 GMT

When did the word “compromise” become vilified?

Compromise is neither good nor bad, it’s something we do every day. It’s decision making. Prioritizing. Deciding that one thing is more important than another. It’s finding the right balance between two competing desires.

Which compromises you make — that’s what matters. Choosing the right compromises is what defines good design.

Companies like to tout products as “uncompromising” or having “no compromises”. That’s impossible. Once you decide on an approach, you inherently decide against other options.

Another word for compromise is “tradeoff”. The word “tradeoff” conveys the relationship between strengths and weaknesses. You are trading a weakness for a strength.

Having an opinionated set of tradeoffs exposes your approach to a set of weaknesses. The more you tip the scale on one side, the weaker something else will be. That’s okay! Making those difficult choices is what people pay you for. You should be proud of your compromises.

My favorite products are opinionated. They make a clear statement about what they are not good at, in favor of being much better at something else.

Appealing to everyone is impossible. If you make something that aims to be good across a broad range of capabilities, you are choosing not to be exceptional at anything in particular. That might be the right compromise for your audience, but it’s definitely a compromise.

Good design is opinionated. Good design is choosing the right compromise for your audience.

ESP32-PICO-KIT-V4 Eagle Library

Greg Herlein — Sun, 30 Dec 2018 20:33:02 GMT

An Eagle part library to support use of the ESP32-PICO-KIT-V4 as a daughterboard.

Using txti.es + Python as a data dashboard

ciesie.com — Sun, 30 Dec 2018 02:00:00 GMT

Have you ever heard about txti.es? It’s a basic web page creator. You can basically create a very simple website by typing content into a form. I thought it might be used as a very simple frontend.

I always dabbled with automation. An example would be a scraping script that gets the job postings from a specific website or one that checks when the bus I take every morning leaves and sends this information by e-mail.

35c3ctf: Collection - an Unintended Solution!

David Buchanan's Blog — Sun, 30 Dec 2018 02:00:00 GMT

Designing an OSR Game Part 4: Alignment, Treasure, and Exploration

Ben Overmyer's Site — Sat, 29 Dec 2018 02:00:00 GMT

Over a month has passed since I last posted about Mysteries of a Broken World. Much has changed since the last time I wrote about the game here. The last three posts in this series are below:

The rules are much more solid now. As I mentioned before, you can view the current state of them here:

Mysteries of a Broken World Playtest Rules

This time around, I'll talk about some changes that have happened as a result of actually digging into the rules.

Alignment

Previously, I somehow completely forgot about alignment. I didn't write about it in blogs or even think about it in notes. Then I started to write monster entries. Since I was using the D&D Rules Cyclopedia as a starting reference point, I hit "Alignment" under the monster statistics and it dawned on me.

So alignment got added, but it's a little different from other role-playing games. Instead of reflecting your character's outlook on life, it reflects primal outside forces' influence on your character. There are four alignments in the game. They are Chaos, Destiny, Void, and Equilibrium. Chaos is not the anarchic evil of old D&D. It's impulsive and mercurial, sure, but not evil. Destiny is closest to old Lawful, but again, it's not "good" like in the old game. Void emphasizes purity and clarity. Equilibrium enforces a balance between all opposites.

Refining Treasure

Treasure in Mysteries of a Broken World is the focus for gaining experience. Monsters have no experience reward, whether for killing or avoiding them. Most monsters, however, will have treasure either on their person or in their nearby lair. Unlike Dungeons & Dragons, treasure is not listed for monsters in abstract classes. Monster entries have a line for "Treasure Type," with values such as "Meager," "All That Glitters," and "Warmonger." Treasure Types still have a table of what they contain, but this named system tries to give Game Masters a better idea at a glance of what the monsters in a given module might possess.

Beginning to Rethink Exploration

Now that the core rules are more or less codified, I'm starting to think about where I want to take exploration. Some games make it a core feature, and others try to sweep it under the rug in favor of more visceral action. In Mysteries, I want exploration to be a primary element of the game. This requires developing rules that encourage focus on exploration.

I'm not quite sure how I'm going to do this yet. Exploration has to be interesting, which means letting players make choices. "We travel by horseback northwest for three days" is pretty boring. Turning that same journey into a game-worthy experience will require thought.

Weeding out misleading misnomers

D13V — Fri, 28 Dec 2018 18:09:35 GMT

When I was a student in MIT, I took a walk on Massachusetts Avenue, across the Harvard bridge. It’s a long bridge which separates Boston from Cambridge, Massachusetts. As I crossed into Boston and I walked about a mile further, I noticed the Christian Science Plaza. It was beautiful. A large ornate church next to reflection pool. A little pocket of spaciousness in crowded Boston. It looked splendid.

I didn’t like getting inside churches, and thought of myself as an atheist. I fit well in Boston - a sciency city hosting MIT, Harvard, BU, Tufts and other top universities. In my ignorance, I assumed that the Christian Science Plaza is some kind of tribute of sort to all these scientists who were also believing and professing Christians. How naive of me.

Only years later, after reading a random thread on the internets, I learned that Christian Science is something entirely different than what I thought it was. Apparently, it is a sect that preaches alternative medicine and quackery. It’s followers believe that if you’re sick, the best way to heal is not to go to the doctor, but to pray.

This is far… far from science, and at the same time far from mainstream Christianity, as I understood it. Yet, for a while Christian Science flew under my bullshit radar, masked by its name.

And Christian science is not the only low flying crap. There are others. Another example is the phrase “Conscious Capitalism”. On the surface, it seems like it’s fixing some of the problems of capitalism. Corporations are often accused, and rightly so, of externalizing the cost, by dumping their negative byproducts on groups of people who don’t have choice, or by destroying the ecosystems of the planet.

Conscious capitalism pretends that it’s a different way to do business, that helps the planet, and aligns incentives. And while to some degree this might be true, it’s a classic marketing technique. These businesses, touted at the forefront of conscious capitalism make their profits not by necessarily finding the better and cheaper and sustainable products and delivering them at a competitive price.

Most of conscious capitalism rely on virtue signaling - whether it is social status, or for intrinsic motivation. People want to feel good about where they spend their money - even if that means spending a lot more of the money. That’s the real business model.

What conscious capitalism really sells vanity, and luxury.

You might say I’m cynical, yet, if you see the bottom line, realize that conscious capitalism only works for selling high end items to affluent customers who want to feel better about their impact on the planet. The really funny, and rather ironic thing is that this kind of decision is often made at the subconscious level. A fully conscious consumer knows that most of the time a more expensive product is harder on the environment. And then they realize they don’t need to spend 12.99 for a small bag of dark chocolate almonds, and that the waste created by it more than negates the good created by getting their organic avocados cheap.

A fully conscious customer would be frank to themselves that they are buying the organic eggplant and the heirloom tomatoes because they taste better. Not because they are more sustainable. When you draw the bottom line, conscious capitalism is really a subliminal luxury.

Phrases like “conscious capitalism” and “christian science” - designed to mislead us, are much more common than we imagine. They can be extra vicious, because they are not a contradictory oxymorons. It is possible to be both a Christian and a scientist, and to approach science with religiousness and theology scientifically. And it is possible to be a conscious capitalist - see the blog of Mr Money Mustache (https://www.mrmoneymustache.com/)

Such phrases are double misnomers. They twist our brains away from the true meaning. We approximate meaning of a phrase by each word’s individual meaning. We are hard wired for associations. Logic is secondary, and we regularly ignore it. And when things are different than what their names imply - that’s doublespeak, and forces us on actively engaging our slower, logical machinery.

We have to exert mental discipline, and a lot of effort, only to avoid getting tricked.

Such doublespeaks are Orwelianisms - forcing us to keep two different interpretations in our heads. One for what things sound like, and another for what they really are. When we are dealing with double misnomers, the number of interpretations becomes four. The more misnomers we have, the number of versions we have to consider grows exponentially. Meanwhile, our brains have limited power. Like… super limited. Our working memory, our RAM, can only keep 5-7 things at the same time, so having even four misnomers can really mess us up.

For double speak to sneak in, we don’t need a double misnomer. Even a single misnomer is enough to force us into doublespeak, and doublethink.

Misleading misnomers are everywhere. All around us, they trick us in all kinds of ways. They are the basis of all misinformation - from deceptive persuasion, to fake news, to propaganda. By weeding them out of our lives, and staying vigilant, we learn to recognize the bigger threats and avoid them.

On Migrating from Google Analytics

Thomas Hunter II — Fri, 28 Dec 2018 02:00:00 GMT

CS1 - What Do We Know?

Acbart ePortfolio — Fri, 28 Dec 2018 02:00:00 GMT

I made a better follow-up to this post based on an awesome journal paper that came out around the same time!

Finally received my 2018 hacktoberfest swag

Shyam Jos — Fri, 28 Dec 2018 02:00:00 GMT

Yay! finally received my 2018 hacktoberfest swag

ASCII art generator bookmarks

xenodium.com @alvaro — Thu, 27 Dec 2018 02:00:00 GMT

𝓔𝓻𝓰𝓮𝓻𝓪𝓽𝓸𝓻/Ergerator (ascii generator).

Malas prácticas de seguridad en bancos: Banco Santander

Daniel Sada Caraveo | Developer Productivity & Culture — Thu, 27 Dec 2018 02:00:00 GMT

No suelo quejarme mucho, sin embargo, varias cosas me han irritado en cómo los bancos que proclaman ser tan conocedores en ciberseguridad tienden a ser los que más caen en falsedades que solo aparentan seguridad. Mas aún, sus malas prácticas suelen llegar al punto que, en lugar de hacer los sistemas más seguros, los hacen más inseguros. No me malinterpreten, – Santander es de los mejores bancos con los que he tratado y la mayoría de estas críticas no son exclusivas de Santander – Sin embargo, del querer ver algo mejor llega la crítica.

Devbot

Anardil — Wed, 26 Dec 2018 10:00:00 GMT

Automate your workflow with Devbot

Osaka travel bookmarks

xenodium.com @alvaro — Wed, 26 Dec 2018 02:00:00 GMT

Tsutenkaku (Osaka) - 2018 All You Need to Know BEFORE You Go (with Photos) - TripAdvisor.

Turning the Static Dynamic with Gatsby + Netlify Functions + Netlify Identity

swyx's site RSS Feed — Wed, 26 Dec 2018 02:00:00 GMT

Gatsby is great for not only static sites but also traditional web applications. You can add authentication and serverless functionality and get up and running incredibly quickly with Netlify - here's how.

Leather Keychains

Zef Houssney — Tue, 25 Dec 2018 21:00:00 GMT

Leather Keychains with brass hardware made by Zef Houssney.

The Absolute Essentials for Bit Manipulation in JavaScript

Lucas F. Costa - Blog — Tue, 25 Dec 2018 14:00:00 GMT

This post aims to demystify bitwise operations in JavaScript, as they can be useful for a myriad of applications. They are also very useful, for example, when building emulators, an exercise I'd defi...

Using OCR to create searchable pdfs from images

xenodium.com @alvaro — Tue, 25 Dec 2018 02:00:00 GMT

Used my phone to take a handful of photos of an article from a magazine. Wanted to convert the images to a searchable pdf on macOS.

This was straightforward, having already installed tesseract.

for i in IMG_3*.jpg; do echo $i; tesseract $i $(basename $i .tif) pdf; done

Should now have a handful of OCR'd pdfs:

ls *.jpg.pdf

IMG_3104.jpg.pdf
IMG_3105.jpg.pdf
IMG_3106.jpg.pdf
IMG_3107.jpg.pdf

Finally, join all pdfs into one. Turns out macOS has a handy python script already installed. We can use it as:

/usr/bin/python "/System/Library/Automator/Combine PDF Pages.action/Contents/Resources/join.py" -o joined.pdf IMG_*pdf

ps. pdfgrep is great for searching pdfs.

Useful references

Ryan Baumann's: Command-Line OCR with Tesseract on Mac OS X.

Audiobook providers bookmarks

xenodium.com @alvaro — Tue, 25 Dec 2018 02:00:00 GMT

Cookbook bookmarks

xenodium.com @alvaro — Tue, 25 Dec 2018 02:00:00 GMT

Emailing pdfs to kindle from mu4e

xenodium.com @alvaro — Tue, 25 Dec 2018 02:00:00 GMT

Wanted to send a pdf to my kindle for some holiday reading. You can easily do this by emailing the pdf to your kindle-bound email address.

Now, I typically attach files when composing mu4e emails by using mml-attach-file, which attaches the file using <#part>…<#/part>. However, the Amazon service did not find the attached pdf, so no pdf was added to my Kindle.

Fortunately, I found a handy Reddit thread, leding me to a working solution. Wrapping the part using <#multipart type=mixed>…<#/multipart> did the job, using mml-insert-multipart, followed by mml-attach-file.

Resulting attachment should look something like:

<#multipart type=mixed>
<#part type="application/pdf" filename="/path/to/file.pdf" disposition=attachment>
<#/part>
<#/multipart>

I should add a convenience elisp function for this, but that's for another time…

Links - December 25, 2018

Avy Faingezicht — Tue, 25 Dec 2018 02:00:00 GMT

My 2018 in review

andrestc.com — Mon, 24 Dec 2018 22:49:09 GMT

As 2018 is coming to and end, I have decided to review my year.

Talks

One of my major goals for 2018 was to focus and improve my public speaking skills. Specifically, I wanted to do my first talk at an international conference.

I created a spreadsheet with a bunch of events that were interesting to me, such as Gophercon US, Gophercon UK, KubeCons and so on. I organized them by CFP dates and tried to come up with a few topics for each of them.

Uber Driver Teaches me Data Analytics

Greg Herlein — Mon, 24 Dec 2018 21:33:02 GMT

An Uber Driver this past week schooled me that data analytics does not require coding.

Nethack Deaths

infrequent oscillations — Sun, 23 Dec 2018 11:48:44 GMT

In which we clean up a messy data set to make more sense of it.

Elf Assembly JIT

Matt Keeter — Sun, 23 Dec 2018 02:00:00 GMT

A Very Merry Compiler

How to Run a Successful Tech Meetup — even if you’re forgetful

Sun, 23 Dec 2018 00:12:03 GMT

A picture taken from the November 2018 LeicesterJS meetup

LeicesterJS is born out of the rise of Javascript now being the de facto programming language for a majority of developers. Additionally, we aimed to bring together a tech community in Leicester.

I have been running LeicesterJS now for over 4 months and this is just the start. Before the first meetup, I was very nervous about getting everything together and although it went off without any dramatic hitches, I have learned a great deal and continue to do so with each and every event. Here is some of the advice I wish I had before planning my first meetup.

Start with the essentials

The barebones requirements from a meetup is a venue and catering. For LeicesterJS, I was able to partner with my current workplace to host the event and foot the bill for the food and drinks provided. In exchange, we give them a plug at the start of each event and they receive exposure to potential hires. Often meetups will be arranged in this manner as it makes organizing the event a lot easier, since you, the organizer, are at the venue ready to prepare anything that is required. If you are looking to arrange your own meetup, kill two birds with one stone and look to see if your employer wants to help.

Next up is the speakers, for this, I turned to developers in my team and asked around if anyone wanted to give a talk. You can also reach out on twitter or to other developers you know in the area. Worst comes to the worst, do the talk yourself!

And with that, we had a meetup! Except for one key thing…

….People

You want your meetup to be a success and part of that success can be measured by the attendance rate. Here is how I approached building a community of people from the ground up

Get support from co-workers — since they know you, it will be easier to break the ice that having a room of strangers
Link up with other meetups in the area via Twitter — I was very fortunate to have the generous support of PHP East Midlands who cancelled their own meetup to join in with the first LeicesterJS (thanks again guys!). It was great to have them along and again, a few more familiar faces
Connect with a tech Slack group — in my case, TechNottingham has a thriving Slack community, I advertised the meetup in their #javascript channel
Personal social media — in addition to using branded “LeicesterJS” twitter account. I further advertised the meetup with my own personal Twitter and LinkedIn. Additionally, you can encourage your co-workers to put the word out too
Word of mouth — you’d be surprised how many people just come to events because they saw it on meetup.com. From my purely anecdotal data gathering, it seems that people mostly came because they saw the event on meetup.
RSVP’s matter — A day before the meetup starts, it is good to encourage people to update their RSVP status if they are no longer planning on attending. Often times, you will only get 60–70% of the RSVP’s you have on meetup. Make sure you keep track of this KPI and encourage the “regulars” to your meetup

Download the full thing here: https://gist.github.com/joshghent/2bb9c6e2ce616e29aaa3c7a2895cb17d

Processes

After the first meetup, I had a number of people come and ask me for the slides to the talks that had been given. I hadn’t even thought about that! There were a number of these sorts of things that I had to get a process nailed down for.

Talk slides — stealing an idea from NottsJS, I decided to create a Github organization and put the slides in a repo. It was the least frictionless way to distribute them. I considered using Google drive or some other file sharing service but this would give me difficult URLs that audience members may have a hard time finding
Talk ideas — after having a number of inquirers about giving talks I again turned to GitHub. The process is now to submit a talk idea as a Github issue on a repo. I have a set template for the issue so there is not any information left out that I may need at the last minute
Somewhere to talk — despite the bounty of slack groups out there, many requested to set up a Slack group to discuss the meetup and the tech community in Leicester at large. This ticks along nicely. I used Slackin on a free Heroku box to create a site for people to get instant invites
Introductions — the first meet had some very basic introduction slides but I’ve gradually fleshed these out to make sure everyone feels at home. Basic things like pointing out the bathrooms and telling people what time the food will be here and the meetup is expected to end can be very useful to people. Another additional tip is I get people to raise their hand if they with me (and therefore, at the office, we were in), I informed other audience members that they can also go to those ones if they need any help

Other Pointers

After getting through the above, it can seem as if there is nothing more to do — just grow the meetup right? I fell into the same trap too. But a further refinement came first by introducing a code of conduct. In this document, it outlines how not only speakers, but all those in attendance, should behave and having clear processes for people who do not. It’s worth grabbing a document that has already been written because in all likelihood you are not a lawyer (but if you are, fill your boots!). Additionally, it’s good to have a point of contact (an email address and a person named) who should be contacted if someone wants to report something. Although you hope that something like that will never happen, there has been far too many cases in the tech industry alone that have highlighted that problems with intolerance or any sort of bad behaviour towards others is unfortunately not uncommon — often it goes unreported, I hope that never happens at LeicesterJS. Nonetheless, it’s essential to have.

Additionally, you can offer yourself as a resource if any potential speakers need help preparing a talk. Since its beginning, I’ve encouraged first-time speakers to give talks. This can be a nerve-racking experience so give generously to offer support with slides, talk topics etc. The meetup group slack channel can be a great place to do this.

A lesson I learned early on was to encourage audience participation. It makes the meetup more active and motivates people to discuss afterward as well. Perhaps recommend that your speakers open the floor so to speak with a discussion. Asking if anyone uses the technology being discussed, or ask them if they use an alternative.

Starting your own tech meetup is an incredibly rewarding pursuit, but don’t underestimate its work. It’s also worth noting that, if you are in a smaller city as I am, it is better to support any existing meetups than to create your own. Ask the organizers if you can assist in any way, perhaps arranging new speakers or handling promotion of the event, they would welcome anyone to lend a hand.

If you are in the Leicester area, come along to LeicesterJS — the meetup is every 3rd Thursday of the month!

Uber, statistics, and a chrome extension

JonLuca's Blog — Sat, 22 Dec 2018 17:39:48 GMT

ka-yich — Sat, 22 Dec 2018 12:40:22 GMT

2018: A year in review

Daniel Immke — Sat, 22 Dec 2018 02:00:00 GMT

It's the end of the year, time for some reflection on how things went!

A functional IDE for React

ferrucc.io — Fri, 21 Dec 2018 21:43:10 GMT

6 years ago this video came out.

At the time I couldn’t code yet.

This concept is still mind blowing to me.

Up until now I always thought about code as a collection of files, connected through imports.

But really we import functions, not files.
We use files to store code because they’re a convenient way to serialize code. They’re not the best representation, in fact functions might provide a better one.

A guide to fighting with a Cisco access point

The Technical Blog of James on purpleidea.com — Fri, 21 Dec 2018 14:45:00 GMT

In my hands is a Cisco AIR-CAP-3502-E-A-K9 that I saved from the landfill. After about eight hours of suffering, and what was surely a cancer causing dose of RF-radiation, I now know:

How to convince this access point to provide me with wifi.
Why Cisco products are probably best left in the garbage.

What follows is my setup guide and review for using a Cisco Aironet device. This guide will likely be useful with other models of Cisco devices, but use your own best judgment.

org tip: convert csv to table

xenodium.com @alvaro — Fri, 21 Dec 2018 02:00:00 GMT

Needed to import some csv data to an org table. Turns out org's got you covered out of the box with M-x org-table-create-or-convert-from-region bound to C-c |.

Mastering Monero first edition has been released

SerHack - Security Researcher — Fri, 21 Dec 2018 02:00:00 GMT

‘Mastering Monero: The future of private transactions’ is your guide through the world of Monero, a leading cryptocurrency with a focus on private and censorship-resistant transactions. This book contains everything you need to know to start using Monero in your business or day-to-day life, even if you’ve never understood or interacted with cryptocurrencies before. -Continue to read the article on SerHack.me

Python AST Element Analysis

Acbart ePortfolio — Fri, 21 Dec 2018 02:00:00 GMT

I decided to clean up some analysis I did a while back about Python Abstract Syntax Trees. The thought was to download a mass of Python code and look at how Python programmers use various syntactic elements. I had thoughts about turning this into a paper, but I figured a blog post might be all I had the investment for. This represents a pretty shallow effort, in the end. But I merged all the code into a Jupyter Notebook, added some notes, and now you can check it out on GitHub: Python AST Element Analysis

Small Computing History Resources

Mike Schaeffer — Fri, 21 Dec 2018 02:00:00 GMT

I've lately run across several interesting small computer history sites. If you have any interest in small computing's emergence from 1980 to 1990 or so, these are worth a look.

In no particular order:

OS/2 Museum - Covers OS/2, but also gets into detail around PC architecture. Among other interesting bits, this is just one of several articles on A20 gate handling, and here's something on the IBM 8514/A.
DTACK Grounded - A newsletter written to promote Hal Hardbergh's side business of attached Motorola 68000 processor boards. Mostly interesting for his commentary on then-crurent events leading up to the emergence and use of 32-bit microprocessors. Notably, this was written at the time of Intel's pivot from the iAPX 432 to the 80386. The commentary on the relative unreliability of DRAM is amusing too.
CRPG Addict - Not sure how he has the time, but the author of this blog has set himself the challenge of playing through and documenting every early CRPG game from the late 70's and well into the 90's.
The Digital Antiquarian - Critical commentary on early small computer gaming. Lots of details about how games came to be made and their content.
Retrocomputing Stack Exchange site - This is currently more like Netflix than anything else. Coverage is spotty, but that doesn't mean you can't find something interesting to read.

2018 in review

Jason Eckert's Website and Blog — Fri, 21 Dec 2018 02:00:00 GMT

At the end of each year, I take some time to reflect on the previous year - it provides perspective, and is a lot of fun!

For me, 2018 was the year I spent a lot of time improving myself physically, challenging myself mentally, and most importantly, enjoying life!

On the personal side, I cut out the bad foods from my diet:

breads
starches
alcohol

On top of this, I upped my indoor and outdoor exercise to lose a whopping 85 pounds since mid 2017 - I feel great, have loads of energy, and no longer have any back pain whatsoever!

Running latest kernels on ARM Scaleway servers

Linux Engineer's random thoughts — Fri, 21 Dec 2018 01:00:00 GMT

Scaleway, a french cloud provider, has been renting baremetal ARM servers for a few years now, and virtual ARM64 servers more recently. They ship with a scaleway-provided kernel and initird, which isn't updated as often as I'd like. The latest ARMv7 (32 bits) kernel, is 4.9.93, while the …

The Editor With No Name - an informal case study into what went wrong with Gutenberg

Sam says you should read this — Thu, 20 Dec 2018 17:42:15 GMT

So WordPress 5.0 has been released. It happened just in time for WordCamp US, conveniently enough, and now "Gutenberg" is just referred to as "The Editor". Of course the "Classic Editor" (which was actually TinyMCE all along) has been given a more recognisable name reminiscent of "Classic Coke", but stock WordPress 5.0 includes the editor formerly known as "Gutenberg" by default.

This has made a lot of people very angry and been widely regarded as a bad move.

But why is the community so angry? Essentially this comes down to one thing - Gutenberg is a prime example of how not to bring a new prod…

Sponsoring platform bookmarks

xenodium.com @alvaro — Thu, 20 Dec 2018 02:00:00 GMT

Artistic/creative bookmarks

xenodium.com @alvaro — Thu, 20 Dec 2018 02:00:00 GMT

Marketing bookmarks

xenodium.com @alvaro — Thu, 20 Dec 2018 02:00:00 GMT

Mastering Monero

SerHack - Security Researcher — Thu, 20 Dec 2018 02:00:00 GMT

Book overview Across these eight chapters, “Mastering Monero” can take a reader from a complete crypto novice to an informed crypto citizen that knows how to protect his or her privacy. This book takes you by the hand on an exploratory adventure, showing you the world of blockchains and crypto, before taking you on a private tour of the Monero protocol and systems. Illustrations throughout the chapters make it easy for readers to understand the content visually. -Continue to read the article on SerHack.me

Review of the iClever Foldable Bluetooth Keyboard

Thomas Hunter II — Thu, 20 Dec 2018 02:00:00 GMT

New Zealand Otago Trip in a 2019 Mustang GT

smcleod.net — Thu, 20 Dec 2018 02:00:00 GMT

In late November of 2018 I was lucky enough to get my hands on the upcoming 2019 Sport spec Mustang GT v8 to be its first driver.

I’m not usually an American Muscle enthusiast, but the new 2019 GT really is quite different from its slightly older counterparts, with a new specially developed 10 speed gearbox (yes, 10 gears - because more is better right?), tuned up suspension and so seemed like it would make for a pretty fun trip.

Handling optional arguments in bash

Pete's Points — Thu, 20 Dec 2018 00:31:37 GMT

I learned a nice way of using bash arrays to hold command line arguments that are conditional or will be computed before passing as arguments to a separate program.

There's 2 basic parts.

Use a bash array to hold them

declare -a opts=(one two three)

Expand them properly using this syntax

"${opts[@]}"

Here's an annotated example:

#!/usr/bin/env bash

# Here's our default arguments we intend
# to pass to curl
declare -a opts=(--silent --fail --output /dev/null)

# Use this for debugging
# opts=(--fail)

if [[ -n "${REMOTE_USER}" ]]; then
# Here we need to change the args, so
# we can prepend some to the variable itself.
  opts=(--user "${REMOTE_USER}:${REMOTE_PASSWORD}" "${opts[@]}")
fi
url="${REMOTE_URL}"

# Now pass the on with proper quoting
# using the [@] syntax

curl "${opts[@]}" "${url}"

Note I omitted the unofficial bash strict mode boilerplate for clarity, but in my production scripts, you'd see that at the beginning.

ka-yich — Wed, 19 Dec 2018 11:56:38 GMT

AWS Certified

Greg Herlein — Wed, 19 Dec 2018 09:50:38 GMT

On Becoming an AWS Certified Solutions Architect (Associate)

Sapiens: A Brief History of Humankind

Apurva Shukla's RSS Feed — Wed, 19 Dec 2018 07:37:27 GMT

⭐ ⭐ ⭐ ⭐ ⭐ This is an expertly crafted piece of work - detailing the formidable task of recounting and analysing all of human history. With…

Bluetooth low energy (BLE) bookmarks

xenodium.com @alvaro — Wed, 19 Dec 2018 02:00:00 GMT

Animated Multiline Link Underlines with CSS

Danny Guo — Wed, 19 Dec 2018 02:00:00 GMT

Avalanche! re:Invent 2018

Greg Herlein — Tue, 18 Dec 2018 20:50:01 GMT

Thoughts on AWS re:Invent 2018

Perfect Advent Calendars

infrequent oscillations — Tue, 18 Dec 2018 07:56:32 GMT

In which we make things symmetrical by brute force.

Fun project bookmarks

xenodium.com @alvaro — Tue, 18 Dec 2018 02:00:00 GMT

echo yang programs everyday obsolete machines to create autonomous art (designboom).

A Mind at Play: How Claude Shannon Invented the Information Age

🦉 olshansky 🦁 — Tue, 18 Dec 2018 02:00:00 GMT

Olshansky's review of A Mind at Play: How Claude Shannon Invented the Information Age by Jimmy Soni

Interesting queries on a football database

infrequent oscillations — Mon, 17 Dec 2018 15:53:00 GMT

In which we query a football database for facts and fun.

Here’s to you

Justin Garrison's Homepage — Mon, 17 Dec 2018 02:00:00 GMT

Here’s to the people I can relate to. The people that I don’t know, but I understand. For those who feel like you don't belong.

A Quest To Find Perfect Editor

Nikhil Soni — Sun, 16 Dec 2018 02:00:00 GMT

I used to use Atom, it’s MIT licensed, got tons of extensions. I was happy, life was good. Then Microsoft baught Github. Soon enough, people started talking about future of Atom. Since they already have something similar called VSCode, I too started worrying about their long term commitment to Atom. Few of my friends use VSCode and I’d heard and seen some very good things about it. Now that both were owned by Microsoft, there was no reason to stick with Atom and not give VSCode a try. So I did. It turned out great. Extension support and community is so great that I installed extension for almost all languages I used to write in that time. I started using it as my main editor and kept searching for something that isn’t owned by MS. I wanted something I can trust to take it to my grave. Options were very clear – Vim or Emacs?

I had tried out these editors before and they require time and effort to get used to, which I had never invested. But, it was never Vim vs Emacs for me. Full disclosure, during my junior year in college, I got a chance to attened a talk by Richard Stallman and I was already influenced by FSF to some extent. After that talk, I tried quiting Facebook (which I eventually did), reading about life without Google and using Emacs as default editor. None of them actually worked out that time.

So now that I was looking for an editor, I reconsidered Emacs, until I accidently pressed some keys (C-x m) and a mail client like buffer opened up. I was already overwhelmed by it and things like these got me off. I just wanted an editor where I can code and not a mail client (it isn’t). That day, I started setting up Vim.

Soon enough, I got used to selecting and deleting words with ved and moving around line like 0 and $ and not being able to use these keys when writing mails or anything that involves typing or editing a little more than a word or two, turned out be an irritating point. That mail client looking like Emacs buffer started to make sense now. An editor that once you get used to will be there for you, whenever or wherever you need to type a word. Need a REST client, want to search through a command’s output or may be just want to write some plain English, it got you covered. Sure, Vim might also provide those things, but I moved back to Emacs.

If you want to move to Emacs, here is my take on setup to smoothly transition from your current editor.

– Nikhil

Everything I've learned in 10 years of Blogging

ferrucc.io — Sat, 15 Dec 2018 21:57:56 GMT

I’m 21, but I’ve been blogging for almost 10 years.

I grew up doing this.

What I’ve written on the internet has reached millions of people. Most of what I’ve written is in Italian, but I was also quite successful when writing in english. My Quora profile, reached 400k people in three months.

Through blogging I had the opportunity to travel the world, getting free tickets to all the biggest music festivals (Ultra, Tomorrowland.. you name it), getting promos from the Majors and interviewing my favorite artists.

Easy Moving From Vscode To Emacs

Nikhil Soni — Sat, 15 Dec 2018 02:00:00 GMT

If you have already decided to go with Emacs as your choice of editor, congratulations! This is going to be great. However, the part after this is what’s hard and takes time. You might already be missing some of the very basic functionalities like opening a project file without typing the full path or searching something project-wide that you are very accustomed to. These features usually comes out of the box or by installing some extensions that do not require any configuration, with most of graphics based editors like Atom, VS Code or Sublime. If that is the case, keep reading. When I first started using Emacs, I spent quite some time reading and trying out stuffs to make these things work and this article is exactly because of that – to make your transition smoother. The focus is NOT on being more productive with Emacs or to give you tricks and fancy lisp code that will make you Emacs ninja overnight (they exists btw). That is something you can focus on once you’re comfortable using Emacs. This is my attempt to make every tyro feel home, to give you a setup that will allow you to do all the stuff that you’re used to.

Emacs is built around the philosophy of extensibility and flexibility – being able to hook your own code at any event of your choice is what makes it awesome. And because of that, the huge community around the editor have highly opinionated setups based on what works for them. There exists modified versions of Emacs like Prelude, Spacemacs, etc. that are loaded with a lot of functionalities you wouldn’t find anywhere else but in my personal opinion, it’s good to know what is cake and what are the cherries.

Before we start, familiarize yourself with basic key combinations and terminology of Emacs. Built-in tutorial explains that very neatly which you can read by C-h t (hold control key and press h, then leave control key and press t). This awesome article from last week’s HN front page is also a great read for this.

Let’s start. After moving from Atom and VSCode to Emacs, I missed these functionalities:

Being able to duplicate current line
Move or drag a line up and down
Use multiple cursors to edit at a time
Auto-completion of variables and functions name
Fuzzy file search i.e. Ctrl-P or Cmd-Shift-o in other editors
Searching a term of string in the whole project
A tree view of the project to explore file structure

Few more things that I had installed extensions for in VSCode:
Goto functions and class definitions
Come back to previous cursor position
Find all the reference of a function
Markdown preview

Surprisingly, some of these are as easy as hooking a couple of lines of code to a keystroke, for some a little more than a couple of lines so we’ll use community developed packages for them. The latest version of Emacs ships with a package manager – package.el. We just need to add MELPA repository to it so it can search packages for us. Add this Emacs Lisp (ref) to your ~/.emacs.d/init.el file to do that.

(require 'package)
(setq package-archives
      '(("GNU ELPA"     . "https://elpa.gnu.org/packages/")
        ("MELPA Stable" . "https://stable.melpa.org/packages/")
        ("MELPA"        . "https://melpa.org/packages/"))
      package-archive-priorities
      '(("GNU ELPA"     . 10)
        ("MELPA Stable" . 5)
        ("MELPA"        . 0)))
(package-initialize)

This will also give priorities to repositories to avoid duplicate listing in case of certain packages being present in more than one. After adding this restart Emacs to reload init.el and do this: M-x package-refresh-contents. Now you are ready to install most of the packages. Note that you can also reload you init.el by doing M-x load-file and then providing file path.

Rest of the article is about enabling above mentioned functionalities one by one.

Add this code to same init.el

 (defun duplicate-line ()
   (interactive)
   (save-mark-and-excursion
     (beginning-of-line)
     (insert (thing-at-point 'line t))))

 (global-set-key (kbd "C-S-d") 'duplicate-line)

Now, you can use C-S-d (Control-Shift-d) to duplicate current line. Change these key combination to whatever suits you.

Add these lines for using C-S-j and C-S-k to move a line up or down one line.

 (defun move-line-down ()
   (interactive)
   (let ((col (current-column)))
     (save-excursion
       (forward-line)
       (transpose-lines 1))
     (forward-line)
     (move-to-column col)))

 (defun move-line-up ()
   (interactive)
   (let ((col (current-column)))
     (save-excursion
       (forward-line)
       (transpose-lines -1))
     (forward-line -1)
     (move-to-column col)))

 (global-set-key (kbd "C-S-j") 'move-line-down)
 (global-set-key (kbd "C-S-k") 'move-line-up)

To get mutiple cursor, I use multiple-cursor package, which can be installed by M-x package-install <RET> multiple-cursor <RET>. Add these key bindings to easily use it.

 (require 'multiple-cursors)
 (global-set-key (kbd "C-|") 'mc/edit-lines)
 (global-set-key (kbd "C->") 'mc/mark-next-like-this)
 (global-set-key (kbd "C-<") 'mc/mark-previous-like-this)
 (global-set-key (kbd "C-c C-<") 'mc/mark-all-like-this)
 (global-set-key (kbd "C-S-<mouse-1>") 'mc/add-cursor-on-click)
 (define-key mc/keymap (kbd "<return>") nil)

These commands do exactly what they look like. To get out of multiple cursors, use C-g, the last line in above code prevents using <RETURN> key to do same. You can read more about it in the project doc page.

To let Emacs auto-complete function names and variables, I use company, which can be installed by M-x package-install <RET> company-mode <RET>. And can be activated by adding this line to init.el:
```
 (add-hook 'after-init-hook 'global-company-mode)
```
I use the combination of projectile and helm package to deal with fuzzy file search. Configurations for helm can be a bit overwhelming so just start with following.
```
 (require 'projectile)
 (setq projectile-indexing-method 'alien)
 (setq projectile-enable-caching t)
 (projectile-global-mode)

 (require 'helm)
 (require 'helm-config)
 (global-set-key (kbd "C-c h") 'helm-command-prefix)
 (global-unset-key (kbd "C-x c"))
 (helm-autoresize-mode 1)
 (global-set-key (kbd "M-x") 'helm-M-x)
 (setq helm-M-x-fuzzy-match t)
 (global-set-key (kbd "C-x C-f") 'helm-find-files)
 (helm-mode 1)
```
This will allow you to use C-c p f for opening your project files just by typing file name and not the whole path. And when you use C-x C-f to open a file that is outside the project directory, you can type any part of the directory or file name or even non contiguous parts separated by space to narrow down the suggestions e.g. if you want to select controllers out of controllers and contol folder name, you can type co s. Use C-j (not TAB) to choose highlighted option.
projectile can also be used to search for something in the whole project: C-c p s s. This command used ag system package, which should definitely have and a ag.el Emacs package. I don’t generally use project wild search and replace but if that’s your thing, this answer from Stackexchange explains a great way of doing so.
Built-in speedbar provides good enough interactive tree view of project. It can be started with M-x speedbar. It’s easier to bind it to some key like F8 using
```
 (global-set-key (kbd "<f8>") 'speedbar)
```
I use dumb-jump package to jump to definitions using C-M-g. Add this line to auto enable it every time:
```
 (dumb-jump-mode)
```
You can jump back to function call by C-M-p.
To find all the references of a function, priviously mentioned project-wide search with projectile works well for me.
I use fymd package to real time markdown preview, with this key binding:
```
(global-set-key (kbd "<f9>") 'flymd-flyit)
```

Most of the snippets on this page are taken from emacsrocks.com, whattheemacsd.com and official documentation of mentioned packages. Both of these resources are great places to learn more about Emacs. Here is my .emacs.d directory, if don’t feel like doing this all by yourself, feel free to clone mine. I have all the defuns defined in ~/.emacs.d/user-lisp/defuns.el file which is imported in init.el by:

(add-to-list 'load-path "~/.emacs.d/user-lisp")
(require 'defuns)

I keep all the references to these defuns in init.el so all the key bindings remain in one file.

I would highly appreciate any feedback you may have or to listen about any other features you miss from your old editor and should be in this list.

[Edit]

Code snippet for duplicating line (first point) has been updated with current cleaner version suggested by Philip K., earlier it was:

(defun duplicate-line ()
   (interactive)
   (let ((col (current-column)))
     (move-beginning-of-line 1)
     (kill-line)
     (yank)
     (newline)
     (yank)
     (move-to-column col)))

– @krsoninikhil

Feliĉan Zamenhofan Tagon

Martin Rue — Sat, 15 Dec 2018 02:00:00 GMT

Hey, it's Zamenhofa Tago, otherwise known as Esperanto Book Day. Read on to learn more about the international language and why it's so cool.

Waspr Project - Baby Steps

CraftyDuck.Rocks — Sat, 15 Dec 2018 02:00:00 GMT

Starting smaller

Hello Flutter

Whacky Labs — Fri, 14 Dec 2018 22:49:54 GMT

Another weekend another tool. The tool for today is Flutter.

Why Flutter?

I had known about Flutter for sometime now, but with the recent news of Flutter 1.0 being out, I got curious about it one more time. Then, recently at work someone ignited my real interest by informing me that Flutter does its own rendering from scratch. I had to try it out.

In the past, I used to wonder a lot that why is that when it comes to cross platform mobile developement, nobody has written a UIKit alternative from scratch or maybe on top of some existing widget library and make it work on all sort of mobile devices. I mean how hard could it be? Given that there are people writing game engines using OpenGL ES, Metal or Vulkan all the time, but why not a UI framework. And that is exactly what Flutter does.

For me it was a love at first sight.

I read a few articles on the official source to get a hang of it. To know more about the vision of the project. I knew this is a product from Google, so I wanted to be sure they’re not baised towards Android. When they’re investing in this tool, what are they getting out of it? Like my initial impression of Xamarin was that Microsoft wants developers to write cross platform apps with Xamarin, and also make a Windows mobile app as a side effect.

But, so far with Flutter this does not seems to be the case, even though they demo a lot with Material designs which looks a lot like Android UI, but then Flutter comes with a rendering engine of its own, and one can render whatever they feel like. There is even a UIKit like widget library which they call as Cupertino.

Then a few hours into making my first Flutter app it occured to me that maybe with Flutter the thing Google is selling is the Dart programming language. I think Google likes to see Dart as a lightweight programming language which is good for quickly iterating and building things as fast as possible. A good candidate for making web and mobile apps. And there is nothing wrong with that. From little experience I had with Dart, I found nothing unlikable about it. At least they did not pick Javascript.

I know most of Game Engines ship a core engine written with a high performance system language like C or C++, and a high level language for writing the actual game, like Lua. Which is also what Flutter does, the core engine is written in C++ and Dart is used to then write the app.

Set up

Whatever they say on the official instructions works out of the box. Although there are quite of bunch of things that have to be installed to get the thing running, but that is expected when installing a tool for cross platform development. The instructions come in 2 flavors, Android Studio and Visual Studio Code (again reflecting the fact that Google does not wants to be biased towards Android). I followed the instructions for Visual Studio Code and it just works.

First run

Okay, even before I get to the part of running the app. I would like acknowledge the fact that how awesome is Visual Studio Code. This is the first time I’m actually using it for anything and I really like it.

Being an iOS developer, I want to give iOS a try first. I created a new project and selected iPhone Simulator (which it magically discovered) And viola!

Next, I lets give Android a run. But can’t find a way to run the app on Android emulator. The drop down shown only the iOS simulator. The faq section says this about switching the target device, but they do nothing:

Clicking on the currently selected device in the status bar
Executing the Flutter: Change Device command
Pressing your custom key binding for the Flutter: Change Device command

Then I discovered, if Flutter can not find an Android emulator, try running it manually by launching the Android Studio > Tool > AVD Manager > Run emulator and then the Android emulator is visible in the status bar.

And now I think, if the main target device is Android, then maybe it is wise to set up the IDE as Android Studio. Anyways, after that part is taken care of, the app runs perfectly on Android emulator as well!

Make the real app

Now, since we have tested the tool and know for sure that the simulators are working perfectly, it’s time to dive a bit deeper and make a real app.

As always, I’ll try to make the Photo app that I try to make every now and then, but probably not today. There seems to be a lot of things that I’ve to learn before I get to that stage. The first stage would be render a list of things and navigate to a details screen when a cell is tapped. But no real networking or concurrency for this stage. I’ll be following the google codelabs part 1 and part 2

So, everything in Flutter is a Widget, which is again very common pattern with Game Engines to have everything as a Node and each node implements a update() and render() functions which are then called by the engine whenever required. In case of Flutter, there is only one function that has to be implemented which is the build() function.

Another concept at the core of Flutter is the react like philosopy, where data flows in one direction so every node or Widget is more or less a stateless entity. Think of it this way, in every UI systems like UIKit or every game engines, at the core there is this tree like data structure, some call it the scene graph other call it the view hierarchy. And then there is is the infinite loop that traverses the tree and updates each node of the tree.

void main()
{
  SceneGraph *rootNode = buildSceneGraph();
  Clock c;
  Event e;
  while(!e.isQuit()) 
  {
    e = nextEvent();
    rootNode->update(c.deltaTime());
    c.tick();
  }
}

So, traditionally each node in the tree had to maintain its own state, which has a tendency to getting messier over time. The react philosophy is more like to not let direct access to nodes, but instead only provide the factory methods which takes in some model data and returns a new node. The tree is the internally managed by the system, and whenever the data changes the factory method would be called to build the new node.

I’m personally a big fan of this philosophy. With this approach, one could freely optimize the tree manipulation as much as they like and on the other hand one could focus only on providing the factory methods. The only part that gets a bit messier is the animation which by definition can not be stateless.

Coming back to Flutter, the build() function is like the factory method that uses some model data and makes a Widget.

Another nice thing about Flutter is that it comes its own package manager. You just have to define the dependencies in the pubspec.yaml and then simply run flutter package get. Nice!

With all this basic understanding of the architecture, writing the first table view is pretty dead simple. I guess this piece of code is self explainatory:

class ListState extends State<Item> {
  final _list = <Item>[];
  Widget _buildWidget() {
    return new ListView.builder(
      padding: const EdgeInsets.all(16.0),
      itemBuilder: (BuildContext _context, int i) {
        if (_needsMoreItems(i)) {
          _data.addAll(_getMoreItems());
        }
        return _buildRow(_data[i]);
      }
    );
  }
}

Since everything is a Widget, it is a piece of cake to add decorations to a Widget

Icon
(
  isSaved ? Icons.favorite : Icons.favorite_border,
  color: isSaved ? Colors.red : null,
)

Another interesting fact is that Flutter builds a Xcode project internally which one can simply open and run. And if one looks close enough, one can find that there is just a single UIView that renders everything.

One question I still had was how does the one maintains a state if required. For example, if say we want to react on a touch interaction we need to use the StatefulWidget which then comes with a createState() function that can provide access to some internal data store where we can read and write data like so:

onTap: () {
  setState((){
    if (isSaved) {
      _saved.remove(word);
    } else {
      _saved.add(word);
    }
  });
}

If you think that is weird, see how the screen transitions are done:

void onDetails() {
  Navigator.of(context)
      .push(MaterialPageRoute<void>(builder: (BuildContext context) {
    final Iterable<ListTile> tiles = _saved.map((WordPair word) {
      return ListTile(title: Text(word.asPascalCase, style: _biggerFont));
    });

    final List<Widget> children =
        ListTile.divideTiles(context: context, tiles: tiles).toList();

    return Scaffold(
        appBar: AppBar(title: const Text('Favorites')),
        body: ListView(children: children));
  }));
}

Conclusion

If you can move out of your comfortable UIKit zone, I think Flutter is a pretty good tool. The performance so far was really nice (at least on the simulators). I can not say much anymore, I would have to complete my photo app to give any good opinion, but I can already say that so far with all the things I’ve tried with cross platform development, Flutter was the best.

The code can be found along with my other prototypes at https://github.com/chunkyguy/PhotoApp. Thansk for reading. See you again later!

Semester Finished

Ashton Wiersdorf on Lambda Land — Fri, 14 Dec 2018 09:08:48 GMT

I finished the semester! This is how I feel:

Don’t you?

Well, I still have finals. But those are easy compared to the projects I’ve had to push out. I’ll probably write about my escapades later. :)

Snowboarding bookmarks

xenodium.com @alvaro — Fri, 14 Dec 2018 02:00:00 GMT

Snowboarding for Geeks (Hacker News).

React Hooks TL;DR

Rodrigo Pombo — Fri, 14 Dec 2018 02:00:00 GMT

React hooks for people in a hurry

Negotiation: from arm wrestling to dancing

D13V — Thu, 13 Dec 2018 21:17:24 GMT

When we hear the word negotiation, we often think of haggling. A series of one-upping offers.

“This couch costs $300”
“I’ll pay $50 for it”
“No way, that’s too low”
“OK, I’ll go to $70”,
“Nah, at least 250”,
“100”, “200” “120”
“200”,
“127 is my final offer”
“…”
“….”
“Okay”, sighing.

It looks like a battle. It feels like two people arm wresting, comparing who has the largest biceps and shoulders. Who is more stubborn.

It’s bruteforce and it’s brute.

But who really “won” the negotiation. Was it the person who kept their position better, or the person who found better compromises. What if the person who compromised more only did it in order to make the other person “feel” they won the negotiation. Meanwhile, they were up already at a big win from the beginning and used the negotiation to stretch it out. Or… what if their goal altogether was to make a deal, any deal, in order to get benefits much larger than the deal itself?

For example, when you take a loan with interest, no matter what the amount of the loan, the agency will make money on it, as long as you sign on to it. For them, not giving you a loan is the losing scenario. They win by making any deal, regardless of the price. Any deal is good deal for them.

That is on the next level arm wrestling here. It’s like Brazilian Jiu Jitsu, BJJ for short. If two people have to resolve a dispute by physical force, someone with 5 years of BJJ will have no trouble overpowering THE BEST arm-wrestler in the world, assuming they don’t know martial arts. And that is regardless of the size. Jiu jitsu is a better system, as it uses the whole body, and not just the arm.

Likewise, in negotiation, a person who approaches it as a haggling competition, will get defeated by anyone more advanced, who can understand positions, interests and emotions and can yield them to outmaneuver the haggler and put them haggling against themselves. The haggler wouldn’t even realize what happened to them.

What good is arm wrestling when you’re getting rear-necked chocked?

But we don’t go around figuratively choking out every person that we have a disagreement with. We don’t have have to make them tap out in submission. Especially when there is any kind of relationship on the line- love, friendship, business relationship, or even parent-child relationship. Because if we choke them at all, we stand to lose relationship.

And because, really, they don’t have to lose, for us to win. It’s not a zero-sum game. In most relationships, there is way to have a win-win.

With relationships, it’s more like a dance than a fight. Each person adapts to the other. One person might lead, and another might follow. Then the roles might switch. The speed and the energy vary and are constantly adjusted. It’s a whole body and whole mind experience. We adapt to them in our posture, position, eye contact, tone of voice, thoughts, emotions, incentives and logic. It takes our whole being to do that dance. If the feet are not moving, we lose the rhythm, and the dance falls apart. We engage our feet, we spin and shake and adapt to the other person. We create the dance together, using our whole beings.

They adapt likewise. Both sides want to keep the dance going. Nobody wants to be choked, or arm wrestled when they’re dancing.

Using an Alfred workflow to assist in switching from Google Chrome to Firefox

Jason Raimondi on Software Engineer — Thu, 13 Dec 2018 10:00:00 GMT

This workflow is meant to help break the habit of using Google Chrome. I tend to launch applications with Alfred. When I want to launch a browser, what is engraved in my brain is CHROME!

Every time: ⌘ + SPACE → “chrome” → ENTER.

To help combat this, I’ve created an Alfred workflow that is triggered by the keyword “chrome” and will launch Firefox.

Now every time I am about to launch Chrome, I see the “Launch Firefox Instead of Chrome” launcher. After selecting it a time or two, Alfred will start to prioritize it above the actual Google Chrome application.

References

ferrucc.io — Wed, 12 Dec 2018 15:31:50 GMT

Blog Posts

The Matrix Calculus You Need For Deep Learning

Design Inspiration

As I want to improve my UX skills for the next projects I’ll be using some standard frameworks like:

Landing Pages

This is a list of landing pages I look to when trying to find inspiration:

Don’t want to miss any of this?

You can find out about everything that gets added here by receiving my monthly email.

Print your shoes

Yasha Solutions — Wed, 12 Dec 2018 06:10:13 GMT

Because why not.Source : https://www.thingiverse.com/thing:285404 Supplies : https://www.gearbest.com/3d-printer-supplies/pp_919228.html?wid=1433363&lkid=15674036

expressjs dynamic routing – reading list

Yasha Solutions — Wed, 12 Dec 2018 05:48:44 GMT

http://expressjs.com/en/guide/routing.html https://alexanderzeitler.com/articles/expressjs-dynamic-runtime-routing/ https://stackoverflow.com/questions/19096453/convert-object-to-express-dynamic-routes https://github.com/OneOfOne/dynamic-routes

how to cut video using nodejs and ffmpeg

Yasha Solutions — Wed, 12 Dec 2018 05:40:46 GMT

https://stackoverflow.com/questions/31046930/how-to-cut-a-video-in-specific-start-end-time-in-ffmpeg-by-node-js#31050135 https://stackoverflow.com/questions/18444194/cutting-the-videos-based-on-start-and-end-time-using-ffmpeg?rq=1 https://stackoverflow.com/questions/40454172/ffmpeg-cut-video-and-include-ending-video-image?rq=1 https://superuser.com/questions/138331/using-ffmpeg-to-cut-up-video#704118 https://medium.com/@brianshaler/on-the-fly-video-rendering-with-node-js-and-ffmpeg-165590314f2

Mongodb – Reference or embed

Yasha Solutions — Wed, 12 Dec 2018 05:37:07 GMT

https://stackoverflow.com/questions/5373198/mongodb-relationships-embed-or-reference https://www.tutorialspoint.com/mongodb/mongodb_relationships.htm https://www.studytonight.com/mongodb/relationships-in-mongodb https://docs.mongodb.com/manual/applications/data-models-relationships/ http://blog.markstarkman.com/blog/2011/09/15/mongodb-many-to-many-relationship-data-modeling/

shitty time reset procedure (src: twitter)

Yasha Solutions — Wed, 12 Dec 2018 05:21:18 GMT

source NB: take with a grain of salt – but still some interesting tips. posted for reference #thread start: if you’re having a shitty time with your mental health and are low functioning i feel u & sympathise. here’s a list of potential actions for u to take to feel better/exist through it. try to pick a few & do them, but if you can’t : that’s valid too. let’s go

Set up a different an SSH key for git push for a give repository

Yasha Solutions — Wed, 12 Dec 2018 05:08:29 GMT

ok so I add the following problem to deal with. Posted here for reference. Source Even if the user and host are the same, they can still be distinguished in ~/.ssh/config. For example, if your configuration looks like this: Then you just use gitolite-as-alice and gitolite-as-bob instead of the hostname in your URL:

UX challenges – 7 rapid prototypinng

Yasha Solutions — Wed, 12 Dec 2018 05:05:11 GMT

Rapid prototyping exercices Exercise 1: Mobile TV programs dashboard Scenario: when users login on the app what will they see, what can they do Goals: Engagement Exercise 2: Rapid prototyping of a live class Scenario: User is watch a live class – with interactive features Goals: interactivity with the teacher and / or other studens, accessibility, access to materials Exercise 3: A memo card : scenario: User selected to watch a series of random memo cards goals: Fun, Interactive Exercise 4: Collaborative social media publishing

Books I'm Reading to Become a Better Writer

Thomas Hunter II — Wed, 12 Dec 2018 02:00:00 GMT

I recently picked up several books on writing. I'd like to share these books with you in the hopes that you might also find them interesting.

What I Listen To

bpev.me — Wed, 12 Dec 2018 02:00:00 GMT

String interning in Go

Artem Krylysov — Wed, 12 Dec 2018 02:00:00 GMT

String interning is a technique of storing only one copy of each unique string in memory. It can significantly reduce memory usage for applications that store many duplicated strings.

The built-in string is represented internally as a structure containing two fields. Data is a pointer to the string data and Len is a length of the string:

type StringHeader struct {
        Data uintptr
        Len  int
}

In Go strings are immutable, so multiple strings can share the same underlying data:

package main

import (
    "fmt"
    "reflect"
    "unsafe"
)

// stringptr returns a pointer to the string data.
func stringptr(s string) uintptr {
    return (*reflect.StringHeader)(unsafe.Pointer(&s)).Data
}

func main() {
    s1 := "1234"
    s2 := s1[:2] // "12"
    // s1 and s2 are different strings, but they point to the same string data
    fmt.Println(stringptr(s1) == stringptr(s2)) // true
}

Most of modern programming languages including Go intern compile-time string constants:

s1 := "12"
s2 := "1"+"2"
fmt.Println(stringptr(s1) == stringptr(s2)) // true

But strings generated at runtime are not interned:

s1 := "12"
s2 := strconv.Itoa(12)
fmt.Println(stringptr(s1) == stringptr(s2)) // false

Implementation #

To implement string interning we need a data structure representing a pool of strings. The pool needs to support two operations: adding a string to the pool and retrieving a string from the pool. A good candidate for the requirements is a hash map:

package main

import (
    "fmt"
    "strconv"
)

type stringInterner map[string]string

func (si stringInterner) Intern(s string) string {
    if interned, ok := si[s]; ok {
        return interned
    }
    si[s] = s
    return s
}

func main() {
    si := stringInterner{}
    s1 := si.Intern("12")
    s2 := si.Intern(strconv.Itoa(12))
    fmt.Println(stringptr(s1) == stringptr(s2)) // true
}

Let's take a look at a few examples where interning could be useful.

Example 1: Text processing #

Lexers, parsers and other text processing tools can greatly benefit from storing only distinct string values in memory.

The following program reads George Orwell's novel 1984 from a file and tokenizes it into a slice of words for further processing:

package main

import (
    "bufio"
    "log"
    "os"
)

func main() {
    f, err := os.Open("1984.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    scanner := bufio.NewScanner(f)
    scanner.Split(bufio.ScanWords)

    var words []string
    for scanner.Scan() {
        words = append(words, scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        log.Fatal(err)
    }

    log.Print(len(words))
}

The book contains 103549 words, the length of all words combined is 483016 bytes. It's important to note that the number of unique words is much smaller - 15585.

If we take a look at a part of the words slice we can see the same words (IS in our example) having different addresses in memory:

fmt.Println(words[1111:1120])
for _, word := range words[1111:1120] {
    fmt.Printf("%x ", stringptr(word))
}

[WAR IS PEACE FREEDOM IS SLAVERY IGNORANCE IS STRENGTH]
     ^                ^                    ^

c0000159fa c0000159fe c000015a00 c000015a05 c000015a0c c000015a10 c000015a17 c000015a20 c000015a28
           ^                                ^                                ^

Let's modify the program to use string interning:

package main

import (
    "bufio"
    "log"
    "os"
)

type stringInterner map[string]string

func (si stringInterner) InternBytes(b []byte) string {
    if interned, ok := si[string(b)]; ok {
        return interned
    }
    s := string(b)
    si[s] = s
    return s
}

func main() {
    f, err := os.Open("1984.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    scanner := bufio.NewScanner(f)
    scanner.Split(bufio.ScanWords)

    si := stringInterner{}
    var words []string
    for scanner.Scan() {
        words = append(words, si.InternBytes(scanner.Bytes())) // intern words
    }
    if err := scanner.Err(); err != nil {
        log.Fatal(err)
    }

    log.Print(len(words))
}

Now the words slice consists of strings that point to the string intern pool. All instances of IS have the same address in memory:

fmt.Println(words[1111:1120])
for _, word := range words[1111:1120] {
    fmt.Printf("%x ", stringptr(word))
}

[WAR IS PEACE FREEDOM IS SLAVERY IGNORANCE IS STRENGTH]
     ^                ^                    ^

c000015220 c0000146c8 c000015223 c000015228 c0000146c8 c000015230 c000015237 c0000146c8 c000015240
           ^                                ^                                ^

The amount of memory required to store the words decreased from 483016 to 119628 bytes which is more than 4x reduction.

Moreover, the []byte map key optimization helps to avoid expensive heap allocations. The Go compiler recognizes si[string(b)] and performs the lookup operation without allocating a new string and producing garbage.

Example 2: Network services #

Another example where string interning can be useful is network services. Interning can be applied to some string responses from databases, gRPC or HTTP servers.

The following snippet caches user information after querying it from a database:

type user struct {
    ID      int
    Country string
}

type userService struct {
    db    *sql.DB
    cache sync.Map
}

func (us *userService) get(id int) (*user, error) {
    if u, ok := us.cache.Load(id); ok {
        return u.(*user), nil
    }
    u := &user{}
    row := us.db.QueryRow("SELECT id, country FROM users WHERE id=?", id)
    if err := row.Scan(&u.ID, &u.Country); err != nil {
        return nil, err
    }
    us.cache.Store(id, u)
    return u, nil
}

There are about 200 countries as of December 2018, so storing only a single copy of each string can save us memory if the number of users we want to keep in cache is large.

The user service is called from an HTTP handler, which means the intern pool is required to be safe for concurrent access from multiple goroutines. Luckily Go 1.9 introduced a concurrent map - sync.Map.

The user service using a thread-safe version of stringInterner:

type stringInterner struct {
    sync.Map
}

func (si *stringInterner) Intern(s string) string {
    interned, _ := si.LoadOrStore(s, s)
    return interned.(string)
}

type userService struct {
    db    *sql.DB
    cache sync.Map
    si    stringInterner
}

func (us *userService) get(id int) (*user, error) {
    if u, ok := us.cache.Load(id); ok {
        return u.(*user), nil
    }
    u := &user{}
    row := us.db.QueryRow("SELECT id, country FROM users WHERE id=?", id)
    if err := row.Scan(&u.ID, &u.Country); err != nil {
        return nil, err
    }
    u.Country = us.si.Intern(u.Country) // intern country
    us.cache.Store(id, u)
    return u, nil
}

Usage in Go standard library #

The Go standard library uses interning in a few places, one of them is net/textproto. textproto.Reader interns common HTTP headers (Content-Type, Host, User-Agent, etc.) to avoid memory allocations.

String comparison #

A decrease in memory usage is not the only advantage of string interning. Interned strings can be compared for equality in constant time. All the compiler needs to do is to check if two pointers are equal (Go source code) instead of going through the characters:

TEXT cmpbody<>(SB),NOSPLIT,$0-0
    CMPQ    SI, DI
    JEQ allsame

I created a benchmark to see the performance difference between interned and non-interned strings:

package main

import (
    "strings"
    "testing"
)

func benchmarkStringCompare(b *testing.B, count int) {
    s1 := strings.Repeat("a", count)
    s2 := strings.Repeat("a", count)
    b.ResetTimer()
    for n := 0; n < b.N; n++ {
        if s1 != s2 {
            b.Fatal()
        }
    }
}

func benchmarkStringCompareIntern(b *testing.B, count int) {
    si := stringInterner{}
    s1 := si.Intern(strings.Repeat("a", count))
    s2 := si.Intern(strings.Repeat("a", count))
    b.ResetTimer()
    for n := 0; n < b.N; n++ {
        if s1 != s2 {
            b.Fatal()
        }
    }
}

func BenchmarkStringCompare1(b *testing.B)   { benchmarkStringCompare(b, 1) }
func BenchmarkStringCompare10(b *testing.B)  { benchmarkStringCompare(b, 10) }
func BenchmarkStringCompare100(b *testing.B) { benchmarkStringCompare(b, 100) }

func BenchmarkStringCompareIntern1(b *testing.B)   { benchmarkStringCompareIntern(b, 1) }
func BenchmarkStringCompareIntern10(b *testing.B)  { benchmarkStringCompareIntern(b, 10) }
func BenchmarkStringCompareIntern100(b *testing.B) { benchmarkStringCompareIntern(b, 100) }

The speed of comparison of interned strings remains constant independently of the number of characters:

BenchmarkStringCompare1-4               500000000            2.93 ns/op
BenchmarkStringCompare10-4              300000000            6.21 ns/op
BenchmarkStringCompare100-4             100000000            13.2 ns/op
BenchmarkStringCompareIntern1-4         1000000000           2.60 ns/op
BenchmarkStringCompareIntern10-4        1000000000           2.60 ns/op
BenchmarkStringCompareIntern100-4       1000000000           2.60 ns/op

Conclusion #

String interning can save memory at a cost of CPU time - every lookup from the intern pool requires hashing the input string (which may not work for CPU bound applications).

Here is a real-life example - memory usage of the application went down from 70% to 45% after I deployed a new version with string interning enabled:

Software Engineering for Academia

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Tue, 11 Dec 2018 22:01:00 GMT

During the course of my research, I tend to write a lot of code. My forays into the world of software development and my usage of C++ taught me the value of writing unit tests, following the mystical best practices (whatever they may be), and caring about the quality of my code. It recently occurred to me that this perspective is not too prevalent in academia because while software engineering seems “nice” and useful, it also seems to detract from making progress, viz. from writing that code and pummelling it into shape until it works. As one of my colleagues so aptly put it:

The code does not have to be maintainable, it just has to survive until the conference deadline has passed.

While that seems about right, I argue that this perspective is short-sighted. Moreover, I claim that following some good practices is bound to pay off for the following reasons:

You will be able to trust your own code better.
You will be able to trust the results of your own code better.
Modifications of said code (such as additional experiments that have been requested by “Reviewer 2”, that fearsome beast whose hooves have trampled many papers already) are easier.
Your results will be stronger and less susceptible to “classical” mistakes, such as incorrect parameters, wrong cross-validation strategies, and so on.

So, without further ado, here are three strategies that will be helpful:

Use a version control system such as git.
Write tests.
Refactor your code whenever it becomes too complicated.

I will not go into detail about the first one—the good people at GitHub have a pretty nice tutorial that covers the basics (and more)!, but let us take a look at the other two instead.

Writing tests

As is so often the case in the software industry, testing (or more precisely, test-driven development) is hailed as the solution to all of your programming woes. I would not go so far. But there are different strategies in testing that will be very helpful in academia:

Regression testing: this refers to writing a test that asserts the basic functionality of your code. The best way to approach this kind of tests is to have manual examples of what your code is supposed to do and check them against what the code actually does. For example, suppose you are writing code that calculates the connected components of a graph. A simple functional test involves taking a simple graph, figuring out its connected components with pen and paper, and checking that the code arrives at the same answer. The trick is not to stop now—instead of concluding that the code is correct, you now write additional code that performs the same verification for you automatically (I will discuss later on how you do this in practice). The idea behind this extra effort is that you might change something in the connected component calculation code in a few weeks and forget to check your results again. However, if you have code available for checking the results automatically, you will catch these regressions easily (hence the name).
Sanity checks: often, our code is too complicated to have examples that we can manually check. In these cases, it might make sense to check simple properties of the results of your code instead. For example, if you have code that should yield a positive definite matrix, a sanity check might be a function that tests whether the eigenvalues of said matrix are all positive. While less powerful than functional tests or regression tests, sanity checks can nonetheless be very helpful in restoring your sanity.
Fuzz testing: this sort of testing refers to test cases in which your function (or model, etc.) is subject to essentially random inputs. When you are writing a routine for calculating the eigensystem of a matrix, for example, your code should be capable of handling any matrix (while raising some errors for those that cannot decomposed, of course). By providing random inputs, you make it less likely that you (inadvertently) cherry-picked some matrix with a special property in your tests. Fuzz testing may also be used to ensure robustness against malicious uses of your functions, but if you really need this, you should not read my blog but study how to write secure code.

Of course, even a test that is well thought-out will might not help in detecting all errors down the line, but overall, writing some test cases is worth the trouble. In most languages, testing frameworks are readily available. Python has the great unittest framework, whereas C++ users have the choice of Google Test, Boost.Test, and many more. If, like me, you would like to roll your own testing framework, look at the test cases of Aleph, my library for topological data analysis. Another blog post of mine explains some design choices and shows you how to combine testing with CMake.

Refactoring

Last, I briefly want to discuss another strategy for combatting the complexities of your code: refactoring. This is merely a fancy word for rewriting your code while not changing its intended output (and at the risk of sounding like a broken record, this is exactly why you need unit tests). Of course, the general idea is to make the code simpler—so that you will understand it even a month or a year after you last touched it. Here are some general ideals you should strive for when refactoring:

Make information local, not global: try to provide a single flow for all your information in your code. This means not using global variables, and relying on small state information whenever possible.
Make your functions do one thing: try to keep your function simple or at least single-purposesebas. Think about splitting up large functions into smaller ones.
Strive for precision: use small classes with descriptive attributes, in particular in languages like Python, to store information about a state in your program rather than using tuples (with hard-coded indices), for example.

Of course, refactoring can also go too far, but if you are doing it to make your life easier and not just to waste some time procrastinating, it will probably he helpful. You should also take a look at Jeff Atwood’s article on “code smells” because recognizing these problematic patterns makes it easier to find parts of your code that benefit from refactoring.

Good luck with your coding efforts, until next time!

In Which the Bees get a New House

one mikro2nd — Tue, 11 Dec 2018 16:31:53 GMT

Rehoused the bees – moved them out of the 5-frame catch-box they’ve been living in since who-knows-when, and into their spiffy new full-size brood-box. It went messily. Despite my inexperience and clumsiness, they moved in quite well and by midday were already calming down. I removed their old house in the evening and the guard-bees were moving in a much more normal fashion after their day on High Alert, though there were quite a few more of them than normal.

Recovering lost Python results in the REPL

utf9k — Tue, 11 Dec 2018 13:00:00 GMT

Did you know that Python stores the previous output as a hidden variable?

3 weird old fashion websites still alive today

Yasha Solutions — Tue, 11 Dec 2018 07:17:33 GMT

Yes, once this was the internet Internet Explorer is EVIL! http://toastytech.com/evil/ ARNGREN – ecommerce from hell http://www.arngren.net/ LingsCar – UK best car shop https://www.lingscars.com/

WebRTC Janus – CPU intensive

Yasha Solutions — Tue, 11 Dec 2018 07:03:20 GMT

WebRTC is great Janus is a cool library (link if you never heard of it : github and good intro link) But boy is it CPU intensive. here are some links for the case you are looking for ways to lower the damage Client CPU benchmark and how to improve it https://groups.google.com/forum/#!msg/meetecho-janus/ydGcbMt7IX0/LOGVIA6yjkAJ Performance analysis of the Janus WebRTC gateway Link To Research Gate Article WebRTC and CPU reduction, settings to tweak

Reducing Chromecast Idle Bandwidth

Carey Metcalfe — Tue, 11 Dec 2018 05:50:00 GMT

Google's Chromecast is a pretty useful device. It plugs into the HDMI port of a TV and allows users to "cast" content to it via a smartphone, Chrome browser, etc. This content is mostly assumed to be streaming services like Youtube, Spotify, or Netflix, but for someone who doesn't really buy into the whole cloud revolution like myself, it's still perfectly capable of playing local content as well.

The one downside of the Chromecast is that when it's not being used, instead of going into a low-power state and/or turning off the TV, it instead enters Ambient Mode. This shows some useful information overlaid on a changing backdrop of featured photos downloaded from Google Photos. The issue with this is that the images are high-resolution, not cached, and are continually being downloaded 24/7 even when the TV is turned off. Although I haven't personally measured it, the general consensus seems to be that it uses around 15GB of data per month from just being plugged in.

What I'll be going over here is reducing the data the Chromecast uses by configuring it to only download some tiny black 1x1 px images to use as a backdrop.

The short version

Make a tiny black PNG (pixel.png)
Make a slightly different tiny PNG to avoid deduplication (pixel_alt1.png)
Upload both to an album in Google Photos
Mark both images as favorites
Configure the Chromecast to only pull images from your specific album every 10 minutes

If you like details, keep reading...

Making a small image

I ended up creating a 1x1 solid black image in GIMP, saving it as a PNG, hacking out as much data as possible using a hex editor, then using pngcrush to attempt to optimize the data that was left.

As I was working on the PNG file, I used a small script based on the Python construct library to visualize the chunks and structure of it so I could figure out what to cut.

To install construct and download the example PNG specification:

python3 -m venv .venv
source .venv/bin/activate
pip install construct
wget "https://raw.githubusercontent.com/construct/construct/abd48c4892ceddc60c11d25f4a955573e2c61111/deprecated_gallery/png.py"

Create pngview.py:

import png  # the png spec downloaded above
import sys

with open(sys.argv[1], 'rb') as f:
    data = f.read()
    print("Size:", len(data), "bytes")
    print(png.png_file.parse(data))

Running it on the PNG exported from GIMP using the default settings gives:

$ ./pngview.py pixel.png
Size: 146 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) truecolor 2
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 2423739358
    chunks = ListContainer: 
        Container: 
            length = 9
            type = b'pHYs' (total 4)
            data = Container: 
                pixels_per_unit_x = 11811
                pixels_per_unit_y = 11811
                unit = (enum) meter 1
            crc = 2024095606
        Container: 
            length = 7
            type = b'tIME' (total 4)
            data = Container: 
                year = 2018
                month = 12
                day = 11
                hour = 5
                minute = 2
                second = 19
            crc = 904567710
        Container: 
            length = 25
            type = b'tEXt' (total 4)
            data = Container: 
                keyword = u'Comment' (total 7)
                text = b'Created with GIM'... (truncated, total 17)
            crc = 1468075543
        Container: 
            length = 12
            type = b'IDAT' (total 4)
            data = b'\x08\xd7c```\x00\x00\x00\x04\x00\x01' (total 12)
            crc = 657729290
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

After removing the unneeded pHYs, tIME, and tEXt chunks using a hex editor, it looks like this:

$ ./pngview.py pixel.png 
Size: 69 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) truecolor 2
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 2423739358
    chunks = ListContainer: 
        Container: 
            length = 12
            type = b'IDAT' (total 4)
            data = b'\x08\xd7c```\x00\x00\x00\x04\x00\x01' (total 12)
            crc = 657729290
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

At this point, all that's left to optimize is the image data itself. This is where pngcrush shines. After running pngcrush -brute -ow pixel.png we see:

$ ./pngview.py pixel.png
Size: 67 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) greyscale 0
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 981375829
    chunks = ListContainer: 
        Container: 
            length = 10
            type = b'IDAT' (total 4)
            data = b'\x08\x1dc`\x00\x00\x00\x02\x00\x01' (total 10)
            crc = 3486004709
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

The 2 byte savings in the data seem to have come from setting the color_type to greyscale instead of truecolor. To view what actually changed in the data, we can un-deflate it using Python:

>>> import zlib
>>> # before pngcrush (12 bytes)
>>> zlib.decompress(b'\x08\xd7c```\x00\x00\x00\x04\x00\x01')
b'\x00\x00\x00\x00'
>>> # after pngcrush (10 bytes)
>>> zlib.decompress(b'\x08\x1dc`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>>

This makes sense since, according to the PNG spec, when using truecolor, each pixel is an RGB triple, requiring 3 bytes. For greyscale, only a single byte representing luminance is needed. The first byte is the filtering method, which isn't relevant here since we only have a single pixel.

Interestingly enough, in both cases we would actually be much better off if we could opt to not use compression, but alas, the spec does not allow for anything except deflate.

But I digress, we now have a black 1x1 px PNG image that's just 67 bytes (pixel.png).

Generating multiple unique small images

So now we have the small PNG we want to display. Since the Chromecast's ambient mode requires at least 2 different images in an album to cycle through, all we need to to do is upload 2 copies of this PNG and we're done right? Almost.

Since Google Photos will automatically deduplicate uploaded images, we need to find a way to make the second image slightly different. Normally this would involve tweaking a comment or something, but in this case, the image has already been stripped down to its bare essentials.

My strategy was attempt to abuse the data compression to see if I could generate an image that compressed the same data into the same number of bytes, but differently. Fortunately, pngcrush can be told to use specific compression strategies (there are currently 177 of them). My hope is that at least one of these will achieve the same results, but in a different way. For starters we'll try the first 9:

for x in $(seq 9); do
    pngcrush -m "$x" pixel.png pixel_alt${x}.png;
done

A quick ls -l reveals that all of the pixel_alt*.png files are still 67 bytes. Now we just need to find one that has different data. sha1sum is the perfect utility for this:

$ sha1sum pixel.png pixel_alt*.png
d99a9d63b1cd9e4b3f823d4d03144ccd95328f48  pixel.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt1.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt2.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt3.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt4.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt5.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt6.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt7.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt8.png
d99a9d63b1cd9e4b3f823d4d03144ccd95328f48  pixel_alt9.png

pixel_alt1.png looks like a good candidate, let's see what changed:

$ diff <(./pngview.py pixel.png) <(./pngview.py pixel_alt1.png)
20,21c20,21
<             data = b'\x08\x1dc`\x00\x00\x00\x02\x00\x01' (total 10)
<             crc = 3486004709
---
>             data = b'\x08[c`\x00\x00\x00\x02\x00\x01' (total 10)
>             crc = 1648381800

There's the same amount of data, but it's different. Let's check if it decompresses to the same image data:

>>> import zlib
>>> zlib.decompress(b'\x08\x1dc`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>> zlib.decompress(b'\x08[c`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>>

Yep, this means that pixel_alt1.png is the exact same image with the exact same size, but won't be deduplicated when uploading it to Google Photos since the compressed data is different.

Final touches

Now that we have 2 different tiny images, we can upload them to Google Photos and put them in an album so they can be pulled down by the Chromecast. Something to note is that you may have to mark the images as favorites to get them to be displayed. There seems to be some sort of AI-fueled "we know better than you" algorithm that initially refused to display my images until I starred them.

Now just set your ambient mode to display the album with your images in it (or your favorites album), set the slideshow speed to its maximum value (change image every 10 mins), and you're done.

From bare-metal to Kubernetes

Boxunix — Tue, 11 Dec 2018 03:02:32 GMT

How migrating Betabrand's bare-metal infrastructure to a Kubernetes cluster hosted on Google Container Engine solved many engineering issues—from hardware failures, to lack of scalability of our production services, complex configuration management and highly heterogeneous development-staging-production environments—and allowed us to achieve a reliable, available and scalable infrastructure.

This post will walk you through the many infrastructure changes and challenges Betabrand met from 2011 to 2018.

Note

Betabrand is a retail clothing company and crowdfunding platform, based in San Francisco.

Scam bookmarks

xenodium.com @alvaro — Tue, 11 Dec 2018 02:00:00 GMT

Passive income bookmarks

xenodium.com @alvaro — Tue, 11 Dec 2018 02:00:00 GMT

About

ferrucc.io — Mon, 10 Dec 2018 15:31:50 GMT

I’m Ferruccio Balestreri

I’ve founded and was the CTO at June. Before that I worked at Intercom as an engineer.

My hobby is to make small projects every couple of months I make something new to have fun and sharpen my skills.

Slow to respond through 2018

blog.harterrt.com — Mon, 10 Dec 2018 10:00:00 GMT

I'm working on an urgent and high priority request for the next few weeks. To make sure I can finish this work in 2018 I'm limiting my meetings and communications for the remainder of the year.

Slack is good for getting my immediate attention, but if your request takes more …

Managing Remote Teams - A Crash Course

Blog posts of Andreas Klinger — Mon, 10 Dec 2018 02:00:00 GMT

Hey folks, I haven’t been blogging for quite some time, so everything is a bit rusty for me, but i thought this article might be u...

Creating WebGL apps with Go

divan's blog — Mon, 10 Dec 2018 02:00:00 GMT

TL;DR In this article I’ll share my experience building an interactive 3D WebGL-based application for peer-to-peer messaging protocol simulation without writing any single line in JS. You’ll learn how GopherJS and Vecty framework can dramatically lower the complexity of building WebGL-enabled web apps in Go.

It’s often said “A picture is worth a thousand words”, but in the era of high-DPI screens and big data, the new idiom is now truer – “3D interactive visualization is worth a thousand pictures”.

XCat 1.0 released or: XPath injection issues are severely underrated

Tom Forbes — Sun, 09 Dec 2018 21:58:01 GMT

I’ve just released xcat 1.0 and it’s demonstration application after like 5 years of on-off development. Feels good! The genesis of xcat was when my boss, Sid, walked up to me out of the blue and asked if I wanted to go on an all expenses paid trip to Amsterdam. Who the hell wouldn’t say yes to that...

SSH Foo

Jason Eckert's Website and Blog — Sun, 09 Dec 2018 02:00:00 GMT

Last week, I gave a presentation at a local LUG (which is a Linux User’s Group, for anyone who hasn’t heard of those before) on the subject of SSH Foo.

In UNIX circles, we refer to tricks of the trade as “Kung Fu” or just “Fu” (which sounds rude), so it is spelled “Foo” ;-)

In this blog post, I’ll summarize my SSH Foo talk! Be aware that this is not an introduction to SSH, but a more advanced look at some interesting SSH features for those who are already comfortable with it. As such, I’ve kept to point form here to allow you to skim to the parts you are most interested in!

AT&T causes mDNS on Linux To Fail

Missives — Sat, 08 Dec 2018 17:44:53 GMT

Today my Jenkins builds were not working because all of the build slaves were offline. Digging around in the logs showed that the couldn’t connect because of name resolution failures. I use mDNS on my network (the slaves are Mac OS X VMs running on a Mac Mini), and so they were named something like … Continue reading

Calculating day of week and Easter date

strchr.com updates — Sat, 08 Dec 2018 13:44:58 GMT

A branchless code for calculating day of week, Easter date, and Jewish holidays.

DWIM ivy quit

xenodium.com @alvaro — Sat, 08 Dec 2018 02:00:00 GMT

"Do-what-I-mean" (DWIM) functions enable us to introduce new Emacs powers to existing workflows without incurring the typical cost of remembering multiple related functions or introducing yet another key binding. DWIM functions invoke other functions, based on current context.

I wanted a small tweak in Ivy's `minibuffer-keyboard-quit' invocation, commonly invoked via C-g key binding:

If we have text selected in minibuffer, deselect it.
If we have any text in minibuffer, clear it.
If no text in minibuffer, quit.

Added `ar/ivy-keyboard-quit-dwim' for this purpose. Binding it to C-g in ivy-minibuffer-map:

(use-package ivy
  :ensure t
  :bind (:map ivy-minibuffer-map
              ("C-g" . ar/ivy-keyboard-quit-dwim))
  :config
  (defun ar/ivy-keyboard-quit-dwim ()
    "If region active, deactivate. If there's content, clear the minibuffer. Otherwise quit."
    (interactive)
    (cond ((and delete-selection-mode (region-active-p))
           (setq deactivate-mark t))
          ((> (length ivy-text) 0)
           (delete-minibuffer-contents))
          (t
           (minibuffer-keyboard-quit)))))

Content Negotiation & Phoenix

denvaar's website — Fri, 07 Dec 2018 08:40:09 GMT

A quick demo exploring how HTTP content negotiation works with Phoenix.

Using Mailgun for a Free Custom Domain Email Address

Danny Guo — Fri, 07 Dec 2018 02:00:00 GMT

Editors, or The Tools of my Trade

Ashton Wiersdorf on Lambda Land — Thu, 06 Dec 2018 08:02:09 GMT

I spend a fair portion of every day writing programs. As with all professions, using the right tools makes a huge difference in my productivity and general happiness. Having good tools helps me keep my gumption up.

One of my favorite books is Zen and the Art of Motorcycle Maintenance. Contrary to what the title suggests, this book is actually not about motorcycles. It’s about a lot of things; one topic is about tools and caring about your trade.

… By far the most frustrating gumption trap is inadequate tools. Nothing’s quite so demoralizing as a tool hang-up. Buy good tools as you can afford them and you’ll never regret it.‌‌(ibid. p.g. 291)

For me, my most important tool is my text editor: manipulating source code is what I spend ALL DAY doing. I’ve selected Emacs as my primary text editor.

Lots of programmers use what’s called an IDE If programming were cooking, then an IDE would be a knife that has a sink, a strainer, and a toaster-oven built into it.

I find IDE’s visually distracting. Everything is done with buttons that you click. Emacs has so much more screen devoted to content.

Emacs and Vim do have steeper learning curves. This is in part because the absence buttons make the features only discoverable via manuals. When working with an IDE, the presense of buttons hints at the existence of certain features.

I find it a shame when people don’t read, prefering a video tutorial or the like. Emacs’s features are very discoverable, but not in the way most people are used to. The apropos-function command is terribly useful—if you think a certain command should exist, searching all available function names for a particular string has helped me find both what I’ve gone looking for, and what I didn’t know I wanted!

The advantage to using keyboard only navigation is that hundreds of commands are immediately available all the time, without having to dig through menus or such. It takes me the same amount of time to access complicated commands as it does moving around a file.

Again, from Zen and the Art of Motorcycle Maintenance:

…One of the first warning signs of impatience is frustration at not being able to lay your hand on the tool you need right away.‌‌(ibid. p.g. 286)

Moving a mouse takes me out of my flow, which slows me down and leads to gumption traps. If something takes longer, I’m more reluctant to do it. That means my productivity drops.

Now, you might have used a word processor for text editing all your life. Or perhaps you’re comfortable with your IDE. I encourage you to stretch beyond what you’re comfortable with, and learn Emacs. An investment in a powerful text editor will change how you consider how you program. Editing will become more fluid, and the barrier of buttons and menus will fade away.

Learning Emacs #

There is an index of learning resources here. If you’re just looking for a cheat sheet, check this link out. Also, see my cheat sheet. (I update this one occasionally.)

If you’re coming from an IDE and miss feature X, you might be able to find the corresponding Emacs feature here.

Picture Credit #

I found this via a Google Image search; the origional file came from this blog post, which I think resonates with mine quite well.

Bitlbee to Pleroma

infrequent oscillations — Wed, 05 Dec 2018 17:50:36 GMT

In which we talk to Pleroma through Bitlbee.

Diffing directories content size

xenodium.com @alvaro — Wed, 05 Dec 2018 02:00:00 GMT

Needed to diff two directories, but only interested in file size changes. diff, find, sort, and stat seem to do the job:

diff <(find dir1 -type f -exec stat -f '%N %z' '{}' \; | sort) <(find dir2 -type f -exec stat -f '%N %z' '{}' \; | sort)

1,3c1,2
< dir1/one.txt 14
< dir1/subdir/file.txt 5
< dir1/three.txt 7
---
> dir2/one.txt 19
> dir2/two.txt 0

Note: Using diff, find, sort, and stat on macOS.

Update 1

I've since learned about mtree (thanks Roman!). A nice utility to add to the toolbox.

mtree -p emacs-25.1 -c -k size -d

#    user: me
# machine: my-machine
#    tree: /path/to/emacs-25.1
#    date: Wed Dec  5 22:21:07 2018
# .
/set type=dir
.               size=1152
# ./admin
admin           size=960
# ./admin/charsets
charsets        size=544
# ./admin/charsets/glibc
glibc           size=3392
# ./admin/charsets/glibc
..
# ./admin/charsets/mapfiles
mapfiles        size=640
# ./admin/charsets/mapfiles
..

Update 2

I've added Emacs ediff to the mix:

(require 'f)

(defun ar/ediff-dir-content-size ()
    "Diff all subdirectories (sizes only) in two directories."
    (interactive)
    (let* ((dir1-path (read-directory-name "Dir 1: "))
           (dir2-path (read-directory-name "Dir 2: "))
           (buf1 (get-buffer-create (format "*Dir 1 (%s)*" (f-base dir1-path))))
           (buf2 (get-buffer-create (format "*Dir 2 (%s)*" (f-base dir2-path)))))
      (with-current-buffer buf1
        (erase-buffer))
      (with-current-buffer buf2
        (erase-buffer))
      (shell-command (format "cd %s; find . -type d | sort | du -h" dir1-path) buf1)
      (shell-command (format "cd %s; find . -type d | sort | du -h" dir2-path) buf2)
      (ediff-buffers buf1 buf2)))

I don’t know what to say…

Wed, 05 Dec 2018 00:12:03 GMT

The issue raised for the event-stream breach. It’s a grizzly flame war that I would not recommend reading

I’m a little late to the party here but after having a couple of conversations at work and with others I wanted to document my thoughts on the recent security issues around the event stream npm package which was used by lots of popular packages such as nodemon.

Surrounding this controversy was many questions about vetting packages more carefully as well as the tendency for node developers to just have a package for basic functionality they could implement themselves (looking at your left-pad).

Whilst all these questions are valid and worth discussing, this is not what I wanted to talk about in this blog post.

I want to talk about giving back to open source.

Most notably around all the discussion around why the original author handed over ownership to “some random”. People questioning how “dare” the author hand over ownership. Others even suggest a conspiracy between the package author and the perpetrator.

I had co-workers approach me saying they couldn’t believe someone did that and how it was “stupid” to hand over permissions to another person.

But was it “stupid” to do so? Well no.

This is open source software that is completely free. It was relied on by hundreds of packages and it alone had over 76M downloads. Yet, despite being depended on so much, it was not financially backed and was a labour of love — as so many open source projects are.

When you are a maintainer of a project, especially if it is popular, there may be many issues but not enough time to fix them. People often simply complain that the free tool they are using often for enterprise software is not working. I myself have handed across ownership of a project and given them full write access. I still take a look at PR’s every now and again but for all intents and purposes, it is their package. I was originally contacted when someone posted an issue about how they wanted to work on it for a class project, I was delighted! Someone wanted to actually spend their time working on a project I originally authored. It never even cross my mind that they would do anything malicious and even if I did, in the politest way possible — it’s not my problem.

Rather than looking at this malicious package and thinking you either need to abandon node js or reject the whole concept of open source, show appreciation for the packages you use. Maybe get your company to give back either by either donating development time or financial resources to it. All other assets in a business are paid for — so why shouldn’t the underlying pieces of your code base be? It is as black and white as this — if the npm library disappeared, a lot of companies would be in serious trouble, yet only a small fraction of libraries receive support.

It is comforting to know that the node ecosystem is not isolated to this issue. Just last year, Equifax blamed Apache struts for a breach to their entire customer base. Apache had fixed the issue but they had failed to update their servers — yet blame was still pinned to that project. From my research, I can find no record of them donating to or sponsoring the Apache foundation — yet their net income last year was $587M.

The message is this, go and check out some packages you use a lot across your projects, see if any of their issues need help. If you do not have development time, check out how you can donate to packages with “npx thanks”. If you cannot do either of those things, leave them a star, there is even an npm package to do this. Just don’t complain about an issue in software you make money off, which you originally got for free.

Proof of authenticity

D13V — Tue, 04 Dec 2018 23:45:39 GMT

The same year I was born, the president of USA, Ronald Reagan, was giving a speech in Germany. I learned about it 31 years later when I saw it on Reddit. As he keeps talking “presidentially,” we can all hear a bang. Unfazed, Reagan comments “Missed me” and tries to continue with the speech. Indeed, it missed him. It wasn’t a gun shot at all, it was a balloon popping. Reagan was shot with a gun six years earlier and still felt the consequences.

What impressed me was the reaction of the crowd. They roared loudly. I couldn’t help but wonder, why would they react to the response, and not to the speech. After all they were there for the speech. Presumably… it was important. But the actions speak louder than words, and the audience showed they value remaining calm in the sense of danger much more than giving a speech. They showed they value the show, the performance. The high stakes.

What is it about the performance that makes it so important as to eclipse the rest of the speech? Can we create that importance and reaction at will? What would a speech be if it has multiple such performances? Would it be as momentous as MLK’s “I have a dream”?

The performance is real. Even though Reagan was a Hollywood actor, in that situation the reaction time was instantaneous. He had a split second to decide if that sound was a source of danger and whether to react. It was all about the timing. If he had taken the time to look around, to get update from Secret Service, if he had paused in any way before remarking “missed me” he wouldn’t have gotten loud ovations. Be reacting instantaneously, he gave the audience a Proof of Authenticity. If the timing was longer, he would appear less confident, and less authentic. It has to be quick enough to be authentic, to be Kahneman-and-Tversky-System-1 reaction, to be a Malcolm-Gladwell-Blink reaction. The quickness of the reaction proves to us, that there is no other though other than the one expressed.

It’s the same as the proof of work principle for cryptocurrencies, looked from a different angle. Proof of work helps us trust that the bits of a bitcoin are real, because the algorithm to compute them is so hard, nobody else would be able to invent them in our lifetime, unless they had access to the massive amounts of compute that the Bitcoin network has. In an oversimplified way - proof of work means that the result takes too long to fake. Proof of authenticy works the same way except that it pushes the amount of time down low, to the atomic limit of human abilities, where we can’t apply filters to mask their authenticity.

We seek that proof of authenticity and regard it highly in the rare chance we notice it.

400 MHz processor in a Macintosh PowerBook G3 PDQ?

David Schlachter — Tue, 04 Dec 2018 19:00:00 GMT

System Profiler reports a 400 MHz processor in a model that only came in 233, 266, and 300 MHz configurations. Is this an overclock, an upgrade card, or something else?

The "Death of Tumblr" is nothing but proof n. 99999....

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 04 Dec 2018 07:00:00 GMT

..that we MUST switch to the RIGHT kind of personal clouds NOW.

ka-yich — Tue, 04 Dec 2018 06:21:20 GMT

Creators responsibility

ciesie.com — Tue, 04 Dec 2018 02:00:00 GMT

Recently I’ve enjoyed reading blog posts by Nikita. You can find the blog here. One blog post I see quoted often is this one (you can read it in 6 languages - that says something). It’s about the disastrous state of current software development.

The author brings a lot of good points. The message is that no one puts effort into making software performant. We solve problems by adding complexity, never actually solving the root cause. This comes at a cost. It’s all a slow, bloated and complicated mess. Hard to disagree.

One million npm downloads

Muffin Man — Tue, 04 Dec 2018 02:00:00 GMT

I want to brag a little - my npm packages have been downloaded more than 1 million timesStats are coming from npm-stat.com. this year!

And I finally got a real domain:

https://muffinman.io/

I've been quiet for the last couple of months. Mostly because I was busy with other things, both work and personal. But I do have a few small side projects I will be sharing soon.

Stay tuned!

Make a Gif of a Website’s Evolution

Baptiste Fontaine’s Blog — Mon, 03 Dec 2018 21:19:00 GMT

For the latest StackExchange “time”-themed contest, I made a gif showing the evolution of StackOverflow from 2008 to today:

(click on the image to play it again)

The first step was to find a decent API for the Internet Archive. It supports Memento, an HTTP-based protocol defined in the RFC 7089 in

Using the memento_client wrapper, we can get the closest snapshot of a website at a given date with the following Python code:

from datetime import datetime, timedelta
from memento_client import MementoClient

mc = MementoClient(timegate_uri="https://web.archive.org/web/",
        check_native_timegate=False)

def get_snapshot_url(url, dt):
    info = mc.get_memento_info(url, dt)
    closest = info.get("mementos", {}).get("closest")
    if closest:
        return closest["uri"][0]

# As an example, let’s look at StackOverflow two weeks ago
url = "https://stackoverflow.com/"
two_weeks_ago = datetime.now() - timedelta(weeks=2)
snapshot_url = get_snapshot_url(url, two_weeks_ago)

print("StackOverflow from ~2 weeks ago: %s" % snapshot_url)

Don’t forget to install the memento_client lib:

pip install memento_client

Note this gives us the closest snapshot, so it might not be exactly two weeks ago.

We can use this code to loop using an increasing time delta in order to get snapshots at different times. But we don’t only want to get the URLs. We wants to make a screenshot of each one.

The easiest way to programmatically take a screenshot of a webpage is probably to use Selenium. I used Chrome as a driver; you can either download it from the ChromeDriver website or run the following command if you’re on a Mac with Homebrew:

brew install bfontaine/utils/chromedriver

We also need to install Selenium for Python:

pip install selenium

The code is pretty short:

from selenium import webdriver

def make_screenshot(url, filename):
    driver = webdriver.Chrome("chromedriver")
    driver.get(url)
    driver.save_screenshot(filename)
    driver.quit()

url = "https://web.archive.org/web/20181119211854/https://stackoverflow.com/"

make_screenshot(url, "stackoverflow_20181119211854.png")

If you run the code above, you should see a Chrome window open, go at the URL by itself, then close once the page is fully charged. You now have a screenshot of this page in stackoverflow_20181119211854.png! However, you’ll quickly notice the screenshot includes the Wayback Machine’s header over the top of the website:

This is handy when browsing through snapshots by hand, but not so much when we access them from Python.

Fortunately, we can get a header-less URL by changing it a bit: we can append id_ to the end of the date in order to get the page exactly as it was when the bot crawled it. However, this means it links to CSS and JS files that may not exist anymore. We can get a URL to an archived page that has been slightly modified to replace links with their archived version using im_ instead.

Page with header and rewritten links:: https://web.archive.org/web/20181119211854/...
Original page, as it was when crawled:: https://web.archive.org/web/20181119211854id_/...
Original page with rewritten links:: https://web.archive.org/web/20181119211854im_/...

Re-running the code using the modified URL gives us the correct screenshot:

url = "https://web.archive.org/web/20181119211854im_/https://stackoverflow.com/"
make_screenshot(url, "stackoverflow_20181119211854.png")

Joining the two bits of code we can make screenshots of a URL at different intervals. You may want to check the images once it’s done to remove inconsistencies. For example, the archived snapshots of Google’s homepage aren’t all in the same language.

Once we have all images, we can generate a gif using Imagemagick:

convert -delay 50 -loop 1 *.png stackoverflow.gif

I used the following parameters:

-delay 50: change frame every 0.5s. The number is in 100th of a second.
-loop 1: loop only once over all the frames. The default is to make an infinite loop but it doesn’t really make sense here.

You may want to play with the -delay parameter depending on how many images you have as well as how often the website changes.

I also made a version with Google (~10MB) at 5 frames per second, with -delay 20. I used the same delay as the StackOverflow gif: at least 5 weeks between each screenshot. You can see which year the screenshot is from by looking at the bottom of each image.

Invited to join the Django software foundation

Tom Forbes — Sun, 02 Dec 2018 21:58:01 GMT

A few days ago I was invited to become a member of the Django software foundation due to my contributions to Django. Awesome! Now I get to hang about in the super-secret mailing list and discuss django-related illuminati business. Removal of core developers The hot news in Django-land is the proposa...

When automation goes horribly right

utf9k — Sun, 02 Dec 2018 13:00:00 GMT

What do you do when a UI element disappears but the service powering it still persists?

All the Light We Cannot See

Apurva Shukla's RSS Feed — Sun, 02 Dec 2018 08:55:05 GMT

⭐ ⭐ ⭐ ⭐ ⭐ One word to describe this novel: illuminating. This story is rich in its atmosphere of history and emotion, tethering the frail…

The Internet of Things? Yes, but ONLY if it is OPEN and ORGANIC

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 02 Dec 2018 08:00:00 GMT

The Internet of Things as advertised today has huge security issues. But they are likely the smallest of its problems.

Swift nil-coalescing operator

xenodium.com @alvaro — Sun, 02 Dec 2018 02:00:00 GMT

Paul Hudson, over at Hacking with Swift, has written The Complete Guide to Optionals in Swift. One of the many highlights is the nil-coalescing operator. If you're a fan of the C-like syntax in ternary operations, you'd enjoy chaining with Swift's nil-coalescing operator:

let players = [ "goose": "run!" ]
let move = players["duck1"] ?? players["duck2"] ?? players["duck3"] ?? players["goose"]
print("\(String(describing: move))")

ps. Swift snippet run on Emacs org babel's ob-swift. See Multiline Swift strings for details.

How to make a wood pen

june.kim — Sun, 02 Dec 2018 02:00:00 GMT

Fountain codes and animated QR

divan's blog — Sat, 01 Dec 2018 16:55:00 GMT

^{(source: Anders Sune Berg)}

In the previous article I’ve described a weekend project called txqr for unidirectional data transfer using animated sequence of QR codes. The straightforward approach was to repeat the encoded sequence over and over until the receiver gets complete data. This simple repetition code was good enough for starter and trivial to implement, but also introduced long delays in case the receiver missed at least one frame. And in practice, it often did, of course.

Brexit means Brexit... and more high-tech pollution too

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 01 Dec 2018 12:00:00 GMT

As a consequence of Brexit, the UK will also build its own GPS-equivalent satellite system. What does this really mean?

Ocado vs Asda (org table)

xenodium.com @alvaro — Sat, 01 Dec 2018 02:00:00 GMT

Someone handed me an Ocado shopping voucher for 30% off. Sounded promising, even for a one-off.

With my Money or Your Life hat on, I took a closer look for potential savings. Results were disappointing, when compared to alternatives like Asda.

Here's a table comparing Ocado (30% off) and Asda (no discount):

                                                                                                                                                                     Ocado    Asda

Coconut Merchant Organic Raw Extra Virgin Coconut Oil 500ml 6.74 KTC 100% pure coconut oil 2.00 Waitrose Love Life Popcorn Maize 510g 1.50 Cypressa Popping Corn 2x500g = 1000g 1.50 Whitworths Ground Almonds 2.00 Whitworths Ground Almonds 1.60 Total £ 5.10 -30% £ 7.17

#+TBLFM: @8$3=vsum(@2$3..@7$3);£ %.2f::@9$2=vsum(@2$2..@7$2) * 0.7;£ %.2f

On the upside, Ocado has plenty of items I cannot find at Asda. May be a good opportunity to get these items at a discount.

Emacs org tables

Small tables are the perfect use-case for Emacs org-mode tables. Been a while since I used one, so great timing for a little refresh.

Here's the org source for the table above (prior to exporting to HTML):

|-------------------------------------------------------------+--------+--------|
|                                                             |  Ocado |   Asda |
|-------------------------------------------------------------+--------+--------|
| [[https://www.ocado.com/webshop/product/Coconut-Merchant-Organic-Raw-Extra-Virgin-Coconut-Oil/372144011][Coconut Merchant Organic Raw Extra Virgin Coconut Oil 500ml]] |   6.74 |        |
| [[https://groceries.asda.com/product/oils/ktc-coconut-hair-oil/910000033621][KTC 100% pure coconut oil]]                                   |        |   2.00 |
| [[https://www.ocado.com/webshop/product/Waitrose-Love-Life-Popcorn-Maize/25130011][Waitrose Love Life Popcorn Maize 510g]]                       |   1.50 |        |
| [[https://groceries.asda.com/promotion/2-for-pound-1.50/ls89129][Cypressa Popping Corn 2x500g = 1000g]]                        |        |   1.50 |
| [[https://www.ocado.com/webshop/product/Whitworths-Ground-Almonds/275684011][Whitworths Ground Almonds]]                                   |   2.00 |        |
| [[https://groceries.asda.com/product/baking-nuts-seeds-fruit/whitworths-ground-almonds/910000797981][Whitworths Ground Almonds]]                                   |        |   1.60 |
|-------------------------------------------------------------+--------+--------|
| Total                                                       |        | £ 5.10 |
|-------------------------------------------------------------+--------+--------|
| -30%                                                        | £ 7.17 |        |
|-------------------------------------------------------------+--------+--------|
#+TBLFM: @8$3=vsum(@2$3..@7$3);£ %.2f::@9$2=vsum(@2$2..@7$2) * 0.7;£ %.2f

Tanzania 1: prep

Micro-optimist — Sat, 01 Dec 2018 02:00:00 GMT

I climbed a mountain with Andrea and Donna. It took a lot of work.

This is a very long blog post, split into five parts. Check out the slide show first. Then, if you aren’t sure how far you’ll get, read parts 3 and 4, then 2, then maybe 1. Part 5 is mostly for me.

1: prep - 2: hike - 3: summit - 4: safari - 5: post-mortem - slide show (upcoming)

Done!

Some time in early 2018, Andrea asked me if I wanted to climb Mount Kilimanjaro. I hadn’t known this was something you could do without significant experience and equipment. It sounded scary and expensive, but after mulling it over for a while, the answer was obvious. I paid my deposit for the seven-day full moon Machame route hike with Kandoo in March, and spent the next six months researching, training, and shopping.

Why did I do it? If you need to ask, I’ll never be able to answer.

Summiting one of the highest mountains on the planet was a few levels beyond any previous adventures of mine. After booking the trip, I started looking at recommended gear lists, training regimens, and general travel advice for Africa. My todo list to prepare for the trip was huge, but it was easy to get started. Early on, the important thing was to start training as soon as possible. My legs needed practice, everything else could wait a while.

For most people doing the Kilimanjaro climb, altitude sickness, or AMS, is the biggest risk, and it’s hard to predict how that will affect you. That means being basically fit is the best thing you can do to prepare, physically. As a bike commuter, my aerobic fitness is fine, so there wasn’t much more I could do to prepare. Instead, the most important thing was to get ready to be on my feet for many hours, for days on end.

I suppose I’d call myself a moderately experienced outdoorsperson. I’ve spent plenty of time car camping, and I’m always up for a strenuous day hike to climb some modest peak. I had no backpacking experience, but I did do a seven-day self-supported bike touring trip with Kevin a few years ago. Kili is an extended backcountry camping trek full of strenuous, high-altitude hikes, and I was comfortable with each aspect of that on its own, except for the altitude.

I’m writing all of these posts after the trip, so it will include a handful of retrospective thoughts. Part 5 is fully focused on that.

Training

I had been in Austin almost ten years, and somehow just recently learned about River Place Nature Trail, the only place to hike near Austin with any appreciable elevation gain. It’s about six miles long, with 800 feet of elevation gain, mostly on well-maintained trail stairs. It’s a bit of a drive, so not usually too busy, but the narrow trail feels full regardless of how many people are there. This place became our weekend hangout for months, replacing all other outdoor activities. I dragged Lisa and Gizmo along a few times. They were NOT into it. They had more fun at Enchanted Rock, which is maybe prettier, but much shorter (length and height), much further away, and harder to get into.

Austin is proud of its hills, but that’s all we got

The River Place hikes were great for consistent training. I tried two other things, briefly. I ran three miles on the Hike and Bike trail downtown. By the second time, I was able to finish the run without stopping or walking. I think I could have kept at it and improved, but I didn’t enjoy it, and I already get plenty of aerobic exercise on my bike, so I gave it up. I also acquired an altitude training mask, and used it twice before deciding it was useless. After my first Colorado trip, I realized how bad those things are at simulating altitude effects.

More challenging, but there was just too much air to breathe

I knew hiking that same trail over and over again wasn’t enough, so I hoped to make a few short trips to climb some more respectable mountains. There are plenty of options in the US, but I’m somehow drawn to Colorado, and I ended up taking two trips there over the summer. I summited Pikes Peak (14114 ft) with Andrea and Donna in July, and Mount Bierstadt (14065 ft) with Cody in August, after a work trip to Denver. Although I’ve done serious climbing hikes in the past, these were my first real experience with AMS. Sleeping at about 6000-7000 feet, with just a few days of acclimatisation, I started to feel the effects around 12000 feet. Mostly, mild headache and shortness of breath. The big problem was my inability to pace myself - I would keep exerting myself at normal levels, when my body couldn’t handle it. I would stop for ten seconds, then push hard for another 30. I knew this was silly, but I also figured it wouldn’t be a problem on Kilimanjaro, since the guides would be setting the pace, instead of me. This turned out to be correct, more or less.

Andrea and Donna on the way to Pikes Peak

The second Colorado trip, I was car camping for three days. This was just weeks before the Kilimanjaro trip, so I had most of my gear already. I brought my day pack, and most of the stuff I planned to carry in it, but not much else - somehow, I missed this opportunity to try out all the rest of my new gear. Anyway, I learned one big thing: the day pack I had been using was fine for a couple hours, but after wearing it all day, it was very uncomfortable.

Colorado car camping con Cody

To summarize:

Find a good place for short hikes near home, and go there a lot.
Summit at least one 14er before the trip.
Do a shakedown camp/hike with as much of the gear you plan to bring as possible.

Gear

On Mount Bierstadt, I met a couple of helpful people. One had done the Kilimanjaro hike several years ago, and gave me two recommendations: get knee-length gaiters, and bring a big daypack. The other guy noticed that I looked like I was stumbling down the path, and decided to keep me company in case the AMS got the better of me. I like to think I was just being lazy and using gravity to assist me in descending faster. In retrospect, I realize it’s probably not a good idea to trust your own judgment in that sort of situation. Anyway, that guy recommended Wilderness Exchange in Denver, so I stopped there on my way to the airport, and replaced my cheap craigslist daypack (a $45 like-new Osprey Mira 26) with a $100 lightly-used Osprey Mutant 38. Incidentally, that’s my new favorite store. The consignment and manufacturer sample sections make it almost a cross between REI and Goodwill.

Anyway, back to the beginning. I read through a handful of gear recommendation lists, and found at least one person’s actual packing list, and I compiled all of those into a series of lists for myself: definitely buy, maybe buy, already have. The basics are critical: good everyday hiking clothes, packable cold-weather gear, duffel bag and day pack, boots. About three years ago, I learned about the magic of Merino wool. I started buying Merino stuff for biking - 2X Icebreaker t-shirts and 5 pairs of ultralight Smartwool socks - most of which I brought to Kilimanjaro. Living in Texas, I don’t have much use for cold-weather clothes, so this was an excuse to acquire some. The majority of new apparel I bought totaled about $300.

About three months before the trip, the sole of one of my hiking boots started coming loose. I was annoyed at the time, but I’m so glad it didn’t happen any later - plenty of time to find new boots and wear them in. I briefly debated trying to use the dying boots on the trip, which would have been an awful idea. I saw 5-10 lost soles over the course of the hike, and if that had happened to me, it would have been worse than anything else that did. So, I donated my seven-year-old boots to Project Shoehorn, and got some new mid-range waterproof Oboz boots. I considered getting higher-end Vasque or Salomon boots. Even though I expect to use them plenty, I didn’t see the point. Maybe next time.

A waterproof duffelbag is suggested, so I got both the Patagonia 60L and 90L, hoping to make the smaller one work. I ended up using the 90L and returning the 60L, but with what I know now, I’m sure I could have managed with the 60L (with my 38L daypack stuffed full for the flights).

Generally speaking, I try to avoid buying new things when possible, preferring hand-me-downs, Craigslist, Goodwill, etc. This kind of trip necessitates some specific items, that fit well, so I mostly gave up on being able to do that. Shortly before the trip, REI announced their new online used gear store, so I was happy to give that a chance. I looked through every available item, and found two things perfect for the trip - a cold-weather sleeping bag, and a decent-sized, waterproof mountaineering day pack. I ordered them, and they just never showed up! This was pretty annoying, and I’m pretty unlikely to use that shop again in the future. Oh well. I already had a three-season sleeping bag, which wasn’t warm enough, so I got a liner, which doesn’t do much. I ended up renting one of Kandoo’s sleeping bags, a Mountain Hardwear Lamina -30. This thing was huge and super warm, some nights I didn’t even zip it up.

We used Kandoo’s Mountain Hardwear Trango 3 tents. These were pretty nice. I’m glad we didn’t have to set them up and take them down ourselves.

Here’s all the new stuff I bought:

amount	item
82.27	smartwool pants base layer
15.36	darn tough hiking socks
8.99	microfiber towel
189.44	oboz hiking boots
9.90	uniqlo heattech undershirt
14.90	uniqlo rain pants
29.90	uniqlo down jacket
45.00	osprey mira day pack
21.60	REI stuff sacks
75.72	peak design capture clip
19.22	cheap wool socks
21.53	patagonia shirt
107.65	osprey mutant 38 pack
5.67	heavy thread (repair kit)
19.99	trekking poles
22.99	down hood
161.29	patagonia duffel 90L
6.50	rite in the rain notebook
48.71	smartwool boxers
250.00	70-300 lens
10.88	bandanas
11.19	inflatable pillow
24.19	black diamond headlamp
37.99	gorillapod
12.93	1.5L nalgene
31.39	buff
9.11	bottle sling
29.99	anker 10Ah battery
45.47	osprey hydration pack
75.72	burton mittens
———–	—————————-
1445.49	total

I bought a lot of it just before the trip, after deliberating on whether I needed it, and exhausting other options besides purchasing new. Most of this is self-explanatory: clothes, bags, shoes, mittens, poles, head lamp. The camera gear makes sense if you’re a photographer. That stuff makes up almost $1300 of the total, leaving just a few odds and ends. Most of that is also obvious, the only exception is the nalgene and the bottle sling - highly recommended for summit night because bladder hoses can freeze. The idea is to put the nalgene in the sling upside down, so the water freezes from the bottom. I used the nalgene bottle plenty, and I’m glad I brought it, but I didn’t use it at all on summit night. I also bought and returned over $600 worth of stuff that didn’t make the cut.

I also borrowed an inflatable sit pad, plus a bunch of stuff sacks and carabiners, from Patsy.

All my stuff, minus my ebook reader. Analysis in part 5

Photography

Photography gear

Half of it I already had - the T6i, Tamron 10-24 f/3.5-4.5, Canon 50 f/1.8, and miscellaneous accessories - and I also got a few new toys. First, a capture clip, at the recommendation of a friend. Kind of pricey, but I’m glad I got it. Second, my first telephoto lens, a Canon 70-300 IS. If a safari isn’t a good enough excuse to get a telephoto, then what is? It’s a low/midrange lens, nothing fancy, but I still got some great shots with it. Finally, a Gorillapod, which I figured I would only use for night photos. I was right, and it was totally worth it. Without it, I wouldn’t have gotten this:

I’m a ghost

I also bought a set of knockoff batteries, to ensure my camera would last until summit night. I was hesitant to use those, for fear of hurting my camera’s precious feelings, but my Canon battery lasted the entire hike anyway. Plus, both my mind and my fingers were unusable at that point, so I gave up on photos at the summit.

Packing

We put some thought into how we would pack for the different phases of the trip. Donna packed a small bag of clothes for each hike day, which I thought was overkill, and unlikely to work for me. Instead, I focused on what to put in each of my main bags for the flight, and for the hike. The safari was an afterthought, since I didn’t know anything about it anyway.

I actually used five distinct bags over the course of the trip: duffel, daypack, passport pouch, light drawstring bag, and a folding reusable shopping bag (to leave stuff behind at the hotel while hiking).

The passport pouch held my passport, yellow card, wallet, giant wad of 10000 shilling notes (basically $5 bills, which is what the ATMs dispense, I guess), and a small knife, flashlight, and pencil, and spare SD cards for my camera. I kept this around my shoulder or in my daypack most of the time, but definitely didn’t have it on my person all the time during the hike, like I should have.

For the flights, the recommendation is to carry the hiking essentials in your carry-on, so I packed my most expensive, hard-to-replace clothes and gear in my daypack (plus camera and lenses), and stuffed everything else in my duffel. I used the drawstring bag for minimal flight essentials (headphones, sleep mask, book, snacks, jacket), which worked well.

For the hike, my daypack would carry water, rain gear, one warmth layer if I wasn’t wearing it, trekking poles, snacks and toiletries, first aid kit and repair kit, and my camera and lenses. Most everything else stayed in the duffel, including all the other camera stuff. I wore hiking pants with some big pockets, which usually ended up full of snacks, toiletries, and lenses, lightening the weight in my daypack a bit.

I unintentionally ended up with a nice rainbow assortment of stuff sacks of various sizes. This was super convenient for organization, especially when trying to find something buried deep in a dark bag. Need underwear or socks? Find the yellow bag, pull it out, get what you need, toss it back in. I’ll continue using that system in the future.

Miscellaneous

My previous experience with bike touring made me worry about my knees. I often make smoothies for breakfast, and I use whey protein powder. A few months before the trip, I switched to collagen, in the hopes that it would strengthen my joints. I don’t know if there’s any science to suggest this would work, and I had no experimental control to evaluate the difference. That said, my knees (and other joints) had absolutely no trouble on the trip.

Some people recommend a level of mental preparation that I felt was unnecessary. One suggestion is a mantra. I didn’t have one, but if I had, it would have been “time is an illusion”.

Documents, etc

I brought the obvious stuff: passport, flight info, insurance info, driver’s license, credit card, debit card. Also, a few pages of info from Kandoo, and prescription info sheets. Other than that, and a small notebook, I tried to minimize paper. Instead I loaded a bunch of info onto my phone. Manuals, all the documents from Kandoo and my insurance, maps of the mountain, a handful of wikipedia page PDFs (like this one, which I read in full, one sleepless night), GPS tracks from my own hikes, and from others’ Kili experiences.

Warning: water-proof notebooks are NOT banana-proof

Mostly, I tried to condense all important information into a few pages of this little waterproof notebook that I got for the trip. Emergency contacts, flight info, travel reference info, notes on meds and AMS, and info about the various climates of the hike. After that, I dedicated one page to each day of the trip, with major itinerary points, todo items, weather forecasts, and a summary of the hike, including a rough elevation profile. Of course it was also nice to keep it accessible for taking notes. Overkill maybe; I didn’t need any of this aside from the basic trip details and emergency info, but I’m glad I had it. I just like knowing what’s going on.

I planned to bring enough American cash to cover all expenses, plus one credit card for emergencies. Andrea changed my mind (“cash isn’t protected from fraud”), and Donna brought cash, and I think Donna did it right. Stopping at an ATM was a hassle and a risk, and they dispensed 10000 shilling notes, which are worth about $5. You want to carry cash on your body, so it should be as value-dense as possible.

Some person or website recommended getting your visa early; I was under the impression this step was required. It saved a few minutes on arrival, but it cost another $30 for shipping, and it required sending my passport in the mail, which was disconcerting. I won’t bother in the future unless it’s strictly required.

Consumables

Unsuspecting prey

I’m a hungry kind of guy, so when the packing lists included “Personal snacks”, but my contact at Kandoo said it wasn’t necessary, I erred on the side of caution. I packed about 30 snack bars, maybe 5000 calories worth, plus a pound of almonds, some candied ginger, and some jelly bellys. It felt like overkill, but I brought it all anyway. Also, a fresh, full-sized bottle of the best hot sauce, chipotle tabasco.

O ye, of little faith

For toiletries, I just brought the usual, plus extra sunscreen and bug spray, two rolls of toilet paper, three packs of wet wipes, and three ounces of hand sanitizer. I’ve almost never used wet wipes or hand sanitizer before, but I figured this was a good time to start.

Finally, a first aid kit and a repair kit. I left out some gauze and medical tape, and instead brought an ace bandage and two cold compresses.

Mostly self-explanatory

Clips, straps, strips, snaps

We had guides who could probably have solved any real problems we had on the hike, so maybe not necessary. I carry stuff like this around on a normal day, so I wasn’t going to Africa without it.

Medical

I got most of the recommended immunizations, yellow fever, hepatitis B, tetanus, and typhoid, with no trouble. Rabies is recommended as well, but when I asked my travel nurse about it in an email, she didn’t even respond, so I didn’t worry about it.

With less than two weeks left before the trip, I spotted a loose pet dog wandering around outside my house, and I went out to try to help him get home. He bit me, making the tiniest puncture wound, that almost certainly carried no risk. Since he escaped and was not found by animal control, when I went to a doctor they had no choice but to start me on full post-exposure rabies immunization. If you’re unfamiliar, that means four immediate shots of immunoglobulin, spread out on the same side of my body as the bite, plus four more of the vaccine, on the other side of my body. Those vaccine shots are supposed to be given on days 0, 3, 7 and 14 after the bite, but on day 14, I would be gone. A few days of mild panic ensued, during which I failed to find any consistent information on how to adjust the schedule. The doctor giving the shots had no idea, I called other doctors and the manufacturer, checked CDC and WHO recommendations, and then gave up on finding an answer. Some internet person told me I should cancel the trip, but I decided to accept the risk of getting rabies from a pet husky in Texas, rather than forfeit thousands of dollars for the trip.

I got three prescriptions. Diamox for AMS, and atovaquone for malaria. With atovaquone, you start taking it two days before entering a risk area, and then continue for seven days after leaving. My travel nurse said only the safari portion of the trip counted as risk days, and I would be fine for the day and a half before the hike, as long as I stayed indoors at night. Although I may not have been at any real risk of contracting malaria, I definitely dealt with mosquitoes in my hotel room before the hike. After that happened, I decided to ask for enough to cover a full trip, if I ever need it in the future. On the other hand, Tommy, the traveling doctor from my group, says he never takes malaria prophylaxis. The last one was ciprofloxacin, an antibiotic for severe traveller’s diarrhea. The side effects of this stuff include “tendon rupture”, usually the Achilles tendon. Probably almost no chance it would happen to me, but I was pretty sure I’d rather suffer diarrhea than risk that anyway.

Finally, a handful of over-the-counter meds: Immodium AD, which I didn’t use, but shared with others. Pepto Bismol, which I started eating like candy after my stomach had one rough day. Ibuprofen, which I was afraid to take, for fear of masking important pain signals. I did take a couple after the summit.

Playlist

Many people prepare a summit playlist, “Ain’t no mountain high enough”, etc. I needed only one song. A few other good tracks came to me during the hike though, so I’ll record for posterity:

Higher and higher
Green typewriters 10, a sleepy track from my favorite band, because of one line: “When you’re ready to come back down, I’ll be waiting”. Probably about drugs, but really, being that far up a mountain is its own high, so not inappropriate.
Anything from The Lion King. I hadn’t realized how appropriate this would have been, as the movie is arguably “set” in Tanzania or maybe Kenya, Kili makes a brief appearance, and “Hakuna Matata” is actually a common Swahili phrase. At least, it is in the tourist experience.
Head, shoulders knees and toes. Yes the kids song. To remind you of everything that hurts. AMS headache, heavy pack on your shoulders, knees do all the climbing, toes compressed in your boots and blistering.

Expenses

The climb itself cost $2695. Depending on how you count it, the trip cost me about $8000 total. That includes everything - the three-day safari plus lodging, flights, BOTH Colorado trips, and all the gear I bought and will continue using (including $300+ of photography stuff).

Obviously, things like gear and flights will vary quite a bit from person to person. Here’s my breakdown:

amount	thing
2695.00	climb
1445.49	gear
1156.51	africa flights
954.00	safari
792.85	pikes peak trip
500.00	incidentals (tips, drinks, food, souvenirs)
105.00	vaccinations
135.81	visa application (30.81 shipping)
100	denver trip (work covered flights, cheap camping)
78.61	travel insurance
50.00	sleeping bag rental
20.00	prescriptions
120.00	misc consumables
———–	—————————————————
8153.27	total

If that seems like a lot… it is. Even without the cost of the Colorado trips and the gear, this was the most expensive thing I’ve ever done, aside from those big life milestones. I think it cost more than my wedding.

Friends: Season 2

🦉 olshansky 🦁 — Sat, 01 Dec 2018 01:06:08 GMT

Olshansky's review of Friends: Season 2

Better Call Saul: Season 4

🦉 olshansky 🦁 — Sat, 01 Dec 2018 00:52:39 GMT

Olshansky's review of Better Call Saul: Season 4

First Man

🦉 olshansky 🦁 — Sat, 01 Dec 2018 00:45:07 GMT

Olshansky's review of First Man

Coming of Age (Album)

Macleod Sawyer | Web Portal Feed — Fri, 30 Nov 2018 19:23:57 GMT

[Computing] Grow-up and grow-down technologies

devever.net/~hl — Fri, 30 Nov 2018 12:54:00 GMT

In the history of computing, there's a pattern that emerges. Technologies can be split into two broad categories: grow-up technologies and grow-down technologies.

WireGuard Setup Guide for iOS

Graham Stevens – Grh.am — Thu, 29 Nov 2018 22:27:00 GMT

16th January 2019 - I have updated the article to include DNS lookups over VPN too, where as previously they would have been going out via your ISP/WiFi/mobile provider.

WireGuard is the new kid on the block when it comes to VPNs. I could try and explain what it is and why you should switch from OpenVPN or IPsec to it, but they do a great job of that themselves:

Law of Christ is Equivalent to the Law of the Ten Commandments

Honza Pokorný — Thu, 29 Nov 2018 16:14:00 GMT

The law of Christ is identical to the law of the ten commandments. Christ did not give another law; Christ gave that law of the ten commandments. To this law He subjected Himself and has perfectly lived according to it, leaving us an example therein. He Himself is also a living law. Christ has never given liberty to transgress any of the ten commandments, be it those that pertain to murder, adultery, or theft, etc. Thus, the law of Christ is the law of the ten commandments.

— Wilhelmus à Brakel, The Christian‘s Reasonable Service, vol. 3 (Grand Rapids, MI: Reformation Heritage Books, 1992), 58.

The Complete Guide to Rails Performance: basic setup

Josh Thompson — Thu, 29 Nov 2018 15:00:00 GMT

You know the feeling.

You are excited to start a guide or a tutorial. You buy it, crack it open, and start working through the environment setup.

Then… something goes wrong. Next thing you know, you’ve spent ~~two~~ ~~three~~ too many hours debugging random crap, and you’re not even done with the introduction to the dang thing.

Oh, this has never happened to you? Must be nice.

I’m working through The Complete Guide to Rails Performance, and I’m thrilled to get learning underway.

I’ve hit a few hiccups, though. I’m not the most sophisticated user out there, so here’s a mess of problems I ran into, and the solutions I did.

Elastic Search

All of my struggles were with ElasticSearch. I’ve never used it before, so this isn’t surprising.

The RubyGems docs recommend the following:

Install Elastic Search:
Pull ElasticSearch 5.1.2 : docker pull docker.elastic.co/elasticsearch/elasticsearch:5.1.2
Running Elasticsearch from the command line:
docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:5.1.2

Docker container, exit code 137

The docker container was instantly exiting with a 137 exit code. That means its out of memory.

I fixed this in the Docker app directly, but just bumping the memory allocated to the containers, and hitting the “save/restart” button.

Now I could run:

bundle exec rake environment elasticsearch:import:all DIR=app/models FORCE=y

Which brought me to:

Elasticsearch…Unauthorized: [401]

I got

Elasticsearch::Transport::Transport::Errors::Unauthorized: [401]

Sigh. Googled around, no quick solutions.

So, I hunted through the Slack group for the course, and saw someone suggest just brew installing ElasticSearch.

Done.

brew install elasticsearch

It installs 6.5.1 by default. This isn’t the recommended version, but… we’ll carry onward.

I can run elasticsearch in the terminal now, and get a lot of output.

Trying bundle exec rake environment elasticsearch:import:all DIR=app/models FORCE=y again, and we get:

> bundle exec rake environment elasticsearch:import:all DIR=app/models FORCE=y
[IMPORT] Loading models from: app/models
[IMPORT] Processing model: Rubygem...
[IMPORT] Done

Woot!

Lets try the tests.

I’ll start memcached. Since it just hangs, filling up a terminal tab, lets send it to the background:

> memcached &

and:

bundle exec rake

Sweet. Just rows upon rows of sweet green dots…. and a failure.

DependencyTest

The error is:

Failure:
DependencyTest#test_: 
  with a Gem::Dependency that refers to a Rubygem that exists and has multiple requirements should create a Dependency referring to the existing Rubygem. 
[/Users/joshthompson/workspace/rails_speed/rubygems.org/test/unit/dependency_test.rb:96]:
Expected: "< 1.0.0, >= 0.0.0"
  Actual: ">= 0.0.0, < 1.0.0"

We’ll table this for now.

Starting the server works fine. I’ll update this post if I sort out the test failure.

If you elect to work through the course, this might save you a bit of hassle. I went way too far down the docker hole before just brew installing it. :(

download Rubygem’s production DB:

$ script/load-pg-dump -c -d gemcutter_production latest_dump

No problems.

When it came time to generate the assets, first take failed:

> RAILS_ENV=production rake assets:precompile
rake aborted!
ArgumentError: Missing `secret_key_base` for 'production' environment, set this string with `rails credentials:edit`

Swap it to > RAILS_ENV=production rake assets:precompile SECRET_KEY_BASE=foo and you’re good to go.

Aaaand…

server error on http://localhost:3000/

A message in the slack group suggested using the following for script/load-pg-dump:

script/load-pg-dump -c -d rubygems_production latest_dump

(it’s rubygems_production instead of gemcutter_production)

still failing. same server error.

Taking a look in log/production.log, I’ve got useful errors like:

F, [2018-11-29T21:30:19.469620 #4540] FATAL – : ActionView::Template::Error (PG::UndefinedTable: ERROR: relation “announcements” does not exist

So:

> RAILS_ENV=production rake db:migrate SECRET_KEY_BASE=foo

but that doesn’t work:

> RAILS_ENV=production rake db:migrate SECRET_KEY_BASE=foo
== 20090527122639 CreateRubygems: migrating ===================================
-- adapter_name()
   -> 0.0000s
-- adapter_name()
   -> 0.0000s
-- adapter_name()
   -> 0.0000s
-- create_table(:rubygems, {:id=>:integer})
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::DuplicateTable: ERROR:  relation "rubygems" already exists
: CREATE TABLE "rubygems" ("id" serial NOT NULL PRIMARY KEY, "name" character varying, "token" character varying, "user_id" integer, "created_at" timestamp, "updated_at" timestamp)

This is probably a horrible fix, but:

$ psql
# enter psql cli
\l
# list available databases
\c rubygems_production
# connect to said db
\dt
# list all relations
drop table rubygems;
# drop the table

re-run migrations and… it now fails on a different relation.

so, running the mother of all horrible commands:

> RAILS_ENV=production rake db:reset SECRET_KEY_BASE=foo DISABLE_DATABASE_ENVIRONMENT_CHECK=1

nukes the production DB, redoes everything. Don’t do this.

RAILS_ENV=production SECRET_KEY_BASE=foo rails s

works, now, but the production DB is empty, so we have to re-import it:

script/load-pg-dump -c -d rubygems_production latest_dump

after a few minutes… boot up the rails server again, and:

damnit. error page. Same complaint about missing relation.

When I swap everything out to working on development environment instead of production, it works. I’ll carry on from here for now.

I asked for a bit of help in the Slack group for the program, and got great help.

Turns out I just needed to do a bundle exec rake db:create RAILS_ENV=production, then re-run the script: script/load-pg-dump -c -d rubygems_production latest_dump

Now I can run the app in production:

RAILS_ENV=production SECRET_KEY_BASE=foo rails s

ka-yich — Thu, 29 Nov 2018 08:50:34 GMT

ka-yich — Thu, 29 Nov 2018 08:50:13 GMT

ka-yich — Thu, 29 Nov 2018 08:49:47 GMT

The winds of change for startups

D13V — Thu, 29 Nov 2018 04:20:17 GMT

Did you know that it is mathematically impossible to predict the weather 30 days in the future. Not just “its really hard and we don’t know how to do it yet”, but rather “nobody will ever, EVER, be able to do this no matter how advanced they are”. Chaos theory proves it. Even if we know physics laws perfectly, even if they are completely deterministic, and we have nearly perfect knowledge of where every single atom is on earth, we will still not be able to know whether it will rain or not in thirty days.

In a chaotic system, by definition, small changes get amplified exponentially over time, and unless we make corrective measurements as we go, we will deviate a lot. Chaotic doesn’t mean anarchic - there are still precise laws that govern that system, but these laws make small changes diverge over time. Chaotic doesn’t even mean probabilistic, or stochastic. It is perfectly possible, and easy at that, to have a deterministic chaotic system.

Weather is determined by the laws of physics, and is still chaotic. And most systems we encounter in life are. Society as a whole is chaotic. The stock market as well. And my main point today - building a company is a very chaotic experience, no matter how organized and diligent you are.

…

When building a company, things invariably will deviate from plan. Projects take longer. We discover new opportunities. Things happen to people. Personalities change. Morale changes. Problems and opportunities alike emerge from places we would have never expected initially.

There is a drift. A wind of change. Sometimes it’s quiet, sometimes it’s gusty. Over time, this wind of change causes expectations to deviate from reality. What was our plan, and our vision will not happen that way. The problem that we are solving will turn out to be way bigger, or way smaller than we expected, and we’ll adjust to something more appropriate.

Some people, such as VCs and investors, might think they can predict early on if a company will be successful. And many may have a good track record, having had successful exits and gained a lot of money. Yet, if you ask them what do the look for as indication of success, they have varied and different opinions. They all profess different leading indicators. They might look at the the founder’s experience, or potential, or attitude, or whether the business model matches the new hotness, or whether the vision is compelling. Those things all matter. Yet they matter far less than everyone gives them credit.

All of these indicators can change. The founder’s attitude and abilities can change, the trends can change, and the vision will almost certainly change. None of these are absolute. None of them are static. When we hear the phrase “vision” in corporate context, many of us roll our eyes, and consider it yet another instance of corporate bullshit. It’s meaningless, because it will be blown away by the winds of change.

What is there to do? Is it all luck? Not all. Definitely some. Quite a bit. They say luck favors the prepared mind. In my opinion, the prepared mind is agile and humble. The prepared mind knows that everything will change. The value delivered might change as the customers change their behaviors. The team’s morale can change at the smallest wins or loses. Partnerships and deals change. The competition changes. The technology changes. Even some of the values and principles which guide the team can change.

We don’t know what the future will bring. But at least we know that.

Python.format

Heitor's log — Thu, 29 Nov 2018 02:00:00 GMT

Fun with Python format().

Growing in your first software development job

Josh Thompson — Wed, 28 Nov 2018 15:00:00 GMT

I started my first software developer role a year ago. (November 2017)

This is tremendously exciting, of course, but introduces its own set of challenges, like:

I finished Turing and I’ve got a job! Oh snap. I just finished a grueling program, and my reward is I’m fit to sit at the same table with people who know so much more than me.

I have to learn a whole new codebase. Or four.

I have to meet the standards of an inexperienced-but-competent professional, which are higher than “eager student”.

So, I’ve been thinking a lot about how to grow as a developer in my job.

What does “grow as a developer” mean?

Part of growing in anything means having goals. I have career goals that may be 5+ years out, but for the next year or two, these are the goals I’m using to drive me towards my goal of being “an experienced developer””

Build deep competency with the “standard” Rails framework. I think Rails scales just fine, and I don’t want to toss a JS-heavy front-end on our application. (I’m ~~secretly~~ pulling for Turbolinks and Stimulus…)
Help guide our product through a few big changes in the coming year, and help drive decisions around those changes
Support other teams within the company
Put up a lot of PRs, close a lot of tickets. Do a lot of “work”
???

These are not great goals, as far as good goal-setting goes, but I am not worrying too much about it, because good habits tend to be more important than good goals. And I’ve got good habits!

Constraints

Here are some constraints I’m operating within:

I work on a 100% distributed team of developers, but most of the rest of the offices works in-office in Pittsburgh. (Project/Product Management, Quality Assurance, DevOps, Support, Managed Services, Sales)
My immediate team is a group of three other developers. Myself and one other dev started at the same time.
The other two developers have been on the team for about three years.
The individuals who built most of the application is no longer on the team.
there is an extreme level of trust passed to all on the team. No one is required to pair with me, or checks in with me every day to make sure I’ve committed X code or fixed Y bugs.
We’re currently without an immediate manager.
I live a balanced life. I spend time with my wife, I rock climb a lot, I’ve gotten back into running, I read a lot. I won’t spend 20 hours a week on top of my job, trying to become a better developer.

These constraints are challenges, indeed. In some ways, it might be easier to waltz into a company that has a robust mentoring program, and batch-hires new developers so everyone is passing through the mentorship pipeline with a buddy.

Sure, that would be cool. But then I’d miss out on all the fun of wrestling with this challenge myself. I’ve rarely found myself in the exact same place, moving in the exact same speed, as many others, so this is a very Josh-friendly opportunity.

What is the range of options available to me?

There’s a few potential avenues I’m exploring right now:

Pairing more with my coworkers
self-study
implement suggestions from other developers
Work-related side-projects
non-work-related side-projects
Meetups
Teaching/helping others

Pairing more with coworkers

A simplistic solution could be to go look at some company who has an established mentorship program, and try to force-fit it into our team of four. If said plan called for 15 hours a week of pairing, I could ask each of the three developers how they felt about pairing with me for five hours a week, each.

That’s almost a workday out of each week that would be lost to pairing. Maybe this is the most effective use of everyone’s time, but I’m not yet convinced that it is.

If it is the best use of time, there’s still a lot of questions about how it would work, like:

how should we pair?
Driver/Navigator? One types, the other adds insights and suggestions?
Do we use my machine or theirs?
Do we work on whatever ticket I’m working on? Whatever ticket they’re working on? both? neither?

We’re also 100% remote, so this pairing would have to be 100% Slack/Zoom/something similar. Anytime the internet dropped many packets, there would be lag and friction.

Since I’ve started, I have set up weekly 1:1s with everyone on my team. Usually technical things come up, and I get help with whatever I’m working on, but a big part of it was I wanted to get to know my team. We don’t see each other often, and I wanted to get to know them. I also sometimes “shadow” one of my coworkers for 30 minutes or an hour a week, to absorb some of how they work, without interrupting their work too much.

Self-study

There’s a lot of great books out there about software development, and classes, videos, and more.

So far, I’ve read (or am reading)

That’ a start, but only that - a start. I don’t yet feel like I’ve read that many books about software development, Ruby/Rails, or programming in general. So, since I love to read, I’ll always have a book or two going on the topic.

Courses

I done most of Avdi Grimm’s Mastering the Object Oriented Mindset

It’s been good so far. I’m learning plenty, and I’m about half-way through it, and have been writing about some of what I’ve learned

I’ve also started Nate Berkopec’s RailsSpeed course - it’s all about speeding up a Rails app, and I’m really excited about it.

Both of these have given me a lot of value. In different ways, but I’m feeling like a wiser, more confident, more competent developer because of it.

So far, though, I’ve not been able to take something directly from one of these courses and apply it at Wombat. I’m not concerned about that at all - I’ve not finished either course, and I’ll have plenty of opportunity to apply this learning.

As I encounter projects at work that are amenable to studying in more depth, and spinning into a one-off tutorial/blog post, I will capture the idea and consider a few hours of effort in that direction.

That is where a few recent blog posts have been born:

These blog posts are a great way for me to build a more robust mental model around the topic, as well as give me something to refer back to if I need to later.

This is a nice “proof of progress”, as well. It’s easy for me to feel like I’ve not made much progress in the last week, and therefore I must have made no progress in the last six months. This is a logical fallacy, but that’s how my brain works. It’s nice to have evidence of work done.

I don’t spend too much time on non-work related projects. I’d love to extend my Turing capstone project, Block Value.

I had a ton of fun with it, and it feels close to being actually good. I’d need to brush up on my Javascript a bit, though. When I’m ready to dig into Javascript, I think this project might be the way I do that. I’d like to drop in OpenStreetMaps instead of Google Maps, and drop in Leaflet.js instead of the crappy Javascript I have in there.

Meetups

I’ve been attending Boulder.rb and Denver.rb (and am considering starting a meetup in Golden!), and I’ve enjoyed those communities. Both are fairly time consuming, as I have to travel to the location mid-day to dodge rush hour, etc.

I’m giving a talk at Boulder.rb in a few weeks, and hope to keep that trend rolling.

Teaching/helping others

I like helping others. In job hunts, salary negotiations, learning programming, rock climbing, and more.

I’ve worked regularly with folks breaking into software development (Turing and non-Turing students) and I enjoy the process. It helps me stretch my empathy/teaching/understanding muscles, and I very much enjoy teaching, and seeing others make progress, so this is probably something I’ll continue for the rest of my career.

I know that helping others is a part of growing in a career, but I’m not yet seeing where this interest of mine ties into helping my own skills grow.

As our team at Wombat grows, I’ll get to step into a bit of a mentorship role there, but this opportunity brings this post full circle. I’m not yet exactly sure what I have to offer. Sure, I’ll be useful in them getting their environment set up, and wrapping their head around the code base, and making good PRs and stuff… but I’ve got a lot more I need to learn.

Additional Resources

I’ve read all of this, and more, in pursuit of upping my skills.

The Cloud Is the New OS - A Developer's Perspective

jorin.me — Wed, 28 Nov 2018 08:00:00 GMT

It already happened a few times in the history of computing that the level of abstraction the majority of us work on has been raised.

My Breakfast: A Recipe and a Ritual

Ben Overmyer's Site — Wed, 28 Nov 2018 02:00:00 GMT

I like cooking. I'm not necessarily good at it, but it's fun to experiment and try new variations.

This is a breakfast I prepare almost every morning. Each time I make it, I change something about it or try and focus on a specific technique. It's like a cooking kata. The repetition of most of it means that I'm not devoting brain time to anything other than the particular focus of the morning.

Scrambling eggs is more of an art than you'd think. The butter is key to making great scrambled eggs. So is the figure-eight pattern. Some people add water or milk to the eggs before scrambling, but this isn't necessary and I feel that it negatively impacts the texture.

Ingredients

2 eggs, scrambled
1 green bell pepper, washed, then julienned or diced
1 large white onion, diced
1 can of black beans, rinsed and drained
1 cup of chunky salsa
2 cloves of garlic, minced
2 Tbsp olive oil
1 Tbsp butter
lime juice
sea salt
various spices

Directions

Step Zero: Prepare all ingredients. Prepare your station. Mise en place is a vital part of this process.

Heat a nonstick pan on medium heat
Add the olive oil and bring up to temperature. Don't let it smoke.
Add the minced garlic. Toast it for just a few moments.
Add the onion. Sauté until caramelized.
Add the bell pepper. Sauté until slightly browned.
Add the black beans. Throw in some spices and lime juice, stir, and let cook for a couple more minutes on medium.
Reduce the heat on the veggies to low.
Heat a second pan on medium heat.
Melt the butter in the second pan. Don't let it brown.
Add two generous pinches of sea salt to the eggs and stir.
Add the eggs to the pan. Stir continuously in a figure-eight motion.
Once the eggs have set, shut off heat to both pans.
Plate. Eggs first, then veggies, then salsa.

Interesting variations that I've tried include:

garnishing with fresh cilantro
using sour cream as a final topping
stirring the eggs in a circular pattern instead of figure-eight
adding sea salt to the veggies at varying times in the cooking
adding fresh spinach to the veggies
replacing black beans with chili beans
replacing black beans with pinto beans
using garam masala for the veggies
putting all the ingredients in a flour tortilla instead of a bowl

I'll continue to experiment and explore the possibilities of this dish.

Using Coroutines

Prasham H Trivedi — Wed, 28 Nov 2018 02:00:00 GMT

Coroutine is most powerful and natural way to process multithreading code. This post is excercept of the talk given to my team.

Black Mirror - technology ethics and can it be used for wrong?

I Learnt — Tue, 27 Nov 2018 14:00:00 GMT

Nearly every technology that has been developed has been used for both good and bad. Developments that look like they could have a massive impact on civilization have usually also spun off a side effect that has a detrimental effect. This could be the latest technologies such as genetic engineering and artificial intelligence or older technologies such as the automobile.

A (long) guide to conversations

Junk Heap Homotopy — Tue, 27 Nov 2018 13:07:12 GMT

NOTE: This was originally a Reddit essay. Please excuse the crude language aka "writing to your audience".I.

Auto deploy Docker containers using a self hosted CI (Drone 1.0.0-rc.1) and a multi stage Dockerfile

Jason Raimondi on Software Engineer — Tue, 27 Nov 2018 10:00:00 GMT

Drone.io is a great, Docker based alternative to Jenkins that is a lot like a self hosted alternative to TravisCI.

For a while, Drone was relatively stable, versioned at 0.8. Recently, Drone is approaching version 1.0.0, so the API has changed a bit. Examples will be using the version 1.0.0-rc.1 markup and may change slightly as we approach the full version 1.0.0

Automating With Drone

To configure drone with your repository, you need to activate the repository inside of your running Drone Server. The installation documentation on the Drone website is straightforward, and will vary depending on the source code management system you use.

Compromised npm Package: event-stream

Thomas Hunter II — Tue, 27 Nov 2018 02:00:00 GMT

Expert Excuses for Not Writing Unit Tests

Ben E. C. Boyter — Tue, 27 Nov 2018 02:00:00 GMT

A while back I collected all of the fake Orly book covers I could find an posted them online https://boyter.org/2016/04/collection-orly-book-covers/

I also mentioned that I was seriously considering writing content for the “Expert Excuses for Not Writing Unit Tests” one. I think I had this idea at the time that it was something I could potentially sell or use to raise my profile. Since I am not a comedy writer there is a market of about 0 people in the universe who would pay for content like this. I have no idea what I was thinking.

How to get Google Home / Google Assistant to call your iOS iPhone

Subclassed — Mon, 26 Nov 2018 16:41:57 GMT

If you have an iPhone, Google Assistant and Google Home won't be able to call or ring your phone out of the box, since it's an Android-only feature. If you are not in the US, you cannot even use the workaround of calling your number. So what can you do, if you tend to misplace your phone around the house ? (ahem)

Get an account on IFTTT
Install the IFTTT app on your phone
create a new applet, click on "this" and select Google Assistant
select "say a simple phrase" and enter the details you prefer (e.g. "make a noise on my phone" or "ping my phone". Note the most common "ring my phone" or "where is my phone" are reserved by Google and won't work).
click on "then" and select VoIP Calls
enter a message the phone will tell you if you pick up (e.g. "Glad you found me!")
Save the applet.

Now, when you say that phrase to Google Home / Goole Assistant, it will ring your phone, so you can dig it out of the sofa or the Lego box (ahem).

Antilibrary - should I finish the book I am reading?

I Learnt — Mon, 26 Nov 2018 14:00:00 GMT

I read a lot of books and once I start reading a book I normally finish it. This article proposes that it may be more valuable to have a bunch of books that are unread. Not sure I completely buy in to the proposition that “Read books are far less valuable than unread ones” however maybe I won’t feel so guilty buying a book and then leaving it unread.

Vegan French Onion Dip

ryanliptak.com — Mon, 26 Nov 2018 02:00:00 GMT

Recipe for simple vegan french onion dip without a spice mix packet

Lazy loading (and preloading) components in React 16.6

Rodrigo Pombo — Mon, 26 Nov 2018 02:00:00 GMT

Fuzzball Desktop Automation

Pete's Points — Sun, 25 Nov 2018 20:53:13 GMT

Here's a screencast of my fuzzball desktop automation system. The accompanying gist is here.

Avahi "Host name conflict" in FreeBSD jail

David Schlachter — Sun, 25 Nov 2018 19:00:00 GMT

Turns out only one FreeBSD jail per network interface can run Avahi at the same time.

Die Metapher der Stadt für das Bauen und Betreiben von Softwareprodukten

Home on The Log Book of Manuel Kießling — Sun, 25 Nov 2018 14:21:00 GMT

In der Branche des Softwarebauens wird gerne mit Metaphern gearbeitet, um die eigene Tätigkeit anschaulich und greifbar zu machen. Dazu gehört auch, den handelnden Personen verschiedene Rollen zuzuweisen. Klassische Beispiele sind die Software-ArchitektInnen, die ähnlich den ArchitektInnen von Bauwerken das große Ganze im Blick haben sollen und dafür zuständig sind, grobe bis detaillierte Pläne und Rahmenbedingungen aufzustellen, in deren Grenzen solide Softwareprodukte einerseits möglichst effizient gebaut werden können, durch die aber andererseits einfache Erweiterungsfähigkeit, eine langfristige Stabilität im Betrieb, gute Wartbarkeit und hohe Sicherheit dieser Softwareprodukte gewährleistet sein soll.

Networking Essentials: Network Security

swyx's site RSS Feed — Sun, 25 Nov 2018 03:43:21 GMT

What is Traffic Engineering?

Hawaii

Avy Faingezicht — Sun, 25 Nov 2018 02:00:00 GMT

A few weeks ago, Hannah had to go to Hawaii for work, and asked if I wanted to tag along. The tickets were cheap, so I said why not! I had low expectations, beacuse how much better could the beaches be than the ones I grew up going to in Costa Rica? I'd say they are different enough, and cool in orthogonal directions - the cultural and historical context are totally different.

2018-11-25

ho.dges.online — Sun, 25 Nov 2018 02:00:00 GMT

❤️

Types of Development

Ben E. C. Boyter — Sun, 25 Nov 2018 02:00:00 GMT

Types of Development #1

Rage driven development

Where an existing application/process causes one to rage so hard you spend the next 6 weeks over-engineering a solution to solve a simple problem. Example Perl.

Types of Development #2

Test driven development

Where previous projects and death marches result in a violent reaction of over-correcting for past failures and everything must be tested dammit. Example includes any project where the goal is 100% test coverage.

Install stand-alone 1Password from App Store on iOS

David Schlachter — Sat, 24 Nov 2018 19:00:00 GMT

How to install 1Password 7 stand-alone version on your iPad or iPhone, without a subscription.

Execute org blocks as root

xenodium.com @alvaro — Sat, 24 Nov 2018 02:00:00 GMT

Been saving admin code snippets in my own org source blocks, some requiring root access. Handy for keeping tiny self-documented scripts to easily bootstrap other machines. TIL org source block's :dir argument can be used to run block as root by using tramp syntax: /:dir sudo::


As user:

#+BEGIN_SRC sh
  whoami
#+END_SRC

#+RESULTS:
: user

As root:

#+BEGIN_SRC sh :dir /sudo::
  whoami
#+END_SRC

#+RESULTS:
: root

Designing an OSR Game Part 3: Names, Backgrounds, and Playtests

Ben Overmyer's Site — Sat, 24 Nov 2018 02:00:00 GMT

This is part three in a series of posts about designing a new OSR game. You can find the previous two here:

Since the last post, I've been doing a fair amount of work on the OSR game. Enough that it now has a title: Mysteries of a Broken World. The logo above is the official one for the line.

That first bit of news is probably the most milestone-y. I've also fleshed out more of the rules, mostly around character creation. While it's going to be a work in progress for quite awhile, you can view the latest playtest version of the rules here:

Mysteries of a Broken World

The above document is regenerated every time I make a change to the manuscript. It has the generation date in the header, for reference.

Everything is still in flux. I have the classes and races roughed out, though I need to work on all of them. I ended up throwing out the magic-via-spellstones idea in favor of a more traditional spell list system. Each magic-using class only has a handful of spell lists that they can access. The lore rationale for this is that some people are attuned to a specific kind of magic, and only those attuned to it can understand and use it.

I added "backgrounds" to the game also. It's meant to give a little color to brand-new characters, and can offer some inspiration for a character's non-adventuring abilities. I may add descriptions for each background to the rules for further inspiration, or I may leave it as just a table. That remains to be seen.

The method of rolling ability scores is firmly in the game now. They are as follows:

Roll 3d6 for each ability, in order.
You may:
1. Swap two numbers once, OR
2. Re-roll all six numbers once

This gives what I feel is an appropriate amount of randomness, while also offering a way out for awful sets.

I'm toying with the idea of having an appendix in the rules that lists out several tables for random generation of character attributes like personality traits and so forth.

For the last bit of news, I'm going to start looking for people to playtest Mysteries soon (in the first couple weeks of December 2018). For inquiries on that, write to ben@overmyer.net with the subject header "Playtesting Mysteries of a Broken World."

Cross platform mobile development with Djinni

Whacky Labs — Sat, 24 Nov 2018 00:54:00 GMT

Motivation

One of my last years goal was to make a Android app. It just occurred to me that even though I had been making app for such a long time, I had never made an Android. The app that I eventually made was far from being anything but glorious. It was just a empty with a label saying Hello World. Nonetheless, it was a milestone of my career. I should also mention, that being a Java illiterate I wrote that app in C++, so most my time was actually spent fighting with the JNI.

To continue the story, this year I want to make a cross platform app. Like one codebase that runs on both iOS and Android. I’ll be using C++ as much as I can. I’ve heard good things about the Djinni from Dropbox, so I’ll try that.

Hello Djinni

I’ve already seen the CppCon 2017 and CppCon 2014 videos already. I found this awesome tutorial which I would be following and see how it goes.

The first part is setting up. If you already have Android studio set up, good, otherwise be prepared to spend half a day setting up the system. But, on the bright side everything works as documented, no random issues. The real game begins when everything is setup.

For testing the water, I cloned the Djinni from the repo and tried running it for a simple interface:

photoapp = interface +c {
  static create(): photoapp;
  get_photoapp(): string;
}

With a simple command

../djinni/src/run \
   --java-out $java_out \
   --java-package $java_package \
   --ident-java-field mFooBar \
   --cpp-out $cpp_out \
   --cpp-namespace $namespace \
   --jni-out $jni_out \
   --ident-jni-class NativeFooBar \
   --ident-jni-file NativeFooBar \
   --objc-out $objc_out \
   --objc-type-prefix $objc_prefix \
   --objcpp-out $objc_out \
   --idl $djinni_file

And after downloading a bunch of more stuff, it chokes on some random Unresolved dependencies error. Retrying with sudo does not help either. From what I understand, the Djinni is build with Scala and there is this sbt that builds the projects. When running the packaged ./src/build, the installer hangs on some dependency issues. Next step could be try building directly with sbt with

$ brew install sbt@1

Next, I would try running the installer directly.

$ sbt

This, launches the interactive shell. After loading the setting for the current Djinni project, the shell waits for me provide inputs.

> help
  tasks                                   Lists the tasks defined for the current project.

This sounds promosing.

  compile          Compiles sources.
[success] Total time: 32 s, completed Nov 24, 2018 3:22:17 PM

Finally, the war with installing Djinni seems to be over. Next up making the actual app.

Setting up the structure

The first question I’ve is to figure out how the file system looks like with a cross platform app. This is how the example app seems to be laid out.

photoapp.djinni 	; Djinni input IDL file
generated-src/		; Djinni output
handwritten-src/	; Shared implementation
android/			; Android app
ios/				; iOS app

I like this, except I would like to wrap all common parts in one directory. Something like:

shared/photoapp.djinni 		; Djinni input IDL file
shared/generated-src/		; Djinni output
shared/handwritten-src/		; Shared implementation
android/					; Android app
ios/						; iOS app

After a simple run, the ls -R looks like:

shared

./shared:
generated-src   photoapp.djinni run_djinni.sh

./shared/generated-src:
cpp  java jni  objc

./shared/generated-src/cpp:
photoapp.hpp

./shared/generated-src/java:
com

./shared/generated-src/java/com:
whackylabs

./shared/generated-src/java/com/whackylabs:
photoapp

./shared/generated-src/java/com/whackylabs/photoapp:
Photoapp.java

./shared/generated-src/jni:
NativePhotoapp.cpp NativePhotoapp.hpp

./shared/generated-src/objc:
PAPhotoapp+Private.h  PAPhotoapp+Private.mm PAPhotoapp.h

My first struggle is what editor to open the files in. I know at some point I’m going to create a Xcode project, so I can do that right now. The problem I have with editing standalone files is that they do not get any Xcode magic, no autocompletion or even good syntax highlighting. It is almost same as editing in a Notepad. We can think of the codebase composed of 3 independent components:

PhotoAppLib: C++ library
PhotoApp iOS: iOS client
PhotoApp Android: Android client

With that in place, this is how the structure looks like:

shared/photoapp.djinni 		; Djinni input IDL file
shared/generated-src/		; Djinni output
shared/lib/					; Shared implementation
android/					; Android app
ios/						; iOS app

Setting up the iOS client

Tackling this part first since I’ve know all ins and outs of Xcode. Also I believe that with every cross platform development everyone has one favorite client, for me that is iOS and Android would be more or less a side product.

A way to set up the whole project could be to simply wrap the library in its own which comes bundled with a bunch of header files. Or another way could be to include all the files required by each clients. I personally like the latter to avoid dealing with link time issues. All the issues would be discovered at compile time and after the compiler is done the rest should sail smoothly.

With all the necessary files in the project, when I do a build I get the error DJICppWrapperCache+Private.h not found. And indeed when I look in my project I don’t find that file. Maybe there is something I missed somewhere, because I do see these files somewhere in the support-lib for Djinni source, but somehow these did not made into the autogenerated project. Could be that I was supposed to bring these in manually, so let’s do that.

Turns out that was indeed the missing puzzle. This is how the iOS project looks like

And here is the text on screen

Setting up the Android client

This would be even more interesting to set up, since I’ve almost 0 experience with Android Studio or even how Java build system and runtime works. But, it seems the out of the box for C++ from Android has greatly improved since the last time I experimented with it. Last time I had to download and install the NDK and it was so much painful as if the Android people did not want anyone to even write C++. This time the set up is almost as clean as in Xcode, you just simply tick the correct boxes and a hello world app template pops up. Nice!

And on top of it the build system is now a rather familiar Cmake. I really love this new Android development environment. This is my CmakeLists.txt:

cmake_minimum_required(VERSION 3.4.1)

# Add shared files
file(
        GLOB photoapp_sources
        ../../shared/support-lib/jni/*.cpp
        ../../shared/generated-src/jni/*.cpp
        ../../shared/src/*.cpp
)

# Provide include files
include_directories(
        ../../shared/support-lib/
        ../../shared/support-lib/jni/
        ../../shared/src/
        ../../shared/generated-src/cpp/
        ../../shared/generated-src/jni/
)

# Create library from the shared files
add_library(
        photoapplib
        SHARED
        ${photoapp_sources}
)

# Link library
target_link_libraries(photoapplib)

And then after reading the same string from the shared library and rendering it on a TextView we get

Thanks

So how do I feel about Djinni so far? I would say, I did like it a lot, if I would every have to do a cross platform mobile app, I would love to do it with Djinni. I also did love the fact that how much improvements have been made in the Android IDE to support C++.

I’ll probably expand this project further to the regular PhotoApp that I try to make with every toolkit that I try, which involves fetching some data, parsing it and rendering it on screen. I’ll also post my notes here on this blog.

The entire project is available for referencing.

Thanks for reading!

Ryanair, Hamiltonian Cycles, and using graph theory to find cheap flights

JonLuca's Blog — Fri, 23 Nov 2018 23:14:51 GMT

You NEVER saw this if you are older than 27. No, 32. No...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 23 Nov 2018 12:00:00 GMT

Global warming? “Same Young, Same Young”.

Inline Swift computed properties

xenodium.com @alvaro — Fri, 23 Nov 2018 02:00:00 GMT

Via objc.io and Max Howell's retweet, TIL about Swift's inline computed properties. Another one to try on Org Babel. ‏

func greetWorld() {
 var message = "hello"
 var betterMessage: String {
   return "\(message) world"
 }
 print(betterMessage)
}

greetWorld()

Multiline Swift strings

xenodium.com @alvaro — Fri, 23 Nov 2018 02:00:00 GMT

Paul Hudson's tweet introduced me to Swift's multiline string indentation control using closing quotes. Neat!

Being an org-mode fan, I thought I'd give Swift multiline strings a try using Org Babel's ob-swift. I get to verify it and document at the same time. Win.

Swift org mode source blocks (ie. BEGIN_SRC/END_SRC) can be added as follows:

#+BEGIN_SRC swift :exports both
  print("""
       Hello World
  """)

  print("""
       Hello World
       """)
#+END_SRC

#+RESULTS:
:      Hello World
: Hello World

By pressing C-c C-c anywhere in the code block, the snippet is executed and its output captured in the RESULT block. Super handy for quickly trying out snippets and keeping as future reference.

As a bonus, the above blocks can be exported to HTML (amongst other formats). With some styling, it looks as follows:

print("""
     Hello World
""")

print("""
     Hello World
     """)

     Hello World
Hello World

The Growth-Share Matrix of Software Development

Who is Nnamdi? — Fri, 23 Nov 2018 01:45:00 GMT

Breaking the Ice for Underwater Hockey

Svedic.org — Thu, 22 Nov 2018 11:11:08 GMT

“This is for you kid,” said my uncle, “have fun!” I was 12 and I got my first mask and fins. I loved it. I would go every day to the seaside, to the rocky pier, and dive next to … Continue reading →

Prolong keyboard backlight timeout

personal code attic — Thu, 22 Nov 2018 02:00:00 GMT

Frustrated by some weird default keyboard backlight timeout on your laptop under Linux? I know how you feel, had the same frustration with my Dell laptop until I found a workaround.

PostgreSQL Upserts - INSERT or UPDATE Made Easy

BHoey — Thu, 22 Nov 2018 01:05:39 GMT

SQL upserts are a combination of an INSERT and/or UPDATE into a single database operation which allows rows to to be added or modified in an atomic, concurrent-safe way. [Continue reading...] PostgreSQL Upserts - INSERT or UPDATE Made Easy

The post PostgreSQL Upserts - INSERT or UPDATE Made Easy appeared first on BHoey.

Prefixing Git Branch With Initials

Caius Theory — Wed, 21 Nov 2018 22:30:40 GMT

Working somewhere where we prefix our branches with the creator’s initials, I sometimes forget to do so.¹ This leads to me having to rename the branch, typing out the whole name again after adding cd/ to the start of it.

Computers are meant to solve repetitive problems for us, so let’s put it to work in this case too. My ~/bin contains git current-branch, which returns the current branch name.

If we hardcode the initials, this becomes a simple command to recall from our history:²

git branch --move --force cd/$(git current-branch)

But computers are supposed to solve all repetitive work, including knowing who I am, right? Correct, my local user account knows my full name, so we can work out my initials from that. Lets lean on the id(1) command to lookup the user’s details then strip it down to just the initials.³⁴

id -F
# => "Caius Durling"

id -F | sed -Ee 's/(^| )(.)[^ ]+/\2/g' | tr 'A-Z' 'a-z'
# => cd

Bingo, we can wrap that into a subshell passed to the branch move command and we’re done in a one-liner.

git branch --move --force "$(id -F | sed -Ee 's/(^| )(.)[^ ]+/\2/g' | tr 'A-Z' 'a-z')/$(git current-branch)"

I don’t follow that policy for my personal repos, or working on forks of other people’s code. And I’m human, so I forget. ↩︎
You can also replace --move --force with -M: git branch -M newname ↩︎
On macOS you can use id -F to return the full name of the user. Doing this on other platforms is left as an exercise for the reader. ↩︎
Yes, this is an incredibly naive way to initialize a name, but it’s good enough for the people I work with. Handling edge cases is left as … you got it, an exercise for the reader. ↩︎

Going beyond the Golden Ratio.

Extreme Learning — Wed, 21 Nov 2018 12:16:42 GMT

I show that for the same reason that the golden ratio, $\phi=1.6180334..$, can be considered the most irrational number, that $1+\sqrt{2}$ can be considered the 2nd most irrational number, and indeed why $(9+\sqrt{221})/10$ can be considered the 3rd most irrational number. This blog post was featured on the front page of Hacker News a …

Continue reading "Going beyond the Golden Ratio."

Google Inbox spoofing vulnerability

Eli Grey — Wed, 21 Nov 2018 10:56:43 GMT

On May 4th, 2017 I discovered and privately reported a recipient spoofing vulnerability in Google Inbox. I noticed that the composition box always hid the email addresses of named recipients without providing a way to inspect the actual email address, and figured out how to abuse this with mailto: links containing named recipients. The link […]

KeePass vs Bitwarden: A Comparison of Free Password Managers

Mike Kasberg's Blog — Wed, 21 Nov 2018 06:00:00 GMT

It’s been nearly 2 years since I originally wrote about How I Manage Passwords with KeePass. That blog entry was inspired by Troy Hunt’s post, “The only secure password is one you can’t remember”. Using KeePass was a wonderful experience, and I’m thoroughly convinced that everyone should use a password manager. The ease of use and level of security a password manager provides is way better than anything else you could do to remember your passwords. One of the most common ways a person can be hacked is by reusing the same password on many websites. The problem is that if any website has a data breach, all the websites you used that password on are compromised. And a password manager solves this problem by using a different password on every site.

Recently, I switched from KeePass to Bitwarden. Now that I’ve used Bitwarden for several weeks, I want to compare the two most popular free password managers.

Update: It’s 2021 now, and I’ve been using Bitwarden for over two years. I’m still incredibly happy with Bitwarden, and I recommend it to anyone I can! It’s a great password manager, and I expect I’ll continue using it for a long time to come.

KeePass

KeePass is a relatively old application. KeePass 1 was originally released for Windows XP in 2003, and KeePass 2 came out of beta in 2009. It shows - KeePass feels like a Windows application from the early 2000’s. (Its icons and toolbar are a bit dated.) Although the application itself is old, it’s still actively developed, and remains one of the most popular password managers. There are a large number of plugins available and because it’s open source, it’s pretty easy to find a KeePass-compatible application for any device or operating system.

Using “KeePass” as your password manager really entails using a combination of several different applications and services to work with a KeePass password database file. I used Dropbox to sync my password database between all my devices. I used Keepass2Android to read and store passwords from my phone. And I used KeePassXC on my Linux and macOS computers. (And if you’re using KeePass, I highly recommend all of these apps.) Although there are some browser plugins available for KeePass or KeePassXC, they always seemed finicky to me and I never found one I liked, so I stuck with the desktop application. If you decide to use KeePass, you’ll have to figure out what works for you. Maybe you prefer to use Google Drive to sync it across all your devices and need to find the best iOS app. But one way or another, you can make it work.

I was originally drawn to KeePass because it’s open source, and that remains one of its strengths. An open source password manager means that security experts (and anyone else who’s interested) can check out the source code and independently verify the strength of the encryption it uses. It also means that the application will always be free, so you won’t get sucked in to an expensive payment plan.

All in all, KeePass has widespread support for different platforms and plugins (although you’ll need to figure out how to sync the file to all your devices yourself). It’s very mature and also very flexible. Although the number of applications and plugins available can be appealing, less technical users might find it confusing and difficult to set up.

Bitwarden

Bitwarden is the new kid on the block, with version 1.0 released near the end of 2016. It was designed - from the beginning - to be used in a multi-device environment and sync passwords over the web, providing an open source solution that can compete with other internet-native password managers. Although KeePass (and its derivatives) are open source, they aren’t designed to operate on the internet so they require additional services (like Dropbox or Google Drive) to make them sync across different devices. Bitwarden is an all-inclusive solution, providing a unified experience across every platform.

Bitwarden provides official desktop applications for Windows, Mac, and Linux; official phone apps for iOS and Android; and official browser plugins for Chrome, Firefox, and Safari (in addition to others). Because all the apps are officially supported by Bitwarden, there’s no need to find a 3rd-party application to support your device. And because Bitwarden has built-in support to synchronize your passwords over the internet, there’s no need to set up additional cloud storage systems. Ultimately, this leads to a smoother user experience that’s also much easier to configure.

There is a for-profit company behind Bitwarden, and they do offer different pricing plans for individuals and businesses. But as long as you’re not using it in a corporate environment, you shouldn’t have to worry about paying anything - it’s free for up to 2 users in an organization. While the pricing plans may make Bitwarden feel proprietary, it is indeed open source. And because it’s open source, it has all the same open source advantages KeePass does. If you were to pay for Bitwarden (in a corporate plan or a premium plan), you’d be paying for the company to manage running the software on the internet for you – the software itself is open source and therefore free. And in a way, this gets you the best of both worlds. You can use open source software and still have a company to lean on for support when you need it.

As I mentioned at the beginning of this article, I’m a Bitwarden user! To me, Bitwarden feels like the new version of KeePass, solving all the problems KeePass has about syncing passwords over the internet to multiple devices, and providing a nice, clean interface to use.

Comparison

I’ve been using Bitwarden for a̶l̶m̶o̶s̶t̶ ̶a̶ ̶m̶o̶n̶t̶h̶ (update: years) now, and I’m really happy with it. I’m using the browser extension and the Android app, and I love how seamless and effortless the experience is. Although my experience with KeePass was good, there were minor annoyances here and there – like trying to find the right plugin or fiddling with the browser extension. With Bitwarden, all that friction is gone. I think the biggest reasons to use KeePass have always been that it’s free (no cost) and open source, so the code can be reviewed for security. I think Bitwarden also meets those criteria, and provides a more modern password management solution with none of the drawbacks of KeePass. The simplicity of configuring Bitwarden is the nail in the coffin for me, and it was really easy to import my passwords.

KeePass	Bitwarden
Stood the test of time.	Built for syncing multiple devices.
Includes password generator.	Includes password generator.
Ported to every platform/OS imaginable.	Support for all major platforms.
Lots of plugins.	Seamless browser extension support.
Requires Dropbox to sync.	Easier to set up.
Harder for beginners to understand.	Professionally maintained.
Different application on each platform.	Seamless UI on all platforms.

While I like KeePass, I think Bitwarden is even better, and I’d encourage you to give it a try if you’re thinking about using a different password manager or trying one for the first time. And if you’re considering switching from a different password manager, like 1Password or LastPass, I think you’ll be pleasantly surprised with how nice it is to use Bitwarden. If you’re ready to give it a shot, check it out at bitwarden.com!

I’m not affiliated with KeePass nor Bitwarden, and I wasn’t compensated in any way to promote one or the other. I just like writing about useful software.

17 Pounds in 44 Days: My Adventure with Keto

Thomas Hunter II — Wed, 21 Nov 2018 02:00:00 GMT

Does confidence cause success?

D13V — Wed, 21 Nov 2018 01:07:47 GMT

Success goes hand in hand with confidence. But which causes which? Does success cause confidence, or is it the other way around?

One way to look at it is to say that successful people gain confidence through their “wins” and display it. We observe their confidence and associate it with success.

Another way is to note, that when we see someone confident, we assume they are successful, they are “doing it right”. We treat them with increased respect and deference, and that might contribute to them becoming successful.

Those two processes are not mutually exclusive. It is possible for both of them to happen at the same time. They both uh… cause each other to happen. There is no chicken and no egg. It is a self-fulfilling prophecy.

But this argument sounds like circular reasoning. If anything, we are always thought that circular reasoning is wrong and fallacious. We might think that in the case that someone is successful and confident, there has to be another phenomenon, which causes both of them to co-occur. According to our understanding, every little action, every movement has consequences, and those consequences trigger other actions after them on so on. Atoms bounce, connect into molecules, move between cells, affect organism and their actions. Aren’t we intricate biological robots? We might think that there has to be a confidence and success atom, or molecule, or cell, or action which is the real reason for the correlation. There must be a reason for the correlation of these two phenomena.

But there ain’t got to be one.

Because quantum physics said so

That’s not how our universe works, even at the most basic level.

Even if we go down to the building blocks of our universe, atoms and photons, we see confidence and success entangled. You either get both, or neither. Not one without the other.

The smallest particles in the universe have quantum behavior. And both-or-neither is the norm there, not the exception.

Part of the reason why is because particles can be in multiple states at the same time. The cat can be both dead and alive at the same time. If a particle can take states A and B, it can also be in combination state of 50% A and 50% B. Or 30% vs 70%. Or any other mix. Physicists use a fancypants word for it - superposition. It means being in multiple states at the same time.

The other part of the reason is that some particles can be entangled with others creating both or neither situations. That same particle we were just talking about can get entangled with another particle which also can be in states A and B. A twisted entanglement. It’s not the case that each particle exists by itself and can be determined independently. The system can only be a combination AA and BB, but not AB or BA. The state can be 50% AA and 50% BB. Which means that the two particles are simultaneously in both states and uncertain, and yet, determining one of them determines the other completely.

This is our universe. This is how it works. There is no hidden magic, hidden state behind it. I’m not making this shit up.

Entanglement is everywhere

If elementary particles exhibit such entanglement, why are we looking for causality in complex social systems?

Developing mutual trust is just like quantum physics. You and I either both trust each other or we both don’t trust each other. We might be uncertain which way things will end up, but they do tend to end up in a situation where both of us trust or mistrust each other. In the long term, it is very unlikely for you to trust me if I don’t trust you.

It’s the prisoner’s dilemma. We either manage to find a way to coordinate our trust, or we don’t. If I don’t trust you, then I’ll take a selfish action, which might cause you to trust me less.

If I show trust towards you, would you trust me more? Would that make either of us more successful and confident?

5 talks to get up to speed on the JAMStack

Yasha Solutions — Tue, 20 Nov 2018 13:52:09 GMT

Never heard of the JAMStack? Maybe not directly – but if you are doing web dev you have for sure heard of some of the players in this field, like Jekyll, Hugo, Jigsaw or GatsbyJS just to name a few. Indeed monoliths CMS like WordPress or Drupal (just to name these two but Wikipedia list easily a few dozen of major players across all stacks and companies) are often being criticised for being slow at scale or hard to customise and the overhead of growing beyond a certain point start to become ROI negative due to the heavy lifting you need to deploy to get the CMS to do what you want.

SRS server – what it is – and install

Yasha Solutions — Tue, 20 Nov 2018 11:52:00 GMT

What is it? source code on github From the website : SRS delivers rtmp/hls/http/hds live on x86/x64/arm/mips linux/osx, supports origin/edge/vhost and transcode/ingest and dvr/forward and http-api/http-callback/reload, introduces tracable session-oriented log, exports client srs-librtmp, with stream caster to push MPEGTS-over-UDP/RTSP to SRS, provides EN/CN wiki and the most simple architecture. Basically it is a simple way to set up and get going with an RTMP server. How to Create an SRS Server I obsviously still need to test it live.

How to Setup OBS with NGINX on Windows for RTMP Streaming + VPS Hosted by Amazon AWS

Yasha Solutions — Tue, 20 Nov 2018 11:43:40 GMT

Links - November 20, 2018

Avy Faingezicht — Tue, 20 Nov 2018 02:00:00 GMT

Writing a lisp compiler from scratch in JavaScript: 1. lisp to assembly

Notes on software development — Tue, 20 Nov 2018 02:00:00 GMT

Next in compiler basics: <! forgive me, for I have sinned >
2. user-defined functions and variables
3. LLVM
4. LLVM conditionals and compiling fibonacci
5. LLVM system calls
6. an x86 upgrade

In this post we'll write a simple compiler in Javascript (on Node) without any third-party libraries. Our goal is to take an input program like (+ 1 (+ 2 3)) and produce an output assembly program that does these operations to produce 6 as the exit code. The resulting compiler can be found here.

We'll cover:

Parsing
Code generation
Assembly basics
Syscalls

And for now we'll omit:

Programmable function definitions
Non-symbol/-numeric data types
More than 3 function arguments
Lots of safety
Lots of error messsages

Parsing

We pick the S-expression syntax mentioned earlier because it's very easy to parse. Furthermore, our input language is so limited that we won't even break our parser into separate lexing/parsing stages.

Once you need to support string literals, comments, decimal literals, and other more complex literals it becomes easier to use separate stages.

If you're curious about these separate stages of parsing, you may be interested in my post on writing a JSON parser.

Or, check out my BSDScheme project for a fully-featured lexer and parser for Scheme.

The parser should produce an Abstract Syntax Tree (AST), a data structure representing the input program. Specifically, we want (+ 1 (+ 2 3)) to produce ['+', 1, ['+', 2, 3]] in Javascript.

There are many different ways to go about parsing but the most intuitive to me is to have a function that accepts a program (a string) and returns a tuple containing the program parsed so far (an AST) and the rest of the program (a string) that hasn't been parsed.

That leaves us with a function skeleton that looks like this:

module.exports.parse = function parse(program) {
  const tokens = [];

  ... logic to be added ...

  return [tokens, ''];
};

The code that initially calls parse will thus have to deal with unwrapping the outermost tuple to get to the AST. For a more helpful compiler we could check that the entire program was actually parsed by failing if the second element of the return result is not the empty string.

Within the function we will iterate over each character and accumulate until we hit space, left or right parenthesis:

module.exports.parse = function parse(program) {
  const tokens = [];
  let currentToken = '';

  for (let i = 0; i < program.length; i++) {
    const char = program.charAt(i);

    switch (char) {
      case '(': // TODO
        break;
      case ')': // TODO
        break;
      case ' ':
        tokens.push(+currentToken || currentToken);
        currentToken = '';
        break;
      default:
        currentToken += char;
        break;
    }
  }

  return [tokens, ''];
};

The recursive parts are always the most challenging. The right paren is easiest. We must push the current token and return all tokens with the rest of the program:

module.exports.parse = function parse(program) {
  const tokens = [];
  let currentToken = '';

  for (let i = 0; i < program.length; i++) {
    const char = program.charAt(i);

    switch (char) {
      case '(': // TODO
        break;
      case ')':
        tokens.push(+currentToken || currentToken);
        return [tokens, program.substring(i + 1)];
      case ' ':
        tokens.push(+currentToken || currentToken);
        currentToken = '';
        break;
      default:
        currentToken += char;
        break;
    }
  }

  return [tokens, ''];
};

Finally the left paren should recurse, add the parsed tokens to the list of sibling tokens, and force the loop to start at the new unparsed point.

module.exports.parse = function parse(program) {
  const tokens = [];
  let currentToken = '';

  for (let i = 0; i < program.length; i++) {
    const char = program.charAt(i);

    switch (char) {
      case '(': {
        const [parsed, rest] = parse(program.substring(i + 1));
        tokens.push(parsed);
        program = rest;
        i = 0;
        break;
      }
      case ')':
        tokens.push(+currentToken || currentToken);
        return [tokens, program.substring(i + 1)];
      case ' ':
        tokens.push(+currentToken || currentToken);
        currentToken = '';
        break;
      default:
        currentToken += char;
        break;
    }
  }

  return [tokens, ''];
};

Assuming this is all in parser.js, let's try it out in the REPL:

$ node
> const { parse } = require('./parser');
undefined
> console.log(JSON.stringify(parse('(+ 3 (+ 1 2)')));
[[["+",3,["+",1,2]]],""]

Solid. We move on.

Assembly 101

Assembly is essentially the lowest-level programming language we can use. It is a human readable, 1:1 representation of the binary instructions the CPU can interpret. Conversion from assembly to binary is done with an assembler; the reverse step is done with a disassembler. We'll use gcc for assembling since it deals with some oddities of assembly programming on macOS.

The primary data structures in assembly are registers (temporary variables stored by the CPU) and the program stack. Every function in a program has access to the same registers, but convention cordons off sections of the stack for each function so it ends up being a slightly more durable store than registers. RAX, RDI, RDX, and RSI are a few registers available to us.

Now we only need to know a few instructions to compile our program (the rest of programming assembly is convention):

MOV: store one register's content into another, or store a literal number into a register
ADD: store the sum of two register's contents in the first register
PUSH: store a register's content on the stack
POP: remove the top-most value from the stack and store in a register
CALL: enter a new section of the stack and start running the function
RET: enter the calling functions stack and return to evaluating from the next instruction after the call
SYSCALL: like CALL but where the function is handled by the kernel

Function calling convention

Assembly instructions are flexible enough that there is no language-defined way to make function calls. Therefore it is important to answer (at least) the following few questions:

Where are parameters stored by the caller so that the callee has access to them?
Where is the return value stored by the callee so the caller has access to it?
What registers are saved by whom?

Without getting too far into the specifics, we'll assume the following answers for development on x86_64 macOS and Linux systems:

Parameters are stored (in order) in the RDI, RSI, and RDX registers
- We won't support passing more than three arguments
The return value is stored in the RAX register
RDI, RSI, and RDX registers are stored by the caller

Code generation

With assembly basics and the function call convention in mind, we've got enough to generate code from the parsed program's AST.

The skeleton of our compile code will look like this:

function emit(depth, code) {
  const indent = new Array(depth + 1).map(() => '').join('  ');
  console.log(indent + code);
}

function compile_argument(arg, destination) {
  // If arg AST is a list, call compile_call on it

  // Else must be a literal number, store in destination register
}

function compile_call(fun, args, destination) {
  // Save param registers to the stack

  // Compile arguments and store in param registers

  // Call function

  // Restore param registers from the stack

  // Move result into destination if provided
}

function emit_prefix() {
  // Assembly prefix
}

function emit_postfix() {
  // Assembly postfix
}

module.exports.compile = function parse(ast) {
  emit_prefix();
  compile_call(ast[0], ast.slice(1));
  emit_postfix();
};

From our pseudo-code in comments it is simple enough to fill in. Let's fill in everything but the prefix and postfix code.

function compile_argument(arg, destination) {
  // If arg AST is a list, call compile_call on it
  if (Array.isArray(arg)) {
    compile_call(arg[0], arg.slice(1), destination);
    return;
  }

  // Else must be a literal number, store in destination register
  emit(1, `MOV ${destination}, ${arg}`);
}

const BUILTIN_FUNCTIONS = { '+': 'plus' };
const PARAM_REGISTERS = ['RDI', 'RSI', 'RDX'];

function compile_call(fun, args, destination) {
  // Save param registers to the stack
  args.forEach((_, i) => emit(1, `PUSH ${PARAM_REGISTERS[i]}`));

  // Compile arguments and store in param registers
  args.forEach((arg, i) => compile_argument(arg, PARAM_REGISTERS[i]));

  // Call function
  emit(1, `CALL ${BUILTIN_FUNCTIONS[fun] || fun}`);

  // Restore param registers from the stack
  args.forEach((_, i) => emit(1, `POP ${PARAM_REGISTERS[args.length - i - 1]}`));

  // Move result into destination if provided
  if (destination) {
    emit(1, `MOV ${destination}, RAX`);
  }

  emit(0, ''); // For nice formatting
}

In a better compiler, we would not make plus a built-in function. We'd emit code for the assembly instruction ADD. However, making plus a function makes code generation simpler and also allows us to see what function calls look like.

We'll define the plus built-in function in the prefix code.

The prefix

Assembly programs consist of a few "sections" in memory. The most important of which are the text and data sections. text is a read-only section where the program instructions themselves are stored. The CPU is instructed to start interpreting from some location in this text section and it will increment through instructions, evaluating each instruction until it reaches an instruction that tells it to jump to a different location to evaluate instructions (e.g. with CALL, RET, or JMP).

To denote the text section we emit .text in our prefix before we emit our generated code.

The data section is for statically initialized values (e.g. global variables). We don't have any need for that right now so we'll ignore it.

Here is a good read with more detail on these (and other) sections.

Additionally, we need to emit an entrypoint (we'll use _main) and add a notice (.global _main) so that the location of this entrypoint is visible externally. This is important because we let gcc handle the hairier parts of generating an executable file and it needs access to the entrypoint.

So far, our prefix looks like this:

function emit_prefix() {
  emit(1, '.global _main\n');

  emit(1, '.text\n');

  // TODO: add built-in functions

  emit(0, '_main:');
}

The last part of our prefix needs to include the plus built-in function. For this, we add the first two parameter registers we agreed on (RDI and RSI) and store the result in RAX.

function emit_prefix() {
  emit(1, '.global _main\n');

  emit(1, '.text\n');

  emit(0, 'plus:');
  emit(1, 'ADD RDI, RSI');
  emit(1, 'MOV RAX, RDI');
  emit(1, 'RET\n');

  emit(0, '_main:');
}

And we're golden.

The postfix

The job of the postfix will be simple, call exit with the value of RAX since this will be the result of the last function called by the program.

exit is a syscall, so we'll use the SYSCALL instruction to call it. The x86_64 calling convention on macOS and Linux for SYSCALL defines parameters the same way CALL does. But we also need to tell SYSCALL what syscall to call. The convention is to set RAX to the integer representing the syscall on the current system. On Linux it will be 60; on macOS it is 0x2000001.

When I say "convention", I don't mean that you really have a choice as a programmer. It was arbitrary when the operating system and standard libraries chose it. But if you want to write a working program that uses syscalls or calls out to (say) glibc, you'll need to follow these conventions.

The postfix then looks like this:

const os = require('os');

const SYSCALL_MAP = os.platform() === 'darwin' ? {
    'exit': '0x2000001',
} : {
    'exit': 60,
};

function emit_postfix() {
  emit(1, 'MOV RDI, RAX'); // Set exit arg
  emit(1, `MOV RAX, ${SYSCALL_MAP['exit']}`); // Set syscall number
  emit(1, 'SYSCALL');
}

And we're set here too.

Putting it all together

We can finally write our Javascript entrypoint and run our compiler against a sample program.

The entrypoint might look like this:

const { parse } = require('./parser');
const { compile } = require('./compiler');

function main(args) {
  const script = args[2];
  const [ast] = parse(script);
  compile(ast[0]);
}

main(process.argv);

And we can call it like so:

$ node ulisp.js '(+ 3 (+ 2 1))'
  .global _main

  .text

plus:
  ADD RDI, RSI
  MOV RAX, RDI
  RET

_main:
  PUSH RDI
  PUSH RSI
  MOV RDI, 3
  PUSH RDI
  PUSH RSI
  MOV RDI, 2
  MOV RSI, 1
  CALL plus
  POP RSI
  POP RDI
  MOV RSI, RAX

  CALL plus
  POP RSI
  POP RDI

  MOV RDI, RAX
  MOV RAX, 0x2000001
  SYSCALL

Generating an executable file from the output

If we redirect the previous output to an assembly file and call gcc on it, we can generate a program we can run. Then we can echo the $? variable to see the exit code of the previous process.

$ node ulisp.js '(+ 3 (+ 2 1))' > program.S
$ gcc -mstackrealign -masm=intel -o program program.s
$ ./program
$ echo $?
6

And we've got a working compiler! The full source of the compiler is available here.

What R package for phylogenetics is the most popular?

Jonathan Chang — Tue, 20 Nov 2018 01:49:00 GMT

While writing my first R package and its associated manuscript, I needed to talk about some other R packages in the phylogenetics research community. The most obvious choice would be to just cite the ones that I actually use, but that doesn’t necessarily mean that other practicing phylogeneticists do the same. I needed to get some stats on which phylogenetics packages were actually popular, but luckily the R ecosystem has the tools to make this easy.

Skip straight to the popularity table.

Instructions

First, let’s install the CRAN Task Views package and the CRAN-logs API package. We’ll use the development version of cranlogs since it hasn’t been updated on CRAN in a while and some stuff has changed.

install.packages("ctv")
devtools::install_github("metacran/cranlogs")
library(ctv)
library(cranlogs)

We can get a list of all the CRAN task views using the available.views function. Annoyingly, there’s no way to filter and extract JUST the Phylogenetics task view, so we’ll have to write a short filter to extract it.¹

available.views()
phylo_ctv <- Filter(function(x) x$name == “Phylogenetics”, available.views())[[1]]

Now we can extract the list of packages that are associated with the “Phylogenetics” task view, and using that list of packages, query the CRAN-logs server to figure out the most popular phylogenetics packages in the last year.

phylo_ctv$packagelist
phylo_packages <- phylo_ctv$packagelist$name
output <- cran_downloads(to = “2018-10-01”, from = “2017-10-01”, packages = phylo_packages)
head(output)

Of course, what about the phylogenetics packages that aren’t in the Phylogenetics task view? Another way to view it is to consider if a package depends on the ape package. Pretty much every phylogenetics package will use ape in some form or another, so it might also be a good proxy of what a “phylo” package is.

Get a list of all the reverse dependencies of ape using devtools.²

    revdep_packages <- devtools::revdep("ape")
all_phylo_packages <- unique(c(phylo_packages, revdep_packages))
output2 <- cran_downloads(to = "2018-10-01", from = "2017-10-01", packages = all_phylo_packages)

  

By default, the output from cranlogs is the number of downloads for a given package on a given date. We want to sum up all of these counts so we have a total number of downloads per package.

Aggregate these data using dplyr.

library(dplyr)
output2 %>% group_by(package) %>% summarise(downloads = sum(count)) %>% arrange(-downloads)

Exercises

Filter the table to only include packages in both the CRAN task view and reverse dependencies list. (This will exclude e.g., ggplot2 and other arguably-peripheral packages.)
Use the lubridate package to find the ten most popular packages by year. (The CRAN logs go back to October 2012.)
Check out Brian O’Meara’s excellent work showing how popular phylogenetics packages changed over time!

Table

Here’s the full version of the table. There’s some packages in here that are only peripherally associated with phylogenetics, but it gives a good picture of what the state of the field looks like. I’ve also annotated each package with which list it came from, the CRAN Task View list or the reverse dependencies list.

	Package	Downloads	CTV?	Revdep?
1	ggplot2	5624177	✅	🚫
2	igraph	1248409	🚫	✅
3	dendextend	440772	✅	✅
4	ape	433337	✅	🚫
5	vegan	426398	✅	🚫
6	ade4	336755	🚫	✅
7	brms	90632	🚫	✅
8	phangorn	78319	✅	✅
9	adegenet	68176	🚫	✅
10	metafor	64462	✅	🚫
11	data.tree	60224	🚫	✅
12	Seurat	55905	🚫	✅
13	MCMCglmm	48496	✅	✅
14	phytools	43721	✅	✅
15	HSAUR2	40079	🚫	✅
16	HSAUR	38960	🚫	✅
17	taxize	34790	✅	✅
18	rncl	31910	✅	✅
19	aqp	30681	🚫	✅
20	pegas	29424	✅	✅
21	RNeXML	29278	✅	✅
22	rotl	28502	✅	✅
23	picante	26858	🚫	✅
24	geiger	26292	✅	✅
25	phylobase	25190	✅	✅
26	HSAUR3	24462	🚫	✅
27	FD	23007	🚫	✅
28	EpiModel	21486	🚫	✅
29	adephylo	20348	✅	✅
30	poppr	19874	🚫	✅
31	vcfR	19346	🚫	✅
32	geomorph	18793	✅	✅
33	adespatial	18084	🚫	✅
34	ggimage	16587	🚫	✅
35	BoSSA	15993	✅	✅
36	asnipe	14109	🚫	✅
37	hierfstat	13967	🚫	✅
38	caper	13847	✅	✅
39	DDD	11690	✅	✅
40	tidygraph	11612	🚫	✅
41	DHARMa	11596	🚫	✅
42	paleotree	11359	✅	✅
43	betapart	11111	✅	✅
44	polysat	10831	🚫	✅
45	phyclust	10631	✅	✅
46	MVA	10280	🚫	✅
47	GUniFrac	9007	✅	✅
48	enveomics.R	8696	🚫	✅
49	AbSim	8432	🚫	✅
50	stylo	8336	🚫	✅
51	phylolm	8172	✅	✅
52	apTreeshape	8101	✅	✅
53	BioGeoBEARS	8007	✅	✅
54	expands	7764	🚫	✅
55	mvMORPH	7579	✅	✅
56	BAMMtools	7344	✅	✅
57	sand	7014	🚫	✅
58	diversitree	6946	✅	✅
59	homals	6933	🚫	✅
60	tidytree	6378	✅	✅
61	convevol	6351	✅	✅
62	ecospat	6320	🚫	✅
63	entropart	6299	🚫	✅
64	phylotools	6182	✅	✅
65	rmetasim	6064	✅	🚫
66	rphast	5890	✅	✅
67	corHMM	5836	✅	✅
68	apex	5750	🚫	✅
69	bayou	5686	✅	✅
70	cati	5572	✅	✅
71	ouch	5520	✅	✅
72	hisse	5267	🚫	✅
73	phyloclim	5223	✅	✅
74	rdryad	5188	✅	🚫
75	dartR	5169	🚫	✅
76	SYNCSA	5102	✅	🚫
77	OutbreakTools	5057	✅	✅
78	TreeSim	4988	✅	✅
79	ALA4R	4943	🚫	✅
80	ips	4873	✅	✅
81	PCPS	4858	✅	✅
82	metacoder	4796	🚫	✅
83	OUwie	4792	✅	✅
84	aphid	4791	🚫	✅
85	brranching	4670	🚫	✅
86	warbleR	4645	🚫	✅
87	MPSEM	4639	✅	✅
88	adhoc	4599	✅	✅
89	distory	4578	✅	✅
90	Momocs	4443	🚫	✅
91	phyloTop	4426	✅	✅
92	ggmuller	4403	✅	✅
93	paleoTS	4392	✅	🚫
94	BIEN	4380	🚫	✅
95	HTSSIP	4198	🚫	✅
96	strap	4153	✅	✅
97	nodiv	4103	🚫	✅
98	BPEC	4095	🚫	✅
99	scrm	4084	🚫	✅
100	FinePop	4031	🚫	✅
101	idendr0	4020	✅	🚫
102	HMPTrees	4015	✅	✅
103	PHYLOGR	4011	✅	🚫
104	evobiR	4000	✅	✅
105	outbreaker	3931	✅	✅
106	nLTT	3925	✅	✅
107	kmer	3891	🚫	✅
108	markophylo	3885	✅	✅
109	DAMOCLES	3879	✅	✅
110	jaatha	3861	✅	🚫
111	TESS	3860	✅	✅
112	SigTree	3823	✅	✅
113	strataG	3786	🚫	✅
114	treeplyr	3732	✅	✅
115	phylogram	3726	✅	✅
116	treebase	3724	✅	✅
117	pmc	3717	✅	🚫
118	surface	3701	✅	✅
119	gamclass	3648	🚫	✅
120	TreePar	3647	✅	✅
121	PBD	3591	✅	✅
122	RAM	3546	🚫	✅
123	Rphylip	3506	✅	✅
124	expoTree	3447	✅	✅
125	HyPhy	3419	✅	✅
126	adiv	3393	✅	✅
127	coalescentMCMC	3384	🚫	✅
128	kdetrees	3330	✅	✅
129	adaptiveGPCA	3325	🚫	✅
130	MAGNAMWAR	3324	🚫	✅
131	phylocanvas	3302	✅	✅
132	iteRates	3301	✅	✅
133	BBMV	3295	🚫	✅
134	CommEcol	3246	🚫	✅
135	netdiffuseR	3191	🚫	✅
136	pastis	3155	✅	✅
137	AnnotationBustR	3148	🚫	✅
138	phyloland	3129	✅	✅
139	phyext2	3119	✅	✅
140	Canopy	2992	🚫	✅
141	RPANDA	2952	🚫	✅
142	BMhyb	2935	🚫	✅
143	phylopath	2912	🚫	✅
144	GLSME	2846	🚫	✅
145	phylotate	2846	🚫	✅
146	phylosignal	2820	🚫	✅
147	shazam	2754	🚫	✅
148	harrietr	2707	🚫	✅
149	prioritizr	2705	🚫	✅
150	BarcodingR	2657	🚫	✅
151	msaR	2552	🚫	✅
152	mvSLOUCH	2522	🚫	✅
153	bcRep	2472	🚫	✅
154	colordistance	2448	🚫	✅
155	treeman	2417	🚫	✅
156	sharpshootR	2409	🚫	✅
157	BMhyd	2408	🚫	✅
158	aptg	2385	🚫	✅
159	qlcData	2383	🚫	✅
160	sensiPhy	2344	🚫	✅
161	GrammR	2229	🚫	✅
162	treespace	2219	🚫	✅
163	dispRity	2217	✅	✅
164	metricTester	2205	🚫	✅
165	evolqg	2181	🚫	✅
166	geomedb	2160	🚫	✅
167	PhyloMeasures	2139	🚫	✅
168	CNull	2107	🚫	✅
169	taxlist	2098	🚫	✅
170	pez	2079	🚫	✅
171	phyreg	2061	✅	🚫
172	structSSI	2047	🚫	✅
173	MiSPU	2039	🚫	✅
174	dcGOR	2031	🚫	✅
175	lefse	2020	🚫	✅
176	SeqFeatR	2006	🚫	✅
177	HAP.ROR	1970	🚫	✅
178	symmoments	1932	🚫	✅
179	genBaRcode	1923	🚫	✅
180	PhylogeneticEM	1912	🚫	✅
181	windex	1900	✅	✅
182	phylocurve	1875	🚫	✅
183	MonoPhy	1865	🚫	✅
184	TreeSimGM	1844	🚫	✅
185	spider	1840	🚫	✅
186	Rsampletrees	1837	🚫	✅
187	Rphylopars	1830	🚫	✅
188	graphscan	1829	🚫	✅
189	recluster	1811	🚫	✅
190	paco	1804	🚫	✅
191	phylosim	1768	🚫	✅
192	ecolottery	1723	🚫	✅
193	outbreaker2	1708	🚫	✅
194	STEPCAM	1697	🚫	✅
195	primerTree	1691	🚫	✅
196	PhySortR	1675	🚫	✅
197	gquad	1673	🚫	✅
198	gromovlab	1669	🚫	✅
199	indelmiss	1666	🚫	✅
200	phybreak	1662	🚫	✅
201	msap	1659	🚫	✅
202	rase	1629	🚫	✅
203	rdiversity	1629	🚫	✅
204	perspectev	1617	🚫	✅
205	ML.MSBD	1614	🚫	✅
206	sidier	1611	🚫	✅
207	pcrcoal	1588	🚫	✅
208	StructFDR	1586	🚫	✅
209	idar	1579	🚫	✅
210	PIGShift	1534	🚫	✅
211	jrich	1511	🚫	✅
212	TotalCopheneticIndex	1509	🚫	✅
213	subniche	1508	🚫	✅
214	Plasmidprofiler	1495	🚫	✅
215	TKF	1487	🚫	✅
216	rwty	1485	🚫	✅
217	TreeSearch	1474	🚫	✅
218	PhyInformR	1402	🚫	✅
219	skeleSim	1400	🚫	✅
220	insect	1383	🚫	✅
221	treeDA	1344	🚫	✅
222	multilaterals	1337	🚫	✅
223	CollessLike	1304	🚫	✅
224	vhica	1299	🚫	✅
225	motmot.2.0	1115	🚫	✅
226	treedater	926	✅	✅
227	ratematrix	813	✅	✅
228	PVR	772	🚫	✅
229	P2C2M	749	🚫	✅
230	metaboGSE	747	🚫	✅
231	RRphylo	692	🚫	✅
232	ggrasp	677	🚫	✅
233	CommT	634	🚫	✅
234	FossilSim	529	🚫	✅
235	POUMM	513	🚫	✅
236	rhierbaps	391	🚫	✅
237	RPS	390	🚫	✅
238	balance	0	🚫	✅
239	hillR	0	🚫	✅
240	kmeRs	0	🚫	✅
241	phylocomr	0	🚫	✅
242	rr2	0	🚫	✅
243	slouch	0	🚫	✅

Note that we can’t use the typical filtering mechanism using the single square bracket [ because of the way lists work. In particular, there’s no good destructuring syntax for lists-of-lists as there are for simple vectors. See ?Extract for more details. ↩
The builtin package tools also has it, but it only returns packages that you have currently installed. ↩

RTMP & Transcoding

Yasha Solutions — Mon, 19 Nov 2018 11:24:57 GMT

Some research I have made recently while working on a broadcasting module of an app. Just saving here some of the best link I have found – as backup and maybe of interest for some people. Obviously if you are looking for a super structured article – I am far from it at this point – but that’s still the filtered version of a few hours of research to find the relevant and usable stuff – and it is practically what I have used to get to the point of a transcoding version of ffmpeg with facebook live (next step is to turn it into a nodejs microservice)

Julia exists, and works at Amazon

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 18 Nov 2018 20:00:00 GMT

“Julia was twenty-six years old, and worked on the novel-writing machines in the Fiction Department…”.

Plotting the Evolution of Movies Durations (1916-2018)

Baptiste Fontaine’s Blog — Sun, 18 Nov 2018 15:09:00 GMT

Two weeks ago I created and put online a box plot of 73k movies durations across the time.

It started with a question: “Are movies getting longer and longer?”. Spoiler: Not really, except maybe in the last couple of years.

I used Wikidata’s online query service to export all movies then filtered those with both a publication date and a duration. This gave me a large JSON which I processed using Python in order to extract a couple numbers for each year: min, max, median, first and third quartiles.

The result fits in a small JSON file, which I then used to build a D3 using a few lines of JS. I used colorbrewer2 to find a colorblind-safe color palette.

You can see the result as well as the JS code on Observable.

To avoid outliers such as “Modern Times Forever” (240 hours) or “The Burning of the Red Lotus Temple”, I used the interquartile range (IQR) to limit the size of the bars: any movie whose duration is below Q1-1.5×IQR or above Q3+1.5×IQR (where Q1 is the first quartile and Q3 the third one) is not shown.

Results

As one can see on the graph, the median duration quickly rises from 50 to 95 minutes from the 1920s to the 1960s, then doesn’t move much except in the last two years.

Limitations

The first obvious limitation is the data: Wikidata has 200k+ movies but only 73k have both a publication date and a duration. It’s not complete enough to let me filter by movie type; e.g. feature film vs. others.

IMDb lists 5.3M titles (most of which are TV episodes), but there’s no way to export them all.

In the end, there’s no way to know how representative Wikidata’s movies dataset is. It does give a hint, but this graph is not a definitive answer to the original question.

See the Python code.

Animated QR data transfer with Gomobile and Gopherjs

divan's blog — Sun, 18 Nov 2018 12:55:00 GMT

TL;DR: a weekend project for transferring data via animated QR codes, written in Go and using fountain erasure codes. The Go code is reused for mobile apps using Gomobile, and in a web application for automating testing QR codes parameters, built with GopherJS and Vecty framework.

I’ll share my experience building it, some code and benchmark results of using animated QR as a data transfer method.

Journal Cover

june.kim — Sun, 18 Nov 2018 02:00:00 GMT

Les bizarreries du langage C

ache: Blog personnel — Sun, 18 Nov 2018 02:00:00 GMT

Le C est un langage à la syntaxe simple. Les seules complexités de ce langage viennent du fait qu'il agit de manière proche de la machine. Pourtant, une partie des syntaxes autorisées par le C n'est pratiquement jamais enseignée. Attaquons-nous à ces cas mystérieux ! 🧞

Networking Essentials: Traffic Engineering

swyx's site RSS Feed — Sun, 18 Nov 2018 00:57:56 GMT

What is Traffic Engineering?

Cicada 3301: An Internet Mystery

Yasha Solutions — Sat, 17 Nov 2018 07:02:16 GMT

Cicada 3301 is an organization who used puzzles to possibly recruit codebreakers/linguists from the public. The first internet puzzle started on January 4, 2012, and ran for approximately one month. A second round began one year later on January 4, 2013, and a third round following the confirmation of a fresh clue posted on Twitter on January 4, 2014.

MongoDB Schema design

Yasha Solutions — Sat, 17 Nov 2018 06:55:57 GMT

MongoDB Schema Design – Many small documents or fewer large documents? Source : Stackoverflow Modeling One-to-Few An example of “one-to-few” might be the addresses for a person. This is a good use case for embedding – you’d put the addresses in an array inside of your Person object. One-to-Many An example of “one-to-many” might be parts for a product in a replacement parts ordering system. Each product may have up to several hundred replacement parts, but never more than a couple thousand or so.

Quickly swapping elfeed filters

xenodium.com @alvaro — Sat, 17 Nov 2018 02:00:00 GMT

I seem to be more efficient in getting through rss feeds by individually browsing through related content. That is, I can get through all Emacs entries a lot faster if I look at Emacs content exclusively, instead of mixing with say BBC news. Elfeed filters are great for filtering related content.

I wanted a way to easily switch through my typical categories of related content by quickly changing elfeed filters using a completion framework.

Emacs's completing-read plays nicely with your favorite completing framework (mine is ivy). With a couple of functions, we can get Emacs to ask us for the filtering category using human-readable options and quickly presenting related content. Binding the new functionality to <tab> is working well for me.

(use-package elfeed :ensure t
  :commands elfeed
  :bind (:map elfeed-search-mode-map
              ("<tab>" . ar/elfeed-completing-filter))
  :config
  (defun ar/elfeed-filter-results-count (search-filter)
    "Count results for SEARCH-FILTER."
    (let* ((filter (elfeed-search-parse-filter search-filter))
           (head (list nil))
           (tail head)
           (count 0))
      (let ((lexical-binding t)
            (func (byte-compile (elfeed-search-compile-filter filter))))
        (with-elfeed-db-visit (entry feed)
          (when (funcall func entry feed count)
            (setf (cdr tail) (list entry)
                  tail (cdr tail)
                  count (1+ count)))))
      count))

  (defun ar/elfeed-completing-filter ()
    "Completing filter."
    (interactive)
    (let ((categories (-filter
                       (lambda (item)
                         (> (ar/elfeed-filter-results-count (cdr item))
                            0))
                       '(("All" . "@6-months-ago +unread")
                         ("BBC" . "@6-months-ago +unread +bbc")
                         ("Dev" . "@6-months-ago +unread +dev")
                         ("Emacs" . "@6-months-ago +unread +emacs")
                         ("Health" . "@6-months-ago +unread +health")
                         ("Hacker News" . "@6-months-ago +unread +hackernews")
                         ("iOS" . "@6-months-ago +unread +ios")
                         ("Money" . "@6-months-ago +unread +money")))))
      (if (> (length categories) 0)
          (progn
            (ar/elfeed-view-filtered (cdr (assoc (completing-read "Categories: " categories)
                                                 categories)))
            (goto-char (window-start)))
        (message "All caught up \\o/")))))

We don't actually need two functions, but ar/elfeed-filter-results-count enables us to list only those feeds that actually have new content. The list will shrink as we get through our content. When no content is left, we get a little celebratory message.

Quadratic Error Functions Explainer

Matt Keeter — Sat, 17 Nov 2018 02:00:00 GMT

Matrix math extravaganza!

Setting up a PiHole

ciesie.com — Sat, 17 Nov 2018 02:00:00 GMT

What’s a PiHole? <

It’s basically a DNS sinkhole. DNS servers resolve domain names into IPs. DNS sinkholes are basically faulty DNS servers. For example when you go to a website your web browser will also download ads. In order to get them it has to resolve the domain name of the ads provider into an IP. That’s what DNS server does. PiHole is just a local DNS server which say ’nah’ when you ask it to serve a content from a blacklisted provider. You can add domains to the blacklist banning their content but you can also whitelist those you want to support.

Here's what a BIG gap between manufacturing and reality looks like

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 16 Nov 2018 20:00:00 GMT

Here’s a report painting a VERY incomplete picture of “what a manufacturing skills gap of more than 2 million people will look like”.

Demistifying blockchain - Part 2

GeekWare - Daniel Pecos Martínez — Fri, 16 Nov 2018 00:10:00 GMT

In the previous post we explained from a theoretical point of view how a block chain works. In this post we will get down to work and will implement a working blockchain in Go.

If you haven’t read it yet, we recommend you to do it now before continuing. It’ll provide you the basic concepts needed to understand the examples below.

(You can find the complete example in this Github repository: https://github.com/dplabs/demistifying-blockchain)

First of all, let’s define what a block looks like for us:

type Block struct {
  Number int
  Timestamp time.Time
  Data string
  Nonce int
  Hash string
  Previous string
}

https://github.com/dplabs/demistifying-blockchain/blob/master/block.go#L14

Where:

Number: is used to identify a block in the chain. In our case it will be the position this block has in the chain. Note that the attribute is just used to have a user friendly reference to the block, not to actually link blocks between them.
Timestamp: when this block was created.
Data: the actual information this block contains.
Nonce: field used to for the hash of the block to meet certain rules. More on this later.
Hash: the result of applying a cryptographic hash function to this block. It has to take into account all the fields in the block (but Hash itself) in order to be able to detect tampering attempts to its content.
Previous: hash of the previous block in the chain. This is the actual field linking blocks in the chain (and in our example, also the position in the array of blocks).

Now let’s define an structure that will help us handle the block chain:

type BlockChain struct {
	Blocks    map[string]*Block
	Challenge string
	LastBlock *Block
}

https://github.com/dplabs/demistifying-blockchain/blob/master/blockchain.go#L8

Where:

Blocks: is the actual collection of blocks. Contiguous blocks are stored in contiguous positions in the array, although this is just an implementation detail, because the actual linkage should be done referencing their hashes.
Challenge: represents the level of difficulty the resulting hash has to meet in order to be considered valid. In our case we will say a block has is valid if it starts with the Challenge substring. The longer this challenge is, the hardest and more time consuming mining a block would be.
LastBlock: is a pointer to the latest block added to the chain, used to make computation easier.

So far we have presented two quite interesting fields: Nonce for a Block, and Challenge for a BlockChain, and the deserve a more in depth explanation. But first let’s talk about what a valid hash or signature for a block is, and for that we have to talk a little bit about Bitcoin.

When Bitcoin was designed, in order to achieve value as a coin, the concept of scarcity had to be taken into consideration: if a resource is easy to get, probably it won’t be valuable at all. The more difficult or unique the resource is, the more value people will give it. But being a digital currency, this was a not easy to solve challenge.

That’s how the concept of mining a block came into the picture: same way as gold is extracted from the ground (with lots of effort), a block has to require a lot of (artificial) effort to be added to the chain. Whomever manages to first mine a block, gets rewarded with coins in exchange for his effort.

In Blockchain, mining a block consist on calculating its hash, as we have explained. But usually hash functions are quite fast, so in order to make it harder (and thus create scarcity), a valid hash has to comply with a set of rules. In our implementation, the rule we’ve chosen is that the resulting hash has to begin with the Challenge substring defined while initializing the chain.

So in order to mine a block in our chain, the resulting signature of a block has to start with a certain string (in our example we’ve chosen it to be '0000'). So the algorithm for mining a block would first calculate current hash of the block and if the results fits the criteria of a valid hash, then we could add a successfully mined block to the chain. Otherwise we would have to change something in the block and repeat the process, and this is important, because if we don’t change anything in the block, the resulting hash will be the same. And thats what Nonce field is for. We will increment this number on every attempt to get a valid hash, until when eventually, we will get a hash matching the criteria. That’s why the longer the Challenge string is, the harder to mine a block will be.

func (block *Block) Mine(challenge string) string {

	fmt.Printf("Mining block %d... ", block.Number)

	tsStart := time.Now()

	block.Hash = ""
	block.Nonce = -1

	for done := false; !done; done = strings.HasPrefix(block.Hash, challenge) {
		block.Nonce = block.Nonce + 1
		block.Hash = calculateHash(block)
	}

	tsEnd := time.Now()
	elapsedTime := tsEnd.Sub(tsStart)

	p := message.NewPrinter(language.English)
	p.Printf("Valid hash found in %s! (hash: %s, nonce: %d)\n", elapsedTime.String(), block.Hash, block.Nonce)

	return block.Hash
}

https://github.com/dplabs/demistifying-blockchain/blob/master/block.go#L27

As you can see in the following execution of our demo, it takes between 10 and 30 seconds to find a valid hash (starting with ‘0000’) for every block. If you try this yourself, you will get different results as the timestamp of a block is taken into consideration for mining it (https://github.com/dplabs/demistifying-blockchain/blob/master/block.go#L51):

$ go run .
Blockchain initialized with challenge '0000'

Mining block 0... Valid hash found in 10.943824518s! (hash: 0000ociyEZOQk8ekRZ5VXcNJPLY=, nonce: 18,158,423)
Mining block 1... Valid hash found in 11.316462656s! (hash: 0000Dk0duKeCCHvDKuyFepXgUNc=, nonce: 15,208,609)
Mining block 2... Valid hash found in 28.805735271s! (hash: 0000O104QmKwfp2tVJ2JowVd9DI=, nonce: 37,840,116)

Resulting blockchain (in reversed order):

Block 2: {
   ts:          2018-11-10 15:17:40.602154 +0100 CET m=+22.260933223
   nonce:       37840116
   hash:        0000O104QmKwfp2tVJ2JowVd9DI=
   previous:    0000Dk0duKeCCHvDKuyFepXgUNc=
}

Block 1: {
   ts:          2018-11-10 15:17:29.285761 +0100 CET m=+10.944444407
   nonce:       15208609
   hash:        0000Dk0duKeCCHvDKuyFepXgUNc=
   previous:    0000ociyEZOQk8ekRZ5VXcNJPLY=
}

Block 0: {
   ts:          2018-11-10 15:17:18.341964 +0100 CET m=+0.000555870
   nonce:       18158423
   hash:        0000ociyEZOQk8ekRZ5VXcNJPLY=
   previous:
}

We encourage you to play with this code. You can download from the repository and compile and execute it yourself: https://github.com/dplabs/demistifying-blockchain

In the following and last post of this series, we will implement some benchmarks to analyze how much time it requires to mine a block based on the difficult we set.

AI deleting human bodies

Blogs on Michail Rybakov — Thu, 15 Nov 2018 16:07:46 GMT

I used neural networks to recognize and remove human bodies from videos, simultaneously trying to fill the void they left. Why? Well, as we are externalizing not just our bodily actions to tools, but our decision process to all-knowing, all-loving AIs, the traditional, perceivable image of what constitutes a human body has become obsolete. It seems oddly fitting in that case, for disembodied neural networks to remove our outdated bodily image.

Mira Hirtz performed for the camera. I then used Mask_RCNN to do body recognition and Generative Image Inpainting with Contextual Attention to fill in the emptiness.

This was a test for Bruno Latour’s upcoming exhibition on the Critical Zone. Made in November 2018.

On Satoshi

nutcroft — Thu, 15 Nov 2018 02:00:00 GMT

"I've been working on a new electronic cash system that's fully peer-to-peer, with no trusted third party."
- Satoshi Nakamoto <satoshi@vistomail.com> Fri Oct 31 14:10:00 EDT 2008

I'm irresistibly amazed at the technological marvel of Bitcoin.

As if the programmer of our simulation descended in their virtual creation with the name of Satoshi Nakamoto. They lived 2 human years, created something that will change the course of humanity, and then reascended back out our universe and into "reality".

The timeline is eerily elegant:

Aug 2008: bitcoin.org domain registration.
Oct 2008: Satoshi posts on cryptography@metzdowd.com one of the most legendary papers of all time, then titled: Bitcoin P2P e-cash paper
Jan 2009: Satoshi releases Bitcoin v0.1 on Sourceforge and mines the genesis block on January 3, 2009, 18:15:05 UTC.
Nov 2009: Satoshi creates the bitcointalk forum.
Dec 2010: The last we hear of Satoshi. Last active on bitcointalk on December 13, 2010. Same day of his final post on the bitcoin mailing list (Bitcoin 0.3.19 is released)

"The Times 03/Jan/2009 Chancellor on brink of second bailout for banks."
Satoshi's proof that the genesis block was mined no earlier than January 3, 2009.

Satoshi was probably male and living in Japan. It has been claimed that when bitcoin.org was registered on August 18, 2008, the registrant used a Japanese anonymous registration service, and hosted it using a Japanese ISP.

Regarding the bitcoin code, it has been deemed by the community that his work resembles someone's with strong theoretical knowledge of cryptography, probably from academia, rather than a software engineer.

Satoshi listed his birthdate as April 5, 1975. Probably a random date, but there is wide speculation as to what this date means. Remember, bitcoin conspiracy theory 101: Satoshi does nothing in random.

Regardless, according to that birthdate, Satoshi would be 34 years old in 2009. His masterful knowledge of C++, cryptography and distributed concepts reveal only part of his genius. What he managed via Bitcoin is the use of existing technologies in such a way that he created something so radical that changed the journey of current technology. Everybody in tech now talks about blockchain. The reason is the work of one person. He devoted—only—2 years of his life for this.

It is, indeed, amazing that every technology Satoshi used as part to form Bitcoin existed for years. However, no one had thought to combine them is such a unique way; at least until then. Historically, it is common for minds at the forefront of human intelligence to discover the same new things independently. Isaac Newton and Gottfried Leibniz independently discovered calculus at the same time, in the mid-17th century, a famous instance of the theory.

But, this is not the case for Bitcoin. No one seems to have been close, even though the impact of this technology is Brobdingnagian.

I secretly hope the real identity of Satoshi Nakamoto will never be known, thus leaving the story of how Bitcoin came to be, an actual legend.

"I would be surprised if 10 years from now we're not using electronic currency in some way, now that we know a way to do it that won't inevitably get dumbed down when the trusted third party gets cold feet."
- Satoshi Nakamoto <satoshi@vistomail.com> Fri Jan 16 11:03:14 EST 2009

---

Notes

The Complete Satoshi. Everything he wrote, for us to obsess over. https://satoshi.nakamotoinstitute.org/
An excellent analysis of Bitcoin in respect to its shortcomings by the one and only, Gwern Branwen. http://www.gwern.net/Bitcoin-is-Worse-is-Better
Domain history of bicoin.org by Security Trails. https://securitytrails.com/domain/bitcoin.org/history/a

Ralph Breaks the Internet

Code & Visuals — Thu, 15 Nov 2018 02:00:00 GMT

The Walt Disney Animation Studios film for 2018 is Ralph Breaks the Internet, which is the sequel to 2012’s Wreck-It Ralph. Over the past two years, I’ve been fortunate enough to work on a number of improvements to Disney’s Hyperion Renderer for Ralph Breaks the Internet; collectively, these improvements make up perhaps the biggest jump in rendering capabilities that Hyperion has seen since the original deployment of Hyperion on Big Hero 6. I got my third Disney Animation credit on Ralph Breaks the Internet!

Over the past two years, the Hyperion team has publicly presented a number of major development efforts and research advancements. Many of these advancements were put into experimental use on Olaf’s Frozen Adventure last year, but Ralph Breaks the Internet is the first time we’ve put all of these new capabilities and features into full-scale production together. I was fortunate enough to be fairly deeply involved in several of these efforts (specifically, traversal improvements and volume rendering). One of my favorite things about working at Disney Animation is how production and technology partner together to make our films; we truly would not have been able to pull off any of Hyperion’s new advancements without production’s constant support and willingness to try new things in the name of advancing the artistry of our films.

Ralph Breaks the Internet is our first feature film to use Hyperion’s new spectral and decomposition tracking [Kutz et al. 2017] based null-collision volume rendering system exclusively. Originally we had planned to use the new volume rendering system side-by-side with Hyperion’s previous residual ratio tracking [Novák 2014] based volume rendering system [Fong 2017], but the results from the new system were so compelling that the show decided to switch over to the new volume rendering exclusively, which in turn allowed us to deprecate and remove the old volume rendering system ahead of schedule. This new volume rendering system is the culmination of two years of work from Ralf Habel, Peter Kutz, Patrick Kelly, and myself. We had the enormous privilege of working with a large number of FX and lighting artists to develop, test, and refine this new system; specifically, I want to call out Jesse Erickson, Henrik Falt, and Alex Nijmeh for really championing the new volume rendering system and encouraging and supporting its development. We also owe an enormous amount to the rest of the Hyperion development team, which gave us the time and resources to spent two years building a new volume rendering system essentially from scratch. Finally, I want to underscore that the research and underpins our new volume rendering system was conducted jointly between us and Disney Research Zürich, and that this could not have happened without our colleagues at Disney Research Zürich (specifically, Jan Novák and Marios Papas); I think this entire project has been a huge shining example of the value and importance of having a dedicated blue-sky research division. Every explosion and cloud and dust plume and every bit of fog and atmospherics you see in Ralf Breaks in the Internet was rendered using the new volume rendering system! Interestingly, we actually found that while the new volume rendering system is much faster and much more efficient at rendering dense volumes (and especially volumes with lots of high-order scattering) compared to the old system, the new system actually has some difficulty rendering thin volumes such as mist and atmospheric fog. This isn’t be surprising, since thin volumes require better transmittance sampling over better distance sampling and null collision volume rendering is really optimized for distance sampling. We were able to work with production to come up with workarounds for this problem on Ralph Breaks the Internet, but this area is definitely a good topic for future research.

Ralph Breaks the Internet is also our first feature film to move to exclusively using brute force path-traced subsurface scattering [Chiang 2016] for all characters, as a replacement for Hyperion’s previous normalized diffusion based subsurface scattering [Burley 2015]. This feature was tested on Olaf’s Frozen Adventure in a limited capacity, but Ralph Breaks the Internet is the first time we’ve switched path-traced subsurface to being to default subsurface mode in the renderer. Matt Chiang, Peter Kutz, and Brent Burley put a lot of effort into developing new sampling techniques to reduce color noise in subsurface scattering, and also into developing a new parameterization that closely matched Hyperion’s normalized diffusion parameterization, which allowed artists to basically just flip a switch between normalized diffusion and path-traced subsurface and get a predictable, similar result. Many more details on Hyperion’s path-traced subsurface implementation are in our recent system architecture paper [Burley 2018]. In addition to making characters we already know, such as Ralph and Vanellope, look better and more detailed, path-traced subsurface scattering also proved critical to hitting the required looks for new characters, such as the slug-like Double Dan character.

When Ralph and Vanellope first enter the world of the internet, there are several establishing shots showing vast vistas of the enormous infinite metropolis that the film depicts the internet as. Early in production, some render tests of the internet metropolis proved to be extremely challenging due to the sheer amount of geometry in the scene. Although instancing was used extensively, the way the scenes had to be built in our production pipeline meant that Hyperion wasn’t able to leverage the instancing in the scene as efficiently as we would have liked. Additionally, the way the instance groups were structured made traversal in Hyperion less ideal than it could have been. After encountering smaller-scale versions of the same problems on Moana, Peter Kutz and I had arrived at an idea that we called “multiple entry points”, which basically lets Hyperion blur the lines between top and bottom level BVHs in a two-level BVH structure. By inserting mid-level nodes from bottom level BVHs in to the top-level BVH, Hyperion can produce a much more efficient top-level BVH, dramatically accelerating rendering of large instance groups and other difficult-to-split pieces of large geometry, such as groundplanes. This idea is very similar to BVH rebraiding [Benthin et al. 2017], but we arrived at our approach independently before the publication of BVH rebraiding. After initial testing on Olaf’s Frozen Adventure proved promising, we enabled multiple entry points by default for the entirety of Ralph Breaks the Internet. Additionally, Dan Teece developed a powerful automatic geometry de-duplication system, which allows Hyperion to reclaim large amounts of memory in cases where multiple instance groups are authored with separate copies of the same master geometry. Greg Nichols and I also developed a new multithreading strategy for handling Hyperion’s ultra-wide batched ray traversal, which significantly improved Hyperion’s multithreaded scalability during traversal to near-linear scaling with number of cores. All of these geometry and traversal improvements collectively meant that by the main production push for the show, render times for the large internet vista shots had dropped from being by far the highest in the show to being indistinguishable from any other normal shot. These improvements also proved to be timely, since the internet set was just the beginning of massive-scale geometry and instancing on Ralph Breaks the Internet; solving the render efficiency problems for the internet set also made other large-scale instancing sequences, such as the Ralphzilla battle [Byun et al. 2019] at the end of the film and the massive crowds [Richards et al. 2019] in the internet, easier to render.

Another major advancement we made on Ralph Breaks the Internet, in collaboration with Disney Research Zürich and our sister studio Pixar Animation Studios, is a new machine-learning based denoiser. To the best of my knowledge, Disney Animation was one of the first studios with a successful widescale deployment of a production denoiser on Big Hero 6. The Hyperion denoiser used from Big Hero 6 through Olaf’s Frozen Adventure is a hand-tuned denoiser based on and influenced by [Li et al. 2012] and [Rousselle et al. 2012], and has since been adopted by the Renderman team as the production denoiser that ships with Renderman today. Midway through production on Ralph Breaks the Internet, David Adler from the Hyperion team in collaboration with Fabrice Rousselle, Jan Novák, Gerhard Röthilin, and others from Disney Research Zürich were able to deploy a new, next-generation machine-learning based denoiser [Vogels et al. 2018] Developed primarily by Disney Research Zürich, the new machine-learning denoiser allowed us to cut render times by up to 75% in some cases. This example is yet another case of basic scientific research at Disney Research leading to unexpected but enormous benefits to production in all of the wider Walt Disney Company’s various animation studios!

In addition to everything above, many more smaller improvements were made in all areas of Hyperion for Ralph Breaks the Internet. Dan Teece developed a really cool “edge” shader module, which was used to create all of the silhouette edge glows in the internet world, and Dan also worked closely with FX artists to develop render-side support for various fracture and destruction workflows [Harrower et al. 2018]. Brent Burley developed several improvements to Hyperion’s depth of field support, including a realistic cat’s eye bokeh effect. Finally, as always, the production of Ralph Breaks the Internet has inspired many more future improvements to Hyperion that I can’t write about yet, since they haven’t been published yet.

The original Wreck-It Ralph is one of my favorite modern Disney movies, and I think Ralph Breaks the Internet more than lives up to the original. The film is smart and hilarious while maintaining the depth that made the first Wreck-It Ralph so good. Ralph and Vanellope are just as lovable as before and grow further as characters, and all of the new characters are really awesome (Shank and Yesss and the film’s take on the Disney princesses are particular favorites of mine). More importantly for a rendering blog though, the film is also just gorgeous to look at. With every film, the whole studio takes pride in pushing the envelope even further in terms of artistry, craftsmanship, and sheer visual beauty. The number of environments and settings in Ralph Breaks the Internet is enormous and highly varied; the internet is depicted as a massive city that pushed the limits on how much visual complexity we can render (and from our previous three feature films, we can already render an unbelievable amount!), old locations from the first Wreck-It Ralph are revisited with exponentially more visual detail and richness than before, and there’s even a full on musical number with theatrical lighting somewhere in there!

Below are some stills from the movie, in no particular order, 100% rendered using Hyperion. If you want to see more, or if you just want to see a really great movie, go see Ralph Breaks the Internet on the biggest screen you can find! There are a TON of easter eggs in the film to look out for, and I highly recommend sticking around after the credits for this one.

Here is the part of the credits with Disney Animation’s rendering team! Also, Ralph Breaks the Internet was my wife Harmony Li’s first credit at Disney Animation (she previously was at Pixar)! This frame is kindly provided by Disney. Every person you see in the credits worked really hard to make Ralph Breaks the Internet an amazing film!

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Carsten Benthin, Sven Woop, Ingo Wald, and Attila T. Áfra. 2017. Improved Two-Level BVHs using Partial Re-Braiding. In Proc. of High Performance Graphics (HPG 2017). Article 7.

Brent Burley. 2015. Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering. In ACM SIGGRAPH 2015 Course Notes: Physically Based Shading in Theory and Practice.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 33.

Dong Joo Byun, Alberto Luceño Ros, Alexander Moaveni, Marc Bryant, Joyce Le Tong, and Moe El-Ali. 2019. Creating Ralphzilla: Moshpit, Skeleton Library and Automation Framework. In ACM SIGGRAPH 2019 Talks. Article 66.

Matt Jen-Yuan Chiang, Peter Kutz, and Brent Burley. 2016. Practical and Controllable Subsurface Scattering for Production Path Tracing. In ACM SIGGRAPH 2016 Talks. Article 49.

Julian Fong, Magnus Wrenninge, Christopher Kulla, and Ralf Habel. 2017. Production Volume Rendering. In ACM SIGGRAPH 2017 Courses. Article 2.

Will Harrower, Pete Kyme, Ferdi Scheepers, Michael Rice, Marie Tollec, and Alex Moaveni. 2018. SimpleBullet: Collaborating on a Modular Destruction Toolkit. In ACM SIGGRAPH 2018 Talks. Article 79.

Peter Kutz, Ralf Habel, Yining Karl Li, and Jan Novák. 2017. Spectral and Decomposition Tracking for Rendering Heterogeneous Volumes. ACM Transactions on Graphics (Proc. of SIGGRAPH) 36, 4 (Aug. 2017), Article 111.

Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chiang. 2012. SURE-based Optimization for Adaptive Sampling and Reconstruction. ACM Transactions on Graphics (Proc. of SIGGRAPH Asia) 31, 6 (Nov. 2012), Article 194.

Jan Novák, Andrew Selle and Wojciech Jarosz. 2014. Residual Ratio Tracking for Estimating Attenuation in Participating Media. ACM Transactions on Graphics (Proc. of SIGGRAPH Asia) 33, 6 (Nov. 2014), Article 179.

Josh Richards, Joyce Le Tong, Moe El-Ali, and Tuan Nguyen. 2019. Optimizing Large Scale Crowds in Ralph Breaks the Internet. In ACM SIGGRAPH 2019 Talks. Article 65.

Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust Denoising using Feature and Color Information. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering) 32, 7 (Jun. 2013), 121-130.

Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Transactions on Graphics (Proc. of SIGGRAPH) 37, 4 (Aug. 2018), Article 124.

Converting docx to pdf on macOS

xenodium.com @alvaro — Wed, 14 Nov 2018 02:00:00 GMT

Wanted to convert a docx document to pdf on macOS. Pandoc to the rescue, but first needed pdflatex installed:

pandoc -t latex some.docx -o some.pdf

pdflatex not found. Please select a different --pdf-engine or install pdflatex

Installed pdflatex on macOS with:

brew install mactex

Can also use HTML5. Install wkhtmltopdf with:

brew install Caskroom/cask/wkhtmltopdf

Convert with:

pandoc -t html5 some.docx -o some.pdf

10 Things I wish I knew before giving my First Tech Talk

Wed, 14 Nov 2018 00:12:03 GMT

Giving the talk — credit https://twitter.com/JamieTanna/status/1029428095223320576

Glossophobia or fear of public speaking is cited as being amongst mankind’s top 10 fears. It related to our inherent fear of failure. Although I have never been afraid of speaking publicly, for even the most experienced speakers, it can be a bit nerve racking at times.

Why did I choose to give a talk then? For one, I wanted the experience, ever since going to my first meetup I thought “that’s really cool to speak about stuff you’re excited about”. In connection with this, I enjoy teaching people, whether that be 1-on-1 or to a group — it’s one of the reasons I contribute to open source, and write blogs. It’s a creative outlet. Overall, my primary objective was simply to share something I’m passionate about and also try and make them laugh — emphasis on the word “try” there.

My first talk was at the NottinghamJS meetup and was titled “Lightning Node Performance”. I’m hugely grateful to the organizers for giving me a platform. Previously they had people from Amazons Alexa division, Microsoft's Machine Learning team and more — so it seemed as if I had big shoes to fill.

But giving the talk is the ending, let’s start at the beginning with things I wish I had known when preparing my first talk.

Preparation took longer than expected

First and foremost, the preparation took a long time. A long time. Initially, I had expected creating the slides and writing the talk to take around 2 days. It actually took over a week — plus all the additions I did late at night and changes to the content of the talk on the day it was supposed to take place. If there is any mistake I made, it’s I severely underestimated the time it would take. It gave me a newfound appreciation for any content I consume, whether that be talks, videos or podcasts. It takes a lot of time to prepare these things. Perhaps why criticism can hurt so much.

Part of the reason the preparation took a long time was I wanted to make sure I was 100% concrete on every last word I said — in case someone picked me up on it and tore the entire talk to shreds. For example, part of my talk was speaking about the NodeJS event loop. Although I know roughly how the event loop works, there were still some questions I did not know. I thought that perhaps someone may ask me about the Node event loop and therefore, I set on down the rabbit hole to explore. This kind of pattern occurred at least 6–7 times when creating the talk and accounted for a large proportion of the time I spent.

The image I created for Node js clusters

Moreover, I wanted to keep the slides almost completely visual. I wanted to keep words off the slides because I have observed people read those rather than listening to you. Finding images for NodeJS clustering is harder than it looks though and so again another time-consuming task was pouring over pages of gifs and images to find one that perfectly encapsulated the subject matter. Often times, I created my own in Photoshop, which again took a large portion of time — primarily due to my appalling photo editing skills.

Choosing a topic is tricky

In connection with preparation time, it also took a long time to come up with a topic. Since I was not from a company, I wasn’t presenting any one particular “thing”. Therefore I went with a more general topic “application performance”. This proved difficult because it’s so broad and had so many subtopics I wanted to cover. For example, I wanted to speak about lambda cold starts, network resilience, asynchronous code in node and much more. Each one in of themselves could have been a talk in their own right. Therefore, a balance had to be struck between covering lots of topics briefly and covering a few topics in depth. I hope I eventually got that balance right, but it’s hard to tell. In the future, I would suggest coming up with a concrete outline in parallel to thinking up a topic.

Not all points are equal

This is a lesson I learned after the fact of presenting the talk. Not every point deserves the same amount of time. Spend more time on the difficult to understand topics and breeze through the small minor points. There is often a sunken cost fallacy at play here, whereby you take lots of time to prepare all the slides so they each deserve their own ceremony. We should try to get rid of this thinking and instead prioritize the points covered. Ordering your points carefully can aid with this. No one wants to be bombarded with lots of heavy topics all in one go, so spread them out and interleave them with smaller, lighter points.

Your delivery can get a bit wooden!

Practice, practice… but not too much

Practising your talk is essential of course but you can practice it too much. At a certain point, your delivery could become too scripted or wooden. Rather than attempting to memorize a script, remember the points you are covering. Then just speak. If you have the subject knowledge then this will produce results. Furthermore, speaking from within rather than from notes will vary your talk in different ways. I found that when practising my talk, I would do it a different way each time, adding anecdotes and talking points and cutting others. This was done at an unconscious level and would not have been achieved if I were reciting verbatim.

There’s an NPM module for that — https://twitter.com/iamdevloper/status/487606612757315584

Don’t fear questions

Questions are fantastic for people to get further insight on what you spoke about and can often reveal places where the talk should have explained a point further or provided a different angle. I didn’t so much fear these questions, more expected the worst. But the questions were about the talk. I did get a couple about technologies I hadn’t heard of but I can hardly be blamed for that — especially in the JS world. Overall, the questions were about the talk and asking me to expand on certain stories I had told about how CloudCall was doing this performance improvement work.

I learned a lot from the whole experience, briefly here are my takeaways.

Upload your slides to GitHub and Slidedeck

One thing people always ask for with talks is where can I get the slides, so make them easily available. Creating a repo called “talks” and upload the file there, and upload them onto Slidedeck for those who may not have powerpoint/keynote.

Visual slides worked well

A picture says a thousand words. Words on slides should be avoided at all costs unless they are used to re-emphasize a point. You can explain much more with visuals. For example, rather than putting a slide with the conclusion from a study, put a nice chart up there with the numbers behind the study.

Avoid lots of code on slides

Code on slides are similar to words on slides. They should be used to make a specific point. Try to keep the code as short as possible, using an extract if possible. It’s not essential that the audience has a complete context around a program.

Slow down delivery

When I gave the talk, I think I rushed a little. It’s a nerves thing I suppose. My advice is to just count in your head 1–5 between points and 1–10 between slides. It will seem like a lifetime from your point of view, but it makes the delivery far more fluid.

Engage with the audience rather than speak to them

My talk was that. A talk. I hope the visuals were enough to keep people engaged but in the future, I will make an effort to ask the audience questions and engage with them further. For example, I may ask the audience if they have any experiences with dealing with X after explaining how I did it.

Since my first talk, I have given a couple of others and want to do more. It’s a good experience but takes a lot of time. Be kind to those who give talks and give constructive feedback, as they have sacrificed a lot of time to deliver this. And if you are interested in giving a talk — do so! Ask the organizers of the event and I’m sure they will be happy to pen you in. If you are in the Leicester, UK area and would like to give a talk, post an issue on the LeicesterJS speaker’s repo and I will get it in the diary — we want to encourage first-time speakers. If you have given a talk, share your experience — it’s good to break down some of the fears people may have.

Four development methods you didn't know you were using!

Sam says you should read this — Tue, 13 Nov 2018 09:29:53 GMT

Waterfall is hard, and agile is confusing, so why should you put your team through them? Instead, you are probably already using some of these alternative development techniques! You won't learn about these in a text book, but you'll find them in use all over the software industry. Let's learn about how we manage small teams:

1. Whack-a-mole development

This is the most common development methodology around. Almost all small teams do this to begin with, and businesses can grow huge off the back of Whack-a-mole development. So what is it?

Whack-a-mole development is a process where your developer…

Wie Studien verdreht werden

Home on The Log Book of Manuel Kießling — Tue, 13 Nov 2018 09:12:00 GMT

Ein schönes Beispiel für professionell gemachte Fake News - oder zumindest grobe Fahrlässigkeit - findet sich unter https://unserplanet.net/e-autos-dreckig. Das Ganze ist auch verbreitet als Facebook-Post unter https://www.facebook.com/unserplanet/videos/188402545408094, unter Einbindung eines grob thematisch verwandten Videos von ZDF heuteplus (dessen Wahrheitsgehalt ich hier nicht beurteilen kann und will). Der verlinkte Artikel sagt: “Eine aktuelle Studie, die vom Trancik Lab des Massachusetts Institute of Technology (MIT) veröffentlicht wurde, hat jedoch ergeben, dass Elektroautos NICHT so grün sind, wie Sie denken und eigentlich schlimmere Umweltverschmutzer sind als Benzin- und Dieselautos.

Faster elfeed browsing with paging

xenodium.com @alvaro — Tue, 13 Nov 2018 02:00:00 GMT

Following up from faster junk mail deletion with mu4e, elfeed is another candidate for enabling actions on pages. In this case, marking rss entries as read, page by Page.

If on use-package, the function can defined and bound to the "v" key using:

(use-package elfeed
  :ensure t
  :bind (:map elfeed-search-mode-map
              ("v" . ar/elfeed-mark-visible-as-read))
  :config
  (defun ar/elfeed-mark-visible-as-read ()
    (interactive)
    (require 'window-end-visible)
    (set-mark (window-start))
    (goto-char (window-end-visible))
    (activate-mark)
    (elfeed-search-untag-all-unread)
    (elfeed-search-update--force)
    (deactivate-mark)
    (goto-char (window-start))))

The Simple Joy of "No Phones"

Macleod Sawyer | Web Portal Feed — Tue, 13 Nov 2018 02:00:00 GMT

“Every time someone in a group of people deploys a screen, the whole group is affected. Each disengaged person in a crowd is like a little black hole, a dead zone for social energy, radiating a noticeable field of apathy towards the rest of the room and what’s happening there.” - The Simple Joy of “No Phones” by David Cain

The rate of smartphone addiction is far more prevalent, and destructive compared to the new WHO (World Health Organization) “Video Game Addiction Disorder”, as a person who was incredibly addicted to their devices to post over a dozen (sometimes hundreds!) of times a day through my social media accounts for nearly a decade, I know first hand the emotional, physical, and mentally destructive turmoil of smart-device addiction and from the hundreds of conversations I’ve had from the average person, psychologists. professors, and user experience developers this is becoming far more prevalent, understood, and accepted - that a smartphone is a “little black hole, a dead zone for social energy, radiating a noticeable field of apathy towards the rest of the room and what’s happening there”.

I have noticed this effect myself - the moment someone begins to tune-out by using their device in a situation, the rest of the group at hand will begin to feel envious of using their own devices as if that minuscule action of one has been amplified to an eleven demanding that everyone in the vicinity join them in using their personal devices. This is similar to cigarette smoking, where if one person lights up - everyone must light up.

I imagine that in another decade or two we’ll look at 2010s-era device use something like we do now with cigarette smoking. I was born in 1980, and I remember smoking sections on planes, which is unthinkable today. I wonder if today’s kids will one day vaguely remember the brief, bizarre time when people didn’t think twice about lighting up a screen in the middle of a darkened concert hall.

In the future, I believe we will begin to have restaurants, concerts, parks, meetings and more begin banning smart-devices - and even begin implementing an automated NFC locking mechanisms that severely limit the interactions of the device (allowing emergency calls, cameras, etc) depending on the situation and the event. Regardless, socially I believe that the tide has already started heading this way - that we will begin phasing out smart devices in public settings.

There is nothing wrong with using your smartphones, enjoy them, use them, they are the greatest communication methods that our species has ever created and has helped create a world more of equality than ever before - but be wise in your usage, and attempt to limit when and where you use them not just for yourself, but for the world around you.

The Tyranny of Metrics

Macleod Sawyer | Web Portal Feed — Mon, 12 Nov 2018 14:00:00 GMT

Facebook live using FFmpeg – and nodejs

Yasha Solutions — Mon, 12 Nov 2018 12:38:13 GMT

Facebook Live using FFmpeg Play h.264 video stream in html page FFmpeg-based Live stream via Python PyLivestream

What Is Clip Art? A Brief History

Sam Solomon — Mon, 12 Nov 2018 02:00:00 GMT

Clip art is the pre-made graphics that came bundled with software like Microsoft Word. A short history of the medium, from mechanical paste-up to the modern web.

An introduction to credit card churning

JonLuca's Blog — Sun, 11 Nov 2018 19:11:41 GMT

Radical Markets: Uprooting Capitalism and Democracy for a Just Society

🦉 olshansky 🦁 — Sun, 11 Nov 2018 02:00:00 GMT

Olshansky's review of Radical Markets: Uprooting Capitalism and Democracy for a Just Society by Eric A. Posner

Demistifying blockchain - Part 1

GeekWare - Daniel Pecos Martínez — Sat, 10 Nov 2018 17:30:00 GMT

Blockchain is one of those buzzwords everyone is listening nowadays, but what it really is?

In this series of posts we will dig one general concepts about BlockChain and a little bit of its relation with BitCoin, as well as we will develop a simple blockchain in Go.

To put it simple, and as its name states, it’s just a chain of blocks. The interesting detail is that is a cryptographic chain providing some characteristics that make them really useful:

Tamper-proof
Distributed
Decentralized

We say a blockchain is tamper-proof because once a block has been added to the chain, nobody can modify it without also modifying its signature or cryptographic hash. But what if the attacker would also modify the signature to make it match its new content? A block stores as part of its data, the signature of the previous block in the chain (and that’s why it’s a chain), so if an attacker would try to replace a block in the chain (even using the correct cryptographic signature), would have to cascade the changes across all blocks from that one on until the end of the chain.

Even though, that’s a plausible attack scenario. How can BlockChain avoid it? Making the chain distributed and decentralized, so an attacker trying to modify the chain, would have to modify all instances ot the chain distributed across the world. And it has to be done at once to more than 50% of the nodes in the network, because being a decentralized network means that there is no golden copy of the chain. All instances in the network are equal.

This also means that whenever someone tries to attach a new node to the end of the chain, the network has to agree to that action. Once the network has acknowledged or achieved consensus, the new block is considered permanently and successfully added to the chain.

These characteristics are what make the Blockchain a secure, read-only and distributed database.

BlockChain is in the bare-bones of Bitcoin, as the cryptocurrency uses a blockchain implementation as the public ledger system, and thus since the boom of cryptocurrencies, this type of databases have caught the interest of more and more researchers and developers.

In the following video you will find a more detailed explanation of what we want to implement in this series of posts. If the concept of what a block chain is is still not clear, please have a look to the video before going further in the series:

In the following post we will start implementing our own blockchain in Go, as well as going more in detail into the characteristics of block mining.

Faster junk mail deletion with mu4e

xenodium.com @alvaro — Sat, 10 Nov 2018 02:00:00 GMT

It's been roughly 5 months since my mu4e email migration. Happy with my choice. Mu4e is awesome.

I now have 4 email accounts managed by mu4e, and unfortunately receiving lots of junk mail.

I regularly peek at junk folders for false positives and delete junk email permanently. I've been wanting a quick way to glance at junk mail and easily delete page by page.

Deleting emails page by page is not supported in mu4e by default. Fortunately, this is Emacs and we can change that™.

There's a handy package by Roland Walker called window-end-visible. We can use it to select mu4e emails by page and subsequently glue it all together to enable deleting emails by page.

(require 'mu4e)
(require 'window-end-visible)

(defun ar/mu4e-delete-page ()
  (interactive)
  (set-mark (window-start))
  (goto-char (window-end-visible))
  (activate-mark)
  (mu4e-headers-mark-for-trash)
  (mu4e-mark-execute-all t)
  (deactivate-mark)
  (goto-char (window-start)))

I'm a use-package fan, so I use it to bind the "v" key to delete visible emails (by page).

(use-package mu4e
  :bind (:map mu4e-headers-mode-map
         ("v" . ar/mu4e-delete-page))

You can fix lumpy Joy-Con using hair dryer

Muffin Man — Sat, 10 Nov 2018 02:00:00 GMT

I love my Nintendo SwitchI have a huge draft about Switch, and how it have put the fun back in gaming for me. I hope to publish it at some point. Unfortunately, tonight I noticed that one of my Joy-Cons has two weird lumps on it.

(This is the image from the internet, not mine.)

On multiple forum threads, people suggested using hair dryer to fix it. But nobody did explain how. I tried it, and in couple of minutes I managed to get it, and here is how I did it.

There are tiny inserts in the cap, and they need to go in to the tiny holes on the joystick. I had three of them fall out. To fix it, just heat it with the hair dryer (I did it for about ten seconds), carefully adjust one insert and rub it a little bit until it falls in. Repeat for each insert.

It was much easier than I thought it would be (before heating it felt impossible). Hopefully they won't fall again, but I will probably get thumb grips to be sure.

Experimenting with ComponentKit

Whacky Labs — Fri, 09 Nov 2018 20:18:00 GMT

It’s a nice Friday afternoon and I just realized that I’ve never used ComponentKit.

One of the reasons that got me really interested in the project is that, I’ve been looking around alternatives to UIKit. Not that I’ve anything against UIKit, but I seen how things out of hands with UIKit pretty soon. And ComponentKit was on my radar for quite some time.

Also, I’m a huge fan of C++, but so far the only ObjectiveC++ I’ve written was just to glue ObjectiveC with C++, like I would start with an AppEngine.cpp and forward all the call from the AppDelegate to the AppEngine. ComponentKit on the other hand, seems to be making full use of C++ and ObjectiveC in the same space. Which sounds weird enough for me to at least try this thing out.

Building an App with ComponentKit

I will be building the PhotoApp that I try to build every now and then but this time with ComponentKit. What the PhotoApp does is that it starts with a loading screen while we fetch a list of photo and when we are done we render them on screen. Next, whenever a user taps on the cell, we display the details screen with the photo in more details and some text.

This is pretty simple app that touches many parts of most of the apps out there. Okay, let’s go!

Set up

The first step is to checkout the repository from github The recommended way is to use Carthage. It’s been a long time since I’ve used Carthage, personally I like to manage all my dependencies manually, which is know is not a popular opinion these days. So, let’s give Carthage a shot.

Following the instructions from https://github.com/Carthage/Carthage#quick-start, this is now my Cartfie looks like:

github "facebook/ComponentKit" ~> 0.24

Now, I can run ` carthage update`… and it does not work

*** Fetching ComponentKit
*** Checking out ComponentKit at "0.24"
*** xcodebuild output can be found in /var/folders/wv/x1yzd_351bbbwgyps66xgytw0000gn/T/carthage-xcodebuild.NGYTMn.log
*** Building scheme "ComponentKit" in ComponentKit.xcodeproj
Build Failed
	Task failed with exit code 65:
	/usr/bin/xcrun xcodebuild -project /Users/sid/Work/Personal/PhotoApp/component-kit/Carthage/Checkouts/ComponentKit/ComponentKit.xcodeproj -scheme ComponentKit -configuration Release -derivedDataPath /Users/sid/Library/Caches/org.carthage.CarthageKit/DerivedData/9.2_9C40b/ComponentKit/0.24 -sdk iphoneos ONLY_ACTIVE_ARCH=NO CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES archive -archivePath /var/folders/wv/x1yzd_351bbbwgyps66xgytw0000gn/T/ComponentKit SKIP_INSTALL=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=NO CLANG_ENABLE_CODE_COVERAGE=NO STRIP_INSTALLED_PRODUCT=NO (launched in /Users/sid/Work/Personal/PhotoApp/component-kit/Carthage/Checkouts/ComponentKit)

This usually indicates that project itself failed to compile. Please check the xcodebuild log for more details: /var/folders/wv/x1yzd_351bbbwgyps66xgytw0000gn/T/carthage-xcodebuild.NGYTMn.log

Life is too short to read a log message. Time for plan B; Manually building the workspace.

The very first thing one notices after the setup is that how much time it takes for the ComponentKit to compile. I’ve been using Swift mostly at work for past 4 years now, and there compile time is unacceptable, and that is why whenever I’m working on something on a Friday evening, I pick something blazing fast, which ObjectiveC is, but ObjectiveC++ is not.

Whenever I’m working on an ObjC project, the very first thing I do is to toggle the TREAT_WARNINGS_AS_ERRORS flag to YES. And I wish every other project does the same, specially a project as widely used as ComponentKit.

Getting started

Okay all code set up and ready to go, the first challenge is to figure out where to start. To be honest, I’ve no clue, so I’ll looking at the sample WildeGuess project. Like, WildeGuessCollectionViewController I’m also planning to start out with a UICollectionView, so that is a good thing.

The most important piece of the system when working with a UICollectionView based layout is CKCollectionViewDataSource. This object is who you talk to whenever the data updates. Probably the first piece of information we want to inform the data source is the number of section we intend to display. Unlike traditional UICollectionViewDataSource, we do not provide a integer value, rather, we provide the change set.

For our case, we already know at the load time that there is going to be one section, which contains the loading view. We need to provide that information as a change set, as the data source starts with zero sections.

    CKDataSourceChangeset *initialData = [[[CKDataSourceChangesetBuilder transactionalComponentDataSourceChangeset]
                                           withInsertedSections:[NSIndexSet indexSetWithIndex:0]]build];
    [_dataSource applyChangeset:initialData mode:CKUpdateModeAsynchronous userInfo:nil];

Before I get my hands dirty with building components, I would also like to plug in the other thing data source might be interested in. And that is a event callback for whenever a cell starts or ends displaying is invoked we want to update our data source with this information. In our case this is the willDisplayCell and didEndDisplayingCell of UICollectionViewDelegate

- (void)collectionView:(UICollectionView *)collectionView
       willDisplayCell:(UICollectionViewCell *)cell
    forItemAtIndexPath:(NSIndexPath *)indexPath
{
    [_dataSource announceWillDisplayCell:cell];
}

- (void)collectionView:(UICollectionView *)collectionView
  didEndDisplayingCell:(UICollectionViewCell *)cell
    forItemAtIndexPath:(NSIndexPath *)indexPath
{
    [_dataSource announceDidEndDisplayingCell:cell];
}

With our basic data source set up, we can now focus on providing the first view. Usually when we start our layout with a UICollectionView the first thing that comes to mind it to have a bunch of UICollectionViewCell and register them and later whenever UIKit needs to render them on screen, we paint them. With ComponentKit things are different, we probably only need to create a UICollectionView and leave the rest to ComponentKit. The starting point of the entire component system seems to be this CKComponentProvider which is a protocol with a class method, and the context is provided as an argument. Neat.

+ (CKComponent *)componentForModel:(id<NSObject>)model context:(id<NSObject>)context;

So, my first attempt would be insert a NSNull and render an PHLoadingComponent for that model

    NSDictionary<NSIndexPath *, NSObject *> *loadingItem = @{[NSIndexPath indexPathForRow:0 inSection:0]: [NSNull null]};
    CKDataSourceChangeset *loadingData = [[[CKDataSourceChangesetBuilder transactionalComponentDataSourceChangeset] withInsertedItems:loadingItem] build];
    [_dataSource applyChangeset:loadingData mode:CKUpdateModeAsynchronous userInfo:nil];

Creating first component

Creating a component does not seems as trivial if you have been writing apps for some time. I know I’m not creating a view from scratch because that is the job of ComponentKit, I probably have to only provide the data for the view. But what data? Let’s start with a prebaked component CKLabelComponent

@interface PHLoadingComponent : CKCompositeComponent
+ (instancetype)newWithTintColor:(UIColor *)color;
@end

@implementation PHLoadingComponent

+ (instancetype)newWithTintColor:(UIColor *)color
{
    return [super newWithComponent:[CKLabelComponent
                                    newWithLabelAttributes:{
                                        .string = @"Loading"
                                    }
                                    viewAttributes:{
                                        {@selector(setBackgroundColor:), color},
                                        {@selector(setUserInteractionEnabled:), @NO}
                                    }
                                    size:{}]];
}

@end

Not bad! So, by simply providing the text data I can get a view. How about making it full screen and replacing the text with a UIActivityIndicatorView.

To be honest the official documentation is outdated as I can not find anything about everything, for example CKFlexboxComponent is not there. So the only way out is to actually check the source code, which is probably even better than looking up the docs.

From what I’ve figured out, the way one creates components for views is by providing the Class type and a bunch of selectors to be invoked. Imagine it this way, if you were to create a custom view with a UIActivityIndicatorView, you might do it like:

-(UIActivityIndicatorView *) makeSpinnerWithColor: (UIColor *)color
{
    UIActivityIndicatorView *spinner = [[UIActivityIndicatorView alloc] initWithFrame:[UIScreen mainScreen].bounds];
    [spinner setColor:color];
    [spinner setActivityIndicatorViewStyle:UIActivityIndicatorViewStyleWhiteLarge];
    return spinner
}

With ComponentKit, you might do it like:

-(CKComponent *) makeSpinnerWithColor: (UIColor *)color
{
    return [CKComponent
            newWithView:{
                [UIActivityIndicatorView class],
                {
                    {@selector(setColor:), color},
                    {@selector(setActivityIndicatorViewStyle:), UIActivityIndicatorViewStyleWhiteLarge}
                }
            }
            size: CKComponentSize::fromCGSize([UIScreen mainScreen].bounds.size)];
}

With that in place, we have a loading screen.

Next step is to get the activity indicator start and stop animating. Which is pretty simple

{@selector(startAnimating), nil}

Rendering list of images

Next step, rendering a bunch of images. The first part is easy, making a network request and get a JSON payload containing a bunch of image URLs.

- (void)updateImages
{
    NSArray *images = _viewModel.list;
    NSUInteger count = images.count;
    NSMutableDictionary<NSIndexPath *, PAListItemViewModel *> *photoList = [NSMutableDictionary dictionaryWithCapacity:count];
    for (NSInteger idx = 0; idx < count; ++idx) {
        [photoList
         setObject:[_viewModel itemAtIndex:idx]
         forKey:[NSIndexPath indexPathForRow:idx inSection:0]];
    }

    CKDataSourceChangeset *loadingData = [[[CKDataSourceChangesetBuilder transactionalComponentDataSourceChangeset] withInsertedItems:photoList] build];
    [_dataSource applyChangeset:loadingData mode:CKUpdateModeAsynchronous userInfo:nil];
}

The tricky part is fetching the images. Normally we have to consider the fact that the user might be scrolling and the cell that started the network request could be out of view already, and we don’t want to render wrong images on wrong cells. So, there has to be a owner of the network callback, from what I’ve read so far that job is usually for the ComponentController, but then I also something called CKNetworkImageComponent.

To make that happen, first step is provide a concrete implementation of CKNetworkImageDownloading which is pretty close to what PANetworkService is doing already. Here is a naive implementation

- (id)downloadImageWithURL:(NSURL *)URL
                    caller:(id)caller
             callbackQueue:(dispatch_queue_t)callbackQueue
     downloadProgressBlock:(void (^)(CGFloat progress))downloadProgressBlock
                completion:(void (^)(CGImageRef image, NSError *error))completion;
{
    [PANetworkService getPhotoWithURL:URL completion:^(UIImage * _Nullable image) {
        dispatch_async(callbackQueue, ^{
            if (image == nil) {
                completion(nil, [NSError
                                 errorWithDomain:@"com.wl.error.image"
                                 code:0
                                 userInfo:nil]);
            } else {
                completion(image.CGImage, nil);
            }
        });
    }];
    return [URL absoluteString];
}

And the second step is then to create the component

@implementation PHPhotoComponent
+ (instancetype)newWithPhoto:(PAPhoto *)photo
             imageDownloader:(PAImageDownloader *)imageDownloader;
{
    UIImage *placeholderImage = [PAImageLoader placeholderImage];
    CKComponent *imageComponent = [CKNetworkImageComponent
                                   newWithURL:[photo imageURLWithResolution:ImageResolutionThumbnail]
                                   imageDownloader:imageDownloader
                                   size:CKComponentSize::fromCGSize(placeholderImage.size)
                                   options:{ .defaultImage = placeholderImage }
                                   attributes:{}];
    return [super newWithComponent:imageComponent];
}
@end

User interactions

Next we want the react on whenever a user taps on the cell, and here is where CKComponentTapGestureAttribute comes into the picture. After playing around for a while, this is what seems to be working

@implementation PHPhotoComponent
+ (instancetype)newWithPhoto:(PAPhoto *)photo
             imageDownloader:(PAImageDownloader *)imageDownloader;
{
    UIImage *placeholderImage = [PAImageLoader placeholderImage];
    CKComponentSize size = CKComponentSize::fromCGSize(placeholderImage.size);
    CKComponent *imageComponent = [CKNetworkImageComponent
                                   newWithURL:[photo imageURLWithResolution:ImageResolutionThumbnail]
                                   imageDownloader:imageDownloader
                                   size:size
                                   options:{ .defaultImage = placeholderImage }
                                   attributes:{}];

    CKComponent *centerYComponent = [CKCenterLayoutComponent
                                     newWithCenteringOptions:CKCenterLayoutComponentCenteringY
                                     sizingOptions:CKCenterLayoutComponentSizingOptionMinimumY
                                     child:imageComponent
                                     size:{}];

    CKComponent *containerComponent = [CKFlexboxComponent
                                       newWithView:{
                                           [UIView class],
                                           {CKComponentTapGestureAttribute(@selector(onTap))}
                                       }
                                       size:{}
                                       style:{}
                                       children:{
                                           {centerYComponent}
                                       }];

    return [super newWithComponent:containerComponent];
}

- (void)onTap
{
    NSLog(@"show details");
}
@end

Whats left is hooking a listener to the event. From what I’ve read about the components is that they should not be set as delegate, as they are by design short lived immutables. So I’m imagining them as being disposed off after a draw cycle. So the question is how do we handle the touch event? To understand let’s see what is CKComponentTapGestureAttribute?

CKComponentViewAttributeValue CKComponentTapGestureAttribute(CKAction<UIGestureRecognizer *> action);

So it a function that takes a CKAction and returns a CKComponentViewAttributeValue. The CKAction on the other hand is a simple wrapper around the target selector pair, maybe we can use that. Or even better if we could construct the CKAction and inject it from outside.

        CKAction<UIGestureRecognizer *>action = CKAction<UIGestureRecognizer *>::actionFromBlock(^(CKComponent *, UIGestureRecognizer *__strong) {
            NSLog(@"did tap");
        });
        
        return [PHPhotoComponent
                newWithPhoto:(PAPhoto *)model
                imageDownloader:(PAImageDownloader *)context
                action: action];

The tricky part is sending the invocation back to the view controller. This whole component creation lives in a class method so has no reference to the view controller. One way to fix this is by passing the view controller as context.

@interface PHRootComponentProvider : NSObject
@end

@implementation PHRootComponentProvider

+ (CKComponent *)componentForModel:(id<NSObject>)model context:(id<NSObject>)context;
{
    if ([model isKindOfClass:[NSNull class]]) {
        return [PHRootComponentProvider loadingComponent];
    } else if ([model isKindOfClass:[PAPhoto class]]) {
        return [PHRootComponentProvider
                componentForPhoto:(PAPhoto *)model
                sender:(PHViewController *)context];
    } else {
        NSAssert(NO, @"Should not happen");
        return nil;
    }
}

+ (CKComponent *)loadingComponent
{
    return [PHLoadingComponent newWithTintColor:[UIColor redColor]];
}

+ (CKComponent *)componentForPhoto:(PAPhoto *)photo sender:(PHViewController *)sender;
{
    CKAction<UIGestureRecognizer *>action = CKAction<UIGestureRecognizer *>::actionFromBlock(^(CKComponent *, UIGestureRecognizer *__strong) {
        [sender onTap:photo];
    });

    return [PHPhotoComponent
            newWithPhoto:photo
            imageDownloader:sender.imageDownloader
            action: action];

}

@end

And then in the view controller

    _dataSource = [[CKCollectionViewDataSource alloc]
                   initWithCollectionView:_collectionView
                   supplementaryViewDataSource:nil
                   configuration:[[CKDataSourceConfiguration alloc]
                                  initWithComponentProvider:[PHRootComponentProvider class]
                                  context:self
                                  sizeRange:[[CKComponentFlexibleSizeRangeProvider
                                              providerWithFlexibility:CKComponentSizeRangeFlexibleHeight]
                                             sizeRangeForBoundingSize:self.view.bounds.size]]];

I’m not sure if this is how the design is intended to be.

Filling up the rest

The rest should not be hard to do. We need another view that shows the details of the photo. We can either build it using the ComponentKit, or could simply have static UIViewController. I like the fact that if we build the detail UI with ComponentKit we would not have to worry about making the UI scrollable in future if it turns out that the UI does not fit one of those small screens. Also, the fact that we can reuse the components.

Although I’m still not sure whether the better design is to have only a single UIViewController and only update the data source or have multiple view controllers. If I were to make a bet, I think UIViewController would be a better choice, as then the view caching could take some benefits or maybe not. Another question is, even if I were to pull the entire thing off with a single view controller, would it make sense to twist my arm around to get the default navigation push/pop animation by writing a fake animator class that reloads the UICollectionView as if it were a push/pop animation.

Conclusion

So what do I think of ComponentKit. I think its a great proof of concept that one could easily write the one directional UI, but I would not force to write the entire app with ComponentKit, but maybe only the critical screens.

I would definetely like to poke around to see how easy it is to build static screens with ComponentKit, like the ones where I know I don’t need a UICollectionView and maybe also do some animation.

The entire code is available at github.com/chunkyguy/PhotoApp sitting along the pure swift and pure Objc implementations.

Scripts for Drilling Down Into Unknown Optimizer Changes

Tanel Poder Blog — Fri, 09 Nov 2018 05:47:32 GMT

When I delivered my Advanced Oracle SQL Tuning training last week, I jumped into a free-form demo to show that the PARALLEL hints don’t really force anything other than adjusting the cost of parallelizable operations for CBO evaluation as usual. If a serial plan (or branch) still has a lower cost than the respective parallel processing cost, then serial execution would be used there. But when demoing this, I got a surprise.

Struct-uring your data

Caius Theory — Fri, 09 Nov 2018 02:31:00 GMT

In Ruby it’s easy to structure data in hashes and pass it around, and usually that leads to errors with calling methods on nil, or misspelling the name of a key, or some such silly bug that we might catch earlier given a defined object with a custom Class behind it. But it’s so much work to create a Class just to represent some grab bag of data we’ve been handed, right? Well, maybe!

Lets say we have some event data that we’re being sent and we want to do some stuff with it in memory, we could just represent this as an array of hashes:

events = [
 {
 name: "Writing",
 duration: 15,
 started: true,
 finished: false,
 },
 {
 name: "Evening walk",
 duration: 60,
 started: true,
 finished: true,
 }
]

This is not a bad way to represent the data, but if we want to start asking questions of it like “find all events currently happening” it becomes trickier. We could filter the collection to just those “in progress” events with the following

events.select { |event| event[:started] && !event[:finished] }

Next time someone reads this though, they have to figure out what it means to have an event that’s started but not finished. Also what happens when someone in future misremembers :finished as :completed when running over the data in new code?¹ Wouldn’t it be better if we could do the following instead?²

events.select { |event| event.in_progress? }

An easy way to do this is to just create a Struct for the event, with the extra method defined internally. Whilst we’re in there, we could add a couple more methods to make us querying the state of boolean attributes nicer to read(eh?)

Event = Struct.new(:name, :duration, :started, :finished) do
 alias_method :started?, :started
 alias_method :finished?, :finished

 def in_progress?
 started? && !finished?
 end
end

And then to create the objects, we can either use the positional arguments to .new (same order as the symbols given to Struct.new), or tap the object and use the setters directly for each attribute.

events = [
 Event.new("Writing", 15, true, false),
 Event.new.tap { |e|
 e.name = "Evening Walk"
 e.duration = 60
 e.started = true
 e.finished = false
 },
]
# => [#<struct Event
# name="Writing",
# duration=15,
# started=true,
# finished=false>,
# #<struct Event
# name="Evening Walk",
# duration=60,
# started=true,
# finished=false>]

Now we can use our easier-to-read code for selecting all in-progress events, or ignoring all of those. Or if we just want to grab all finished events, we now have a method to call—Event#finished?—that conveys the intent of what it returns without having to look up the data structure of the hash to work out if that field is a String or Boolean.³

For super-powered structs, you don’t even need to assign them to a Constant. You can just assign them to normal variables and use them locally in that scope without needing to define a constant.

class Grabber
 def call
 result = Struct.new(:success, :output) do
 alias_method :success?, :success
 end

 if (data = grab_data)
 result.new(true, data)
 else
 result.new(false, nil)
 end
 end
end

That’ll handily return you an object you can interrogate for success? and ask for the output if it was successful. And no Constants were created in the making of this method. 🎉

Keep an eye out for where you can Struct-ure your data. It might be more often than you expect.

It would always returns true - !nil. ↩︎
That’s events.select(&:in_progress?) for the golfers amongst you. ↩︎
…? methods in ruby are truthy/falsy by convention. ↩︎

On Migrating from Wordpress to Static Markdown

Thomas Hunter II — Fri, 09 Nov 2018 02:00:00 GMT

Designing an OSR Game Part 2: Freeholds and Fatality

Ben Overmyer's Site — Fri, 09 Nov 2018 02:00:00 GMT

This is part two of a series on a new Old School Renaissance tabletop role-playing game that I'm designing. You can read part one over here.

Designing a New OSR Game

Freeholds: Backdrop For Adventure

I'll get back to character creation in a bit. For a moment, I want to focus on another aspect of gameplay. After creating your characters, your group will need to create your Freehold. In the context of the game, a Freehold is an independent town of moderate size. It's self-sufficient, and large enough to be able to supply plenty of new characters should/when your original characters die. It's also large enough to have multiple concerns beyond simple survival.

Freeholds are living communities. When creating your Freehold, you roll for or choose its Assets, Pillars, Authorities, and Tensions. Each of these helps determine the fabric of the community your party is devoted to. They also change over time, not just in reaction to what your party does, but also in accordance with the community's own character and how it interacts with other Freeholds.

Assets are things that enhance a Freehold's abilities. They could include things like herds of cattle, skilled artisans, well-trained city guard, and so on.

Pillars are things that form the bedrock of the community, and would exist even without the Freehold itself. They could include things like gold mines, permanent magical spells, a calm natural harbor, and so on.

Authorities are things that drive the action of the community. They could include things like a major temple, a hereditary dukedom, a merchant guild, and so on.

Tensions are things that both drive the action of the community and constantly threaten to change its nature. They could include things like a religious divide, feuding noble families, a quarrel between guilds, and so on.

Each of these four facets of your Freehold help determine what is available to your characters at any given time, and what potential adventures your characters might be a part of. They also give your characters a reason to be invested in your community.

Fatality: Death Is A Constant

This world is dangerous. While the worst of the apocalyptic chaos is over, your characters will still need to deal with major threats. They may not be able to do it alone. And even when successful, the cost may be high.

Like games from the first age of role-playing, death in this one is easy to come by. Characters don't get much more individually powerful over time. This makes banding together far more important. This also drives the necessity for character creation to be quick and painless, so that when a player's character meets an untimely end, he can roll up a new one on the spot.

Unlike some popular fantasy role-playing games, death is more than just a speed bump in this game. Resurrections are so rare as to be legendary, and perhaps even feared. Necromancy is more common, but not without a terrible price.

Digital nomading in October in Thessaloniki, Greece

Posts by Petr Stribny — Thu, 08 Nov 2018 18:33:44 GMT

Cool Python Libraries: TQDM and Resampy

bastibe.de — Thu, 08 Nov 2018 16:52:00 GMT

In my recent post about appreciation for open source software, I mentioned that we should praise our open source heros more often. So here are two lesser-known libraries that I use daily, and which are unabashedly awesome:

TQDM

TQDM draws text progress bars for long-running processes, simply by wrapping your iterator in tqdm(iterator). And this, alone, would be awesome. But, TQDM is one of those libraries that aren't just a good idea, but then go the extra mile, and add fantastic documentation, contingencies for all kinds of weird use cases, and integration with notebooks and GUIs.

I use TQDM all the time, for running my scientific experiments and data analysis, and it just works. For long-running tasks, I recommend using tqdm(iterator, smoothing=0, desc='calculating'), which adds a meaningful description to the progress bar, and an accurate runtime estimate.

Resampy

Resampy resamples numpy signals. Resample your data with resample(signal, old_samplerate, new_samplerate). Just like with TQDM, this simple interface hides a lot of complexity and flexibility under the hood, yet remains conceptually simple and easy to use.

But beyond simplicity, resampy uses a clever implementation that is a far cry better than scipy.signal.resample, while still being easy to install and fast. For a more thorough comparison of resampling algorithms, visit Joachim Thiemann's blog.

Wrapped in a blanket of brotherly love

Blogs on Michail Rybakov — Wed, 07 Nov 2018 16:07:46 GMT

Some important, appropriate digital developments across Africa

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 07 Nov 2018 10:00:00 GMT

This is “just” a short list of some digital communities and projects for sustainability and social development in Africa that I think everybody should know. I aim to follow this field much more regularly from now on.

First steps running a Cloud Task on Spring Cloud Data Flow

Jonathan Jones homepage: planetjones.net — Wed, 07 Nov 2018 02:00:00 GMT

Everyone is talking cloud native. Spring might not be the newest kid on the block, but Spring Cloud Data Flow holds bags of promise. Here I give an introduction to Spring Cloud Data Flow and write my first Cloud Task.

Shared base fixture

Blog on blog.gnoack.org — Tue, 06 Nov 2018 08:44:44 GMT

A diagram of how different tests share common test setup with a shared base fixture

Q: How do you know you can trust your test set-up?
A: You reuse a well-known test set up in every test.

Example code

def test_simple():
  req = set_up_working_request()
  resp = invoke(req)
  assert resp.status == OK

def test_invalid_page_size():
  req = set_up_working_request()
  req.page_size = -5
  resp = invoke(req)
  assert resp.status == INVALID_ARGUMENT

In the test,

We call set_up_working_request() and we know this results in “working” behavior. Then we purposefully inject a defective negative page size into the working request.
We exercise the system under test in the same way.
We expect a different result.

Because the only difference in set up between test_simple and test_invalid_page_size was in the differing page size, we can easily reason that the different response status is resulting from just that.

This works well together with these techniques:

Structure your test methods with separate set up, exercise and verify phases.¹
Use explicit helper functions for set up and possibly name them according to expected behavior such as “working” in the example above.
Keep the shared set up code in test methods as short as possible, so that it’s easy to spot the equal parts.
1:N: Use one base scenario (e.g. “working”) and use its set up as a base for multiple derived scenarios.

This is called the “Four-Phase test” in the xUnit test pattern book. (Meszaros07, p358) ↩︎

Working with vultr's ipv6-only instances

xenodium.com @alvaro — Tue, 06 Nov 2018 02:00:00 GMT

Having recently read Your Money or Your Life, I've been cutting down on personal expenses wherever possible. Specially recurring expenses which include monthly charges from VPS hosting. Let's reduce those charges…

My VPS needs are fairly small (mostly hobby and tinkering). Vultr† has a plan for $2.50/month (not seen anything cheaper). The caveat for the price, you get ipv6 access only (ie. 0000:1111:2222:3333:4444:5555:6666:7777:8888).

So far so good, but my ISP doesn't yet support ipv6:

$ ping6 0000:1111:2222:3333:4444:5555:6666:7777:8888
$ ping6: UDP connect: No route to host

Fortunately, we can still work with ipv6 by using a tunnel (TIL about Hurricane Electric's tunnel broker). After signing up and creating a tunnel, they conveniently show you "Example Configurations" from the "Tunnel Details" menu. In my case, macOS:

ifconfig gif0 create
ifconfig gif0 tunnel <ipv4 client broker IP or DCHP internal IP> <ipv4 server IP>
ifconfig gif0 inet6 <ipv6 client broker IP> <ipv6 server IP> prefixlen 128
route -n add -inet6 default <ipv6 server IP>

Note: If behind router, use the DHCP internal IP.

After configuring with ifconfig, all is good. Yay!

$ ping6 0000:1111:2222:3333:4444:5555:6666:7777:8888
PING6(56=40+8+8 bytes) 2001:111:22:aaa::2 --> 0000:1111:2222:3333:4444:5555:6666:7777:8888
16 bytes from 0000:1111:2222:3333:4444:5555:6666:7777:8888, icmp_seq=0 hlim=52 time=270.019 ms
16 bytes from 0000:1111:2222:3333:4444:5555:6666:7777:8888, icmp_seq=1 hlim=52 time=290.834 ms
16 bytes from 0000:1111:2222:3333:4444:5555:6666:7777:8888, icmp_seq=2 hlim=52 time=311.960 ms
16 bytes from 0000:1111:2222:3333:4444:5555:6666:7777:8888, icmp_seq=3 hlim=52 time=330.902 ms

I'm an ipv6 noob. I mostly need ssh access. My typical usages need small tweaks.

For ssh:

ssh -6 username@0000:1111:2222:3333:4444:5555:6666:7777:8888

For scp:

scp -6 file.txt username@\[0000:1111:2222:3333:4444:5555:6666:7777:8888\]:/remote/dir/

† I get $10 credit if you use this affiliate link. Thank you.

The art of Sales, (Not So) Secrets from Old School Marketers

ferrucc.io — Mon, 05 Nov 2018 12:44:01 GMT

Marketing on the internet isn’t much different from Marketing on any other medium.

So let’s go back a hundred years to get some inspiration.

Inspiration from the 1920s

After selling 100 million books of his series Little Blue Book marketer Julius Hadleman wrote a book analysing all the data he had collected.

The book title is “The First Hundred Million” and it came out in 1928 (email me if you need help finding it) and features an interesting collection of proven Titles that work and insights on what we’re interested in reading.

Bad Blood: Secrets and Lies in a Silicon Valley Startup

🦉 olshansky 🦁 — Mon, 05 Nov 2018 02:00:00 GMT

Olshansky's review of Bad Blood: Secrets and Lies in a Silicon Valley Startup by John Carreyrou

Thunderbolts and Lightning

The David Brownman Blog — Mon, 05 Nov 2018 02:00:00 GMT

In which a surge protector does not protect as much as is expected

Read the whole thing.

Networking Essentials: Software Defined Networking

swyx's site RSS Feed — Sun, 04 Nov 2018 23:11:45 GMT

Why Software Defined Networking is taking the networking world by storm

Why do you still propose to regulate, break or clone Facebook, Google & C.?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 04 Nov 2018 10:00:00 GMT

In the last months, I have seen several experts suggest three kinds of solutions to the problems created by today’s social networks. Here I suggest, again, that those solutions are wrong, and then make a request.

Shaving bookmarks

xenodium.com @alvaro — Sun, 04 Nov 2018 02:00:00 GMT

8 best safety razors (The Independent).

Buy it for life bookmarks

xenodium.com @alvaro — Sun, 04 Nov 2018 02:00:00 GMT

The Monster in the Basement

Thomas Hunter II — Sun, 04 Nov 2018 02:00:00 GMT

The Guten, the Berg, and the Ugly

Sam says you should read this — Sat, 03 Nov 2018 11:58:47 GMT

Or: How I learned to stop worrying, and love like accept Gutenberg.

For those not in the loop, Gutenberg is the controversial new editor for WordPress. Despite not using WordPress on this blog, for reasons poorly explained elsewhere, I do use WordPress for client work. With Gutenberg becoming the default editor in the fast-approaching version 5.0 of WordPress, I figured that I should install the plugin version, and at least try to understand what it's all about.

Why a new editor?

Because the old editor is crap.

Ok, now that I've offended everyone, it's not crap, but it's also not sufficient for b…

Introducing React-Scroll-To v2

Dylan Paulus' Blog — Sat, 03 Nov 2018 02:00:00 GMT

The latest version of a React component I maintain just released, v2 of react-scroll-to. First, it started out as a component I needed, but over time a few people started using it. Because it was initially created for my specific use-case it started to show its warts when new features were ask for. The goals for v2 were two-part: introduce a more flexible API and implement some highly asked for features (scroll by a ref and smooth scrolling).

So what is react-scroll-to? React-scroll-to is a pretty simple component. It allows the control of scrolling either the brower's window, or a component (overflow) to a specific point.

We'll look at a few examples to show off what react-scroll-to can do.
(As a note, in this post I'll only be showing render props, but react-scroll-to also supports higher order components. Examples can be found at: https://github.com/ganderzz/react-scroll-to)

Basics

import React, { Component } from "react";
import { ScrollTo } from "react-scroll-to";

export default class MyComponent extends Component {
  render() {
    return <ScrollTo>{({ scrollTo }) => <a onClick={() => scrollTo({ y: 500 })}>Scroll to Bottom</a>}</ScrollTo>;
  }
}

In the component above we're importing the ScrollTo component from react-scroll-to. This is a render prop, so we'll pass a function as its children. ScrollTo's children has an object as its parameter--this is so the consumer of this component can choose, exactly, what properties they care about. Generally we'll only need scrollTo.

scrollTo() is a function, when called, will either:

Scroll the brower's window based on the given parameters (in this case, scroll 500px from the top of the page)
Scroll all <ScrollArea> components, inside <ScrollTo />, to the given parameters (We'll see this later)

Note: react-scroll-to uses absolute positions--the ability to use relative positioning is coming soon!

When a user clicks on the link, in the example, we'll see the window scroll down 500px!

Scroll Area

The <ScrollArea> component gives scrollTo the ability to scroll components, instead of the window.

import React, { Component } from "react";
import { ScrollTo, ScrollArea } from "react-scroll-to";

export default class MyComponent extends Component {
  render() {
    return (
      <ScrollTo>
        {({ scrollTo }) => (
          <>
            <a onClick={() => scrollTo({ y: 500 })}>Scroll to Bottom</a>

            <ScrollArea>My long content...</ScrollArea>
          </>
        )}
      </ScrollTo>
    );
  }
}

Since a <ScrollArea/> component lives as a child component of <ScrollTo />, whenever we call scrollTo(), it'll scroll all of the <ScrollArea /> components. So in this example when the link is clicked only the "My long content..." area will scroll.

But what if we have multiple <ScrollArea /> components?

import React, { Component } from "react";
import { ScrollTo, ScrollArea } from "react-scroll-to";

export default class MyComponent extends Component {
  render() {
    return (
      <ScrollTo>
        {({ scrollTo }) => (
          <>
            <a onClick={() => scrollTo({ y: 500 })}>Scroll to Bottom</a>

            <ScrollArea>My long content...</ScrollArea>

            <ScrollArea>My long content 2...</ScrollArea>
          </>
        )}
      </ScrollTo>
    );
  }
}

In the case where more than one <ScrollArea /> components exist in a <ScrollTo />, both components will scroll.

What if we only want to scroll one of the <ScrollArea /> components?

import React, { Component } from "react";
import { ScrollTo, ScrollArea } from "react-scroll-to";

export default class MyComponent extends Component {
  render() {
    return (
      <ScrollTo>
        {({ scrollTo }) => (
          <>
            <a onClick={() => scrollTo({ id: "content-two", y: 500 })}>Scroll to Bottom</a>

            <ScrollArea>My long content...</ScrollArea>

            <ScrollArea id="content-two">My long content 2...</ScrollArea>
          </>
        )}
      </ScrollTo>
    );
  }
}

By adding an id property to a <ScrollArea /> we are able to reference that component in our scrollTo() function. scrollTo() then takes another field, 'id', that'll scroll any <ScrollArea /> component that matches that id (id's must be unique).

V2 Features

scrollTo and ScrollArea have both existed in v1; let's look at branch new features.

Smooth Scrolling

Smooth scrolling couldn't be simpler. By adding a smooth: true property to the scrollTo() object, react-scroll-to will automatically apply smooth scrolling.

import React, { Component } from "react";
import { ScrollTo } from "react-scroll-to";

export default class MyComponent extends Component {
  render() {
    return (
      <ScrollTo>
        {({ scrollTo }) => (
          <a onClick={() => scrollTo({ x: 200, smooth: true })}>
            {/* Oh yeah, we can scroll by x also! */}
            Scroll to Bottom
          </a>
        )}
      </ScrollTo>
    );
  }
}

Scroll by Ref

Maybe instead of using a <ScrollArea /> component, we have a ref to a component we want to scroll. Or maybe instead we want to scroll a component outside <ScrollTo />'s children. Now we have the ability by providing the ref through the scrollTo() argument object!

import React, { Component } from "react";
import { ScrollTo } from "react-scroll-to";

export default class MyComponent extends Component {
  myRef = React.createRef();

  render() {
    return (
      <ScrollTo>
        {({ scrollTo }) => (
          <a onClick={() => scrollTo({ ref: this.myRef, y: 200, smooth: true })}>
             Scroll to Bottom
          </a>
        )}
      </ScrollTo>

      <div ref={this.myRef}>
        I'm outside ScrollTo!
      </div>
    );
  }
}

Conclusion

V2 of react-scroll-to provides a lot of behind-the-scenes structural changes that'll allow for a better developer experience while implementing new features. We also added many new features that have been in demand. As things solidify, we hope to expose hooks in react-scroll-to to allow even more flexibility.

Check it out: react-scroll-to
Examples: Click Here

On Gryphons and Their Kin

Ben Overmyer's Site — Sat, 03 Nov 2018 02:00:00 GMT

As one of the founders of Silver Gryphon Games I have a fondness for gryphons. It seems only natural, then, that I write up a few variations of the standard gryphon (or griffon, griffin, or gryfon, as you like) for OSR uses.

Gryphon

Common

Armor Class: 5 Hit Dice: 7 Move: Normal move: 90' (30'); flying: 240' (80') Attacks: 2 claws/1 bite Damage: 1-8/1-8/2-12 No. Appearing: 1 (1-2) Save As: Fighter: 3 Morale: 9 Treasure Type: V Alignment: Neutral

Gryfalcon

Armor Class: 5 Hit Dice: 6 Move: Normal move: 90' (30'); flying: 280' (100') Attacks: 2 claws/1 bite Damage: 1-6/1-6/2-8 No. Appearing: 1 (1-2) Save As: Fighter: 3 Morale: 9 Treasure Type: U Alignment: Neutral

Opinicus

Armor Class: 5 Hit Dice: 7 Move: Normal move: 120' (40'); flying: 240' (80') Attacks: 2 claws/1 bite Damage: 1-8/1-8/2-12 No. Appearing: 1 (1-2) Save As: Fighter: 3 Morale: 9 Treasure Type: V Alignment: Neutral

Demigryph

Armor Class: 5 Hit Dice: 7 Move: 120' (40') Attacks: 2 claws/1 bite Damage: 1-6/1-6/1-10 No. Appearing: 1 (1-2) Save As: Fighter: 3 Morale: 9 Treasure Type: U Alignment: Neutral

Hippogriff

Gryphons are powerful predators, but generally avoid fights unless very hungry. They are found in all manner of climates, but tend to prefer mountainous areas. All variations of gryphon except the demigryph will build their nests (called aeries) in high-up places that are easily defended and difficult for non-flyers to reach. Gryphons are social, intelligent creatures and will sometimes be found in small communities of gryphons. They mate for life, and if their mate dies, they will rarely take another mate, instead living the rest of their life alone. Some cultures use gryphons as fighting-beasts or as mounts. Because they are intelligent creatures, gryphons in this capacity are usually willing participants in this arrangement, but not always. A person having a gryphon as a mount will usually have a strong bond with that creature, and vice versa.

Common: The common variety have the body, tail and back legs of a lion, and the head, wings, breast, and front legs of an eagle. They stand roughly 6-7' high at the shoulder, making them a little larger than most horses.

Gryfalcon: Gryfalcons are very similar to common gryphons, but instead of bearing the likeness of an eagle, resemble a falcon. They are typically smaller and more lithe than their common kin.

Opinicus: The opinicus has the full lower body of a lion, including the front legs, and the head and wings of an eagle. Unlike common gryphons, they do not have tufted ears. Opinici tend to be slightly more aggressive than other gryphons.

Demigryph: A demigryph (or keythong) looks very similar to a common gryphon, but has no wings. Demigryphs are fonder of plains and hills than their kin, and are more likely to interact with humans and other sentient species.

Hippogriff: Hippogriffs have the body, back legs, and tail of a horse instead of a lion. They are otherwise similar to common gryphons. These creatures are rarely seen outside of forested mountains.

Writing test cases with given, when and then

Posts by Petr Stribny — Fri, 02 Nov 2018 20:32:23 GMT

Kill the crunch time heroics

ntietz.com blog - technically a blog — Fri, 02 Nov 2018 02:00:00 GMT

Crunch time has an allure: it feels like if you just push hard enough, you can get more done. You can push hard and get that next release done on time, get those new features out, earn more revenue for your company. Engineers are under immense pressure to deliver more and do it now, and we also feel special: we feel unique, like we are not subject to the fatigue that others experience, or that this project is different and we can do it even when exhausted.

None of this is true, though. We are not special. We are all humans, and for all of us, crunch time is expensive. Some people literally die as a result of crunch time; most of us just end up as worn out shells in poor health, making poor products.

This happens because crunch time is inherently very fatiguing, physically and mentally. During crunch time, you don't get the sleep that your body needs to maintain itself and to recharge the brain, so you end up grouchy and tired and sloppy. During this time, you make big sacrifices: you see your kids less and your partner less; you give up some of your hobbies; you sacrifice your health by giving in to temptations and unhealthy food and alcohol. This all can go to extremes, as happened in 2013 at Bank of America, where an intern died after working for three straight days.

The costs to personal lives are bad enough, but you'd expect them to be justified with something really valuable to the business. I mean, why else would a business press people to do this to themselves? Unfortunately, it isn't so. Long-term, crunch time kills businesses. It leads to lower morale and disengaged employees. Quality declines and passes into negative returns. The mistakes people make while they are fatigued are incredibly expensive, dropping databases or crashing cars or opening gaping security holes.

Look, I know these costs. In the face of these, I decided recently to do some crunch time. We were working on a plan for a prototype and there was a board meeting coming up, and my teammate and I decided to go for it: "let's try to ship a prototype ahead of the board meeting!" It was an ambition born out of a threefold desire: to elevate our company at the board meeting; to elevate the reputation of our engineering team internally; and honestly, to make ourselves look damn good by shipping something incredible in a short timeframe.

We did it. We worked overtime and gave up our hobbies and time with our partners so that we could write a lot of code that ended up looking really slick, working pretty well, and impressing our audience. I actually feel a lot closer to my teammate now, because he and I went through some tough stuff together and really got in sync.

But was it worth it? Definitely not. Looking back, I think that with some extra care (people shielding us from meetings; turning off Slack for us for that time; etc.) we could have gotten the prototype out on the same schedule without the overtime. Even if we missed that deadline, the days after that crunch time were essentially sick days for us both (we could not think clearly enough to write good code) and the week after it was not a great week, either. And the sacrifice to ourselves, to our partners, to our cats - that was something we took too lightly.

The software industry glorifies crunch time. We have this hero mythos where the cowboy coder goes into a cave with a bottle of Mountain Dew and emerges, a sleepless night later, with a beautiful, functional prototype. It's time to kill this hero. This is not how humans work: humans need sleep; humans need downtime to recharge; and humans deserve time for their friends, family, and hobbies.

Let's all fight back against crunch time. You deserve your own time and your own energy, and your business and product will be better for it. Incentives are aligned here: everyone wins when you avoid long hours and crunch time. Let's say no, together.

Links - November 1st, 2018

Avy Faingezicht — Thu, 01 Nov 2018 02:00:00 GMT

I guess this is what happens when I don't share what I've been reading in over two months. Don't miss out on the counterpart post, which is just links to some of the best podcasts I've listened to recently.

Links - November 1st, 2018 (Podcast edition)

Avy Faingezicht — Thu, 01 Nov 2018 02:00:00 GMT

I guess this is what happens when I don't share what I've been listening to in over two months. Don't miss out on the counterpart post, which is just links to some of the best articles I've read recently.

VC is like Soviet Statism

ferrucc.io — Thu, 01 Nov 2018 00:37:21 GMT

VCs do everything you would expect the USSR would do, while being a private company.

VC

Financing projects without asking too many questions. Sometimes people refer to this as VC welfare. If a company puts the right combination of “disruptive”, “blockchain” and “AI” in their slides they receive funding. Showing some APIs, an app with users (real or fake) the money flows him.

With no real technical evaluation.

USSR

Financing projects without asking too many questions. Sometimes people refer to this as Social welfare. If a company puts the right combination of “surplus”, “proletarians” in their presentations they receive funding. Showing some building, a product and have employees (working or not) the money flows him.

My New Mac Setup (2018-2020)

swyx's site RSS Feed — Wed, 31 Oct 2018 18:53:20 GMT

quick checklist for setting up a new macbook

Karatara Fires

one mikro2nd — Wed, 31 Oct 2018 18:35:35 GMT

So here we are… Fires again. Last year our neighbourhood was one of the safest places to be while fire raged all over the Eden District, destroying nearly 2000 buildings. Many friends lost their homes. This year it looked to be our turn. Maybe. At first the fire was well away from us – out on the other side of Karatara. For several days we’ve been breathing smoke, lungs feeling like you’ve smoked a whole pack of cigarettes, eyes gritty and smarting all the time, and that unnerving red glow all around the night sky, but no real hazard to us.

Why you should avoid ORMs (with examples in Node.js)

Thomas Hunter II — Wed, 31 Oct 2018 02:00:00 GMT

Designing a New OSR Game

Ben Overmyer's Site — Wed, 31 Oct 2018 02:00:00 GMT

So lately I've been delving back into the world of OSR role-playing games. In my past several years of playing only modern games, I'd forgotten what it felt like to play an OSR game. I miss the simplicity and the speed of combat. I especially miss the very real sense of danger that my character would die, and I'd have to start over from scratch.

I decided to look at the original source material, a few recent entries into the genre, and some other inspirational material and come up with my own OSR rules.

This blog post represents the first part of that journey.

Simplicity

One of the core themes of OSR games is simplicity. Character classes have very few details to them, and certainly not a deep progression arc. The game is less about characters becoming more mechanically powerful and more about putting these characters into new situations and seeing what they do. In that sense, at least, it's a role-play heavy style.

Another expression of the theme of simplicity is in the resolution mechanic. If you take into account the optional rules from the original game (and here I'm referring specifically to Moldvay Basic), there were attribute tests, but they were simple d20 roll-under-ability-score affairs. There were no bonuses to such rolls. Combat was a little weird with the AC to-hit scheme. I don't miss that part, at least.

A New Game Begins

So for this post, let's say that this game of mine will begin with the basics and riff on that. The original game included only a handful of classes, and three races were classes in themselves. In my game, let's make this a little more complex.

There are classes, and there are races. Classes remain very simple, with only a couple special abilities to give them a bit of flavor. Races are added to the mix, so that each character has both a race and a class, but they also are very simple. This gives any given character less than half a dozen special abilities from both choices, and this doesn't change beyond first level.

"What about spells?" I hear you say. Well, in this system, spells are going to be treated much like equipment. Each spell is a limited-use magic item, probably in the form of a spell-gem or something similar. Magic-using classes will be able to collect and use these in different ways. Non-magic-users won't even be able to hold the things.

"Does that go for divine magic too?" No, divine magic - called miracles in this game - uses something much more freeform. A user of divine magic prays to a deity to enact a miracle through them. This will be a unique roll - I'm toying with a few ideas here.

Most tests not covered by special abilities or other game rules will be roll-under ability score tests. I'll use the original game's 3d6 ability score generation, and d20 roll-under test. That seems just fine.

Saving throws are reduced to a single type. It's basically a hail-Mary play that can save your character from something awful by making it less awful, but still bad. It's also a roll-under target, but it's set, not random. You also only get one per game session.

I like weapons all doing the same damage, so we're going back to that. We'll make things more interesting, though, by making all weapons have a special feature. That will keep them from feeling same-y.

More on these topics is coming in further blog posts.

The biggest mistake Postgres ever made

CRAIG KERSTIENS — Tue, 30 Oct 2018 22:55:56 GMT

Postgres has experienced a long and great run. It is over 20 years old and has a track record of being safe and reliable (which is the top thing I care about in a database). In recent years it has become more cool with things like JSONB, JIT support, and a powerful extension ecosystem. But, Postgres has made some mistakes along the way, the most notable being the name.

Postgres gets its name from Ingress. Ingress was one of the first databases and was lead by Michael Stonebreaker who won a Turing award for Postgres and other works. Ingress began in the early 70s at UC Berkeley, which is still to this day known as a top university when it comes to databases. Out of Ingress came a number of databases you’ll still know today such as SQL Server and Sybase. It also as you may have guessed by now spawned Postgres which means Post-Ingress.

In the early days of Postgres there was no SQL. No not NoSQL, there was not SQL. Postgres had its own query language. It wasn’t until 1995 that Postgres received SQL support, and with its addition of SQL support it updated its name to PostgreSQL.

You see, with Postgres becoming PostgreSQL we began a journey of Postgres being mispronounced for its forseeable future and it is still currently the case. Is it really that big of an issue? Well it’s big enough that the PostgreSQL website has a FAQ including “How to pronounce PostgreSQL”. As it stands today there are two generally accepted names:

post-GRES-que-ell
Postgres

With one of the above there is far less confusion. And in fact I’m not the only one to share this opinion. Tom Lane is a major contributor to every Postgres release for more than the last decade. He’s one of the top 10 contributors to open source in general having worked on the JPEG/PNG/TIFF image formats before coming over to database land. Tom has this classic email in the PostgreSQL mailing list:

 [>> Can i get data in postgre from non-postgre db?
> The name is PostgreSQL or Postgres, not postgre.
It might help to explain that the pronunciation is "post-gres" or
"post-gres-cue-ell", not "post-gray-something".
I heard people making this same mistake in presentations at this
past weekend's Postgres Anniversary Conference :-( Arguably,
the 1996 decision to call it PostgreSQL instead of reverting to
plain Postgres was the single worst mistake this project ever made.
It seems far too late to change now, though.
regards, tom lane

The best part is this was mail was 2006, when it was arguably too late to change the name, and here we are in 2018 with the same issue.

Personally I may start calling it Postgre just to emphasize a point, but for the rest of you just going with Postgres is probably a safe choice.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

The End of a Red Giant

The Technical Blog of James on purpleidea.com — Tue, 30 Oct 2018 14:39:00 GMT

As I’m sure you’ve now heard, Red Hat is being acquired by IBM.

Reactions have ranged from:

The first two emotions everyone experienced.

to:

The classic humourous response.

to:

The classy response.

to:

The thing we're all curious about.

to everything in between. If someone were to leak me a copy of the memo-list 2018-October.txt.gz mailman archive, I’d sure love to see that! (Of course I’m joking and I wouldn’t actually ask anyone to do this.)

Evolutive Integration Testing of HTTP APIs with OpenAPI

Juan Cruz Viotti — Tue, 30 Oct 2018 02:00:00 GMT

This article describes an experimental approach to describing HTTP API integration tests

Networking Essentials: Content Distribution

swyx's site RSS Feed — Mon, 29 Oct 2018 06:46:33 GMT

How CDNs, BitTorrent and Distributed Hash tables work

The waves of compulsion

D13V — Mon, 29 Oct 2018 04:50:34 GMT

I was talking to someone who used to work in finance in New York. They told me they left to write fiction. I thought it’s their dream to be a writer. Then they also told me they don’t enjoy writing that much, with all the editing and revising. Yet, they feel a strong compulsion to write, to discover the characters.

I thought I know what they were talking about. Years ago, I had became a compulsive math solver. At the sight of an interesting math problem I’m was a bull that sees the red cape and has to charge. I still am. Few weeks ago I saw a thumbnail of a Youtube video which described a problem, with the caption “You should be able solve this”. I charged.

I have other compulsions too. I like to write, and to code. Both are similar to math. When it has been too long without one or the other. I get restless.

I cook compulsively. Over the last year I’ve gradually been perfecting a whole wheat seeded sourdough.

And exercising. I am now addicted to it. I lift weights, train Brazilian Jui Jitstu, bike long distances and occasionally try surfing.

These are all excellent compulsions. I call them compulsions as they are stronger than a mere habit.

But there is a dark side. I have disruptive compulsions. Wasting a lot of time on the internet. A lot. And procrastinating. Clicking on another youtube video, I become passenger in my body, which is running on autopilot. It feels restraining, as I realize I it is doing something I don’t want to be doing, and yet I do not stop it. I can’t. I’m caught in the whitewater of a mighty breaker.

These days most of my time is spend on my compulsions.

I believe that to live a happy and rewarding life, I need to develop the positive habits, and enhance my appreciation and productivity. I’ve spent a lot of time trying to get myself to develop beneficial habits, with moderate success. Developing habits is hard. It takes mountainfuls of energy and effort. Listening to motivational podcasts. Struggling and fighting with myself.

But developing compulsions is easier. All I need is to get hooked… just get hooked and give in to it. With time, it becomes a habit too, but it is stronger. Getting hooked is like catching a wave. It takes initial effort to get into it, then it becomes really exciting, as the wave picks up. For a little while nothing else exists. And then, it’s all about keeping the balance and going with it.
Not that know much about surfing.

A rogue wave of compulsion would pick me up and point me towards the rocks, where it might smash me. When I find myself on it, try to remember and look for any other waves that I could catch. They don’t have to be the best waves. They just have to take me away from the rocks, so I have a chance to catch an even better ones, to take me where I want to go, and bring me closer to the people I love.

This is a great article and a great exploration.

🦉 olshansky 🦁 — Mon, 29 Oct 2018 03:59:46 GMT

This is a great article and a great exploration. Your pros and cons list sums everything up, but I’m was curious you’d every leverage this Postgres features for a system you use for work that has to be maintable in the long-term?

Making high-fidelity audio sound like it came through the phone

JonLuca's Blog — Mon, 29 Oct 2018 03:17:38 GMT

Rust bookmarks

xenodium.com @alvaro — Mon, 29 Oct 2018 02:00:00 GMT

Networking Essentials: Rate Limiting and Traffic Shaping

swyx's site RSS Feed — Sun, 28 Oct 2018 23:19:10 GMT

How Computer Networks shape, police, measure, and limit the traffic that flow across them.

Hangman in Haskell

Anardil — Sun, 28 Oct 2018 09:00:00 GMT

Implementing a simple game in functional style

Visualizing SQL Plan Execution Time With FlameGraphs

Tanel Poder Blog — Sun, 28 Oct 2018 07:26:02 GMT

Update: I wrote a follow-up article about adding Loop Counters and Row Counts to SQL Plan FlameGraphs. I also show some old examples of “Bookshelf Graph” visualizations there. Check it out after reading this post first.

Introduction

Brendan Gregg invented and popularized a way to profile & visualize program response time by sampling stack traces and using his FlameGraph concept & tools. This technique is a great way for visualizing metrics in nested hierarchies, what stack-based program execution uses under the hood for invoking and tracking function calls. If you don’t know what FlameGraphs are, I suggest you read Brendan’s explanation first.

Fonts bookmarks

xenodium.com @alvaro — Sun, 28 Oct 2018 03:00:00 GMT

Essential Firefox Tweaks

Thomas Hunter II — Sun, 28 Oct 2018 03:00:00 GMT

Generating Harry Potter Spell Names and Kelewan House Names

Ben E. C. Boyter — Sun, 28 Oct 2018 03:00:00 GMT

Occasionally I try to branch out from working on https://searchcode.com/ https://searchcodeserver.com/ or https://github.com/boyter/scc and find something that I find interesting to toy around with. Usually I go for something that will only take a few hours to play around with.

I had recently re-read a book series I remember loving a while ago which is the Empire Trilogy of books by Feist and Wurts https://en.wikipedia.org/wiki/Empire_Trilogy One of the things that stuck out to me as I was reading was that the names seemed to be a blend of Korean/Japanese/Chinese/Aztec and seems to follow a standard structure.

The Party of Lincoln

The David Brownman Blog — Sun, 28 Oct 2018 03:00:00 GMT

The GOP claims to be the party of Lincoln Let's examine that claim.

Read the whole thing.

GraphQL Interfaces vs Unions

Dan Palmer — Sun, 28 Oct 2018 02:00:00 GMT

GraphQL’s type system allows us to make many invalid states impossible to represent, which improves the usability and reliability of our APIs. Two features of the type system that contribute significantly to this are Interfaces and Unions, however they can be used to address similar design considerations so it’s not always obvious which is the right option.

In this post we’ll look at several examples from the Thread API, and explore whether using an interface or a union is the right option. It’s not always obvious, and in some cases we got it wrong the first time, but after reading this post we hope you’ll have more tools to hand to help you choose the option that’s the best fit in each circumstance.

FatturaElettronica for .NET v0.9 has been released

Nicola Iarocci — Sat, 27 Oct 2018 10:05:25 GMT

FatturaElettronica for .NET v0.9 has been released. The companion Extensions package also hits v0.4.

The main new feature is the FromJson() extension method which allows, you guessed it, for deserialization of a JSON stream into a Fattura class instance:


    var fattura = Fattura.CreateInstance(Instance.Privati);
    fattura.FromJson(new JsonTextReader(new StringReader(json)));
    // or, if FatturaElettronica.Extensions v0.4 is being used:
    fattura.FromJson(json);

    // Invoice is now ready for inspection.
    foreach (var documento in fattura.Body)
    {
        var dati = documento.DatiGenerali.DatiGeneraliDocumento;
        Console.WriteLine($"fatt. num. {dati.Numero} del {dati.Data}");
    }

I should probably mention that FatturaElettronica.Core 0.4 is also out, and that that is the actual place where all the new juice comes from, courtesy of Emanuele Zavallone. You don’t have to directly install Core, it will be pulled down for you by the main packages.

Convert Markdown text to HTML and to plaintext in JavaScript

Posts by Petr Stribny — Fri, 26 Oct 2018 22:34:37 GMT

Trade wars will boost digital manufacturing. But not at home

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 26 Oct 2018 21:00:00 GMT

Last summer, professor J. Pearce argued that “Trade wars will boost digital manufacturing - at consumers' own homes with personal 3D printers”. I hope that that 3D printing-enhanced boost happens, but I also believe, and hope, that it will not happen where prof. Pearce says.

Dealing with Unreliable Software

bastibe.de — Fri, 26 Oct 2018 12:45:00 GMT

About a year ago, I started working on a big comparison study between a bunch of scientific algorithms. Many of these have open-source software available, and I wanted to evaluate them with a large variety of input signals. The problem is, this is scientific code, i.e. the worst code imaginable.

Things this code has done to my computer:

it crashed, throwing an error, and shutting down nicely
it crashed with a segfault, taking the owning process with it
it crashed, leaving temporary files lying around
it crashed, leaving zombie processes lying around
it spin-locked, and never returned
it spin-locked, and forked uncontrollably until all process decriptors were exhausted
it spin-locked, and ate memory uncontrollably until all memory was consumed
it crashed other, unrelated programs (no idea how it managed that)
it corrupted the root file system (no idea how it managed that)

Note that the code did not do any of this intentionally. It was merely code written by non-expert programmers, the problems often a side effect of performance optimizations. The code mostly works fine if called only once or twice. My problems only become apparent if I ran it, say, a few hundred thousand times, with dozens of processes in parallel.

So, how do you deal with this? Multi-threading is not an option, since a segfault would kill the whole program. So it has to be multi-processing. But all the multi-processing frameworks I know will lose all progress if one of the more sinister scenarios from the above list hard-crashed one or more of its processes. I needed a more robust solution.

Basically, the only hope of survival at this point is the kernel. Only the kernel has enough power to rein in rogue processes, and deal with hard crashes. So in my purpose-built multi-processing framework, every task runs in its own process, with inputs and outputs written to unique files. And crucially, if any task does not finish within a set amount of time, it and all of its children are killed.

It took me quite a while to figure out how to do this, so here's the deal:

# start your process with a new process group:
process = Popen(..., start_new_session=True)

# after a timeout, kill the whole process group:
process_group_id = os.getpgid(process.pid)
os.killpg(process_group_id, signal.SIGKILL)

This is the nuclear option. I tried SIGTERM and SIGHUP instead, but programs would happily ignore it. I tried killing or terminating only the process, but that would leave zombie children. Sending SIGKILL to the process group does not take prisoners. The processes do not get a chance to respond or clean up after themselves. But you know what, after months of dealing with this stuff, this is the first time that my experiments actually run reliably for a few days without crashing or exhausting some resource. If that's what it takes, so be it.

Playing Characters You Hate

Ben Overmyer's Site — Fri, 26 Oct 2018 03:00:00 GMT

When you play a role-playing game, you adopt the persona of someone other than you. In American RPGs, this often just means that you're running a fictional version of yourself with a different race and a lot of swords, spells, or guns. But what if the character you play is intentionally your anathema? For example, what if you personally believe wholeheartedly in giving others second chances, but your character believes that if someone fails, they are irredeemably flawed? For your next character, instead of reaching for your comfort zone, try out something brand new. Imagine the group of people you most hate in the real world. Take the second worst after that, and build your character's defining traits based on them. This is not new advice. But here's where it gets interesting: Figure out why your character genuinely believes they have the best possible outlook. Play them that way. Their defining traits that you hate so much may change in subtle ways. Or they might not. Either way, you might gain an interesting insight into the “other side.”

CanvasRecorder.js

XOSH.ORG — Fri, 26 Oct 2018 03:00:00 GMT

I made some visualizations in HTML5 canvas and wanted to record them to display in blog. Could not find a simple way to do that. CCapture.js added too much performance overhead which made animations slow. CanvasRecorder.js uses built in MediaRecorder which is apparently the simplest and most effitient way to record whatever is happening on canvas. NOTE: I have only tested it with Chrome and it should work fine with Firefox

Visualize sorting algorithms side by side

XOSH.ORG — Fri, 26 Oct 2018 03:00:00 GMT

A few weeks ago I found out this imgur gallery of sorting algorithms through a reddit post. The patterns which emerged while sorting random data looked very beautiful and amazing with the only catch that they were not interactive. You can not play with them. I decided make them in JavaScript to see what many different sorting algorithms look like, the patterns they create, while also learning them in the process.

Avoid multitasking to write better code

ntietz.com blog - technically a blog — Fri, 26 Oct 2018 03:00:00 GMT

Multitasking is incredibly alluring. Why go slowly, doing one thing at a time, if you could get a second thing done? Why not fill those five seconds while your code compiles with reading an article about the latest web frameworks?

In fact, multitasking is hiding everywhere in your daily work. Any time you switch from one task to another with the intention of going right back, that counts as multitasking. You might do it without realizing, because...

while your code is compiling, you switch to your browser to check Twitter.
while you are coding, you check Slack briefly to see if anything is going on.
during a meeting, you check Hacker News for anything interesting.

The siren song of multitasking is strong, but the costs are high. Computers are designed for multitasking and parallelism, but humans are not. For us, context switching is very expensive: every time you switch, you lose track of where you were in the previous task, and it can take you 15 minutes to get the state of your codebase back into your head when you switch back. Is it worth spending minutes getting back to where you were just to save the mere seconds you wait for something else to happen? It certainly is not.

This constant context switching also drives down your quality. Every time you context switch, you drop ideas and you drop your focus and this means you cannot engage in very deep thought. You might be able to produce simple CRUD apps this way, but even then, you will miss subtleties in your data model or the domain you are solving for or in how your users will engage with the application. The quality of what you ship decreases when you work in short bursts.

Instead of working in bursts, the way our phones have trained us to do, it is better to spend that time simply idle or bored (in shame, I admit that I looked at Twitter three times while writing this post). Boredom is critical to getting good work done, because the downtime lets your brain explore the non-obvious paths that you may not go down otherwise. This is how you find major bugs, deficiencies in your architecture, or unexpected user experience issues before you ship it.

Multitasking will also make you ship your code late. When you provide estimates, those estimates are usually given optimistically. They are written with the assumption that the requirements are complete, no unexpected complexity is hiding in the problem, and most importantly, that your time will be allocated in sufficiently large chunks to the task. To see this is true, think of a programming task that would take you two hours to complete. How long would it take you if you can do it all in one shot? What about if you can only work in 30 minute chunks? What if you can only work in one minute chunks? If you are limited to one minute chunks and the task has any reasonable complexity, you might never finish it.

It is very rare that you truly need to multitask or switch between tasks rapidly. (If your job is one where you are expected to respond to instant messages instantly, I am truly sorry, and you need to know that there are better opportunities out there. I'm always happy to help point people toward better jobs.) In almost all of your daily work, you can afford to let things go. Those posts on Twitter will still be there when you are at a good stopping point. Those Slack messages will still be there when you are at a good stopping point with your code. That HackerNews post will still be there when your meeting is over. The task you are working on, or the peers you are working with, deserve your full attention, and giving it will let you ship higher quality software on a more reliable schedule.

How to save hundreds of dollars on theology books

Honza Pokorný — Fri, 26 Oct 2018 03:00:00 GMT

We, as theology nerds, love our books. Especially by authors who have long passed on to glory. And who wouldn’t want to save some money?

Here is what you do:

Step 1: Buy an Amazon Kindle

The base model is $79. You can find it on sale during the usual sale events. Or, for a more comfortable reading experience, you can go for the Paperwhite, which is about twice as much. Both are excellent options. The advantage of the Paperwhite is the backlight, and the higher screen resolution.

Step 2: Acquire books

Head over to Monergism’s huge list of free theology ebooks. Scroll through the list, find something you like, download it to your Kindle.

Step 3: Profit

If there are 400 books, and if you downloaded each one, and if the printed paper copy cost at least $1 (which it probably costs a lot more), you are saving at least $300.

If we assume that each book on the list has an average price of $30, you are saving about $12,000.

Exploring calendar/booking apps in reactjs

Yasha Solutions — Thu, 25 Oct 2018 13:06:28 GMT

Looking to build a calendar app in React. You are not alone. I did some research to see what the world published on this so far, and here is what I have found. Room Booking System A room booking system built with MongoDB, Express, Node.js and ReactJS. This one has a lot going on. But it’s interesting to have a look at it. It seems like it’s a uni project or something… at least it’s organised as such.

Geopolitics For Fun & Profit

Privacy, Power, & Protection In The Cyber Century — Thu, 25 Oct 2018 10:38:04 GMT

1750 words, 7 minutes. Sketch by Sir William Rothenstein, 1933. “Who rules East Europe commands the Heartland. Who rules the Heartland commands the World-Island. Who rules the World-Island commands the world." - Sir Halford Mackinder, Democratic Ideals and Reality. 1919. Do you work in technology, are you building a company? You should think about how your product or service fits with the wider world because aligning with large movements is a winning system.

Mipmapping with Bidirectional Techniques

Code & Visuals — Thu, 25 Oct 2018 03:00:00 GMT

1. Introduction
2. Texture Caches and Mipmaps
3. Mipmap Level Selection and Ray Differentials
4. Ray Differentials and Path Tracing
5. Ray Differentials and Bidirectional Techniques

6. Camera-Based Mipmap Level Selection
7. Results
8. Additional Renders
9. References

Introduction

One major feature that differentiates production-capable renderers from hobby or research renderers is a texture caching system. A well-implemented texture caching system is what allows a production renderer to render scenes with potentially many TBs of textures, but in a reasonable footprint that fits in a few dozen or a hundred-ish GB of RAM. Pretty much every production renderer today has a robust texture caching system; Arnold famously derives a significant amount of performance from an extremely efficient texture cache implementation, and Vray/Corona/Renderman/Hyperion/etc. all have their own, similarly efficient systems.

In this post and the next few posts, I’ll write about how I implemented a tiled, mipmapped texture caching system in my hobby renderer, Takua Renderer. I’ll also discuss some of the interesting challenges I ran into along the way. This post will focus on the mipmapping part of the system. Building a tiled mipmapping system that works well with bidirectional path tracing techniques was particularly difficult, for reasons I’ll discuss later in this post. I’ll also review the academic literature on ray differentials and mipmapping with path tracing, and I’ll take a look at what several different production renderers do. The scene I’ll use as an example in this post is a custom recreation of a forest scene from Evermotion’s Archmodels 182, rendered entirely using Takua Renderer (of course):

Texture Caches and Mipmaps

Texture caching is typically coupled with some form of a tiled, mipmapped [Williams 1983] texture system; the texture cache holds specific tiles of an image that were accessed, as opposed to an entire texture. These tiles are typically lazy-loaded on demand into a cache [Peachey 1990], which means the renderer only needs to pay the memory storage cost for only parts of a texture that the renderer actually accesses.

The remainder of this section and the next section of this post are a recap of what mipmaps are, mipmap level selection, and ray differentials for the less experienced reader. I also discuss a bit about what techniques various production renderers are known to use today. If you already know all of this stuff, I’d suggest skipping down a bit to the section titled “Ray Differentials and Bidirectional Techniques”.

Mipmapping works by creating multiple resolutions of a texture, and for a given surface, only loading the last resolution level where the frequency detail falls below the Nyquist limit when viewed from the camera. Since textures are often much more high resolution than the final framebuffer resolution, mipmapping means the renderer can achieve huge memory savings, since for objects further away from the camera, most loaded mip levels will be significantly lower resolution than the original texture. Mipmaps start with the original full resolution texture as “level 0”, and then each level going up from level 0 is half the resolution of the previous level. The highest level is the level at which the texture can no longer be halved in resolution again.

Below is an example of a mipmapped texture. The texture below is the diffuse albedo texture for the fallen log that is in the front of the scene in Figure 1, blocking off the path into the woods. On the left side of Figure 2 is level 1 of this texture (I have omitted level 0 both for image size reasons and because the original texture is from a commercial source, which I don’t have the right to redistribute in full resolution). On the right side, going from the top on down, are levels 2 through 11 of the mipmap. I’ll talk about the “tiled” part in a later post.

Before diving into details, I need to make a major note: I’m not going to write too much about texture filtering for now, mainly because I haven’t done much with texture filtering in Takua at all. Mipmapping was originally invented as an elegant solution to the problem of expensive texture filtering in rasterized rendering; when a texture had detail that was more high frequency than the distance between neighboring pixels in the framebuffer, aliasing would occur when the texture was sampled. Mipmaps are typically generated with pre-computed filtering for mip levels above the original resolution, allowing for single texture samples to appear antialiased. For a comprehensive discussion of texture filtering, how it relates to mipmaps, and more advanced techniques, see section 10.4.3 in Physically Based Rendering 3rd Edition [Pharr et al. 2016].

For now, Takua just uses a point sampler for all texture filtering; my interest in mipmaps is mostly for memory efficiency and texture caching instead of filtering. My thinking is that in a path tracer that is going to generate hundreds or even thousands of paths for each framebuffer pixel, the need for single-sample antialiasing becomes somewhat lessened, since we’re already basically supersampling. Good texture filtering is still ideal of course, but being lazy and just relying on supersampling to get rid of texture aliasing in primary visibility is… not necessarily the worst short-term solution in the world. Furthermore, relying on just point sampling means each texture sample only requires two texture lookups: one from the integer mip level and one from the integer mip level below the continuous float mip level at a sample point (see the next section for more on this). Using only two texture lookups per texture sample is highly efficient due to minimized memory access and minimized branching in the code. Interestingly, the Moonray team at Dreamworks Animation arrived at more or less the same conclusion [Lee et al. 2017]; they point out in their paper that geometric complexity, for all intents and purposes, has an infinite frequency, whereas pre-filtered mipmapped textures are already band limited. As a result, the number of samples required to resolve geometric aliasing should be more than enough to also resolve any texture aliasing. The Moonray team found that this approach works well enough to be their default mode in production.

Mipmap Level Selection and Ray Differentials

The trickiest part of using mipmapped textures is figuring out what mipmap level to sample at any given point. Since the goal is to find a mipmap level with a frequency detail as close to the texture sampling rate as possible, we need to have a sense of what the texture sampling rate at a given point in space relative to the camera will be. More precisely, we want the differential of the surface parameterization (a.k.a. how uv space is changing) with respect to the image plane. Since the image plane is two-dimensional, we will end up with a differential for each uv axis with respect to each axis of the image plane; we call these differentials dudx/dvdx and dudy/dvdy, where u/v are uv coordinates and x/y are image plane pixel coordinates. Calculating these differentials is easy enough in a rasterizer: for each image plane pixel, take the texture coordinate of the fragment and subtract with the texture coordinates of the neighboring fragments to get the gradient of the texture coordinates with respect to the image plane (a.k.a. screen space), and then scale by the texture resolution. Once we have dudx/dvdx and dudy/dvdy, for a non-fancy box filter all we have to do to get the mipmap level is take the longest of these gradients and calculate its logarithm base 2. A code snippet might look something like this:

float mipLevelFromDifferentialSurface(const float dudx,
                                      const float dvdx,
                                      const float dudy,
                                      const float dvdy,
                                      const int maxMipLevel) {
    float width = max(max(dudx, dvdx), max(dudy, dvdy));
    float level = float(maxMipLevel) + log2(width);
    return level;
}

Notice that the level value is a continuous float. Usually, instead of rounding level to an integer, a better approach is to sample both of the integer mipmap levels above and below the continuous level and blend between the two values using the fractional part of level. Doing this blending helps immensely with smoothing transitions between mipmap levels, which can become very important when rendering an animated sequence with camera movement.

In a ray tracer, however, figuring out dudx/dvdx and dudy/dvdy is not as easy as in a rasterizer. If we are only considering primary rays, we can do something similar to the rasterization case: fire a ray from a given pixel and fire rays from the neighboring pixels, and calculate the gradient of the texture coordinates with respect to screen space (the screen space partial derivatives) by examining the hit points of each neighboring ray that hits the same surface as the primary ray. This approach rapidly falls apart though, for the following reasons and more:

If a ray hits a surface but none of its neighboring rays hit the same surface, then we can’t calculate any differentials and must fall back to point sampling the lowest mip level. This isn’t a problem in the rasterization case, since rasterization will run through all of the polygons that make up a surface, but in the ray tracing case, we only know about surfaces that we actually hit with a ray.
For secondary rays, we would need to trace secondary bounces not just for a given pixel’s ray, but also its neighboring rays. Doing so would be necessary since, depending on the bsdf at a given surface, the distance between the main ray and its neighbor rays can change arbitrarily. Tracing this many additional rays quickly becomes prohibitively expensive; for example, if we are considering four neighbors per pixel, we are now tracing five times as many rays as before.
We would also have to continue to guarantee that neighbor secondary rays continue hitting the same surface as the main secondary ray, which will become arbitrarily difficult as bxdf lobes widen or narrow.

A better solution to these problems is to use ray differentials [Igehy 1999], which is more or less just a ray along with the partial derivative of the ray with respect to screen space. Thinking of a ray differential as essentially similar to a ray with a width or a cone, similar to beam tracing [Heckbert and Hanrahan 1984], pencil tracing [Shinya et al. 1987], or cone tracing [Amanatides 1984], is not entirely incorrect, but ray differentials are a bit more nuanced than any of the above. With ray differentials, instead of tracing a bunch of independent neighbor rays with each camera ray, the idea is to reconstruct dudx/dvdy and dudy/dvdy at each hit point using simulated offset rays that are reconstructed using the ray’s partial derivative. Ray differentials are generated alongside camera rays; when a ray is traced from the camera, offset rays are generated for a single neighboring pixel vertically and a single neighboring pixel horizontally in the image plane. Instead of tracing these offset rays independently, however, we always assume they are at some angular width from main ray. When the main ray hits a surface, we need to calculate for later use the differential of the surface at the intersection point with respect to uv space, which is called dpdu and dpdv. Different surface types will require different functions to calculate dpdu and dpdv; for a triangle in a triangle mesh, the code requires the position and uv coordinates at each vertex:

DifferentialSurface calculateDifferentialSurfaceForTriangle(const vec3& p0,
                                                            const vec3& p1,
                                                            const vec3& p2,
                                                            const vec2& uv0,
                                                            const vec2& uv1,
                                                            const vec2& uv2) {
    vec2 duv02 = uv0 - uv2;
    vec2 duv12 = uv1 - uv2;
    float determinant = duv02[0] * duv12[1] - duv02[1] * duv12[0];

    vec3 dpdu, dpdv;

    vec3 dp02 = p0 - p2;
    vec3 dp12 = p1 - p2;
    if (abs(determinant) == 0.0f) {
        vec3 ng = normalize(cross(p2 - p0, p1 - p0));
        if (abs(ng.x) > abs(ng.y)) {
            dpdu = vec3(-ng.z, 0, ng.x) / sqrt(ng.x * ng.x + ng.z * ng.z);
        } else {
            dpdu = vec3(0, ng.z, -ng.y) / sqrt(ng.y * ng.y + ng.z * ng.z);
        }
        dpdv = cross(ng, dpdu);
    } else {
        float invdet = 1.0f / determinant;
        dpdu = (duv12[1] * dp02 - duv02[1] * dp12) * invdet;
        dpdv = (-duv12[0] * dp02 + duv02[0] * dp12) * invdet;
    }
    return DifferentialSurface(dpdu, dpdv);
}

Calculating surface differentials does add a small bit of overhead to the renderer, but the cost can be minimized with some careful work. The naive approach to surface differentials is to calculate them with every intersection point and return them as part of the hit point information that is produced by ray traversal. However, this computation is wasted if the shading operation for a given hit point doesn’t actually end up doing any texture lookups. In Takua, surface differentials are calculated on demand at texture lookup time instead of at ray intersection time; this way, we don’t have to pay the computational cost for the above function unless we actually need to do texture lookups. Takua also supports multiple uv sets per mesh, so the above function is parameterized by uv set ID, and the function is called once for each uv set that a texture specifies. Surface differentials are also cached within a shading operation per hit point, so if a shader does multiple texture lookups within a single invocation, the required surface differentials don’t need to be redundantly calculated.

Sony Imageworks’ variant of Arnold (we’ll refer to it as SPI Arnold to disambiguate from Solid Angle’s Arnold) does something even more advanced [Kulla et al. 2018]. Instead of the above explicit surface differential calculation, SPI Arnold implements an automatic differentiation system utilizing dual arithmetic [Piponi 2004]. SPI Arnold extensively utilizes OSL for shading; this means that they are able to trace at runtime what dependencies a particular shader execution path requires, and therefore when a shader needs any kind of derivative or differential information. The calls to the automatic differentiation system are then JITed into the shader’s execution path, meaning shader authors never have to be aware of how derivatives are computed in the renderer. The SPI Arnold team’s decision to use dual arithmetic based automatic differentiation is influenced by lessons they had previously learned with BMRT’s finite differencing system, which required lots of extraneous shading computations for incoherent ray tracing [Gritz and Hahn 1996]. At least for my purposes, though. I’ve found that the simpler approach I have taken in Takua is sufficiently negligible in both final overhead and code complexity that I’ll probably skip something like the SPI Arnold approach for now.

Once we have the surface differential, we can then approximate the local surface geometry at the intersection point with a tangent plane, and intersect the offset rays with the tangent plane. To find the corresponding uv coordinates for the offset ray tangent plane intersection planes, dpdu/dpdv, the main ray intersection point, and the offset ray intersection points can be used to establish a linear system. Solving this linear system leads us directly to dudx/dudy and dvdx/dvdy; for the exact mathematical details and explanation, see section 10.1 in Physically Based Rendering 3rd Edition. The actual code might look something like this:

// This code is heavily aped from PBRT v3; consult the PBRT book for details!
vec4 calculateScreenSpaceDifferential(const vec3& p,            // Surface intersection point
                                      const vec3& n,            // Surface normal
                                      const vec3& origin,       // Main ray origin
                                      const vec3& rDirection,   // Main ray direction
                                      const vec3& xorigin,      // Offset x ray origin
                                      const vec3& rxDirection,  // Offset x ray direction
                                      const vec3& yorigin,      // Offset y ray origin
                                      const vec3& ryDirection,  // Offset y ray direction
                                      const vec3& dpdu,         // Surface differential w.r.t. u
                                      const vec3& dpdv          // Surface differential w.r.t. v
                                      ) {
    // Compute offset-ray intersection points with tangent plane
    float d = dot(n, p);
    float tx = -(dot(n, xorigin) - d) / dot(n, rxDirection);
    vec3 px = origin + tx * rxDirection;
    float ty = -(dot(n, yorigin) - d) / dot(n, ryDirection);
    vec3 py = origin + ty * ryDirection;
    vec3 dpdx = px - p;
    vec3 dpdy = py - p;

    // Compute uv offsets at offset-ray intersection points
    // Choose two dimensions to use for ray offset computations
    ivec2 dim;
    if (std::abs(n.x) > std::abs(n.y) && std::abs(n.x) > std::abs(n.z)) {
        dim = ivec2(1,2);
    } else if (std::abs(n.y) > std::abs(n.z)) {
        dim = ivec2(0,2);
    } else {
        dim = ivec2(0,1);
    }
    // Initialize A, Bx, and By matrices for offset computation
    mat2 A;
    A[0][0] = ds.dpdu[dim[0]];
    A[0][1] = ds.dpdv[dim[0]];
    A[1][0] = ds.dpdu[dim[1]];
    A[1][1] = ds.dpdv[dim[1]];
    vec2 Bx(px[dim[0]] - p[dim[0]], px[dim[1]] - p[dim[1]]);
    vec2 By(py[dim[0]] - p[dim[0]], py[dim[1]] - p[dim[1]]);

    float dudx, dvdx, dudy, dvdy;

    // Solve two linear systems to get uv offsets
    auto solveLinearSystem2x2 = [](const mat2& A, const vec2& B, float& x0, float& x1) -> bool {
        float det = A[0][0] * A[1][1] - A[0][1] * A[1][0];
        if (abs(det) < (float)constants::EPSILON) {
            return false;
        }
        x0 = (A[1][1] * B[0] - A[0][1] * B[1]) / det;
        x1 = (A[0][0] * B[1] - A[1][0] * B[0]) / det;
        if (std::isnan(x0) || std::isnan(x1)) {
            return false;
        }
        return true;
    };
    if (!solveLinearSystem2x2(A, Bx, dudx, dvdx)) {
        dudx = dvdx = 0.0f;
    }
    if (!solveLinearSystem2x2(A, By, dudy, dvdy)) {
        dudy = dvdy = 0.0f;
    }

    return vec4(dudx, dvdx, dudy, dvdy);
}

Now that we have dudx/dudy and dvdx/dvdy, getting the proper mipmap level works just like in the rasterization case. The above approach is exactly what I have implemented in Takua Renderer for camera rays. Similar to surface differentials, Takua caches dudx/dudy and dvdx/dvdy computations per shader invocation per hit point, so that multiple textures utilizing the same uv set dont’t require multiple redundant calls to the above function.

The ray derivative approach to mipmap level selection is basically the standard approach in modern production rendering today for camera rays. PBRT [Pharr et al. 2016], Mitsuba [Jakob 2010], and Solid Angle’s version of Arnold [Georgiev et al. 2018] all use ray differential systems based on this approach for camera rays. Renderman [Christensen et al. 2018] uses a simplified version of ray differentials that only tracks two floats per ray, instead of the four vectors needed to represent a full ray differential. Renderman tracks a width at each ray’s origin, and a spread value representing the linear rate of change of the ray width over a unit distance. This approach does not encode as much information as the full ray derivative approach, but nonetheless ends up being sufficient since in a path tracer, every pixel essentially ends up being supersampled. Hyperion [Burley et al. 2018] uses a similarly simplified scheme.

A brief side note: being able to calculate the differential for surface normals with respect to screen space is useful for bump mapping, among other things, and the calculation is directly analogous to the pseudocode above for calculateDifferentialSurfaceForTriangle() and calculateScreenSpaceDifferential(), just with surface normals substituted in for surface positions.

Ray Differentials and Path Tracing

We now know how to calculate filter footprints using ray differentials for camera rays, which is great, but what about secondary rays? Without ray differentials for secondary rays, path tracing texture access behavior degrades severely, since secondary rays have to fall back to point sampling textures at the lowest mip level. A number of different schemes exist for calculating filter footprints and mipmap levels for secondary rays; here are a few that have been presented in literature and/or are known to be in use in modern production renderers:

Igehy [1999] demonstrates how to propagate ray differentials through perfectly specular reflection and refraction events, which boil down to some simple extensions to the basic math for optical reflection and refraction. However, we still need a means for handling glossy (so really, non-zero surface roughness), which requires an extended version of ray differentials. Path differentials [Suykens and Willems 2001] consider more than just partial derivatives for each screen space pixel footprint; with path differentials, partial derivatives can also be taken at each scattering event along a number of dimensions. As an example, for handling a arbitrarily shaped BSDF lobe, new partial derivatives can be calculated along some parameter of the lobe that describes the shape of the lobe, which takes the form of a bunch of additional scattering directions around the main ray’s scattering direction. If we imagine looking down the main scattering direction and constructing a convex hull around the additional scattering directions, the result is a polygonal footprint describing the ray differential over the scattering event. This footprint can then be approximated by finding the major and minor axis of the polygonal footprint. While the method is general enough to handle arbitrary factors impacting ray directions, unfortunately it can be fairly complex and expensive to compute in practice, and differentials for some types of events can be very difficult to derive. For this reason, none of the major production renderers today actually use this approach. However, there is a useful observation that can be drawn from path differentials: generally, in most cases, primary rays have narrow widths and secondary rays have wider widths [Christensen et al. 2003]; this observation is the basis of the ad-hoc techniques that most production renderers utilize.

Recently, research has appeared that provides an entirely different, more principled approach to selecting filter footprints, based on covariance tracing [Belcour et al. 2013]. The high-level idea behind covariance tracing is that local light interaction effects such as transport, occlusion, roughness, etc. can all be encoded as 5D covariance matrices, which in turn can be used to determine ideal sampling rates. Covariance tracing builds an actual, implementable rendering algorithm on top of earlier, mostly theoretical work on light transport frequency analysis [Durand et al. 2005]. Belcour et al. [2017] presents an extension to covariance tracing for calculating filter footprints for arbitrary shading effects, including texture map filtering. The covariance-tracing based approach differs from path differentials in two key areas. While both approaches operate in path space, path differentials are much more expensive to compute than the covariance-tracing based technique; path differential complexity scales quadratically with path length, while covariance tracing only ever carries a single covariance matrix along a path for a given effect. Also, path differentials can only be generated starting from the camera, whereas covariance tracing works from the camera and the light; in the next section, we’ll talk about why this difference is critically important.

Covariance tracing based techniques have a lot of promise, and are the best known approach to date for for selecting filter footprints along a path. The original covariance tracing paper had some difficulty with handling high geometric complexity; covariance tracing requires a voxelized version of the scene for storing local occlusion covariance information, and covariance estimates can degrade severely if the occlusion covariance grid is not high resolution enough to capture small geometric details. For huge production scale scenes, geometric complexity requirements can make covariance tracing either slow due to huge occlusion grids, or degraded in quality due to insufficiently large occlusion grids. However, the voxelization step is not as much of a barrier to practicality as it may initially seem. For covariance tracing based filtering, visibility can be neglected, so the entire scene voxelization step can be skipped; Belcour et al. [2017] demonstrates how. Since covariance tracing based filtering can be used with the same assumptions and data as ray differentials but is both superior in quality and more generalizable than ray differentials, I would not be surprised to see more renderers adopt this technique over time.

As of present, however, instead of using any of the above techniques, pretty much all production renderers today use various ad-hoc methods for tracking ray widths for secondary rays. SPI Arnold tracks accumulated roughness values encountered by a ray: if a ray either encounters a diffuse event or reaches a sufficiently high accumulated roughness value, SPI Arnold automatically goes to basically the highest available MIP level [Kulla et al. 2018]. This scheme produces very aggressive texture filtering, but in turn provides excellent texture access patterns. Solid Angle Arnold similarly uses an ad-hoc microfacet-inspired heuristic for secondary rays [Georgiev et al. 2018] . Renderman handles reflection and refraction using something similar to Igehy [1999], but modified for the single-float-width ray differential representation that Renderman uses [Christensen et al. 2018]. For glossy and diffuse events, Renderman uses an empirically determined heuristic where higher ray width spreads are driven by lower scattering direction pdfs. Weta’s Manuka has a unified roughness estimation system built into the shading system, which uses a mean cosine estimate for figuring out ray differentials [Fascione et al. 2018].

Generally, roughness driven heuristics seem to work reasonably well in production, and the actual heuristics don’t actually have to be too complicated! In an experimental branch of PBRT, Matt Pharr found that a simple heuristic that uses a ray differential covering roughly 1/25th of the hemisphere for diffuse events and 1/100th of the hemisphere for glossy events generally worked reasonably well [Pharr 2017].

Ray Differentials and Bidirectional Techniques

So far everything we’ve discussed has been for unidirectional path tracing that starts from the camera. What about ray differentials and mip level selection for paths starting from a light, and by extension, for bidirectional path tracing techniques? Unfortunately, nobody has a good, robust solution for calculating ray differentials for light path! Calculating ray differentials for light paths is fundamentally something of an ill defined problem: a ray differential has to be calculated with respect to a screen space pixel footprint, which works fine for camera paths since the first ray starts from the camera, but for light paths, the last ray in the path is the one that reaches the camera. With light paths, we have something of a chicken-and-egg problem; there is no way to calculate anything with respect to a screen space pixel footprint until a light path has already been fully constructed, but the shading computations required to construct the path are the computations that want differential information in the first place. Furthermore, even if we did have a good way to calculate a starting ray differential from a light, the corresponding path differential can’t become as wide as in the case of a camera path, since at any given moment the light path might scatter towards the camera and therefore needs to maintain a footprint no wider than a single screen space pixel.

Some research work has gone into this question, but more work is needed on this topic. The previously discussed covariance tracing based technique [Belcour et al. 2017] does allow for calculating an ideal texture filtering width and mip level once a light path is fully constructed, but again, the real problem is that footprints need to be available during path construction, not afterwards. With bidirectional path tracing, things get even harder. In order to keep a bidirectional path unbiased, all connections between camera and light path vertices must be consistent in what mip level they sample; however, this is difficult since ray differentials depend on the scattering events at each path vertex. Belcour et al. [2017] demonstrates how important consistent texture filtering between two vertices is.

Currently, only a handful of production renderers have extensive support for bidirectional techniques; of the ones that do, the most common solution to calculating ray differentials for bidirectional paths is… simply not to at all. Unfortunately, this means bidirectional techniques must rely on point sampling the lowest mip level, which defeats the whole point of mipmapping and destroys texture caching performance. The Manuka team alludes to using ray differentials for photon map gather widths in VCM and notes that these ray differentials are implemented as part of their manifold next event estimation system [Fascione et al. 2018], but there isn’t enough detail in their paper to be able to figure out how this actually works.

Camera-Based Mipmap Level Selection

Takua has implementations of standard bidirectional path tracing, progressive photon mapping, and VCM, and I wanted mipmapping to work with all integrator types in Takua. I’m interested in using Takua to render scenes with very high complexity levels using advanced (often bidirectional) light transport algorithms, but reaching production levels of shading complexity without a mipmapped texture cache simply is not possible without crazy amounts of memory (where crazy is defined as in the range of dozens to hundreds of GB of textures or more). However, for the reasons described above, standard ray differential based techniques for calculating mip levels weren’t going to work with Takua’s bidirectional integrators.

The lack of a ray differential solution for light paths left me stuck for some time, until late in 2017, when I got to read an early draft of what eventually became the Manuka paper [Fascione et al. 2018] in the ACM Transactions on Graphics special issue on production rendering. I highly recommend reading all five of the production renderer system papers in the ACM TOG special issue. However, if you’re already generally familiar with how a modern PBRT-style renderer works and only have time to read one paper, I would recommend the Manuka paper simply because Manuka’s architecture and the set of trade-offs and choices made by the Manuka team are so different from what every other modern PBRT-style production path tracer does. What I eventually implemented in Takua is directly inspired by Manuka, although it’s not what Manuka actually does (I think).

The key architectural feature that differentiates Manuka from Arnold/Renderman/Vray/Corona/Hyperion/etc. is its shade-before-hit architecture. I should note here that in this context, shade refers to the pattern generation part of shading, as opposed to the bsdf evaluation/sampling part of shading. The brief explanation (you really should go read the full paper) is that Manuka completely decouples pattern generation from path construction and path sampling, as opposed to what all other modern path tracers do. Most modern path tracers use shade-on-hit, which means pattern generation is lazily evaluated to produce bsdf parameters upon demand, such as when a ray hits a surface. In a shade on hit architecture, pattern generation and path sampling are interleaved, since path sampling depends on the results of pattern generation. Separating out geometry processing from path construction is fairly standard in most modern production path tracers, meaning subdivision/tessellation/displacement happens before any rays are traced, and displacement usually involves some amount of pattern generation. However, no other production path tracer separates out all of pattern generation from path sampling the way Manuka does. At render startup, Manuka runs geometry processing, which dices all input geometry into micropolygon grids, and then runs pattern generation on all of the micropolygons. The result of pattern generation is a set of bsdf parameters that are baked into the micropolygon vertices. Manuka then builds a BVH and proceeds with normal path tracing, but at each path vertex, instead of having to evaluate shading graphs and do texture lookups to calculate bsdf parameters, the bsdf parameters are looked up directly from the pre-calculated cached values baked into the micropolygon vertices. Put another way, Manuka is a path tracer with a REYES-style shader execution model [Cook et al. 1987] instead of a PBRT-style shader execution model; Manuka preserves the grid-based shading coherence from REYES while also giving more flexibility to path sampling and light transport, which no longer have to worry about pattern generation making shading slow.

So how does any of this relate to the bidirectional path tracing mip level selection problem? The answer is: in a shade-before-hit architecture, by the time the renderer is tracing light paths, there is no need for mip level selection because there are no texture lookups required anymore during path sampling. During path sampling, Manuka evaluates bsdfs at each hit point using pre-shaded parameters that are bilinearly interpolated from the nearest micropolygon vertices; all of the texture lookups were already done in the pre-shade phase of the renderer! In other words, at least in principle, a Manuka-style renderer can entirely sidestep the bidirectional path tracing mip level selection problem (although I don’t know if Manuka actually does this or not). Also, in a shade-before-hit architecture, there are no concerns with biasing bidirectional path tracing from different camera/light path vertex connections seeing different mip levels. Since all mip level selection and texture filtering decisions take place before path sampling, the view of the world presented to path sampling is always consistent.

Takua is not a shade-before-hit renderer though, and for a variety of reasons, I don’t plan on making it one. Shade-before-hit presents a number of tradeoffs which are worthwhile in Manuka’s case because of the problems and requirements the Manuka team aimed to solve and meet, but Takua is a hobby renderer aimed at something very different from Manuka. The largest drawback of shade-before-hit is the startup time associated with having to pre-shade the entire scene; this startup time can be quite large, but in exchange, the total render time can be faster as path sampling becomes more efficient. However, in a number of workflows, the time to a full render is not nearly as important as the time to a minimum sample count at which point an artistic decision can be made on a noisy image; beyond this point, full render time is less important as long as it is within a reasonable ballpark. Takua currently has a fast startup time and reaches a first set of samples quickly, and I wanted to keep this behavior. As a result, the question then became: in a shade-on-hit architecture, is there a way to emulate shade-before-hit’s consistent view of the world, where texture filtering decisions are separated from path sampling?

The approach I arrived at is to drive mip level selection based on only a world-space distance-to-camera metric, with no dependency at all on the incoming ray at a given hit point. This approach is… not even remotely novel; in a way, this approach is probably the most obvious solution of all, but it took me a long time and a circuitous path to arrive at for some reason. Here’s the high-level overview of how I implemented a camera-based mip level selection technique:

At render startup time, calculate a ray differential for each pixel in the camera’s image plane. The goal is to find the narrowest differential in each screen space dimension x and y. Store this piece of information for later.
At each ray-surface intersection point, calculate the differential surface.
Create a ‘fake’ ray going from the camera’s origin position to the current intersection point, with a ray differential equal to the minimum differential in each direction found in step 1.
Calculate dudx/dudy and dvdx/dvdy using the usual method presented above, but using the fake ray from step 3 instead of the actual ray.
Calculate the mip level as usual from dudx/dudy and dvdx/dvdy.

The rational for using the narrowest differentials in step 1 is to guarantee that texture frequency remains sub-pixel for the all pixels in screen space, even if that means that we might be sampling some pixels at a higher resolution mip level than whatever screen space pixel we’re accumulating radiance too. In this case, being overly conservative with our mip level selection is preferable to visible texture blurring from picking a mip level that is too low resolution.

Takua uses the above approach for all path types, including light paths in the various bidirectional integrators. Since the mip level selection is based entirely on distance-to-camera, as far as the light transport integrators are concerned, their view of the world is entirely consistent. As a result, Takua is able to sidestep the light path ray differential problem in much the same way that a shade-before-hit architecture is able to. There are some particular implementation details that are slightly complicated by Takua having support for multiple uv sets per mesh, but I’ll write about multiple uv sets in a later post. Also, there is one notable failure scenario, which I’ll discuss more in the results section.

Results

So how well does camera-based mipmap level selection work compared to a more standard approach based on path differentials or ray widths from the incident ray? Typically in a production renderer, mipmaps work in conjunction with tiled textures, where tiles are a fixed size (unless a tile is in a mipmap level with a total resolution smaller than the tile resolution). Therefore, the useful metric to compare is how many texture tiles each approach access throughout the course of a render; the more an approach accesses higher mipmap levels (meaning lower resolution mipmap levels), the fewer tiles in total should be accessed since lower resolution mipmap levels have fewer tiles.

For unidirectional path tracing from the camera, we can reasonably expect the camera-based approach to perform less well than a path differential or ray width technique (which I’ll call simply ‘ray-based’). In the camera-based approach, every texture lookup has to use a footprint corresponding to approximately a single screen space pixel footprint, whereas in a more standard ray-based approach, footprints get wider with each successive bounce, leading to access to higher mipmap levels. Depending on how aggressively ray widths are widened at diffuse and glossy events, ray-based approaches can quickly reach the highest mipmap levels and essentially spend the majority of the render only accessing high mipmap levels.

For bidirectional integrators though, the camera-based techinque has the major advantage of being able to provide reasonable mipmap levels for both camera and light paths, whereas the more standard ray-based approaches have to fall back to point sampling the lowest mipmap level for light paths. As a result, for bidirectional paths we can expect that a ray-based approach should perform somewhere in between how a ray-based approach performs in the unidirectional case and how point sampling only the lowest mipmap level performs in the unidirectional case.

As a baseline, I also implemented a ray-based approach with a relatively aggressive widening heuristic for glossy and diffuse events. For the forest scene from Figure 1, I got the following results at 1920x1080 resolution with 16 samples per pixel. I compared unidirectional path tracing from the camera and standard bidirectional path tracing; statistics are presented as total number of texture tiles accessed divided by total number of texture tiles across all mipmap levels. The lower the percentage, the better:

16 SPP 1920x1080 Unidirectional (PT)
    No mipmapping:                       314439/745394 tiles (42.18%)
    Ray-based level selection:           103206/745394 tiles (13.84%)
    Camera-based level selection:        104764/745394 tiles (14.05%)

16 SPP 1920x1080 Bidirectional (BDPT)
    No mipmapping:                       315452/745394 tiles (42.32%)
    Ray-based level selection:           203491/745394 tiles (27.30%)
    Camera-based level selection:        104858/745394 tiles (14.07%)

As expected, in the unidirectional case, the camera-based approach accesses slightly more tiles than the ray-based approach, and both approaches significantly outperform point sampling the lowest mipmap level. In the bidirectional case, the camera-based approach accesses slightly more tiles than in the unidirectional case, while the ray-based approach performs somewhere between the ray-based approach in unidirectional and point sampling the lowest mipmap level in unidirectional. What surprised me is how close the camera-based approach performed compared to the ray-based approach in the unidirectional case, especially since I chose a fairly aggresive widening heuristic (essentially a more aggressive version of the same heuristic that Matt Pharr uses in the texture cached branch of PBRTv3).

To help with visualizing what mipmap levels are being accessed, I implemented a new AOV in Takua that assigns colors to surfaces based on what mipmap level is accessed. With camera-based mipmap level selection, this AOV shows simply what mipmap level is accessed by all rays that hit a given point on a surface. Each mipmap level is represented by a different color, with support up to 12 mipmap levels. The following two images show accessed mipmap level at 1080p and 2160p (4K); note how the 2160p render accesses more lower mipmap levels than the 1080p render. The pixel footprints in the higher resolution render are smaller when projected into world space since more pixels have to pack into the same field of view. The key below each image shows what mipmap level each color corresponds to:

In general, everything looks as we would expect it to look in a working mipmapping system! Surface points farther away from the camera are generally accessing higher mipmap levels, and surface points closer to the camera are generally accessing lower mipmap levels. The ferns in the front of the frame near the camera access higher mipmap levels than the big fallen log in the center of the frame even though the ferns are closer to the camera because the textures for each leaf are extremely high resolution and the fern leaves are very small in screen-space. Surfaces that are viewed at highly glancing angles from the camera tend to access higher mipmap levels than surfaces that are camera-facing; this effect is easiest to see on the rocks in bottom front of the frame. The interesting sudden shift in mipmap level on some of the tree trunks comes from the tree trunks using two diffrent uv sets; the lower part of each tree trunk uses a different texture than the upper part, and the main textures are blended using a blending mask in a different uv space from the main textures; since the differential surface depends in part on the uv parameterization, different uv sets can result in different mipmap level selection behavior.

I also added a debug mode to Takua that tracks mipmap level access per texture sample. In this mode, for a given texture, the renderer splats into an image the lowest acceessed mipmap level for each texture sample. The result is sort of a heatmap that can be overlaid on the original texture’s lowest mipmap level to see what parts of texture are sampled at what resolution. Figure 5 shows one of these heatmaps for the texture on the fallen log in the center of the frame:

Just like in Figures 3 and 4, we can see that renders at higher resolutions will tend to access lower mipmap levels more frequently. Also, we can see that the vast majority of the texture is never sampled at all; with a tiled texture caching system where tiles are loaded on demand, this means there are a large number of texture tiles that we never bother to load at all. In cases like Figure 5, not loading unused tiles provides enormous memory savings compared to if we just loaded an entire non-mipmapped texture.

So far using a camera-based approach to mipmap level selection combined with just point sampling at each texture sample has held up very well in Takua! In fact, the Scandinavian Room scene from earlier this year was rendered using the mipmap approach described in this post as well. There is, however, a relatively simple type of scene that Takua’s camera-based approach fails badly at handling: refraction near the camera. If a lens is placed directly in front of the camera that significantly magnifies part of the scene, a purely world-space metric for filter footprints can result in choosing mipmap levels that are too high, which translates to visible texture blurring or pixelation. I don’t have anything implemented to handle this failure case right now. One possible solution I’ve thought about is to initially trace a set of rays from the camera using traditional ray differential propogation for specular objects, and cache the resultant mipmap levels in the scene. Then, during the actual renders, the renderer could compare the camera-based metric from the nearest N cached metrics to infer if a lower mipmap level is needed than what the camera-based metric produces. However, such a system would add significant cost to the mipmap level selection logic, and there are a number of implementation complications to consider. I do wonder how Manuka handles the “lens in front of a camera” case as well, since the shade-before-hit paradigm also fails on this scenario for the same reasons.

Long term, I would like to spend more time looking in to (and perhaps implementing) a covariance tracing based approach. While Takua currently gets by with just point sampling, filtering becomes much more important for other effects, such as glinty microfacet materials, and covariance tracing based filtering seems to be the best currently known solution for these cases.

In an upcoming post, I’m aiming to write about how Takua’s texture caching system works in conjunction with the mipmapping system described in this post. As mentioned earlier, I’m also planning a (hopefully) short-ish post about supporting multiple uv sets, and how that impacts a mipmapping and texture caching system.

Additional Renders

Finally, since this has been a very text-heavy post, here are some bonus renders of the same forest scene under different lighting conditions. When I was setting up this scene for Takua, I tried a number of different lighting conditions and settled on the one in Figure 1 for the main render, but some of the alternatives were interesting too. In a future post, I’ll show a bunch of interesting renders of this scene from different camera angles, but for now, here is the forest at different times of day:

References

John Amanatides. 1984. Ray Tracing with Cones. ACM SIGGRAPH Computer Graphics (Proc. of SIGGRAPH) 18, 3 (Jul. 1984), 129-135.

Laurent Belcour, Cyril Soler, Kartic Subr, Nicolas Holzschuch, and Frédo Durand. 2013. 5D Covariance Tracing for Efficient Defocus and Motion Blur. ACM Transactions on Graphics 32, 3 (Jun. 2013), Article 31.

Laurent Belcour, Ling-Qi Yan, Ravi Ramamoorthi, and Derek Nowrouzezahrai. 2017. Antialiasing Complex Global Illumination Effects in Path-Space. ACM Transactions on Graphics 36, 1 (Jan. 2017), Article 9.

Per Christensen, Julian Fong, Jonathan Shade, Wayne Wooten, Brenden Schubert, Andrew Kensler, Stephen Friedman, Charlie Kilpatrick, Cliff Ramshaw, Marc Bannister, Brenton Rayner, Jonathan Brouillat, and Max Liani. 2018. RenderMan: An Advanced Path-Tracing Architeture for Movie Rendering. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 30.

Per Christensen, David M. Laur, Julian Fong, Wayne Wooten, and Dana Batali. 2003. Ray Differentials and Multiresolution Geometry Caching for Distribution Ray Tracing in Complex Scenes. Computer Graphics Forum (Proc. of Eurographics) 22, 3 (Sep. 2003), 543-552.

Robert L. Cook, Loren Carpenter, and Edwin Catmull. 1987. The Reyes Image Rendering Architecture. ACM SIGGRAPH Computer Graphics (Proc. of SIGGRAPH) 21, 4 (Jul. 1987), 95-102.

Frédo Durand, Nicolas Holzchuch, Cyril Soler, Eric Chan, and François X Sillion. 2005. A Frequency Analysis of Light Transport. ACM Transactions on Graphics (Proc. of SIGGRAPH) 24, 3 (Aug. 2005), 1115-1126.

Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomáš Davidovič, Andrea Weidlich and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 31.

Iliyan Georgiev, Thiago Ize, Mike Farnsworth, Ramón Montoya-Vozmediano, Alan King, Brecht van Lommel, Angel Jimenez, Oscar Anson, Shinji Ogaki, Eric Johnston, Adrien Herubel, Declan Russell, Frédéric Servant, and Marcos Fajardo. 2018. Arnold: A Brute-Force Production Path Tracer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 32.

Larry Gritz and James K. Hahn. 1996. BMRT: A Global Illumination Implementation of the RenderMan Standard. Journal of Graphics Tools 1, 3 (Jul. 1996), 29-47.

Paul S. Heckbert and Pat Hanrahan. 1984. Beam Tracing Polygonal Objects. ACM SIGGRAPH Computer Graphics (Proc. of SIGGRAPH) 18, 3 (Jul. 1984), 119-127.

Homan Igehy. 1999. Tracing Ray Differentials. In Proc. of SIGGRAPH (SIGGRAPH 1999). 179–186.

Wenzel Jakob. 2010. Mitsuba Renderer.

Christopher Kulla, Alejandro Conty, Clifford Stein, and Larry Gritz. 2018. Sony Pictures Imageworks Arnold. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 29.

Mark Lee, Brian Green, Feng Xie, and Eric Tabellion. 2017. Vectorized Production Path Tracing. In Proc. of High Performance Graphics (HPG 2017). Article 10.

Darwyn Peachey. 1990. Texture on Demand. Pixar Technical Report 217.

Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation, 3rd ed. Morgan Kaufmann.

Matt Pharr. 2017. The Implementation of a Scalable Texture Cache. Physically Based Rendering Supplemental Material.

Dan Piponi. 2004. Automatic Differentiation, C++ Templates and Photogrammetry. Journal of Graphics Tools 9, 4 (Sep. 2004), 41-55.

Mikio Shinya, Tokiichiro Takahashi, and Seiichiro Naito. 1987. Principles and Applications of Pencil Tracing. ACM SIGGRAPH Computer Graphics (Proc. of SIGGRAPH) 21, 4 (Jul. 1987), 45-54.

Frank Suykens and Yves. D. Willems. 2001. Path Differentials and Applications. In Proc. of Eurographics Workshop on Rendering (Rendering Techniques 2001). 257–268.

Lance Williams. 1983. Pyramidal Parametrics. ACM SIGGRAPH Computer Graphics (Proc. of SIGGRAPH) 12, 3 (Jul. 1983), 1-11.

Develop your snaps faster

kyrofa's blog — Thu, 25 Oct 2018 03:00:00 GMT

Snaps bundle their dependencies, which take time to fetch and unpack, and they’re typically made up of multiple parts, each of which need to be built and installed into the snap. Once you finish this process, you need to install the snap and test it out. Once you notice that your code is terrible and doesn’t work the way you thought, you have to make a change and go back through that process again.

Do aliens play video games

D13V — Thu, 25 Oct 2018 00:52:22 GMT

Do aliens play video games?

If we meet an alien, from a different planet, and are able to talk with them, how relate-able would they be?

Would they feel pride for their nation, town or sports team?

Would they feel ambition, after being rewarded for achievements?

Would they keep playing video games once they start it, or binge-watch TV shows?

Would they be content to sit down and do nothing?

We might imagine them similar to us in many ways, being driven by the same incentives to live well. An alien, transported to earth and given a human skin could try to blend in. They would likely want to maximize their own prosperity by earning more money and resources, getting in a safe position for themselves and their close ones, and affecting the community wellbeing. Those are all common sense incentives, and in these regards, an alien would blend in.

But I bet, most likely, no alien would ever like to play video games. It’s because video games reflect not a common sense incentive, but a very human one.

Games are designed to be addictive… for us. We have chemical systems in our brains and body - endorphins, dopamine, serotonin, oxytocin, cortisol. When we play a video game we get rewarded with dopamine, which makes us feel good, but is also highly addictive. And we keep on playing, not because it is good for us in some abstract sense, but because it gives us pleasurable chemicals in our brain.

Our reward circuits have been hijacked by the video game. The game is optimized for stimulating our chemical circuits. And so are optimized the binge watchable TV series. And so is optimized social media. And so is optimized our To-Do list. All of these act similar to drugs, in that they rely on the quirks of our chemical systems in order to get a hold of our habits.

Drugs and alcohol deserve a special mention though, as they even go and mess up with our chemistry directly, instead of merely adapting to it. And our chemistry matters a lot. It determines our actions far more than any rational arguments. I experienced this firsthand when I became a compulsive problem-solver. I also experienced it when dealing with alcohol

What is rationality? 

I’d like to compare chemical decision making with “rational” decision making to drive the point further. There are at least two different definitions of rationality. Both definitions are compelling, and neither one takes into account our chemical undercurrents.

One of the most commonly used definitions of rationality, is the Von Neumann & Morgenstern utility maximization, which I would refer to as “classical” rationality. According to it, we assign a utility function to outcomes and assume the superposition of outcomes has a weighted average of the utilities, then we come up with some predictions that we can empirically test. Classical rationality a really neat theory. Just four common sense axioms and ta-da! From these four axioms we can derive how people should behave in any situation.

Just like with classical mechanics, classically rationality is a good approximation to behavior. When economists talk about rational actors, that’s exactly what they mean by “rational,” and they often tie it to maximizing money. For example, we would look for discounts or buy the same item from a cheaper place because we want to maximize the utility of money. In some situations, we will choose the more expensive, more convenient option because we also value our time. If we assign a price for the value our time, we can derive that price based on the tradeoffs we make.  Some economists point to those cases, and claim “see, humans can be rational”. Yet others, such as Dan Ariely in Predictably Irrational point at examples and studies which show that people are indeed not rational. And he’s not the only one finding holes in classical rationality. Nassim Taleb in Skin in the game defines as rational the actions which maximize long term survival. He then goes and spends time, showing examples of people and institutions which, in the absence of any threat, start acting in corrupt and selfish way. Skin in the game rationality can be useful when analyzing the incentives of public figures to serve the common good.

Skin in the game rationality is a novel concept but nonetheless effective in some situations that classical rationality fails to explain behavior, especially in building systems of actors with diverging interests. When a system doesn’t involve skin in the game, it can promote people who can talk a lot but not get stuff done. When a system involves skin in the game, it promotes people who realize that in order for them to remain at a position of authority, they need to deliver results. When they don’t deliver, they lose their skin.

Chemical rationality is who we are

Would these two definitions of rationality apply to aliens? My opinion is emphatic yes! Both the classical, and the skin in the game rationality are math, and math transcends not only solar systems and galaxies, but also multiverses and living-inside-a-simulation. In the beginning there was math. Math is abstract, and not rooted in reality, and thus universal.

Aliens would be willing to both maximize profit according to classical rationality, and also survival according to skin-in-the-game rationality. Yet, these rationalities might apply to a different degree. Classical rationality is motivated by the emotion of greed, and skin in the game rationality is motivated by the emotion fear. Depending on how much greed and fear aliens have, their actions would be more or less classically rational, or skin in the game rational. We can not expect their society or lack thereof to resemble ours in any way. Hell no. Aliens would have different inclinations for social bonding, for addictive behavior, for pride in achievement, for pain tolerance.

And for video games. What defines us as humans are our erratic habits and tendencies. Hormones and chemicals are what provides the rewards for our actions, and these rewards are a basis for our rationality.

Our proclivities are our signature as a species.

ka-yich — Wed, 24 Oct 2018 09:27:04 GMT

ka-yich — Wed, 24 Oct 2018 09:25:31 GMT

Classes? Where we’re going, we don’t need classes — React 16.6

Rodrigo Pombo — Wed, 24 Oct 2018 03:00:00 GMT

Sales for Cybersecurity Saas

ferrucc.io — Tue, 23 Oct 2018 22:12:03 GMT

It’s been almost I month after I started working on Magehash

This is my first cybersecurity product I’ve ever developed so I decided to share what I learned so far!

1) Selling Pain Points

We can divide products into three different groups: candies, painkillers and vitamins.

We don’t want to have much to do with candies.

Vitamins are interesting.

We love Painkillers.

For Cyber security software it’s hard to be a painkiller as what we’re generally trying to do is prevent attacks. Generally these kinds of products are like vitamins, they make us stronger, but we don’t really notice the effect they have on our health.

Setting up Github SSH

I Learnt — Tue, 23 Oct 2018 14:00:00 GMT

Just a reminder of the starting point for setting up SSH with Github so I don’t have to keep looking for it each time.

Managing Linux Users & SSH keys using Ansible

Mon, 22 Oct 2018 21:30:00 GMT

Today I was assigned a task to create user accounts on an EC2 instance (Ubuntu) and also add SSH public keys to the respective user account’s authorized key list. The EC2 instance would act as a gateway to access the internal network. (This is a basic setup in which the user creates an SSH tunnel to access resources on the internal network. It’s not a foolproof security solution but controls external access to some extent)

Fun PresentationFramework side-effect of bad equality

Liza Shulyayeva — Mon, 22 Oct 2018 18:50:57 GMT

I looked into a pretty fun bug a few days ago. The symptom: a WPF DataGrid populated with an ICollectionView was seeing inconsistent row copy behaviour. You would select a row by clicking on it and press Ctrl+C, but instead of copying the row you selected it would copy the row your cursor was hovering over.

Is prayer a vibration?

Honza Pokorný — Mon, 22 Oct 2018 17:00:00 GMT

Recently, I came across an item someone in my Facebook timeline shared. The author was thinking about what prayer is. Here is what it said:

Do you pray?

I love this interpretation of a Prayer. What is a prayer? Prayer doesn’t only happen when we kneel or put our hands together and focus and expect things from God. Thinking positive and wishing good for others is a prayer. When you hug a friend. That’s a prayer. When you cook something to nourish family and friends. That’s a prayer. When we send off our near and dear ones say, “Drive safely” or “Be safe”. That’s a prayer. When you are helping someone in need by giving your time and energy. You are praying. Prayer is a vibration. A feeling. A thought. Prayer is the voice of love, friendship, genuine relationships. Prayer is an expression of your silent being. Keep praying always…

The interesting part is that this person is a professed Christian. How can someone who regularly attends a church service where the Word of God is preached arrive at a definition like this one? The modern evangelical church has lost its rigor in teaching pure doctrine to its members. There is often very little structure and precision. I’d suggest that we have failed our people if we neglect to explain such basic concepts as prayer.

However, since the Reformation, there has been a strong tradition of teaching doctrine in the form of catechism: prepared questions and answers that the students were expected to memorize. From a young age, children were taught how to articulate the core tenets of their faith. The catechism explains doctrines of Scripture, sin, justification, sanctification, repentance; it covers the ten commandments, and the Lord’s prayer. This practice is unfortunately foreign to the modern evangelical church.

Why am I bring all this up? Here is Question 98 of the Westminster Shorter Catechism, it asks “What is prayer?”. Here is the answer succinct answer which wonderfully summarizes what prayer is according to the Scriptures:

Prayer is an offering up of our desires unto God, for things agreeable to his will, in the name of Christ, with confession of our sins, and thankful acknowledgment of his mercies.

If you have memorized a catechism, you will have a framework in your mind by which you can assess new ideas coming at your from social media. It can serve as a useful sieve for quickly determining what is pure doctrine, and what is of the world.

And lastly, here is an example of how to catechize your children on the Lord’s day.

NVMe SSD Upgrade Guide for an Early 2015 MacBook Pro

Graham Stevens – Grh.am — Mon, 22 Oct 2018 14:00:00 GMT

It’s early 2016, and my employer comes up with a scheme that can help us purchase a laptop and spread the cost over 12 months. For those of us in the UK, it is managed by the same company that also do the Cycle To Work Scheme vouchers etc.

At the time, I had a capable but old MacBook Pro (2011 I believe), so the temptation of an upgrade was too great, and I jumped at the opportunity for a shiny new retina MacBook Pro (Early 2015). As it turns out, they were just drop-shipping the standard builds, which meant any customisation (upgrading the RAM etc.) was out of the question. Entry-level 128GB SSD, 8GB RAM it is then!

Custom assertions using Rust macros

Phil Booth's Blog — Sun, 21 Oct 2018 20:35:40 GMT

Recently I wrote about how Rust macros make it easy to refactor repetitive code that might otherwise become annoying in a strongly-typed language. Continuing the theme from that post, I’ve noticed another use case where macros can be beneficial: writing custom assertions in tests.

In the unicode-bom crate, there’s a bunch of tests that assert the different Unicode byte-order marks are parsed correctly and, just as importantly, that a number of similar byte sequences are not incorrectly identified as byte-order marks.

The cruft in these assertions isn’t huge, but it’s enough to reduce readability for the key parts of each assertion. Here’s what one of them looks like without the assistance of a macro:

assert_eq!(Bom::from(vec![0u8, 0u8, 0xfeu8, 0xffu8].as_slice()), Bom::Utf32Be);

That may not seem too bad at first glance, but in context it swiftly becomes a wall of impenetrable boilerplate:

assert_eq!(Bom::from(vec![0u8, 0u8, 0xfeu8].as_slice()), Bom::Null);
assert_eq!(Bom::from(vec![0u8, 0u8, 0xfeu8, 0xfeu8].as_slice()), Bom::Null);
assert_eq!(Bom::from(vec![0u8, 0u8, 0xfeu8, 0xffu8].as_slice()), Bom::Utf32Be);
assert_eq!(Bom::from(vec![0x0eu8, 0xfeu8].as_slice()), Bom::Null);
assert_eq!(Bom::from(vec![0x0eu8, 0xffu8, 0xfeu8].as_slice()), Bom::Null);
assert_eq!(Bom::from(vec![0x0eu8, 0xfeu8, 0xffu8].as_slice()), Bom::Scsu);

However, a macro to tidy it up is pretty simple:

macro_rules! assert_bom {
    ([$($byte:expr),*], $bom:ident) => {
        assert_eq!(Bom::from(vec![$($byte as u8),*].as_slice()), Bom::$bom)
    }
}

With that in place, the assertions from earlier now look like this:

assert_bom!([0, 0, 0xfe], Null);
assert_bom!([0, 0, 0xfe, 0xfe], Null);
assert_bom!([0, 0, 0xfe, 0xff], Utf32Be);
assert_bom!([0x0e, 0xfe], Null);
assert_bom!([0x0e, 0xff, 0xfe], Null);
assert_bom!([0x0e, 0xfe, 0xff], Scsu);

The benefit here is that a reader’s eye is only confronted by the key part of each assertion. All you can see are the byte sequences and the expected result, which means it’s easy to zero in on any problems if you’re debugging a failing test. And because you’re already used to the standard assertion macros like assert! and assert_eq!, a name like assert_bom! is immediately intuitive.

Holiday to Italy visiting Pompeii, Herculaneum, Vesuvius and the Amalfi Coast

Sam Hooke — Sun, 21 Oct 2018 19:37:00 GMT

This year we went on holiday to Italy for a week, staying near Naples. There are many wonderful sights in the area, such as Mt. Vesuvius, Pompeii, Herculaneum and the Amalfi Coast. Off the beaten track, the food is cheap and pizza is unlike anywhere else we’ve experienced – which makes sense given it’s the region that invented the stuff!

We arrived on Wednesday 3rd October, and following is a brief day-by-day account.

Erratic scratches

ka-yich — Sun, 21 Oct 2018 02:15:01 GMT

Erratic scratches

Disconnected projections

Unbounded geometries

Lurching unmovement

This strange brown chaos everywhere

Transformed

Into infinite order

Into endless fields

Of quietly synchronized

Orange turning cubes

ka-yich — Sun, 21 Oct 2018 02:09:58 GMT

ka-yich — Sun, 21 Oct 2018 02:07:44 GMT

Treasure Hunt

Heitor's log — Sat, 20 Oct 2018 03:00:00 GMT

I hunted a treasure. Treasure hunting is actually a sport called geocaching.

2018-10-20

ho.dges.online — Sat, 20 Oct 2018 03:00:00 GMT

On NYC, Tokyo and Seoul

Notes on software development — Sat, 20 Oct 2018 03:00:00 GMT

I’ve lived in NYC for the past year — moved here after years in Philly and after growing up in a rural community a few hours west of there. My wife is South Korean and last week concluded my second trip to the suburbs of Seoul to visit her family. We finished up that trip with a week in Tokyo.

Long a mecha and Godzilla fan, I was struck by a city not significantly more modern, or significantly more “Eastern”, than NYC. In contrast, the lesser known Seoul is more modern than both cities and shares as much “Eastern” vibe as Tokyo.

I’d go so far as to say that Seoul is the most livable of the three for anyone of a similar background. There are a few concrete areas that led me to this including transportation, apartments, WiFi/cafes, food, and language.

I'll conclude with a few tourist recommendations and a list of books to read on South Korea and Japan if you share my enthusiasm for comparing life in different cities.

Transportation

NYC is one of the few cities in the world with a subway that runs 24/7. Tokyo and Seoul do not share this trait despite being many decades newer. (Tokyo and Seoul were heavily damaged during World War II and the Korean War, respectively.) And despite being built later, Tokyo subway cars are even less wide than NYC subway cars (~8.2ft vs. ~8.5ft).

In contrast, Seoul subway cars are ~10.2ft wide. The difference may seem slight but it is noticeable during rush hour when in Seoul there is space for four people to stand in the aisle versus room for perhaps two in a Tokyo or NYC subway car.

Seoul subway car, source: Travel Blog

The Seoul subway system is also the most advanced in terms of safety. All stations have a floor-to-ceiling barrier with doors that only open when a train arrives. Most stations in Tokyo have a ~3ft tall barrier that does the same, though some stations have no barrier. In NYC there are no barriers anywhere.

Concerning innovation, Seoul and Tokyo both have multiple driverless subway lines whereas NYC has none. But in terms of complexity the NYC subway is the simplest because you pay only once. Seoul and Tokyo subways are slightly more complex in that you swipe your card when you enter and exit (or transfer).

Taxis

It was jarring to be greeted by the very 90s, vaguely British Toyota Crown taxi cabs that dominate the streets of Tokyo.

Source: Phil Eaton

These cabs have no integrated navigation unit but a modern unit was typically mechanically attached. We saw a few of the recently approved Toyota JPN Taxi, but they only account for around 10 percent of cabs. (The integrated navigation is massive, perhaps 10-inch screens.) In contrast, Seoul has a variety of modern cabs all with integrated navigation — the most common of which is the Hyundai Sonata.

Source: The Seoul Guide

Although Japanese car companies pioneered integrated navigation in the 90s, it appears to have been the standard for South Korean car companies for the past 10-20 years.

And then there’s NYC with its primary mix of Crown Victorias and Priuses with multiple 4-inch smartphones mechanically attached for navigation.

Source: New York Post

Living

South Korea has no concept of the suburb oriented around single-family houses. Drive an hour or two out from Seoul or Busan and see the same massive, modern apartment complexes that are found in the city center. After that it's the stark farms of Kansas. Japan appears more like the US in that the city graduates steadily to suburb and farm.

Apartments in Seoul, source: Japan Times

In general, buildings in South Korea are fairly homogeneous. Even the downtown areas of Seoul have little architectural creativity. Tokyo and NYC are both diverse in building styles and sizes. However, NYC takes the cake for ubiquity of massive towers. In fact, the first time my South Korean father-in-law visited Manhattan he was blown away by this mass.

New York City, source: Wikipedia

The most popular neighborhoods in Tokyo seem more developed than their Seoul counterparts, the mass of stores and crowds extends further. And while the average age of buildings in Tokyo seems younger than the average age of buildings throughout Seoul (including less desirable areas), the developed areas (including buildings and streets) of Seoul are significantly cleaner and more modern. In contrast, and on average, Tokyo buildings seem as old as NYC buildings.

Tokyo, source: Fodors

Air quality

Air quality in NYC and Tokyo is high, pollution is low. But in recent times, air quality in Seoul has deteriorated with dangerous levels of fine dust from factories in South Korea and China. It is not clear when or how the South Korean government will address this.

WiFi/Cafes

My idea of a good cafe is a decent ratio of seats to traffic, available electrical outlets, and decent WiFi. NYC and Tokyo have some similarities: chain coffee shops are larger and non-chains are often pretty small. Tokyo differs from NYC in that there are few electrical outlets and in the existence of interior smoking sections. (Tokyo bans smoking while walking but designates areas like parks or inner rooms in restaurants or cafes.)

But the WiFi in Tokyo is abysmal. Many cafes do not have it (though the trend is to provide) and even the chains that do provide it have terrible speeds reaching peaks of 5Mbps down. In NYC WiFi is available near ~20Mbps down at most chains and ~5Mpbs at smaller non-chains.

In contrast, South Korea is the jewel of cafe culture. Unlike how in the US coffee shop size decreases as population increases, coffee shop sizes in South Korea are oddly enormous everywhere. South Korea is rich with local shops, domestic chains (including the exported Paris Baguette and Tous Les Jours), and foreign chains (South Korea has the highest number of Starbucks Reserve stores per capita of any country).

Starbucks Reserve in Seoul, source: Pulse News

From Jeju Island to Seoul we never worried about a seat or an outlet at a cafe. Furthermore, the WiFi in South Korea is incredible. My tech-hopeless in-law’s basic internet plan got 80Mbps down and the small cafes near their apartment got at least 40Mbps down.

NYC falls closer to Seoul in terms of ubiquity and speed of WiFi and has the added benefit of fast city-provided, outdoor WiFi surprisingly fast and available throughout the city. NYC is much worse in terms of daylight. Most cafes close between 8-10pm whereas cafes in Seoul and Tokyo easily stay open past 11pm.

Caveat

It’s not exactly fair to exclude internet cafes, prevalent in both Seoul and Tokyo (oddly even NYC has a few). At an internet cafe in Tokyo you can expect abundant outlets and excellent WiFi (I saw peaks of 40Mbps down). I did not visit an internet cafe in Seoul but I expect it to be similar. In both Seoul and Tokyo you can easily find 24/7 service (with showers!?).

I did not include internet cafes above because I find them slightly less convenient for tourists. Though credit is due: unlike American Chinatown internet cafes, the ones we visited in Tokyo were very clean, spacious and warm.

Internet cafe in Shinjuku, source: Rakutama

Food

Dining out in NYC is similar in cost to other major US cities. The quality is usually pretty good. Tokyo was about as expensive as food in NYC and generally as high quality. For instance, most dinners in NYC and Tokyo cost about $40-60 for two people. In contrast, most entrees in Seoul are sold for two and the dinner in total was often about $20-40. Restaurants on average seemed to be lower quality in Seoul compared to Tokyo and New York, but there are still more than enough high quality options.

Language

I am biased having a better knowledge of Korean than Japanese and a South Korean partner to fall back on. But I believe South Korea is the more friendly place for an English speaker in that it is more dedicated to providing English translations and that the written language is simpler. In both cities the penetration of English-speaking natives (and quality of speech and comprehension) is indistinguishable and decent.

To the first point, even the oddest locations and obscure signage had English translations in South Korea (not just Seoul) — not so even within Tokyo.

To the second point, Japanese has three writing systems (kanji, hiragana, and katakana). Kanji (characters originating from Chinese) cannot be replaced in writing by phonetic counterparts in hiragana or katakana. So you have little choice but to memorize all important characters, disregarding the fact that many characters can be broken down. Then you must also memorize the alphabetic systems of hiragana and katakana.

In contrast, Korean has two writing systems (hangul and hanja) where hanja (characters originating from Chinese) is primarily used in formal settings (government forms, academic books, etc.) and can be replaced with the phonetic equivalent in hangul.

This makes it much simpler to memorize and read Korean compared to Japanese.

Assorted recommendations

For New Yorkers, don’t stay in the recommended areas of Shinjuku/Shibuya/Roppongi unless you’re the type who’d enjoy staying around Times Square. These three areas of Tokyo are just as obnoxious albeit much safer. I also don’t recommend the Harajuku area; it is extra. There’s no real equivalent level of crazy in Seoul although Hongdae comes close.

In a future Tokyo trip I’d stick to the Meguro Station area including Ebisu and Daikanyama. They are beautiful, quiet neighborhoods with lots of restaurants and cafes beside the Meguro river. Areas along the Sumida River are also beautiful and quiet. Ginza/Tokyo Station is also a fun-but-not-obnoxious area to visit.

Ebisu, source: Homeaway

I cannot recommend the Edo-Tokyo Museum enough, it is the best city museum I've visited. Tsukiji is also a must see, reminding me how much I miss going to Reading Terminal Market each weekend in Philly.

In Seoul I’d recommend Yeonnam-Dong, Itaewon (which is much nicer than it’s made out to be), and Gwanghwamun. Mapo-Gu in general is a great region of Gangbuk as is the area below it (near Yeouido) in Gangnam.

Yeonnam-Dong, source: Phil Eaton

I recommend visiting the National Museum of Korea in Seoul as well as Hangang Park and Gyeongui Line Forest Park. The areas around the Tancheon stream flowing South to Bundang are also beautiful.

Tancheon near Bundang, source: Misadventures of an Awkward American

Conclusion

I came to Tokyo with the expectation of a highly modern city fused with Eastern culture. But it is difficult to see many ways it is ahead of NYC technically and it is very similar to NYC culturally. In some ways Tokyo even seems a little stuck in the past or just... off. Why are all vending machines [e.g. for tickets, ordering food, etc.] mechanical and not touch screens? The National Museum of Science is awfully old and ugly, the National Diet Building the same.

So on the one hand I’d like to let the next person down lightly on the excitement of Japan. It is a world-class city with great restaurants, live music and refined culture but all-in-all very similar to NYC. On the other hand I recommend Seoul for a cheaper, cleaner, more English-speaker friendly, and genuinely novel city with splashes of "Eastern" romantic elements like Tokyo.

Cherry blossoms in Seoul, source: English Spectrum

Everything you need to know on Magecart Attacks

ferrucc.io — Fri, 19 Oct 2018 22:12:03 GMT

In the last month a group of hackers called Magecart came back striking in full force.

They first appeared in the news back in 2015 when RiskIQ found out they injected code in Magento’s “Magecart” shopping software. Thus the name.

The attacks they organized have caused massive damages to hundreds, likely even thousands of companies like British Airways, Ticketmaster and even Newegg.

This group is specialized in card skimming payment forms on the internet.

Routes of RESTful services with nested resources

Qubyte Codes — Fri, 19 Oct 2018 22:00:30 GMT

Nested resources are common in real life, and when you're a programmer working on RESTful APIs you may have noticed that there are often trade-offs when it comes to formatting the routes of nested resources.

Let's say we're building an API for carpenters. Their guild wants a way to help the carpenters manage their clients, and the items which they're creating for their clients (jobs). A set of routes for this might look like

GET  /carpenters
POST /carpenters
GET  /carpenters/:carpenterId

GET  /clients
POST /clients
GET  /clients/:clientId

GET  /jobs
POST /jobs
GET  /jobs/:jobId

where segments preceded by a colon are placeholders for a UUID. The first route of each group gets a collection, the second creates a new member, and the third gets a member by its ID. There may well be other operations like the third which act on individual members of a collection, such as PUT for replacing and DELETE for removal.

The problem with this layout is that, as a carpenter, you're rarely going to want to see clients which aren't your clients. There may also be lots of jobs, and making a request for a list of jobs could select a huge number of them. Usually you'll want to filter it by the client. We can use query parameters to supply additional context, but this seems a bit clumsy. Let's have another go

GET  /carpenters
POST /carpenters
GET  /carpenters/:carpenterId

GET  /carpenters/:carpenterId/clients
POST /carpenters/:carpenterId/clients
GET  /carpenters/:carpenterId/clients/:clientId

GET  /carpenters/:carpenterId/clients/:clientId/jobs
POST /carpenters/:carpenterId/clients/:clientId/jobs
GET  /carpenters/:carpenterId/clients/:clientId/jobs/:jobId

It's pretty clear now that these are nested resources. It's no longer possible to select a collection without refining it by its parent resources.

The problem now is that it's a bit annoying to have to use all the parent IDs of a list or a member. I recently realised (and I'm probably late to the party) that a good solution is a compromise between the two

GET  /carpenters
POST /carpenters
GET  /carpenters/:carpenterId

GET  /carpenters/:carpenterId/clients
POST /carpenters/:carpenterId/clients
GET  /clients/:clientId

GET  /clients/:clientId/jobs
POST /clients/:clientId/jobs
GET  /jobs/:jobId

In this arrangement, we need to use a route with a parent ID when listing a resource or creating a new member. When we already have an ID of a resource we don't need to include its parent in the path. This includes other singular operations such as PUT and DELETE.

Finally, a word on nesting. The implicit assertion made above is that a child resource has a single parent, which may not be the case. Think of the solution above as a filtering mechanism. i.e. a client may have contracted more than one carpenter. You may also want to list jobs for a carpenter, rather than for a client. If you find yourself writing too many routes, using query parameters may be the lesser evil. Be pragmatic!

Git - what to do when it goes wrong

I Learnt — Fri, 19 Oct 2018 14:00:00 GMT

This has a great summary of what to do when things go wrong when using Git. Added here so I know where to look in moments of panic.

Cruelty by Edward St Aubyn

ho.dges.online — Fri, 19 Oct 2018 03:00:00 GMT

Three reasons why incubators breed failure

Jason Eckert's Website and Blog — Fri, 19 Oct 2018 03:00:00 GMT

These past few years, I’ve provided various types of support for some tech incubators in the Kitchener-Waterloo region where I live (it’s a complex arrangement that I won’t go into within this post, but mostly involves helping people with technology implementation issues related to software). It’s not hard to find tech incubators nowadays - they’ve become quite popular this past decade, and larger cities often have several incubators of various sizes.

Originals: How Non-Confomists Move The World

Macleod Sawyer | Web Portal Feed — Thu, 18 Oct 2018 15:00:00 GMT

Test-Driven Development is not about tests

Lucas F. Costa - Blog — Thu, 18 Oct 2018 15:00:00 GMT

Counterintuitively, test-driven development is not about tests. Test-driven development is about being able to take gradual and small steps towards a solution. Taking small steps allows you to solve...

Implementing Support for Continuity Camera

Build and Run — Thu, 18 Oct 2018 14:39:43 GMT

One of macOS Mojave's user-facing features is Continuity Camera. It allows you to capture images or documents from your iOS devices.
While Apple prominently showcases the feature on the Mojave product page, documentation for developers is very thin. There's only a short mention of the related APIs in the "What's new in Cocoa" WWDC 2018 session.
Some of AppKit's default view classes support the feature out of the box (e.g. NSTextView), but it's also possible to add capturing to your own controls.

Small sample app with automatically inserted Continuity Camera menu.

Apple implemented Continuity Camera on top of the Services infrastructure of macOS and NSResponder. Captured image data is passed in via NSPasteboard.
To make the "Import from iPhone or iPad" menu item appear in a menu, we have to implement validRequestor(forSendType:returnType:) in a NSResponder subclass (e.g. in a view controller). From there, we need to return an instance that implements the NSServicesMenuRequestor protocol.

I created a small sample project that provides a simple implementation of the above methods. It consists of a view controller that hosts an NSImageView and a button. The button has an empty menu connected, which gets popped-up when the button's action is performed. The view controller is a subclass of NSResponder and it also implements NSServicesMenuRequestor. So when the macOS Services architecture traverses the responder chain, it asks if we accept images from the pasteboard via validRequestor(forSendType:returnType:). As we indicate true by returning self, the menu gets populated with the Continuity Camera menu items. Those items dispatch their actions via the Responder Chain, so it is important that the view controller that handles the passed in image is firstResponder. Otherwise, menu validation fails and the Continuity Camera items are disabled. To ensure that, we call self.view.window?.makeFirstResponder(self) before showing the menu.

After the user captured an image on the selected iOS device, we need to handle the returned image in readSelection(from pasteboard: NSPasteboard) (which is part of NSServicesMenuRequestor). The sample app reads the captured image from the pasteboard and displays it in an image view. You can download the Xcode project here.

The call flow for Continuity Camera integration. Left: APIs to implement. Right: calls performed by AppKit.

Special thanks go out to Michael Schwarz. He implemented Continuity Camera support for MindNode for Mac and shared his insights with me.

React: Controlling Rendering with Keys

Dylan Paulus' Blog — Thu, 18 Oct 2018 03:00:00 GMT

key, the property we throw some value into to suppress warnings. This property is used by React to determine when a component is deleted, updated, or created. We generally don't interact much with the key prop--except in loops.

My team recently ran into an issue lately--how do we rerender all child components when a state has changed? The simplest approach... you guessed it, key!

Let's take a look.

class Parent extends Component {
  state = {
    key: true,
    count: 0,
  };

  handleChildUnmount = () => {
    this.setState((prevProps) => ({ count: prevProps.count + 1 }));
  };

  toggleKey = () => {
    this.setState((prevProps) => ({ key: !prevProps.key }));
  };

  render() {
    const { key, count } = this.state;

    return (
      <div>
        <button onClick={this.toggleKey}>Toggle Child Key</button>
        <Child key={key} count={count} onUnmount={this.handleChildUnmount} />
      </div>
    );
  }
}

class Child extends Component {
  componentWillUnmount() {
    this.props.onUnmount();
  }

  render() {
    return <div>Total Renders: {this.props.count}</div>;
  }
}

In this example we are toggling the key prop of our Child component between true and false. We then keep track every time the Child unmounts through a callback to increment the Parent's state. When we toggle the key prop, we will see the counter increment each time. This is, because React uses the key to determine if a component has changed. Since we are changing the key each time React will see our component has updates, and rerender the Child component.

This is the easiest way to cause a rerender of a child component (and all children of the child), and shows the power of understanding thekey prop!

Note: Even if a child component returns false in a shouldComponentUpdate, changing the key prop will cause it to rerender anyway (Try it in the CodeSandbox above!).

2018-10-18

ho.dges.online — Thu, 18 Oct 2018 03:00:00 GMT

The Big Bang Theory: Season 10

🦉 olshansky 🦁 — Wed, 17 Oct 2018 16:39:24 GMT

Olshansky's review of The Big Bang Theory: Season 10

Movie 43

🦉 olshansky 🦁 — Wed, 17 Oct 2018 16:30:05 GMT

Olshansky's review of Movie 43

Open source UI Kits pour le Web

Yasha Solutions — Wed, 17 Oct 2018 11:49:53 GMT

UI Kit Site: https://getuikit.com/ GitHub: https://github.com/uikit/uikit Stream Ui Kit Site: https://htmlstream.com/templates/stream-ui-kit Basé sur Bootstrap. One Nexus Site: http://www.onenexus.io/ Basé sur Synergy, un framework front-end pour la génération de composant UI et Kayzen, un systeme de grille CSS – tous réalisé par la même équipe. Milligram Simple Juste 2kb https://milligram.io/ Tabler Un des meilleurs UI de dashboard Open Source. Basé sur Bootstrap Le code est clean La logique est claire et bien conçue.

Open source UI Kits & themed frontend for the web

Yasha Solutions — Wed, 17 Oct 2018 11:26:36 GMT

Sleek Dashboard Demo: Sleek Demo site/ GitHub: https://github.com/tafcoder/sleek-dashboard Bootstrap 4 Admin Dashboard Template and UI Kit – and yeah sleeeeek indeed – like I really love it. Great work by tafcoder. UI Kit Site: https://getuikit.com/ GitHub: https://github.com/uikit/uikit A lightweight and modular front-end framework for developing fast and powerful web interfaces. Well suited for website, landing pages etc. Simple, straightforward. Also, nice extra: Autocomplete plugins for both Atom and Sublime Text 3.

imenu on Emacs eshell

xenodium.com @alvaro — Wed, 17 Oct 2018 03:00:00 GMT

imenu navigation is one of those Emacs gems I didn't discover until much later on. It does what you'd expect in all types of modes. In rare instances, I've found specific modes missing imenu support. Fortunately, this is Emacs and you can fix that.

Eshell has a handy feature to jump back and forth over previous prompts using M-x eshell-previous-prompt (C-c C-p) and M-x eshell-next-prompt (C-c C-n). Upon learning about these two functions, my immediate reaction was to try imenu. Surprisingly, it didn't "just work", but a tiny bit of elisp brought balance back to the Emacs universe.

In an eshell mode hook function, one can set the imenu-generic-expression to help it find your favorite prompt:

(setq-local imenu-generic-expression
                  '(("Prompt" " $ \\(.*\\)" 1)))

Ah it's the little things…

ps. If wondering why my imenu experience looks a little different, that's because I'm using Abo Abo's wonderful counsel and M-x counsel-semantic-or-imenu.

Sometimes to learn you also need to do

I Learnt — Tue, 16 Oct 2018 14:00:00 GMT

I am back on the learning Scala trail and it is clear that just reading books and articles or watching videos won’t cut it. It needs to be hands-on coding.

The secret history of Ctrl+Alt+Del

I Learnt — Tue, 16 Oct 2018 14:00:00 GMT

The background to why the famous “three finger salute” was chosen to be so difficult to do.

Engineering Leadership 321

Greg Herlein — Tue, 16 Oct 2018 07:30:29 GMT

I’ve come to firmly believe that Engineering Leadership has a 3-2-1 principle to create Engineer happiness, which is a key prerequisite for high performance Engineering teams.

Hiking in Essertines - Switzerland

Heitor's log — Tue, 16 Oct 2018 03:00:00 GMT

Switzerland is amazing for Hiking. Here's how one went.

3 Tips from Kent C Dodds for People Just Getting Started

swyx's site RSS Feed — Tue, 16 Oct 2018 02:54:50 GMT

advice for beginners from a podcast

Mapplication

mbutler — Mon, 15 Oct 2018 21:19:55 GMT

Mapplication is a tool I built with my co-workers (Rob Shepard, Ethan DeGross, and Alyssa Varner) as a way to rapidly prototype maps for researchers and scholars. It accepts a CSV file upload and outputs a leaflet map. It converts the spreadsheet into valid GeoJSON, allows the users to pick a marker color and map […]

Reflections of a Floating World (Album)

Macleod Sawyer | Web Portal Feed — Mon, 15 Oct 2018 19:30:59 GMT

Calm Black Water (Album)

Macleod Sawyer | Web Portal Feed — Mon, 15 Oct 2018 19:28:32 GMT

Property testing

bfdes.in — Mon, 15 Oct 2018 03:00:00 GMT

An introduction to property-based testing and its applications in standard library algorithm validation

Oil, Climate and Africa

one mikro2nd — Sun, 14 Oct 2018 20:40:16 GMT

So here’s a peculiar think: America (the USA, but Canada, too) got rich off sucking the oil under their continent out of the ground (well, digging in the case of Athabasca), turning it into fuel and releasing it into the atmosphere where it’s become the biggest single existential threat facing humankind to date. (Biome-killing meteorites possibly excepted.) Yes, it’s arguable. It’s complex, nuanced and not as simple as I’ve baldly put it here, but I believe most sensible people will get my drift.

Swing

rob.sh — Sun, 14 Oct 2018 18:00:00 GMT

Inspired by Becca on the swing at Bernal Heights.
October 2018, spray paint on layered card.

Appreciation for Open Source and Commercial Software

bastibe.de — Sun, 14 Oct 2018 13:20:00 GMT

I recently released my first-ever piece of commercial software, a plugin for the X-Plane flight simulator. I wrote this primarily to scratch my own itch, but thought other users might like it, too, so I put it up on the store. What struck me however, were the stark difference between the kinds of responses I got to this, as compared to my open source projects: They were astonishingly, resoundingly, positive!

You see, I have a bunch of open source projects, with a few thousand downloads per month, and a dozen or so issues on Github per week. Most of my interactions with my users are utilitarian, and efficient. Someone reports a bug or asks for help, I ask for clarification or a pull request, we iterate a few times until the issue is resolved. The process is mechanical and the tone of our conversation is equally unemotional. This is as it should be.

After having released my flight simulator plugin, however, people thanked me! They congratulated me! They extolled about the greatness of what I had built! And they did this despite the fact that the initial release had quite a few major bugs, and even flat-out did not work for some people. Yet even people who couldn't get it to work were grateful for my help in resolving their issue!

This blew my mind, in comparison with the drab "I found a bug", "Could you implement…" I was used to from my open source work. There, the feedback I got was mostly neutral (bug reports, feature requests), and sometimes even negative ("You broke something!"). So I release my software for free, as a gift, and get average-negative feedback. My commercial work, in contrast, costs money, and yet the feedback I get is resoundingly positive! I can not overstate how motivating it is to get praise, and love, from my users.

I think this is a huge problem for our Open Source community. I had my run-ins with burnout, when all the pull requests came to be too much, and I started dreading the little notification icon on Github. And I think the negativity inherent in bug reports and feature requests has a huge part to do with this. In the future, I will try to add more praise to my bug reports from now on, just to put things into perspective.

But I think we should go further than that. We should create tools for praising stuff, beyond the impersonal Stars on Github. We should be able to write reviews on Github, and recommendations, and blog posts about cool libraries we find.

I recently got my first github issue that was just a thank-you note. I loved it! We need more positivity like that.

Encrypted disk image on macOS

xenodium.com @alvaro — Sun, 14 Oct 2018 03:00:00 GMT

Hundred years are nothing

D13V — Sat, 13 Oct 2018 05:10:30 GMT

Hundred years are nothing. Nada.

Yes… I know. A lot can happen during a hundred years. Take a look the twentieth century with the two world wars, and the growth of computers and the Internet and the enormous leaps in biology.

And the twentieth century is not the only one with drastic changes. Major changes have been happening every century all throughout the last few thousand years. And that’s my point.

The total amount of changes is so high, that even talking to someone from the beginning of the twentieth century would be much more comfortable to us than anyone who lived five hundred or two thousand years ago.

That’s what a roman toilets look like. The did number two and socialized at the same time. Today, most people around the world prefer to poop in solace.

Somewhere along the time we went through a lifestyle bottleneck and decided that we don’t feel comfortable with simultaneously hanging out with people and relieving ourselves.

People back in the day were weird. Not to say people today aren’t weird, but the magnitude of weariness is a lot smaller than looking historically.

Taking time into perspective, all human beings alive today are brothers and sisters. We’re siblings, Millennials and Baby Boomers. We’re pretty much the same. We are much closer than we realize, should we only step back and see how far along we’ve come.

We ARE closer than ever, and yet, still diving ourselves and in-fighting.

The human mind is relative. We will find minor differences to fight about no matter how similar we are.

(credit XKCD 1095)

Only by looking in perspective we unite… against a common enemy.

You and I might support different football teams and get in a fight. And yet, we would unite in our love for football against Tiffany who just doesn’t get it. And we might argue with Tiffany person about whether football is a worthwhile way to spend your time, but we might unite with her against Ronald and Marcia who’d vote for the opposite party. And yet, no matter how hard we debate, we’d unite if a natural disaster occurs, and help each other.

Because when the bigger enemy arrives, they take us by surprise. The show us how petty most of our quarrels are.

Would they matter a thousand years from now? Or even hundred years from now?

No.

Sheffield travel bookmarks

xenodium.com @alvaro — Sat, 13 Oct 2018 03:00:00 GMT

Headsphones bookmarks

xenodium.com @alvaro — Sat, 13 Oct 2018 03:00:00 GMT

Setting your pc for remote access

ciesie.com — Sat, 13 Oct 2018 03:00:00 GMT

I’m not sure I need it. Maybe you will. It is an exercise in excess. I rarely need something from my home PC because nowadays it’s so easy to store things in the magical cloud. Still, it’s a nice exercise.

I have set up remote access on Void Linux and I will describe the process of doing so for that OS. You might encounter some problems if you are using Windows… but if you do you have bigger things to worry about (writing this around the time M$ update for Windows bricked some users’ computers.

Primitive Obsession & Exceptional Values

Josh Thompson — Fri, 12 Oct 2018 16:00:00 GMT

I’ve been working through Avdi Grimes’ Mastering the Object Oriented Mindset course.

One of the topics was using “whole values”, instead of being “primative obsessed”. The example Avdi gave was clear as day.

He used a course with a duration attribute to show the problem.

course.duration 
=> 3

3 what? weeks? days? months?

Of course, you could write a method like:

course.duration_in_weeks
=> 3

But now you’ll have trouble rendering this all over the place. You’d have conditionals every time you wanted to render courses in weeks (if it makes sense), or in months (if appropriate), or of course, days.

So, the solution is to use “Whole values”. This means an attribute should be a complete unit, in and of itself, and should need no further refining to be usable.

So, you should be able to do something like this:

course.duration
=> Months[3]
shorter_course.duration
=> Weeks[5]

So, here’s the basics of this Duration class, that your units (like Days, Weeks, and Months) inherit from:

class Duration
  attr_reader :magnitude
  
  def initialize(magnitude)
    @magnitude = magnitude
    freeze
  end
  
  def inspect
    "#{self.class}[#{magnitude}]"
  end
  
  def to_s
    "#{magnitude} #{self.class.name.downcase}"
  end
  
  alias_method :to_i, :magnitude
end


class Days < Duration; end
class Weeks < Duration; end
class Months < Duration; end

And it delivers pretty cool stuff:

main:0> Days.new(3)
=> Days[3]
main:0> Days.new(3).to_s
=> "3 days"
main:0> length = Weeks.new(3)
=> Weeks[3]

But having the option to call course.duration and get Weeks[3] as a response is… amazing. Or course.length.to_s and get 3 weeks. Super cool.

Avdi walked through the example code, but I was partial to having it available for playing around myself. So, I built a very simple test file.

Check out the full test suite, if you’re interested

The above gist also has the code that makes it all pass. I’m going to highlight just a few of the tests below:

def test_duration_is_months_object
  assert_instance_of Months, @math.duration
end

This test (and a few others) make it explicit that when you call @math.duration you don’t expect a primitive back - you expect an instance of the Months class. Super cool.

def test_duration_inspect
  assert_equal "Months[4]", @math.duration.inspect
end

We can “convert” our Duration value into a primitive (a string) by calling #inspect on it. Other than this, though, the duration value lives as its own object.

The tests test some helper methods that Avidi mentioned, to make it a bit easier to render the course.duration in a view:

def render_course_info(course)
  "#{course.name} (#{render_value(course.duration)})"
end

def render_value(value)
  case value
  when Months
    "#{value.to_i} gruling months"
  when Weeks
    "#{value.to_i} delightful weeks"
  when Days
    "a paultry #{value.to_i} days"
  end
end

Exceptional Values

So, we’ve got this method that takes input as a string, like “12 months”, and tries to convert it to Months[12].

If you are accepting data from a user, you’ll need to plan on invalid input, like “99 blinks”.

Here’s the first take of the conversion method:

def Duration(raw_value)
  case raw_value
  when Duration
    raw_value
  when /\A(\d+)\s+months\z/i
    Months[$1.to_i]
  when /\A(\d+)\s+weeks\z/i
    Weeks[$1.to_i]
  when /\A(\d+)\s+days\z/i
    Days[$1.to_i]
  else 
    nil
  end
end

This kind of works, but nil isn’t a great place-holder. Now your view logic needs to do all sorts of special work to handle if there are nil values, which of course there will be all the time, because if you call:

course = Course.new
course.name = "math"
course.duration = "12 days"
course.save

The course will have nil values auto-assigned simply because the user has not filled it in yet.

Anyway, so, as you might expect from someone talking about “whole values”, there’s a “whole value” implementation of an exception:

def Duration(raw_value)
  case raw_value
  when Duration
    raw_value
  when /\A(\d+)\s+months\z/i
    Months[$1.to_i]
  when /\A(\d+)\s+weeks\z/i
    Weeks[$1.to_i]
  when /\A(\d+)\s+days\z/i
    Days[$1.to_i]
  else 
    ExceptionalValue.new(raw_value, reason: "unrecognized format")
    # we create a new Exceptional Value object if we get unrecognized input
  end
end

Here’s what that object might look like, using this exceptional value:

math = Course.new("Math")
math.duration = "a blink of an eye"
=> <struct Course
 name="Math",
 duration=<ExceptionalValue:0x00007fca79021188 @raw_value="a blink of an eye", @reason="unrecognized format">>

pretty cool, huh?

check out the gist for tests and class code. Don’t judge me for sticking like 40 classes in the same file…

📸 Iceland

bpev.me — Fri, 12 Oct 2018 03:00:00 GMT

Using Bitbucket As Private Maven Repository

Prasham H Trivedi — Fri, 12 Oct 2018 03:00:00 GMT

Sharing code from Private libraries is a difficult task. In this post we will see how we can use BitBucket as Private Maven Repository.

SIGSEGV1 qualification CTF

Linux Engineer's random thoughts — Fri, 12 Oct 2018 01:00:00 GMT

After my r2con r2wars writeup, here's another writeup of a "challenge". This challenge is the Capture-The-Flag (CTF) pre-qualifications for the SIGSEGV1 conference in Paris. It felt a bit weird to have a conference registration limited to those who pass a certain challenge, but I was curious about what it would …

Using the 'timer' observable from Rxjs and Chrome Headless to hack a contest at work

Jason Raimondi on Software Engineer — Thu, 11 Oct 2018 10:00:00 GMT

I am going to begin by saying, I couldn’t not do it. It was calling to me like Ring calls to Sauron.

Official Contest Rules

The official rules read: “Everyone has their own tagged link that points to the landing page — each unique click = 1 point”. That is very interesting… I present to you jasonraimondi/hack-contest.

The Code

import * as puppeteer from 'puppeteer';
import { Observable, Subscription, timer } from 'rxjs';

const link = 'http://bit.ly/2N9xokm';

async function addPageview() {
  const browser = await puppeteer.launch({
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox'
    ]
  });
  const page = await browser.newPage();
  await page.goto(link);
  console.log(`Navigated to ${link}`);
  await browser.close();
}

const intervol = 15000;
const intervolTimer = timer(0, intervol);
const intervolTimerSubscription = intervolTimer.subscribe(addPageview);

We are using puppetter, a Headless Chrome instance to spin up a new browser for us every time we want to view the page. The ‘timer’ function from the Rxjs library is what is powering our loop. We are then able to add a subscription to the timer that will execute every intervol.

Tips on Improving FabricJS Speed

Dylan Paulus' Blog — Thu, 11 Oct 2018 03:00:00 GMT

FabricJS is an awesome library that handles Canvas interactions and rendering. Inherently it is a pretty performant library, but recently we ran into an issue where we needed to render ~8mb of json data (10k objects). Given any framework, it is no easy task to render this much data. The biggest culprit while rendering a lot of objects is renders (or rerenders). When the canvas has to show any change, it renders the entire canvas. This includes clearing the canvas, looping through each object, and drawing it. In this post I'll go through some of the techniques we used to improve interaction and rendering speed.

Cut out the basics

FabricJS has a small post about optimizing performance. All great tips, but the one to look at is setting renderOnAddRemove to false. Adding this property to the canvas will prevent Fabric from re-rendering if a shape is added or deleted. This gives huge speed improvements on initial loads while adding a lot of shapes to the canvas.

const canvas = new fabric.Canvas("myId", { renderOnAddRemove: false });

Caution: canvas.renderAll(); will have to be called whenever we want a change to show on the canvas

Enliven over loadFromJSON

loadFromJSON is a really helpful function when we want to deserialize JSON to the canvas, but it comes with some draw-backs. First, loadFromJSON will clear the canvas before it loads the JSON objects. This can be bad if we added shapes to the canvas before loading then JSON. Second, it will call renderAll once the JSON is finished. This is bad when we have additional operations to perform after our JSON object is loaded (ie. adding more shapes or selecting then).

Alternatively we can use a function called enlivenObjects.

const canvas = new fabric.Canvas("myId", { renderOnAddRemove: false }););
// enlivenObjects([array of json objects], [callback])
fabric.util.enlivenObjects([{}, {}, {}], (objs) => {
    objs.forEach((item) => {
        canvas.add(item);
    });
    canvas.renderAll(); // Make sure to call once we're ready!
});

enlivenObjects takes in an array of the serialized FabricJS JSON (found in: [json].objects), and returns the Fabric object representation of it. We use a callback to get the parsed objects. This is similar to how loadFromJSON handles loading behind the scenes! The bonus here is there are no unneeded renders and canvas clearing. We can control when the canvas gets rendered!

Don't Canvas Everything

The more shapes on the canvas, the more work Fabric has to do on a render. One way around this is to take out any shapes that don't need to be manipulated or animated. We achieve this by converting those un-interactive layers to an image canvas.toDataURL("png"); and rendering it as an HTML <img /> tag. Then, overlay the canvas on top of the image; only using the canvas for the interactive bits while showing everything.

Generate Oracle SQL Monitoring Reports as HTML using SQL Developer v18.3 (no Flash needed)

Tanel Poder Blog — Thu, 11 Oct 2018 00:00:52 GMT

The Oracle SQL Developer team has released version 18.3 of this tool. My favorite new feature in this version is its ability to save SQL Monitoring reports in HTML format! Adobe Flash is finally not needed for sharing graphical SQL Monitoring reports.

Here’s an example output, it looks pretty nice and clean:

The report is reasonably navigatable, has tooltips (useful for identifying wait classes for example) and you can also rearrange columns by dragging them left or right with your mouse. The developers have helpfully put the ESTIMATED ROWS and ACTUAL ROWS columns next to each other for making identifying optimizer cardinality misestimates easier (btw, in the Flash version you can move columns around too but these two are not side by side by default). The navigation and scrolling is much smoother as there’s no Flash plugin and iframe/embed in the way.

Getting rid of hiccups

Jason Raimondi on Software Engineer — Wed, 10 Oct 2018 10:00:00 GMT

I found this the other day when I had the hiccups a few different days in a row. Posting it here because it worked amazing and I do not ever want to forget this technique.

The 100% Guaranteed way to get rid of hiccups. Instantly. Every time.

Today I share with you his ancient secret.

Fill your lungs with your mouth. Very important you use the mouth.
Swallow TWICE in a row. If you have difficulty doing this generate saliva in your mouth, it’ll help you get the second swallow easier.
Exhale SLOWLY through your nostrils.

Hiccups are your diaphragm spasming out of your normal inhale-exhale cycle. It can usually be fixed within a few breathes by mindfully focusing on my breathing and taking slower, fuller, deeper breathes to force your diaphragm back into sync.

Optimizing your Dockerfile build layers

Jason Raimondi on Software Engineer — Wed, 10 Oct 2018 10:00:00 GMT

Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
802b00ed6f79: Downloading [================================================>  ]  21.63MB/22.49MB
5291925314b3: Download complete
bd9f53b2c2de: Download complete

The number of layers a container is using is related to the number of RUN and COPY/ADD commands that were used in the Dockerfile during that containers build.

Too many layers and your machines take a large amount of storage and can take a long time to download and extract. Too few layers and the machine takes F O R E V E R to rebuild.

Inconvenient Truth, 2018 version

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 10 Oct 2018 06:00:00 GMT

Combine headlines about the economy with those about climate change, and something uncomfortable emerges.

Show at NUPOC

John Jacobsen — Wed, 10 Oct 2018 03:00:00 GMT

I am pleased to announce:

Of the Flesh but not the World...

A show of nine works in a small gallery space at the Feinberg School of Medicine at the Northwestern University Prosthetics-Orthotics Center (NUPOC).

The works include four paintings made from life at NUPOC over the course of the last few months. There are also paintings and drawings done from life at the Palette and Chisel Academy of Fine Art.

Plagiocephaly Helmet Casts, oil on panel, 12" x 9"

Location and Hours

Feinberg School of Medicine
Northwestern University Prosthetics-Orthotics Center (NUPOC)
680 North Lake Shore Drive, Suite 1100
Chicago, IL 60611
312-503-5700

Dates: November 2 through the end of 2018.

Hours: Monday-Friday, 9 AM - 4:30 PM

A quick Google will show a number of parking options nearby. The building is about four blocks from Navy Pier; the closest El stop is Chicago Ave., on the Red Line.

Portrait of J., oil on panel, 11.5" x5.5"

Artist's Statement

"Of the flesh but not the world? How peculiar. How can that be? It sounds like maths, only more practical somehow." – Thomas Pynchon, Against the Day

For the past six months I have had the privilege of painting from life roughly once per week at the Northwestern University Prosthetics-Orthotics Center (NUPOC). NUPOC carries out research and training in the design, manufacture, and application of orthotic and prosthetic devices. Aside from representing a fascinating synthesis of engineering and medicine, it is a visually rich environment hosting hundreds of artifacts, all of which relate in some way to the human form.

These artifacts appeal to both the engineer and the artist in me. The anatomical models, common in both medical schools and art studios, are not only helpful objects of study (a figurative artist ought to know how the body is put together), but beautiful objects of contemplation in their own right. In addition, NUPOC's collection of historical and current prostheses and orthoses, ranging from the crude to the intricate, is visually fascinating… any of these objects could be the cornerstone of a striking still life painting.

In addition to the four paintings done at NUPOC, there are figurative paintings and drawings based on live models. They represent a subset of many studies carried out over the past few years at the Palette and Chisel Academy of Fine Arts, a century-old Chicago institution where one can find people drawing and painting from life nearly every day of the year. Working from life and studying anatomy are centuries-old strategies for turning paint into convincing illusions - to create bodies that are "of the flesh, but not the world."

From the standpoint of engineering, I find the devices at NUPOC intriguing for their mix of high- and low-tech. 3D printers and advanced polymers contrast with mechanical joints, hooks, levers and cables. Automated mass-production of some components exists in conjunction with highly customized fittings of others, to adapt to every individual's unique morphology and requirements. High-tech is not necessarily better tech: advanced prostheses with multiple digits controlled by nerve induction may be less appropriate than a more straightforward, affordable device that can be maintained by its wearer "in the field."

More generally, the artifacts represent a literal meeting of the human organism, with its animal origins, and the engineered world. Limbs lost, or never present, can be augmented in capability, at times far past their "baseline" functionality. What could be more human than to build devices to mold or extend the function of our bodies challenged by trauma, age, genetics, or other vicissitudes? What are the limits of our flesh, assisted or no? These are interesting questions for a figurative artist to ponder.

Many thanks to the staff at NUPOC for their assistance and hospitality.

NUPOC Leg Model ("Legcorché"), oil on linen, 7"x28"

Gift to NUPOC with RJ Garrick

For news and updates

Say hi on email, or check me on Instagram (eigenhombre).

Valley of Genius: The Uncensored History of Silicon Valley (As Told by the Hackers, Founders, and Freaks Who Made It Boom)

🦉 olshansky 🦁 — Wed, 10 Oct 2018 03:00:00 GMT

Olshansky's review of Valley of Genius: The Uncensored History of Silicon Valley (As Told by the Hackers, Founders, and Freaks Who Made It Boom) by Adam Fisher

Self-sufficient virtual organizations, and how to build them up

Zefram's Blog — Tue, 09 Oct 2018 06:50:00 GMT

I believe one of the most, if not the most, important applications of blockchain and smart contact technology will be the establishment of self-sufficient organizations that are completely virtual:

They will use virtual currencies like Ether and ERC-20 tokens to handle all transfers & storage of value.
Their collaboration & making of crucial decisions will be handled by smart contacts.
Their members will consist of pseudo-anonymous individuals from around the world who collaborate mostly remotely.
They will produce products that are purely digital in nature, and use the profits to cover their operational expenses, as well as provide their members with a non-negligible salary so that they may work for the organization on a part/full-time basis.

The main challenge of accomplishing such a feat is this: how may we replicate the highly complex, fuzzy, and subjective business logic needed for most organizations using immutable, simple, strict, and objective smart contacts?

Those little links between Climate Change, income and digital buzzwords

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 09 Oct 2018 06:00:00 GMT

A report issued yesterday about climate change “paints a far more dire picture of its immediate consequences than previously thought”. It also contains, between the lines, some links and implications that I have not seen in mainstream media yet.

San Francisco Walks

Avy Faingezicht — Tue, 09 Oct 2018 03:00:00 GMT

This set of photos is a mixed bag. Some of them have been sitting in my archive for almost a year, and some I took just a couple of weeks ago. They're also a mixed bag in terms of quality, but editing is not my strong suit - I have a really hard time cutting.

Induction and Side-Effects

Ashton Wiersdorf on Lambda Land — Mon, 08 Oct 2018 21:23:28 GMT

Today in my proofs class (MATH 290 at BYU) we talked about the concept induction. I like this, because it sounds a lot like recursion.

On the Wikipedia article, there’s an excerpt from a book that illustrates the principle with an analogy using a ladder:

Mathematical induction proves that we can climb as high as we like on a ladder, by proving that we can climb onto the bottom rung (the basis) and that from each rung we can climb up to the next one (the step). — Concrete Mathematics, page 3 margins

The problem with this, however, is that climbing a ladder has side-effects! Namely, when you climb up a step, you get tired. Eventually, there comes a point where you get so tired that you collapse from exhaustion, fall off the ladder and smack into the hard, uncaring ground of reality below.

This will happen to you too if your recursive functions have side effects: you code will be really hard to debug, and eventually, a bug somewhere in the state of your system will deftly shove your process off the call stack to crash on the cold, hard, uncaring silicon below.

Photo by Robin Joshua on Unsplash

Log file switch (checkpoint incomplete) wait events and LGWR waiting for checkpoint progress

Tanel Poder Blog — Mon, 08 Oct 2018 07:00:08 GMT

I was recently involved in troubleshooting an interesting performance issue with some non-intuitive background process & wait event behavior. I reproduced the problem in my own custom-tailored environment (Oracle 18.3 on Linux), here’s the starting point:

After starting a transaction-heavy benchmark, soon after the transaction rate drops and the database sessions are mostly waiting for some Configuration wait class events, instead of getting work done.

Just looking into the high level wait class names is not enough, so let’s drill down with ashtop.sql to see more:

Set Phasers to Blind

taylor.town — Mon, 08 Oct 2018 03:00:00 GMT

It's crazy that there's a Wikipedia Page entitled Protocol on Blinding Laser Weapons.

Why is it so crazy? Because most of the world believes it's better to be dead than blind.

Protecting Civilians

Okay, well, maybe it's not so crazy.

The Convention on Certain Convential Weapons concluded in 1980. Many nations came together to ban weapons like Booby-Traps and Incindiary Weapons. These weapons were found to cause too much "collateral damage". Hooray for protecting civilians!

Of course we don't want to harm innocent people, but imagine if we started replacing bullets with lasers -- to blind our enemies rather than kill them.

Blinded enemies are no longer viable opponents. Blinding weapons can remove combatants them from war without killing them.

Maiming soldiers is way better than creating orphans and widows, right?

Humans favor sight over all other senses -- seriously, a huge proportion of our brains are dedicated to visual/spatial processing.

So why would you want to purposefully destroy somebody's most precious connection to the world? Shouldn't we rather aim for the limbs?

Unfortunately, our current method is "bombard them with bombs and bullets and maybe they'll survive". We currently have no way of purposefully aiming for limbs during war. Right now, every combatant plays a game of roulette with prizes ranging from deafness to burns to limb loss to chronic respiratory illness to death. Wouldn't most soldiers rather play a game of "blind or not-blind"?

Becoming blind may reduce your quality of life, but it doesn't reduce the quantity of your life, and it doesn't result in chronic pain.

Blinding weapons aren't better because they maximize damage -- it's because they minimize randomness.

As a brief warning, conflict epidemiology is notoriously difficult. So be fairly skeptical of casualty counts.

The ICRC contends that most victims of war are not permanently injured:

Sixty percent of war casualties both survive and fully recover over time. With blinding from lasers there would be no recovery and no prosthetic device can replace sight.

The ICRC is very wrong.

In all of the US wars ever, about half all casualties were deaths (1.3M dead, 1.4M wounded). And how many of those 1.4M wounded do you really think fully recovered?

But let's pretend few survivors are permanently wounded, so that our recover:maime:kill ratio is 40:10:50.

Let me now remind you that our best attempt at being humane is "bombard them with bombs and bullets and maybe they'll survive".

Once again, non-lethal weapons that permanently injure others are terrible, terrible feats of engineering. But blinding weapons could change the recover:maime:kill ratio to something like 10:70:20. Wouldn't that be a signifcant improvement?

Jab, Jab, Hook!

The ICRC argues that blinding weapons are particularly harmful when they're used in combination with lethal weapons:

Blinding lasers would not actually save lives as they are intended to be used in addition to other weapons. They might even have the effect of increasing mortality rates as b linded opponents would not be able to defend themselves and thus be easily targeted by other weapons. As it is unlikely that an attacker would be able to assess at a distance whether an opponent has been rendered out of action by blinding, he would also use his other weapons. The result would therefore be just as many deaths and many more blind, thus increasing the suffering which results from battle.

The argument here is that blinding rays would be used as jab, to be followed up with lethal hooks.

Death is unavoidable in war, but it is in every military's best interest to create as many survivors as possible. Why? Because wounded casualties are more burdensome than dead ones.

It's also in every military's best interest to reduce the destruction of enemies' equipment and supplies. When you destroy something that can be reused, you're harming both sides of the conflict! If blinding lasers are used as an initial jab, then it incentivizes less-catastrophic secondary action.

In other words, if a significant portion of your opponents have been blinded, it opens up more options of secondary non-lethal force. And all combatants are incentivized to preserve enemies' lives, supplies, and equipment.

Conclusion

Imagine if US police officers used lasers instead of bullets -- how many deaths could be prevented? 1/3 of gunshot wounds are lethal (this is just counting the people that survive long enough to be rushed to the hospital). What if police officers were equipped with weapons less lethal than guns, but with more range than pepper-spray?

Imagine if the US used lasers rather than atomic bombs. Imagine if we blinded hundreds of thousands of Japanese civilians, rather than killing them? Wouldn't that something like that still do the trick? Of course, blinding an entire city would cause a few casualties (e.g. people driving), but once again, it's better than exposing them to radiation.

Of course, war is terrible and we should stop fighting forever.

But until we figure out how to achieve world-peace, we'll be the blind leading the blind.

5 Reasons to Contribute to Open Source

Mike Kasberg's Blog — Sun, 07 Oct 2018 19:00:00 GMT

It’s October, and Hacktoberfest is in full swing! It’s a great time to contribute to open source. I love open source software and the open source community, and I think contributing to open source software has lots of benefits for professional software engineers. Here’s 5 reasons you should contribute to open source software.

#1 Try Something New

Whether you want to learn a new language or try a new framework, open source projects provide a great opportunity to learn about new technologies. Want to see the latest version of Angular or React in action? Want to play around with some projects in Go or Scala? By working on open source projects, you can gain experience with technologies you’re interested in but might not be exposed to at your day job. In fact, it’s a great way to research technologies to see if they might fit well with your business needs. And if you contribute a pull request, you’ll likely get some feedback from people who are experts in the technology you want to learn.

#2 Collaborate More Effectively

When you work on an open source project, you probably won’t ever meet the people you work with in person. All of your interaction will be online, via Github and perhaps IRC or email. Because you can’t talk to the team face-to-face, written communication is critically important. Most open source projects have guidelines that promote best practices for communication, such as providing clear, minimal examples of bugs and writing good commit messages. Open source forces you to learn to communicate effectively in writing because there are no other options for communication. Practices from open source projects can help even in-person teams work more efficiently, and the communication skills you pick up working on open source will make you a better developer.

#3 You Have To Test Your Code

Most open source projects are pretty well tested. They have to be in order to keep their products from breaking while accepting pull requests from dozens (or hundreds) of different people. If you submit a pull request and your code isn’t tested, you’ll probably be asked to test it (and commit the tests) before it can be merged. In fact, many projects use tools like Codecov to automatically request changes on pull requests that aren’t sufficiently tested. Contributing to open source projects like these forces you to thoroughly test your code. Moreover, many of the testing practices they use (CI testing, code coverage analysis, style checkers, etc) are also useful tools for closed-source development.

#4 Learn From Other Developers

When you work on open source software, you get to interact with some of the best developers around. You can learn a lot from the code they write as well as the way they manage the project. How do they ensure their code is sufficiently tested? How do they make the project accessible to newer developers without letting the quality slip? Maybe you’ll be exposed to some new design patterns or libraries that you haven’t seen before. Do they use JUnit or TestNG? Maybe they’re using Hamcrest for their assertions? How do they handle dependency injection, and what patterns do they use to test their date logic? There’s no better way to learn than to see these patterns in action.

#5 It Makes You A Better Developer

Contributing to open source is a great way to make yourself a better developer. You’ll learn new skills, get new ideas, and have fun along the way. If you haven’t worked on any open source projects lately, give it a try! There are still 3 weeks left in Hacktoberfest 2018, so it’s a great time to start! (And a big thanks to Digital Ocean, GitHub, and Twilio for sponsoring the event!)

Store Passwords With KeePass

jorin.me — Sun, 07 Oct 2018 09:00:00 GMT

Let me show you why KeePass is more fun than your cloud-based password manager.

Networking Essentials: Congestion Control

swyx's site RSS Feed — Sun, 07 Oct 2018 05:52:44 GMT

Bottlenecks inevitably arise in networks. How do we deal with them in TCP? How about in practical streaming applications like Youtube and Skype?

The Problem with Podcasts

taylor.town — Sun, 07 Oct 2018 03:00:00 GMT

Los Angeles and Riverside are separated by a minimum of 90 minutes of suburbs and shopping centers. Taking the CA-60, suburbia is interrupted by hills peppered in weed-like plants. I'm reluctant to call the flora "weeds", because half are tree-sized and the other half are dead.

My pilgrimages to Riverside usually allot 240 minutes for phone-calls, listening, and aimless thought. The human mouth is comfortable producing 150 English words-per-minute (WPM), but our ears can comprehend upwards of 500 WPM. I usually listen to podcasts 2.0-2.5 times faster than they're recorded, which gives me 500 minutes (75,000 words) of podcasts on each round-trip. That's about eight podcast episodes or one whole book!

Today my podcast backlog seemed particularly unappealing. I didn't want to hear about ice-baths, or identity politics, or nootropics trends, or cryptocurrency, or crime drama, or whether a "Devil's Triangle" could conceivably refer to anything but a MMF threesome. Luckly, I have audiobooks on standby for emergencies like this.

I'm a huge fan of the gritty cyberpunk universes crafted by authors like Phillip K. Dick and Neal Stephenson. Books like Snowcrash consistently warp my views of capitalism, society, technology, drugs, and The Self. So I guess it was finally time for "Distrust That Particular Flavor" by William Gibson, which is apparently not a novel -- it's an essay collection. I love essay collections! It was a surprise, but a welcome one.

Anyway, Gibson is a genius. And I don't use that word lightly. These essays induced frission. These essays demanded quiet reflection. These essays will forever change the way I see Japan, writing, futurology, dystopias, and the internet. But this rant is not about how amazing Gibson is -- it's about what we're losing in this golden-age of podcasts.

Some podcasts are just unprinted audiobooks with advertising. Stories like S-Town and The RFK Tapes take advantage of phone-interviews and historical audio, which doesn't translate to text or video very well. And series like Dan Carlin's Hardcore History have carved out a niche where books are too long and articles are too short. Carlin creates 4-hour masterpieces, recounting the past with fresh perspectives and a voice carrying conviction and emotion. In these audiobook-esque experiences, the creators use audio to forge artistic information experiences in a deeply satisfying way. It is a careful craft, like film and literature.

But most podcasts aren't audiobooks. They're weekly radio shows. Tim Ferris and Joe Rogan and Preet Bharara and Sam Harris have taught me a lot about the world -- I'm very grateful for that. But I'm starting to think that interviews are not optimal for communicating truly deep ideas.

Conversation is good for certain things. It's good for covering current events from a singular perspective. It's good for debates (which usually devolve into mutual confirmation-bias masturbation). It's good for exploring cursory ideas, which the listener can later use The Internet for dissecting in greater detail.

But it's hard to go deep in a few hours. It's not enough time to open up to thousands of strangers about your trauma (unless it's rehearsed). It's not enough time to catalog complexity. And it's not enough time to find mutual ground on contentious issues. It's just not enough time for our inefficient mouths and minds.

Books are focused. Authors spend years crafting a singular narrative, to communicate a very specific idea. Podcast interviews are diffuse. Conversations ramble and reclarify and misstep and interrput and drift. They demand low-resolution regurgitations of their guests' ideas. And they're incentivized to blast through as many low-resolution sound-bites as possible, because listeners often want the "greatest hits". Conversations are unexplored territory. You have to phrase things for an interviewer and an audience in real-time. Text is forgiving until it becomes permanent -- you can write draft, after draft, after draft, until your paragraphs approximate that uncomfortable idea deep down in your soul. It's difficult to write about difficult things. But it's often impossible to talk about them.

Podcast guests must be careful about what "facts" they use. During interviews, hosts and guests use stats from their faulty memories. And even if they remember their "facts" correctly, there is no easy way to find a reference to the exact thing they mentioned. And if the "fact" was wrong, then most shows don't bother even to "fact"-check.

Podcasts need to be more entertaining than books in order to survive. A book just needs to supply the information it promised on its cover -- the entire text is purchased up-front. And if you purchased the book, you're probably already interested in its contents, and gave the author permission to go as deep as possible. Podcasts, on the other hand, need to keep users engaged for advertisers. They need you to come back every week. They need novelty. They need excitement. Because they need to gain and retain as many listeners as possible, which stifles depth. Podcasts will always devolve into pulp and click-bait given enough time, because that what advertisers want.

Anyway, I'm not saying that all books are better than all podcasts. I'm saying that podcasts aren't a viable replacement for audiobooks and other long-form content. Podcasts shouldn't be the main source of nutrition in your information diet.

Ultimate Member User Profile & Membership Plugin STORED XSS

SerHack - Security Researcher — Sat, 06 Oct 2018 03:00:00 GMT

Ultimate Member is one of the many user profile & membership plugins for WordPress. The plugin makes it a breeze for users to sign-up and become members of your website. The plugin allows you to add user profiles to your site and is suitable for creating advanced online communities and membership sites. Lightweight and highly extendible, Ultimate Member will enable you to create almost any type of site where users can join and become members with absolute ease. -Continue to read the article on SerHack.me

(Mis)adventures of Building a Chat Bot

Playground — Sat, 06 Oct 2018 03:00:00 GMT

In a Telegram group of three, a fourth member was added - GB - a Telegram bot. GB is the most useless, unavailing and noisy bot there is, but it’s fun. I started working on it because, I have always been interested in building chat bots, but never had the chance. Secondly, and this is a rather ambitious thinking, I want to build a NLP system able to understand and act on the natural language. Lastly, this was more of a realization when I started my work on GB, opportunity to build NLP tools and techniques for Hindi+English (or Hinglish). In the process, I’ll be working on both basic NLP and some advanced NLP techniques. And, I’ll try to implement everything from scratch.

Eve 0.8.1 has been released

Nicola Iarocci — Fri, 05 Oct 2018 10:01:01 GMT

Just a quick note to report that Eve v0.8.1 is out and ready to rock. It brings a number of fixes, some improvements as well as a couple of new features. Check out the changelog for details.

Join the newsletter to get an email alert when a new post surfaces on this site. If you want to get in touch, I am @nicolaiarocci on twitter.

Command/Query Bus with IoC Container

Jason Raimondi on Software Engineer — Fri, 05 Oct 2018 10:00:00 GMT

Actions can either be a Command or Query.

Application Core

ApplicationCore is our gateway into our bus. With the ApplicationCore, we can dispatch our Actions.

const logger = LoggerService.createFromWinstonLogger();
const applicationCore = new ApplicationCore(logger);

applicationCore.displatchCommand(
  new CreateUser({
    email: body.email,
    passwordString: body.password,
    firstName: body.firstName,
    lastName: body.lastName,
  })
).then(response => console.log(response));

Actions

Actions can be either Commands or Queries. Commands change state, queries retrieve data.

Actions need to register their respective handlers with the @Action(ActionHandler) decorator. This registers the Action/ActionHandler pair so when a new action is dispatched we know which handler to execute.

macOS app bookmarks

xenodium.com @alvaro — Fri, 05 Oct 2018 03:00:00 GMT

React: Not Everything is State

Dylan Paulus' Blog — Fri, 05 Oct 2018 03:00:00 GMT

When working with React we get introduced to state as a way if storing component's data. There is another way that generally gets overlooked: class properties.

Example

class AnnoyingButton extends React.Component {
  intervalId = null; // Here!
  id = generateUniqueId(); // Here!

  state = {
    isDisabled: false,
  };

  componentDidMount() {
    this.intervalId = setInterval(() => {
      this.setState(({ isDisabled }) => ({
        isDisabled: !isDisabled,
      }));
    }, 100);
  }

  componentWillUnmount() {
    console.log(`Unmounting ID: ${this.id}`);
    clearInterval(this.intervalId);
  }

  render() {
    return <button disabled={this.state.isDisabled} />;
  }
}

Nothing special about the component; the interesting bits are the id and intervalId variables. We need some way of preventing memory leaks when the component unmounts. In componentDidMount we save the interval's ID to a class property. Then, use the property to clear the interval and console.log the component's unique ID.

You might be thinking, "Why chouldn't we just store this information in the component's state?" React re-renders the component, and any child components, when the state changes. If the render method never uses a piece of the state it could be causing unwanted renders--making your application slower!

When to use class properties: If state needs to be stored, but is never used as a part of the view/render of a component.

Using bulk update in Django 2.2

Tom Forbes — Thu, 04 Oct 2018 21:36:02 GMT

My work on adding bulk_update() to Django has been merged and will be released in Django 2.2! Like my filtered aggregates feature it relies heavily on the wonderfully versatile CASE statement to achieve some pretty good speedups for certain use-cases. The development documentation gives a pretty cle...

how to load initial data and test data in Django 2+

Subclassed — Thu, 04 Oct 2018 15:42:41 GMT

There are two ways to automatically load data in Django:

for data you need while running tests, place xml/json/yaml files in yourapp/fixtures.
for data you need while setting up the database from scratch, or at specific points in time, you must create a Migration

This is a bit annoying, because chances are these locations will get out of sync sooner or later, and it duplicates effort if you do reproducible builds, docker, and stuff like that.

The solution is to create a migration that actually loads fixtures. So:

Create your fixtures: manage.py dumpdata --output yourapp/fixtures/yourmodel.json yourapp.YourModel
Create an empty Migration: manage.py makemigrations --empty yourapp

Edit the resulting migration (the last file created under yourapp/migrations, making it look like this:

from django.db import migrations

def load_fixtures(apps, schema_editor):
    # This is what will be executed by the migration
    from django.core.management import call_command
    # this is the equivalent of running manage.py loaddata yourmodel.json
    for fixture_name in ['yourmodel']: # add any additional model here
        call_command("loaddata", fixture_name)
    # add other calls if you have multiple models

def rollback(apps, schema_editor):
    # This will be executed if you rollback the migration, so you want to clean up
    for model_name in ["YourModel"]:  # add any additional model here
        model = apps.get_model("yourapp", model_name)
        model.objects.all().delete()

class Migration(migrations.Migration):
    dependencies = [
      # ... don't touch anything here ...
    ]

    operations = [
        migrations.RunPython(load_fixtures, rollback),
    ]
# -*- coding: utf-8 -*-

Profit

Note that this does not remove the option to have data that is available only in certain situation: just don't list the fixtures you don't want in the migration, and vice-versa.

`ls` command to show directory contents

Josh Thompson — Wed, 03 Oct 2018 14:00:00 GMT

I like to use the tree command on my local machine when trying to peek into the structure and contents of a given directory.

tree -L 2

will [L]ist recursively everything [2] levels deep from your current directory. The output is nicely formatted like this:

> tree -L 2
.
├── cargo
│   ├── ARCHITECTURE.md
│   ├── CONTRIBUTING.md
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── LICENSE-APACHE
│   ├── LICENSE-MIT
│   ├── LICENSE-THIRD-PARTY
│   ├── README.md
│   ├── appveyor.yml
│   ├── rustfmt.toml
│   ├── src
│   ├── target
│   └── tests
├── fastly
│   └── fastly-test-blog
├── get-pip.py
├── learning_aws
├── learning_elixir
│   ├── hellow
│   ├── sample01.exs
│   └── sling_clone
├── learning_over_the_wire
│   ├── bandit
│   └── leviathan
├── learning_react
│   └── react_tutorial
├── learning_ruby_rails
│   ├── InstaClone
│   ├── blocks_practice.rb
│   ├── chris_pine_ruby_lessons
│   ├── eloquent_ruby
│   ├── email_sender

If you’ve SSHed into a linux box, however, and you’re trying to look around a bit, you won’t have tree available to you. How can you list out the contents of directories?

Easy. The good ‘ol ls command:

ls -1d ./*/*

results in:

> ls -1d ./*/*
./cargo/ARCHITECTURE.md
./cargo/CONTRIBUTING.md
./cargo/Cargo.lock
./cargo/Cargo.toml
./cargo/LICENSE-APACHE
./cargo/LICENSE-MIT
./cargo/LICENSE-THIRD-PARTY
./cargo/README.md
./cargo/appveyor.yml
./cargo/rustfmt.toml
./cargo/src
./cargo/target
./cargo/tests
./fastly/fastly-test-blog
./learning_elixir/hellow
./learning_elixir/sample01.exs
./learning_elixir/sling_clone
./learning_over_the_wire/bandit
./learning_over_the_wire/leviathan
./learning_react/react_tutorial
./learning_ruby_rails/InstaClone
./learning_ruby_rails/blocks_practice.rb
./learning_ruby_rails/chris_pine_ruby_lessons
./learning_ruby_rails/eloquent_ruby
./learning_ruby_rails/email_sender

ls is obviously “list”. What about that -1d thing?

according to the man page:

-1   (The numeric digit ``one''.)  Force output to be one entry per line.  This is the
      default when output is not to a terminal.
         
-d   Directories are listed as plain files (not searched recursively).

to achieve a certain number of directories in depth, just repeat the ./*./* pattern, one /* per directory depth you’d like to view.

The output isn’t always particularly readable - sometimes you get a ton of results.

If this is the case, you could always pipe the output to head, or grep the output. That said, I’ve used this general pattern regularly, and it’s been helpful.

List subdirectories only n level deep (unix.stackexchange.com)

Base64 Basics

denvaar's website — Wed, 03 Oct 2018 08:40:09 GMT

Base64 is an encoding scheme used to change binary data into text.

How to Setup Discourse on DigitalOcean

Sam Solomon — Wed, 03 Oct 2018 03:00:00 GMT

I have spent a good portion of my evenings during the last two weeks working with a piece of community software called Discourse.

Ripping DVDs in linux with FFmpeg

Shyam Jos — Wed, 03 Oct 2018 03:00:00 GMT

In this tutorial, I will show you a simple way to RIP DVD into a single file using FFmpeg command line tool in Linux.

Install FFmpeg in Ubuntu/Debian/Linux Mint #

sudo apt-get install ffmpeg

Now copy contents of your DVD to local directory and cd into VIDEO_TS directory

Note: If you want to RIP the DVD directly without copying it to the local directory first , then you need to modify last part of the below commands to : /home/your-username/myvideo/rip.mp4

1. RIP DVD into a single file #

Now enter below command to start ripping process in optimum quality, If you want to control the size (eg: limit size to 700MB) of the ripped file, then use the third command

cat VTS_0*_*VOB | ffmpeg -i - -vcodec h264 -acodec mp2 rip.mp4

2. RIP DVD with deinterlace option #

If you want to enable deinterlace option run below command

cat VTS_0*_*VOB | ffmpeg -i - -vf yadif -vcodec h264 -acodec mp2 rip.mp4

3. RIP DVD with quality option #

Use below command to reduce size and quality of output file, Here the ‘-crf’ option sets the quality of the ripped file. The range of the CRF scale is 0–51, where 0 is lossless , 23 is the default, and 51 is the worst quality possible.

cat VTS_0*_*VOB | ffmpeg -i - -c:v libx264 -crf 23 rip.mp4

Above command will read all video files in a folder and pipe the stream to FFmpeg command, after the successful completion of the command you can find the ripped file with name rip.mp4

Hammer Factories

Dan Stroot — Wed, 03 Oct 2018 03:00:00 GMT

There is a post from 2005 by Benji Smith in the old (and now closed) Joel on Software Discussion Group. It is titled "Why I Hate Frameworks". But I know it as "the hammer factory" post. It is just brilliant, even 13 years later. I am reproducing it here for posterity in case the old Joel on Software Discussion Group ever disappears.

This post Hammer Factories first appeared on Dan Stroot's Blog

Fix Missing Headers Compiling on macOS Mojave

Donat Studios — Wed, 03 Oct 2018 00:22:40 GMT

Update It would appear that the latest version of Xcode has removed the macOS_SDK_headers_for_macOS_10.14.pkg file. I am currently searching for alternative solutions.

While trying to compile PHP extensions after installing macOS Mojave, I was butting up against a ton of missing header file errors including php.h and zlib.

After several hours of beating my head against my desk and Googling/Stack Overflowing I found the solution.

Ends up you just need to force re-install the header files. Special thanks to @sfdye for the solution!

$ sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

If you get an error about that file not existing, you probably simply don't have the Xcode command line tools installed which can be remedied simply as follows:

$ xcode-select --install

Blogtober 2018, the second

infrequent oscillations — Tue, 02 Oct 2018 21:44:19 GMT

In which we fail dismally at all the 'obers.

You Don't Understand Compound Growth

Who is Nnamdi? — Tue, 02 Oct 2018 11:13:00 GMT

How the cloud grew up around Linux

Jason Eckert's Website and Blog — Tue, 02 Oct 2018 03:00:00 GMT

Open source software has had a powerful effect on computing. When the ARPANET/NSFNET was sold to different companies in the 1990s to create the commercial Internet, nobody expected it to grow as fast as it did. It was the open source Apache Web server running on the open source Linux (and BSD UNIX) operating system that ultimately allowed the Internet to grow quickly and attract the attention of the world. Apache on Linux allowed anyone with a computer and Internet connection to implement a Web server to serve Web pages as part of the World Wide Web of Internet resources. In short, if there was no open source Web server or operating system freely available in the 1990s, the rise of the Internet would have been dramatically slower or even non-existent until the 2000s. The Internet and the World Wide Web grew up around open source and Linux.

Blogtober 2018, the first

infrequent oscillations — Mon, 01 Oct 2018 22:31:06 GMT

In which we try to follow some 'obers.

JSON.stringify removes undefined, how to keep it

Muffin Man — Mon, 01 Oct 2018 03:00:00 GMT

This is something I keep rediscovering, because I keep forgetting it. JSON.stringify will omit all object attributes that are undefined.

In most cases, it doesn't really matter, because if we parse that string back, and try to access that attribute - it will be undefined by design. Check the example below:

const user = { name: 'Stanko', phone: undefined };
user.phone; // -> undefined

const stringifiedUser = JSON.stringify(user); // -> "{\"name\":\"Stanko\"}"

const parsedUser = JSON.parse(stringifiedUser) // -> { name: "Stanko" }

// At the end it behaves the same
parsedUser.phone; // -> undefined

Why should we care then? #

In most scenarios you shouldn't. But for me, one case keeps popping up - sending http requests. Request body is a string, so we need to stringify our data.

In certain cases we want server to be aware that some data has been explicitly removed, so it can be removed from the database as well. This is where dropping undefined can cause problems.

Few days ago, one of my clients had a question about React Final Form. Problem was that Final Form returns undefined for the values that have been removed by the user. As you can imagine, this was a problem, when they stringified form values undefined fields were omitted and server wasn't aware that the field was removed.

Using `replacer` parameter #

Luckily JSON.stringify accepts replacer function as a second parameterThird one is space, number of spaces or a string to be used for indentation. Function accepts two parameters, current key and value being stringified. This allows us to replace any value, in our case undefined.

We just need to check if the value is undefined and return null:

const user = { name: 'Stanko', phone: undefined };

const replacer = (key, value) =>
  typeof value === 'undefined' ? null : value;

const stringified = JSON.stringify(user, replacer); // -> "{\"name\":\"Stanko\",\"phone\":null}"

This is one example where replacer comes in handy. It can also be practical when stringifying complex objects and getting TypeError: Converting circular structure to JSON. It this case, we need to write custom replacer which will take care of circular references.

This post ended up being a bit longer than I planned, but I hope it was useful.

Self-Control is an Anti-Virtue

taylor.town — Mon, 01 Oct 2018 03:00:00 GMT

When we're young, we often get scolded for self-control issues.

If you talk too much, you're told to "hold your tongue". If you cry in public, you're told to "control your temper". And so on.

The logic is something as follows:

You often want to do things that offend others.
If you offend others, they won't like you.
If others don't like you, then you'll be miserable and lonely.
You can use self-control to offend others less often.
Thus, if you don't want to be miserable and lonely, use self-control to avoid offending others.

It's not very formal, but I actually think the logic is sound enough -- if you want to be likable, then don't do unlikable things!

But I'm starting to think that there are unintended consequences from living this way.

Don't get me wrong -- it's good to be considerate of others. But if you frequently supress your true feelings with self-control, when do your "bad" desires ever get addressed? I know that there's a camp that says "fake it 'til you make it", but I'm not so convinced anymore.

Many of my friends and family are a bit more than mildly racist. When I hear implications of deeper, darker feelings, I coax them into being explicit: "…and what exactly do you mean by that?" (but not in a judgemental way). When probed enough, most people find fear and self-hatred in that tangled ball of feelings. And they didn't want to talk about it in the first place because they were trying to be "politically correct". With enough practice, we move from "Mexicans are taking our jobs" to "I'm scared that my grandchildren won't have job security". But to make this jump, most people need to be unabashedly "politically incorrect". In order to change, they need to be honest with themselves and others. Otherwise, all contradictory evidence gets labeled as "irrelevant" or as "fake news". If you are truly convinced that Mexicans are The Problem, then any intention to help your grandchildren directly will seem misguided.

And so here's the pickle: we show social-restraint to avoid criticism. But with enough constructive criticism, we wouldn't need to restrain ourselves. But criticism is uncomfortable for everybody involved, so we continue restraint.

Short of threat of physical force, all beliefs are fair-game. If you don't test your thoughts against others, then how will you know which ones are wrong? And if you know it's wrong, then maybe saying it out-loud will force you to discover how bad it feels to say it, and how terrible it feels others. Sometimes seeing the consequences of a bad belief inspires change.

But I don't know -- what if all your peers are impressionable and your bad beliefs spread and mutate and become terrible monstrosities? Oof. Maybe I overestimate people's ability to change. Tell me I'm wrong, I guess 💁

I suppose this advice is more about listening than speaking. Become someone who others can expose bad ideas to. Most people will provide enough self-criticism -- just be there.

Never Do Things You Dislike

Self-control also has weird affects on long-term goal-planning.

When you consistently do things you don't like, why would you expect to end up somewhere better?

Most medical students loathe medical school. But they endure medical school because one day they'll get to be Dr. LastName. But many of them discover -- surprise -- that they hate being doctors! If you hated the training, why would you expect to love the outcome‽

If you dislike the training, the outcome is unpredictable. If you practice violin for 25 years, you may enjoy being an accomplished violinist. But why on earth wouldn't you take the time to learn to love violin‽ Or just choose a different instrument‽ If you practice something you enjoy, you are guaranteed to like the outcome.

If you dislike it, here are your choices:

Learn to love it.
- You will love the outcome.
Find a lovely alternative.
- You will love the outcome.
Change nothing.
- You may love the outcome.
Abandon it.
- You may love the outcome.

Lately, I've been having a lot of trouble going to the gym because, well, I hate it there. So let's plot out my options using the guide above:

Learn to love it.
- I love learning and science and challenges! I would probably enjoy weight-training a lot more if I reframed it as an experiment.
- I could also post constant reminders around my apartment to remind myself why I wanted to do weight-training. But human-minds are really good at filtering out reminders. I should really figure out why I stopped caring about my goals.
- I could pair weight-training with things that I already love, like podcasts and close friends!
Find a lovely alternative.
- Rather than lifting weights, I may really enjoy something dextrous and skill-based like rock-climbing or swimming! Unfortunately, my goal is to gain mass, and I think it will be particularly difficult for me to gain weight with that type of training.
- I could find new, interesting exercises! I've been really focused on barbell exercises, so maybe I focus more on machines? It's something to consider.
Change nothing.
- I may actually learn to love weight-lifting if I keep doing it! I really enjoyed it when I started. The problem is, what if I don't start loving it? What if I could spend my time bulking up faster on an activity that I actually enjoy? Seems stupid to me. I should change something.
Abandon it.
- Nah, exercise is important. But maybe I should stop wanting to bulk up so badly? Maybe I should change my goals? Maybe I'm more of a marathon-type person? Seems like an unsatisfying option.

Anyway, it's actually kind of fun exploring yourself this way! I learned a lot about myself just writing that simple list.

Life is like a bunch of forking roads. If you apply judicious self-control, you may end up where you want to go, but you may hate the entire journey there. Or even worse, you may hate the entire journey there, and also hate the destination. Take some time to look at other options!

Self-Flagellation

I have a lot of things to say about Christianity, but I guess I'll save it for another time.

Feeling Bad About What I Do All-Alone All By Myself

I don't remember what I was originally going to write here, but I love this subtitle 🐸

TL;DR

Self-control is weird and sometimes bad.
- Knock down others' social-restraints -- give everybody safe spaces for bad ideas.
- Stop doing things you dislike and you'll probably be happier.

Trifecta of Obers

infrequent oscillations — Sun, 30 Sep 2018 20:01:54 GMT

In which we make plans we fail to keep.

Doing the same thing and expecting different result

D13V — Sun, 30 Sep 2018 06:12:13 GMT

Einstein’s had a lot of witty and profound sayings, but “The definition of insanity is doing the same thing over and over again and expecting a different result” is not one of them. And yet, these words had often been misattributed to him. It is not exactly clear who came up with the phrase, but it is a fact that a lot of people got convinced that originated behind the famous mustache of Einstein.

He was one of the giants of science in the 20th century, (Einstein, not the ’stache) which means that the phrase has the sound, the ring of something smart. It sounds like something a physics genius might say. The phrase sounds sciencey.

And it had me believing it. Einstein, Meinstein or whoever said it, I subscribed to it. Until my work experience proved it otherwise.

Back when I worked for Twitter, I was running A/B tests. A properly conducted A/B test is one of the closest things you get to science when building a product. It lets you sift the noise and luck out so you can arrive at a piece of knowledge. If I change the product from A to B, I will get X% of my customers better by Y% with statistical significance. That’s the type of sentence you can confidently claim and bet a lot of money on after an A/B test.

And yet, A/B tests lack one crucial property of science. They are not reproducible. If you run an A/B test today, and run the same A/B test six months from now, you might get a different result. Allowing for a different result is emphatically not insanity. I get it, running the same experiment is likely to result in the same effect. But likely ain’t certain.

I saw the same experiment get run twice, the two runs being about a year apart. The results were different. The first time around, the experiment didn’t show any improvement. And yet, half a year later, it did work. The circumstances had changed. What we were trying to test worked better, even though the idea was the same. One year later, it showed positive improvement. It was green.

Same idea. Different result. This is counterintuitive AF. It was surprising after the fact, but even harder to imagine a priori.

Because we tend to overgeneralize from limited experiences and discard ideas that failed once before. Once an idea has been tried unsuccessfully, we tend to give it a black stamp in the passport and ban it from entering our heads again. We want knowledge that we can trust, and it sometimes takes a single foul for us to discredit an idea. One strike. Out.

Unscientific. Because science is all about reproducibility. The definition of science is doing the same thing over and over and knowing that you’ll get the same result. You can prove scientifically that if you drop an apple, in the absence of interference from other forces, it will fall down. It would be incredibly stupid and yes… insane to believe that the apple may not fall down. It will be non-scientific, which means that it is not the way the universe we live in works.

In our universe, gravity force always tries to pull objects down. And science is the way of determining the rules of the universe. But there are limits to science even in a deterministic system.

For example, take magnets. It is reasonable to expect that the North Pole of a compass will always be pointing at the North Pole of the Earth. It has always been pointing the same way. People have used it to navigate around the globe, for centuries, even millennia. And yet, it might flip around. It hasn’t always been that way. The magnetic field of Earth has reversed its direction multiple times, at random intervals

The magnetic reversal of the poles is due to being part of a chaotic dynamic system. Molten metals and whatnot are spinning and twirling and splashing and splashing inside the Earth’s core. If they spin mostly in circles, their electric charge generates a powerful magnetic field. It’s like a cup of coffee spinning after adding milk or sugar, but it is possible to spin it the other way rather quickly by changing the direction in which we stir it with a spoon.

In a chaotic system like that, the further you go out in the future, the lower your ability to predict what would happen. And the catch is that anything could happen. When dealing with chaotic systems, like the weather or politics, “expecting the unexpected” is not insanity. It’s common sense for the long run.

And we are often part of the systems. We adapt, learn new skills, and forget things such as what we had for lunch yesterday. Which means that if we’ve tried to do something before, and it didn’t work, but we’ve changed and we are trying it again… well, we might not be trying exactly the same idea. Not in the scientific sense of it being the same. Of course, if there are a lot of similarities, then we might have a strong prior expectation that the result will be the same. But there always are small differences, and often they matter a lot.

“Try harder” is a phrase many of us have heard, whether from a coach or a boss or a parent. The idea is that the reason for not succeeding before is our level of effort. Some tasks require a lot of strength, concentration and force us to muster our entire self. To “try harder.”

And sometimes the change in the chaotic system is imperceptible to us. We might be in a negotiation and keep on asking our counterpart to agree on something. They might be using our persistence to evaluate how much we care about the topic, and the amount we care might be a factor in them deciding whether to acquiesce to our demands, and by persisting on a topic, we might be more convincing. Such persistence only works sometimes though, as it can be interpreted as badgering, or bullying and could alienate the other side. Even if it gives us a win, it might be a pyrrhic win, dragging us down in the long term.

So… what’s the whole point. Was Meinstein right to point at insanity, where others might see persistence? I think not. I believe unless scientifically proven wrong, we shouldn’t give up easily. To paraphrase another big physicist, David Deutsch “Everything that’s not forbidden by the laws of physics is possible.”

To overgeneralize failure is a logical fallacy. To tell ourselves stories, claiming that things are impossible. These stories are lies.

And we can stop lying to ourselves.

Gaming bookmarks

xenodium.com @alvaro — Sun, 30 Sep 2018 03:00:00 GMT

FontStruct - Break the Font InfraSTRUCTure!

SerHack - Security Researcher — Sun, 30 Sep 2018 03:00:00 GMT

The website FontStruct allows even novice users to create their own personal font from scratch. A very simplified procedure: you sign up and you are ready to compose your font. Everything is offered through a simple interface as a free service. Not bad for a free service! -Continue to read the article on SerHack.me

Waspr Project - Start failing

CraftyDuck.Rocks — Sun, 30 Sep 2018 03:00:00 GMT

First ideas, bad ideas

Why Clojure?

Brian Sunter — Sun, 30 Sep 2018 03:00:00 GMT

What makes Clojure interesting and well suited for modern software development?

I2c Tools for Project RBOT

Greg Herlein — Sat, 29 Sep 2018 18:44:43 GMT

I2C Tools I am Using for the RBOT Project

Solving My Email Problem - Followup

Beetle Space — Sat, 29 Sep 2018 10:00:00 GMT

My last post got quite a few comments on Hacker News. They did force me to think a little, and here I’ll gather some of my thoughts.

“I have an email workflow that is different from yours”

If you’re happy with your email setup, I am by no …

Install docker on debian with Ansible

Yasha Solutions — Sat, 29 Sep 2018 08:59:28 GMT

Here is a playbook for Ansible to install Docker and Docker Compose on Debian. You can of course, use a role from Galaxy. You have Gerling’s version or Debops (Ansible stack for Debian). But here I wanted to implement the install guide from Docker – and translate this into an Ansible playbook (which then should be turn into a custom role for more reusability). The official guide give use the following install steps:

Lua bookmarks

xenodium.com @alvaro — Sat, 29 Sep 2018 03:00:00 GMT

Skin product bookmarks

xenodium.com @alvaro — Sat, 29 Sep 2018 03:00:00 GMT

Sustainability bookmarks

xenodium.com @alvaro — Sat, 29 Sep 2018 03:00:00 GMT

Source Fabric. Find a Manufacturer. Raise Money to Fund Production. (Factory45).

Investment platform bookmarks

xenodium.com @alvaro — Sat, 29 Sep 2018 03:00:00 GMT

Empathy is the new black

Yasha Solutions — Fri, 28 Sep 2018 13:08:24 GMT

For what it’s worth, I have been noticing a raising interest for empathy in the past few years. It’s been popping everywhere from a variety of fields, either in business and entrepreneurship, human resources and management to more more remote fields like design and product development as well as the obvious therapy and education fields. Science seems to have discovered a couple of years ago that we are wired for it:but science isn’t really everything there is to it – I mean look at this article about this lady being a lead researcher on empathy, and she still managed to bullied colleagues and staff.

Rails Migration: When you can't add a uniqueness constraint because you already have duplicates

Josh Thompson — Fri, 28 Sep 2018 13:00:00 GMT

I get to occasionally contribute to the Wombat Security dev blog. I wrote the following for development.wombatsecurity.com.

For work, I picked up a bug where a CSV export was creating duplicate rows when it shouldn’t have been.

We generate the CSV export with a big ol’ SQL statement in some Elixir workers, and got the bug reproduced and found the problem was with how we did some joins on other columns.

We had something in the statement like this:

LEFT OUTER JOIN `reports` ON `reports`.`id` = `people`.`report_id`
LEFT JOIN `addresses` ON `addresses`.`people_id` = `people`.`id`
LEFT JOIN `favorite_colors` ON `favorite_colors`.`people_id` = `people.id`

In this highly contrived example, I found out that we’re expecting a single row in the favorite_colors table for each person, but we were getting multiple favorite_color rows.

Every time we had a duplicate row on that joined table, the LEFT JOIN created two rows in the export, even though there should have been one.

Thoughtbot has an amazing article about the exact problem we were having.

In that article, they describe the problem with uniqueness validations like so:

https://robots.thoughtbot.com/the-perils-of-uniqueness-validations

We need to enforce uniqueness at the database level. Not the model level.

What’s the easy fix?

The “easy” fix is simple - Add an index to the column you want to enforce uniqueness upon, then add a uniqueness constraint:

(since this column already happened to have an index in our database, we have to remove it and re-add it in the migration)

class AddUniquenessConstraintToFavoriteColors < ActiveRecord::Migration
 def change
   remove_index :favorite_colors, :person_id
   add_index :favorite_colors, :person_id, unique: true
 end
end

Lets make this a bit more explicit. We’ll define an up and down migration. (you’ll see why in a moment)

Up/down migration

I am going to be explicit about the up and down, for the rollback:

class AddUniquenessConstraintToFavoriteColors < ActiveRecord::Migration
  def up
    remove_index :favorite_colors, :person_id
    # you can't MODIFY the index, just remove it, then re-add it with the changes
    add_index :favorite_colors, :person_id, unique: true, algorithm: :inplace
    # unique forces uniqueness (duh)
  end
  
  def down
    remove_index :favorite_colors, :person_id
    # removing the index that has uniqueness constraint
    add_index :favorite_colors, :person_id
    # adding one without the constraint
  end
end

This migration will work, if you run it against a table that doesn’t have duplicate values in the given column.

But the whole reason I was digging into all this was because I had a table with duplicate values. This is a table that now has many rows of data, and more than a few duplicates.

If you run the above migration, and there are duplicates on the table, you’ll get a lovely error like so:

== 20180914203948 AddUniquenessConstraintToFavoriteColors: migrating ========================
-- remove_index(:favorite_colors, :person_id)
   -> 0.0388s
-- add_index(:favorite_colors, :person_id, {:unique=>true, :algorithm=>:inplace})
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Mysql2::Error: Duplicate entry '44' for key 'index_favorite_colors_on_person_id': CREATE UNIQUE INDEX `index_favorite_colors_on_person_id`  ON `favorite_colors` (`person_id`) ALGORITHM = INPLACE

Clean up duplicates and add uniqueness constraint

The big question was:

How do we clean up the duplicates, and run the migration, without duplicates showing up between when we clean them up and run the migration?

My first thought was a rake task to find and prune duplicates, but this is a relatively active table, and if even a few seconds elapsed between the rake task and migration, we might get another duplicate on it, which would prevent the migration from running.

The Solution

The rake task was not a good solution. Fortunately, I work with many people who are much smarter than I, and they put me on the right track. I needed to update the migration to run a query that would:

Find all duplicate rows
Get the ID’s of those duplicate rows
Trim one of the row IDs of the “duplicates” list
Delete all the remaining IDs.

So, the SQL query gets built up in three pieces.

I’ll give you the full SQL query, then we’ll dig into the component pieces. The following query creates a list of IDs that we can then use ActiveRecord/Sequel to delete. (More on the migration in a minute):

SELECT substring(dups.id_groups, position(','IN dups.id_groups) + 1)
  FROM 
    (SELECT  group_concat(fc.id) AS id_groups
    FROM favorite_colors fc
    WHERE EXISTS
      ( 
      SELECT 1
      FROM favorite_colors tmp
      WHERE tmp.person_id = fc.person_id
      LIMIT 1, 1
     )
    GROUP BY fc.person_id
) AS dups

Lets unpack it. First, here’s a little SQL you could run locally to generate results to play with:

CREATE TABLE favorite_colors (
  ID INT AUTO_INCREMENT PRIMARY KEY ,
  person_id INT,
  color VARCHAR(255)
  );
INSERT INTO `favorite_colors` (`ID`, `person_id`, `color`)
VALUES
	(1, 1, 'yellow'),
	(2, 1, 'yellow'),
	(3, 2, 'blue'),
	(4, 2, 'blue'),
	(5, 3, 'green'),
	(6, 2, 'purple'),
	(7, 3, 'yellow'),
	(8, 4, 'black');

OK, first, lets find the duplicate rows. There’s a lot of them in here. The following query comes from StackOverflow, and served me well:

SELECT  * 
FROM    favorite_colors fc
WHERE   EXISTS
        (
        SELECT  1
        FROM    favorite_colors tmp
        WHERE   tmp.person_id = fc.person_id
        LIMIT 1, 1
        )
ORDER BY fc.person_id

So, obviously, that WHERE EXISTS piece is interesting. But that depends on the SELECT 1... LIMIT 1, 1, which is unfamiliar to me.

Limit count, offset

SELECT 1
FROM favorite_colors
LIMIT 1, 1

This is the most basic version of the above statement. The SELECT piece just inserts a 1 in the table. It could easily be SELECT "FOOBAR". The prior WHERE EXISTS is just looking for a value from the statement.

The LIMIT 1, 1 is curious. It’s LIMIT <count> <offset>. So it’s limiting the count to one, and it’s offsetting it by 1. PostgreSQL has the best docs on this function.

This means it returns one row, offset from the first row by 1. So, if you ran the following query:

SELECT  *
FROM    favorite_colors
LIMIT 2, 4

You’d get four rows, skipping the first two rows, and including the subsequent four.

Here’s the results of:

SELECT  *
FROM    favorite_colors
LIMIT 2

id	person_id	color
1	1	yellow
2	1	yellow

And if we run

SELECT  *
FROM    favorite_colors
LIMIT 2, 2

we get:

id	person_id	color
3	2	blue
4	2	blue

So, LIMIT 1, 1 takes the results of the prior WHERE and basically “hides” the first match, and leaves only the second match. The temporary result would be empty when the EXISTS statement checked the subquery, and the row would be deemed “not a duplicate”.

`EXISTS`

In the docs, we learn:

If a subquery returns any rows at all, EXISTS subquery is TRUE, and NOT EXISTS subquery is FALSE. For example:
SELECT column1 FROM t1 WHERE EXISTS (SELECT * FROM t2);

I still don’t fully grasp the nuance of the total SQL statement, but I feel like I’m approaching comprehension. Either way, I am pleased to know how to return the complete rows of duplicates from a table where there are duplicate values in a given column.

OK, so we’ve got duplicates. How do we make them usable?

Get IDs from results

We don’t want to SELECT *, now, we want to SELECT group_concat(favorite_colors.id) AS id_groups.

SELECT  group_concat(fc.id) AS id_groups
FROM favorite_colors fc
WHERE EXISTS
  ( 
  SELECT 1
  FROM favorite_colors tmp
  WHERE tmp.person_id = fc.person_id
  LIMIT 1, 1
 )
GROUP BY fc.person_id

returns:

id_groups
1,2
3,4,6
5,7

The GROUP BY table.column_with_duplicates is important to split the groups by unique value. Without the group by, you’d get one long string of joined IDs, which would be totally useless.

Next, we want to trim off the first ID in each of these groups. So, here’s the full query:

SELECT substring(dups.id_groups, position(','IN dups.id_groups) + 1)
  FROM (
    SELECT  group_concat(fc.id) AS id_groups
    FROM favorite_colors fc
    WHERE EXISTS
      ( 
      SELECT 1
      FROM favorite_colors tmp
      WHERE tmp.person_id = fc.person_id
      LIMIT 1, 1
     )
    GROUP BY fc.person_id
) AS dups

And this returns:

id_groups
2
4,6
7

And the results of this SQL statement can now be deleted from the database via ActiveRecord. Here’s our full migration:

class AddUniquenessConstraintToFavoriteColors < ActiveRecord::Migration
  disable_ddl_transaction!
  
  def up
    results = execute <<-SQL
    SELECT substring(dups.id_groups, position(','IN dups.id_groups) + 1)
      FROM 
        (SELECT  group_concat(fc.id) AS id_groups
        FROM favorite_colors fc
        WHERE EXISTS
          ( 
          SELECT 1
          FROM favorite_colors tmp
          WHERE tmp.person_id = fc.person_id
          LIMIT 1, 1
         )
        GROUP BY fc.person_id
    ) AS dups
    SQL

    results.each do |id_array|
      # I know find_in_batches would normally be a better fit
      FavoriteColors.where(id: id_array).delete_all
    end
    remove_index :favorite_colors, :person_id
    add_index :favorite_colors, :person_id, unique: true, algorithm: :inplace
  end
  def down
    remove_index :favorite_colors, :person_id
    add_index :favorite_colors, :person_id
  end
end

In Conclusion

You can use this pattern to add a uniqueness constraint to a table that already has duplicate values. This will clean out duplicates, but leave original values, and will prevent additional duplicates from being written to the table.

In hindsight, this was a relatively straight-forward migration. I had not found any resource online that talked about the process of adding a uniqueness constraint if the table already had data that violated the constraint, so I hope that this write-up might help someone else in a similar spot.

Update: Do not leave a column without an index at any point in the migration

I made a mistake in all this.

Take a look at what we ran:

results.each do |id_array|
  # I know find_in_batches would normally be a better fit
  FavoriteColors.where(id: id_array).delete_all
end
remove_index :favorite_colors, :person_id
add_index :favorite_colors, :person_id, unique: true, algorithm: :inplace
end

See how we remove an index, then add it?

This means if the migration fails while adding the new index, and we have to re-run the migration, we’re now executing the query without an index on the favorite_colors column.

If you’re doing a big ol’ query against a few hundred thousand rows of data, with the index, it’ll take a few seconds. Without the index, it could take… many minutes.

Here’s what you should do:

results.each do |id_array|
  # I know find_in_batches would normally be a better fit
  FavoriteColors.where(id: id_array).delete_all
end
# rename existing index
rename_index :favorite_colors, :favorite_colors_non_unique

# add the uniqueness index (if we didn't rename the existing index, we'd get an error saying "an index by this name already exists")
add_index :favorite_colors, :person_id, unique: true, algorithm: :inplace

# This line won't get run unless the above index has been successfully added. We're removing it by the name we've given it:
remove_index :favorite_colors_non_unique

end

Useful additional resources

Solving My Email Problem

Beetle Space — Fri, 28 Sep 2018 10:00:00 GMT

Email is not a messaging protocol. It’s a TODO list. Or at least my inbox is a TODO list and email is the protocol for putting stuff on it. Here’s the problem. It is a s**tty TODO list. Any one of you can put something on my …

Minimalist bookmarks

xenodium.com @alvaro — Fri, 28 Sep 2018 03:00:00 GMT

Creating a grid based on Geohashes

Tafkas Blog — Fri, 28 Sep 2018 03:00:00 GMT

When dealing with geospatial data it is sometimes useful to have a grid at hand that represents the given data. One way to create a grid like this is to use Geohashes. GeoHashes are a hierarchical spatial data structure which subdivides space into buckets of grid shape, which is one of the many applications of what is known as a Z-order curve, and generally space-filling curves. A Geohash is an encoded character string that is computed from geographic coordinates.

Stanley Glacier

Dustin.Lammiman — Thu, 27 Sep 2018 19:46:10 GMT

Length: 8.4km (Round Trip)
Elevation Gain: 365m
Time: 3-4 Hours
Trailhead: Stanley Glacier Parking Lot on Highway 93 (map). Note: the parking lot fills up fast. Go early, or you may need to park along the side of the highway.
Type: Out & Back
Rating: Moderate

Some hikes are all about the beautiful view at the destination, but with Stanley Glacier you enjoy beautiful and changing scenery for the duration of the hike. You will climb towards the toe of Stanley Glacier as you travel alongside a waterfall and the burned but beautiful remnants of a forest fire.

stop and remove all docker containers

Yasha Solutions — Thu, 27 Sep 2018 14:59:33 GMT

Simple. You can make an alias and it to your bash profile :

Lettura di fatture elettroniche con firma digitale in .NET (.p7m)

Nicola Iarocci — Thu, 27 Sep 2018 12:23:25 GMT

Sorry folks. Because of its target audience, this post is in Italian.

Ho appena rilasciato FatturaElettronica.Extensions. Si tratta di un package che estende FatturaElettronica.NET aggiungendo (per ora) un solo extension method: ReadXmlSigned

Il metodo si affianca all’esistente ReadXml ed accetta un file in formato standard fattura elettronica già firmato digitalmente (estensione .p7m), lo legge, verifica che le firme siano valide, quindi lo carica in un oggetto FatturaElettronica che lo rappresenta interamente:

Gautrain API reverse engineering

Negative Feedback — Wed, 26 Sep 2018 16:19:18 GMT

Gautrain API reverse engineering

The iOS 12 update contained a very cool feature: Siri Shortcuts. This replaces the Workflows app. I was using Workflows to send my wife an SMS with my estimated time of arrival at home when I left work. With the new Shortcuts app I could have this all happen automatically, and it also supports reading from Web APIs.
The first part, however, is to figure out how to get data from the Gautrain website.

mitmproxy

mitmproxy or Man In The Middle Proxy is a piece of software you can run on your computer which will stand between a device and the internet and allow you to inspect what is going on. I might cover installation in another post, but basically I just followed instructions to get it going and then fired up the Gautrain app to see what it was doing.

Sequence of events with Gautrain App

First endpoint to check service is live

https://api.gautrain.co.za/user-service/api/0/mobile/isLive/1.0.0

Returns true or false.
The app then hits

https://api.gautrain.co.za/transport-api/api/0/agencies

And gets a whole list of agencies, including this one:

[
....
{
        "culture": "en",
        "description": null,
        "href": "https://platform.whereismytransport.com/api/agencies/edObkk6o-0WN3tNZBLqKPg",
        "id": "edObkk6o-0WN3tNZBLqKPg",
        "name": "Gautrain"
},
...
]

The important part here is "name": "Gautrain" and that id which is used later.
Also interesting is that they are clearly using whereismytransport.com, which is the first time I've heard of this service.
The app then asks ask for a list of stops:

https://api.gautrain.co.za/transport-api/api/0/stops/gautrain

The response is a list of stops like this one:

[
...
{
    "agency": {
        "culture": "en",
        "description": null,
        "href": "https://platform.whereismytransport.com/api/agencies/edObkk6o-0WN3tNZBLqKPg",
        "id": "edObkk6o-0WN3tNZBLqKPg",
        "name": "Gautrain"
    },
    "code": null,
    "geometry": {
        "coordinates": [
            28.23794,
            -25.74762
        ],
        "type": "Point"
    },
    "href": "https://platform.whereismytransport.com/api/stops/_rkqSHvRE0Scvbcsuy0EVw",
    "id": "_rkqSHvRE0Scvbcsuy0EVw",
    "modes": [
        "Rail",
        "Bus"
    ],
    "name": "Hatfield"
},
...
    {
    "agency": {
        "culture": "en",
        "description": null,
        "href": "https://platform.whereismytransport.com/api/agencies/edObkk6o-0WN3tNZBLqKPg",
        "id": "edObkk6o-0WN3tNZBLqKPg",
        "name": "Gautrain"
    },
    "code": null,
    "geometry": {
        "coordinates": [
            28.05693,
            -26.10858
        ],
        "type": "Point"
    },
    "href": "https://platform.whereismytransport.com/api/stops/jXU-OlvxukW8wfc7JeVeXw",
    "id": "jXU-OlvxukW8wfc7JeVeXw",
    "modes": [
        "Rail",
        "Bus"
    ],
    "name": "Sandton"
}
...
]

I've kept the important ones for me - I'm going from Hatfield to Sandton and back, so I need those coordinates.
Right, so the next thing they hit is

https://api.gautrain.co.za/transport-api/api/0/journey/create

This is the first request which has parameters. You can see that they are using the coordinates for Hatfield station and Sandton station as the start and end points.

{
    "geometry": {
        "coordinates": [
            [
                28.23794,
                -25.74762
            ],
            [
                28.05693,
                -26.10858
            ]
        ],
        "type": "MultiPoint"
    },
    "maxItineraries": 5,
    "omit": {
        "agencies": [
            "A1JHSPIg_kWV5XRHIepCLw",
            "CA3o3RuuGUmoDKfyAPNTsg",
            "NfBxKfzMA0exbwToc-7o2g",
            "XxDwxnin_Eu_T7_wBJIJRA",
            "Hwe1673sC0yuT6fyANnDZQ",
            "DcXaTl-5hkGRHKenAIi_5w",
            "WaasqFZwEEa5VqbIAJKsJQ",
            "wZfSY3o0LUGHiqeoAQDeIQ",
            "W7A63uTwIECgQvU9MxcCIw",
            "Ka2d4V1g3E65VqheANhvVw",
            "FTGTH38Tm0C5O6gAAQifSQ"
        ],
        "modes": []
    },
    "only": {
        "agencies": [
            "edObkk6o-0WN3tNZBLqKPg"
        ],
        "modes": []
    },
    "profile": "ClosestToTime",
    "time": null,
    "timeType": "DepartAfter"
}

So that list of agencies to omit is obviously from the original request. I'm hoping I don't need that. For my purposes I just need to know when the next train will arrive and how long it will take to get to the destination.
The response from this request is quite huge, but the first bit shows the important parts:

{
    "agencies": null,
    "earliestDepartureTime": null,
    "geometry": {
        "coordinates": [
            [
                28.23794,
                -25.74762
            ],
            [
                28.05693,
                -26.10858
            ]
        ],
        "type": "MultiPoint"
    },
    "href": "https://platform.whereismytransport.com/api/journeys/ZPtEXRtgeEq1mqljAE4new",
    "id": "ZPtEXRtgeEq1mqljAE4new",
    "itineraries": [
        {
            "arrivalTime": "2018-09-22T05:29:52Z",
            "departureTime": "2018-09-22T04:56:20Z",
            "distance": {
                "unit": "m",
                "value": 51662
            },
            "duration": 2012,
    ...
    Lots more output

So when is the next train?

I'm only really interested in when the next train will arrive at Sandton and I don't want to type all that stuff into a mobile app, I want a minimal version.

Further investigation yields the simplified request:

{"geometry":
 {"coordinates":
  [[28.23794,-25.74762],
  [28.05693,-26.10858]],
  "type":"MultiPoint"},
  "profile":"ClosestToTime",
  "maxItineraries"5,
  "timeType":"DepartAfter",
  "time":null,
  "only":{"agencies":["edObkk6o-0WN3tNZBLqKPg"],
  "modes":[]}}

This is something I can put into the automation app eally easily. More info on that to follow.
For posterity, this is that call being checked via curl:

curl -d '{"geometry":{"coordinates":[[28.23794,-25.74762],[28.05693,-26.10858]],"type":"MultiPoint"},"profile":"ClosestToTime","maxItineraries":5,"timeType":"DepartAfter","only":{"agencies":["edObkk6o-0WN3tNZBLqKPg"]}}' -H "Content-Type: application/json" -X POST https://api.gautrain.co.za/transport-api/api/0/journey/create

Guess it's also time to create an account at whereismytransport.com!

Sign Into Gmail Without Signing Into Google Chrome

MattCrampton.com — Wed, 26 Sep 2018 08:00:00 GMT

Sign Into Gmail Without Signing Into Google Chrome Unfortunately, Google has made changes to Chrome since this blog post was posted which removed the options noted below. Sorry :( I'll update the blog post if I find another way. Recently, Google released an update to Google Chrome that effectively forces you to sign into the Google Chrome browser when you sign in to your Gmail account. If you're like me and either use multiple google accounts for different purposes or just dont like Google looking all your browsing history against your gmail account, this update is problematic. And I'm not alone in thinking so, see a few recent articles complaining about the new update... Google to give Chrome users an opt-out to ‘forced login’ after privacy backlash Why experts are freaking out over the new way Google Chrome sign-in works Google’s Forced Sign-in to Chrome Raises Privacy Red Flags To de-couple the login of your Gmail account and your Google Chrome login status requires digging into ...

Networking Essentials: DNS

swyx's site RSS Feed — Wed, 26 Sep 2018 07:12:12 GMT

Domain names, you buy them, you sell them, but how do they work?

Networking Essentials: Router Design

swyx's site RSS Feed — Wed, 26 Sep 2018 06:33:05 GMT

A small detour into Router Hardware!

It's 2018 already!

Erico Notes — Wed, 26 Sep 2018 05:43:44 GMT

So 2016 passed, and then 2017 and it's now 2018. Life just goes faster the older you get... Time for some updates.

update time!

So I decided to give a pause on A Glass of Lores - starting out building a full blown RPG and engine as my first project maybe wasn't very smart. The story keeps refining slowly on my Google docs... But, cool thing. I started to make a new game! And I am NOT making an engine this time.

This game is Future Flashback and I've been screaming a lot about it on social networks, so if you never heard about it, please click through the website, there is a lot of material there.

I usually throw a single thing I made or some instruction here, but instead I will just flow through some things...

book I've read: Driving Results through Social Networks by Robert J Thomas

It states the need to align the culture of an organisation with its business strategy, and the importance of finding the influencers in the company social network. It also presents the hypothesis that innovation usually arrives from teams instead of a single person, so to favour innovation you have to have more teams, and offer the idea that if people in the organisation can have more connections, you have a bigger flow of ideas that are possible to come to fruition - you need a multitude of disciplines to generate good profitable innovation.

Also the network will have people working against innovation, which is important to take note. Also a risk is presented when a single person gets too many connections but accepts all incoming demands, becoming a bottleneck in the network. One interesting passage attribute the managers the work of exception-handlers, and so the less exceptions and more routines the organisation encounters, less managers will be necessary.

Creepy writing

Some years ago, I've read Writing Ethnographic Fieldnotes , which talks about writing fieldnotes from observed behaviours, experiences and interactions with people and between people. I found this a good hobby for when I am alone travelling, eating and only have my phone, so I take the opportunity to write about some relation I see around me. My Google Docs file where I write this is called Everyday Scenes. Here is an excerpt:

The woman waits patiently. Sit shrunked in her chair, she swipes throgh group conversations in her phone. The man she was waiting appears, and sits besides here. She doesn't realize he is there until he touches her neck. She breaks from her phone. From their gestures, they appear late to the cinema, and they rush on the movies direction. They looked very happy.

Coding things

Future Flashback is made in Adventure Game Studio, so lots of recently things have been AGS related. You can skim through either my github profile or my brand new portfolio page. That page is made using Jekyll, I used a prebuilt theme and just customised some details, which is why I was able to create it in a weekend.

I want to write more here! Hopefully before 2019!

Why (and how) to read books

Notes on software development — Wed, 26 Sep 2018 03:00:00 GMT

The last time I read for fun was in elementary school. Since college I knew I must read more, but I never forced myself to build the habit. Then three years ago I spent time around my brother and a coworker who were avid readers. This "peer pressure" helped me get started.

Since I started, I've seen concrete improvements in vocabulary. I find myself using words I didn't know I knew. I question my choice of words more. And I understand coworkers a little better. Perhaps it is only personal style, but I've also become more aware of hyperbole in my speech and have begun to tone that down.

Specifically, books provide more density of information than I can pull together myself. I've also benefited heavily from reading books on tools I use daily. Contrary to being boring, a book on a topic with which I'm familiar has been a (often needed) break from books on topics with which I am unfamiliar. The former category might include books on CSS, Bash, Emacs, Python, Scheme, data modeling, Linux/FreeBSD system administration, mystery novels, and so on. The latter category might include books on Common Lisp, system architecture, the implementation of Linux/FreeBSD, behavioral psychology, management, stock/bond markets, the history of Argentina/Chile/South Korea/Japan, sci-fi novels, and so on.

Reading diversely exposes how little I know. And that can be depressing. But I'm fairly confident reading books is the fastest way to grow.

Tactically speaking, I started slowly with few books and the ones easiest for me to read. The first year I read two books, both technical. The second year I read nine books and was able to start branching out beyond technical books. Last year I read a much more diverse set of forty books. And this year I followed suit with forty-one so far (on track for fifty-five or so).

I keep track of books I'm reading and books I want to read in Goodreads. I particularly enjoy their reading challenge system that lets you know if you are on track to meet your reading goal for the year.

Kernel Recipes 2018 liveblog

Linux Engineer's random thoughts — Wed, 26 Sep 2018 01:00:00 GMT

I had a lot of good feedback from the previous years live-blogs of the Embedded and Kernel Recipes conferences. So much in fact, that this year I'm now doing the liveblog directly on the official website of kernel recipes.

I hope you enjoy it !

I also live-tweeted my impressions of …

Warehouse locations with k-means

Bytepawn - Marton Trencseni — Wed, 26 Sep 2018 01:00:00 GMT

Sometimes, the seven gods of data science, Pascal, Gauss, Bayes, Poisson, Markov, Shannon and Fisher, all wake up in a good mood, and things just work out. Recently we had such an occurence at Fetchr, when the Operational Excellence team posed the following question: if we could pick our Saudi warehouse locations, where would be put them? What is the ideal number of warehouses, and, what does ideal even mean? Also, what should our “delivery radius” be?

The Swiss Far West and GPX fun

Heitor's log — Tue, 25 Sep 2018 03:00:00 GMT

GPX fun in Python.

Summer Haute Route - Part 8: Trient and back to Champex

Wilderness Prime — Tue, 25 Sep 2018 03:00:00 GMT

Our hike concludes, back at Champex.

The trail wound on, up into the hills once again for the last leg. To Trient, then down into the valley where we returned to the beautiful Lac Champex. This ends a magnificent 21 day hike from Zermatt to Chamonix and all the way round Mont Blanc. I've seen beautiful mountains all over the world, but nothing beats the European Alps for charm.

Networking Essentials: Naming, Addressing, and Forwarding

swyx's site RSS Feed — Sun, 23 Sep 2018 18:39:34 GMT

How IP Addressing evolved over time, how it works today, and how we can transition to IPv6 in future

Spring in the backcountry

Michael Honey: Writing - michaelhoney.com — Sun, 23 Sep 2018 11:13:11 GMT

Recover from Time Machine's "backup already in use"

xenodium.com @alvaro — Sun, 23 Sep 2018 03:00:00 GMT

Started seeing "backup already in use" error from my daily Time Machine backups, against my Synology. Disabling and re-enabling AFP did the job (via Synology -> Control Panel -> Files Services -> Enable AFP service).

The Listening Room, or habits to unlearn

Avy Faingezicht — Sun, 23 Sep 2018 03:00:00 GMT

I’m not sure how I found Magritte in the first place, but as a teenager, his _Son of Man_ became my avatar on every online platform where I spent afternoons and evenings during the mid-aughts. I pondered endlessly on his playful compositions, full of contradictions, false premises, and recursive repetitive themes. The motifs in Magritte’s art — paintings within paintings, windows confounding what was inside with what was outside, ambiguous and impossible light sources — invite us to question the automatic assumptions we make about what is in front of our eyes, the nature of reality, and what the objects we interact with on our daily lives actually represent. Images are treacherous, etc, etc. This weekend I had a chance to think about these things once again (with fresh eyes, and with a few more years of experiences behind me) at SFMOMA’s "_Magritte: The Fifth Season_." While I was familiar with many of the pieces presented in The Fifth Season, a pair of images I did not remember seeing before hit close to home. _The Listening Room_ (1952) and its remake bearing the same name, _The Listening Room_ (1958), were displayed in a room together with the better known _Personal Values_. In these three pieces, the viewer is presented with incongruously large everyday objects: a wine glass the size of an armoire, a comb larger than the bed it sits on, apples that take up whole rooms. These are surreal images, which evoke an eerie, frightening effect the longer you look at them. The artist asks us to question whether what we see is really there, and, perhaps most importantly, makes us wonder what’s not shown, what’s hiding behind these disproportionate objects. Not to be too literal, but my brain quickly locked onto the engorged apples of _The Listening Room_ as representations of Apple and the growing effect my job there has had on who I am, how others see me, and how I see myself. You might have heard me say before that I landed at Apple by chance - that I was trying to go down the startup route, and instead ended up joining one of the largest corporations in the world. Opportunity showed itself, and it seemed too good to pass up. For better and for worse, the decision has defined me and will continue to do so whenever I decide it’s time for whatever comes next. In organizational behavior theory, it is well understood that a company’s culture, enshrined in its processes, its jargon, and its incentive structures, is ultimately embodied in its people. In a recent Twitter thread Keith Rabois argued that the cachet of a company’s founder being an ex-Googler has changed over time, and how people who land PM roles there learn the wrong lessons for when they quit to start their own companies. The stemming conversations made me wonder about the lessons I’ve learned during my tenure at Apple, which, crazily, is reaching three and a half years now. In my case, by learning to navigate Apple's corporate life, I’ve picked up its habits and its way of approaching problems. I’ve even warmed up to its siloed functional structure and was surprised when I found its DRI model is not shared in other organizations. Expanding that circle to not just concepts I've incorporated from Apple, but to Silicon Valley's tech culture at large, I am continuously amazed by all the ways in which the technology we build is broken, an idea diametrically opposed to the techno-optimism that surrounds me. This thought takes an outsized amount of space in my head, and I come back to it often. I’m convinced that it’s only due to the miracle of emergent behavior that anything works, and that any company can build something that, at least temporarily, works. If engineers at Apple, Google, FB, etc, with all that supporting infrastructure, have so much trouble making things work, how are people who don’t have these tools at hand able to solve problems? Obviously this is hyperbolic, and there are uncountable ways in which being constrained, and in which not having said tools is an advantage, but the longer I spend at Apple, the more this feeling of learned helplessness takes over me. That makes me worry. I have habits to unlearn. Seeing how the beast works from inside and absorbing the lessons of how it cracks its challenges continues to be fascinating. I’m sure some of the practices that led to its success have rubbed off on me and will come in handy in the future. However, I am afraid that, much like Magritte tricks us with _The Listening Room’s_ apples, I've tricked myself into thinking that these ideas (the Apple way of solving problems, my opposition to Silicon Valley's all-encompassing techno-optimism, etc) take up room in my identity and the stories I tell about myself. My most direct environment has taken up the room that should be inhabited by other ideas - ideas that I'll need when I start building something of my own - so I'm trying to be more thoughtful about what ideas to keep, and what to discard. Unlearning things is hard, though. _The Listening Room_ is not about the apple, but about the room it has overtaken. This post is not about what's next, but about the realization that if we’re not deliberate about what habits we pick up, and what ideas we celebrate, we end up giving inordinate amounts of attention to the wrong abstractions. Rooms become useless when overtaken by objects - any object. Here’s to unlearning things.

Lessons from launching on Product Hunt

ferrucc.io — Sat, 22 Sep 2018 22:18:03 GMT

Two months ago I hadn’t launched any solo project that got users.

Until lately everything I had ever done was with someone else, usually with a compatible skillset. The problem with this approach is that if someone on the team loses interest or is busy the project dies out.

As this happened a couple of times I decided to take some time and work on projects by myself.

The products I launched

Klipped.in (https://klipped.in)

Illegal streams, decrypting m3u8's, and building a better stream experience

JonLuca's Blog — Sat, 22 Sep 2018 18:14:32 GMT

A first stab at gene expression

Liza Shulyayeva — Sat, 22 Sep 2018 17:00:47 GMT

In the last post I talked a bit about how I might approach expressing genes in SnailLife Go, and how this worked in the PHP prototype of the simulation.

How to reference hidden elements in Vue.js 2

Posts by Petr Stribny — Sat, 22 Sep 2018 16:58:25 GMT

Moon Jar

Matt Keeter — Sat, 22 Sep 2018 03:00:00 GMT

A glowing, charming gift

Summer Haute Route - Part 7: Les Houches, Chamonix

Wilderness Prime — Sat, 22 Sep 2018 03:00:00 GMT

We continue through Les Houches, Chamonix and on to Argentiere.

We can see Les Houches and Chamonix in the distance for the long descent from the mountains. We had to make up a day so we bypassed the normal hiking trail along the Petit Balcon Sud, and walked through the centre of Chamonix. I've lived here for four winters but this was my first time visiting in the summer.

Firefox imgur failure

Missives — Fri, 21 Sep 2018 22:20:20 GMT

My imgur had been acting up for a while now. In my Firefox it wouldn’t render the actual image but rendered most of the rest of the page just fine: If I went to i.imgur.com (adding the .png or .jpg to the url) then the image loaded just fine. So it wasn’t getting blocked by … Continue reading

Laptop Stand

june.kim — Fri, 21 Sep 2018 03:00:00 GMT

Summer Haute Route - Part 6: Refuge Des Mottets, Les Contamines

Wilderness Prime — Fri, 21 Sep 2018 03:00:00 GMT

Over the Col de la Seigne and into France towards Les Contamines.

The valley to the west of Courmayeur gets plusher and wetter the further we go. Lush green marsh land and cattle grazing.

We leave the bog behind and head up the mountain to the Col de la Seigne, and to the France border.

It's a long and steep descent to the stoutly constructed Refuge de la Mottets. One of the more comfortable mountain huts we've visited.

The next day we're hiking into easier terrain, and we soon reach the village of Les Contamines-Montjoie.

Postgres 11 - A First Look

CRAIG KERSTIENS — Thu, 20 Sep 2018 23:55:56 GMT

Postgres 11 is almost here, in fact the latest beta shipped today, and it features a lot of exciting improvements. If you want to get the full list of features it is definitely worth checking out the release notes, but for those who don’t read the release notes I put together a run down of some what I consider the highlight features.

Quitting Postgres

This is a small usability feature, but so long over due. Now you can quit Postgres by simply typing quit or exit. Previously you had to use Ctrl + D or \q. As a begginer it’s one thing to jump into a psql terminal, but once in if you can’t figure out how to quit it’s a frustrating experience. Small usability features, such as this and watch in an earlier release, are often lost in the highlighted features which talk about performance or new data types. Improvements like this really go a long way for making Postgres a better database for everyone.

Fear column addition no more

Brandur had a great in depth write-up on this feature already, but it falls somewhere into the category of the above as well as a performance improvement. Previously when you added a new column that was NOT NULL with a default value Postgres would have to take a lock and re-write the entire table. In a production environment on any sizable table for all practical purposes the result was an outage. The work around was to break your migrations apart to be a several step process.

With Postgres 11 you can add a new column to a table that is not null with a default value. The new row will get materialized on your database without requiring a full re-write. Here is to having to think less about your migrations.

Of course performance is a highlight

No Postgres release would be complete without some performance improvements. This release there are really two areas that feature key improvements around performance.

Parallelism continuing to mature

We first saw parallelism support back in PostgreSQL 9.6. At the time it was primarily for sequential scans, which if you used parallelism for your sequential scans was great, but overall that was a narrow focus. PostgreSQL 10 parallelism because much more useful, and with PostgreSQL 11 it just keeps getting better. Some of the highlights for parallelism include:

Parallel hash joins
Parallel append
Parallel index creation - We’ve talked about how great this can be over on the Citus blog. With it natively in Postgres it makes it even easier for people to leverage.

If you want to dig deeper into all the parallelism support in Postgres this presentation by PostgreSQL committer Thomas Munro at PostgresOpen Silicon Valley from a few weeks ago is a great resource.

Postgres gets a JIT

Just in time compilation is going to be a big deal for Postgres for the coming years. We have the initial support for it now in PostgreSQL 11. Even in this initial implementation of JIT support you can see a nearly 30% speedup on certain queries, such as highlighted here by the TPC-H benchmark

It is still early days for the just in time query compilation support, so expect the improvements here to be even better in PostgreSQL 12 and 13.

Statistics keep getting better

In Postgres 10 we saw a feature that few have probably used CREATE STATISTICS. You see under the covers Postgres keeps a lot of information about your database which it uses to determine the query plan it will use when executing a query. Most statistics were single column ones previously, now with CREATE STATISTICS you could define a correlation between two separate columns. With Postgres 11 now you can create statistics based on expression indexes giving you even more cases where they can help the performance of your app.

Keeping standbys warm

pg_prewarm has been great for warming up a replica’s cache so that should you have a failover you’re not failing over to a cold cache. However up until PostgreSQL 11 you’d have to manually run it yourself or setup some scheduler such as pg_cron, now you can configure pg_prewarm to run all on it’s own at a regular interval.

And more

PostgreSQL 11 is packed with more features than I’ve seen in a release before, though I think I’ve also said that before. It will be exciting to see several of these features such as the JIT support, statistics, and others as it is still in the early days for them. Meanwhile we have a great set of new features to improve user experience as well as help performance with parallelism. If you’re curious to get your hands on these give the beta a try and send your feedback to the PostgreSQL community.

Ten ways to improve any engineering class [Video]

David Schlachter — Thu, 20 Sep 2018 20:00:00 GMT

Ten easily-applied ideas taken from the literature on teaching in engineering. Presentation won second place in the Reg Friesen competition at the 68th Canadian Chemical Engineering Conference.

The embarrassing absurdity of Article 11, explained

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 19 Sep 2018 09:00:00 GMT

Article 11 of the recently approved EU Copyright Directive is going to be fun. Eventually.

Color Notations

John Jacobsen — Wed, 19 Sep 2018 03:00:00 GMT

I have been experimenting with notations for color mixtures, so I can make notes about colors on the go without actually carrying a whole set of oil paints everywhere.

Here's an example from a very quick sketch during tonight's commute home:

Though it may look black and white, there are a dozen colors captured in this quick sketch. In this post, I briefly describe the notation.

The letters are based on my oil painting palette of late: usually,

name	provider	pigment number	code
cremnitz white	RGH	PW 1	w
nickel (titanium) yellow	Williamsburg	PY 53	n
winsor (primary) yellow	Holbein	PY 74	y
raw sienna	Gamblin	PBr 7	s
transparent earth yellow	Gamblin	PY 42	t
venetian red	Gamblin	PR 101	v
cadmium red light	Gamblin	PR 108	c
pyrrole red	Holbein	PR 254	p
brown pink	Williamsburg	PR 101	-
spanish earth	Williamsburg	PR 102	a
quinacradone violet	Gamblin	PV 19	-
dioxazine purple	Gamblin	PV 23	-
cobalt blue	Gamblin	PB 28	o
(real) manganese blue	Vasari	PB 33	m
prussian blue	Gamblin	PB 27	r
ivory black	Williamsburg	PBk 9	b
viridian	Gamblin	PG 18	i
terre verte	Gamblin	PY 43, PG 18, PBk 9	e
raw umber	Williamsburg	PBr 7	u

Since there are fewer than two dozen colors, a single letter can be assigned to each pigment (the rightmost column shows my assignations as of this morning). I'd like to whittle the number of colors down even further, but I'm hooked on the various properties of each of these. Almost all of them are pure pigments… the only mixture is terre verte, which I love for its transparency, mass tone and utility in mixing cool midtones.

Occasionally I'll throw in another pigment for special occasions, or start out with just a few for underpaintings, but usually I lay most of these out for any given painting session.

The notation, then, is very similar to that used for writing chemical compositions, which you might remember from high school chemistry: $\mathrm{H_2O}$ is water, and $\mathrm{C_{18}H_{32}O_2}$ is linoleic acid, one of the components of linseed oil.

For paint mixtures, we apply a similar rule: one letter per "element" (paint color/pigment), with an optional numerical suffix to indicate a correspondingly higher proportion (for simplicity we eliminate the subscripting of the numbers).

For example, king's blue, which is typically white plus cobalt blue, is, in my system,

w2o

… i.e., roughly two parts Cremnitz (lead) white per part of cobalt blue.

Not every color I might use is assigned a letter. In this case, just write the pigment code in parentheses. For example, for an exact match for commercial king's blue, I would use titanium white (PW-6) instead of the more transparent lead white; in this case, king's blue is

(PW6)2o

Now let's talk through the sketch at the top, location by location. The head is mostly backlit (contre-jour) with light from the window and some interior light inside the train.

Hair on top of head, b4uw: four parts black, plus raw umber and white.
Fairly flat area of color on forehead, eta: terre verte plus transparent earth yellow plus Spanish earth.
Left upper highlight, w4(PV19) - mostly white with a bit of quinacradone violet.
Forward/top plane of nose, w(PV19)s: white, quinacradone violet, raw sienna.

For the ear, we use a variant of the notation which has light, mid tone, and dark values, stacked (in parentheses):

w(PV19)

… which means,

Light: venetian red + white
Mid tone: pyrrole red + nickel titanium yellow
Dark: quinacradone violet + white

For the lips I could see, roughly, light and dark values:

w2(PV19)

w1a1(PV19)1 [the 1's are superfluous]

i.e., two portions white per quinacradone violet for light, and white, Spanish earth and quinacradone violet for the dark.

Are you serious about all this?

This may seem like a lot of work, or hard to remember. The notation is highly personal, though – your palette and your notation will be different, and you will remember yours better than mine. I do find it interesting that in just a few minutes I was able to make a study recording a dozen color decisions with just paper and pencil. One could work up a simple portrait on this basis alone (especially if the sketch were a little more complete).

This will be hard to do unless you've painted enough with your palette to have some knowledge about how your colors behave when mixed with each other. It is a good way to practice thinking about color when out in the world and away from the easel. Like preparing for drawing from memory, it's something you can do just about anywhere.

This is a work in progress: since I used quinacradone violet multiple times in my sketch, I am hereby officially dedicating the letter q to that pigment. Maybe Williamsburg brown pink can be k (originally reserved for my chalk putty medium, which maybe I don't need a letter for). Dioxazine purple can be d. As my palette evolves (as they tend to), I expect to swap around my letter assignations … and will have to remember to write down a key now and then so I can still "decode" my notations in years to come.

Sloc Cloc and Code Revisited - Optimizing an already fast Go application

Ben E. C. Boyter — Wed, 19 Sep 2018 03:00:00 GMT

Update 2019-03-13

This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links.

I don’t want to make any false claims about the impact of scc and the blog post about it https://boyter.org/posts/sloc-cloc-code/ but following its release both tokei and loc were updated with impressive performance improvements. In addition a new tool polyglot http://blog.vmchale.com/article/polyglot-comparisons was released which also claimed performance as its main feature. Lastly the tool gocloc https://github.com/hhatto/gocloc appears to be getting updates as well. All good stuff.

The People v. O.J. Simpson: American Crime Story: The People v. O.J. Simpson

🦉 olshansky 🦁 — Wed, 19 Sep 2018 02:25:51 GMT

Olshansky's review of The People v. O.J. Simpson: American Crime Story: The People v. O.J. Simpson

Recovering data off a bootlooping Nexus 5

Carey Metcalfe — Wed, 19 Sep 2018 01:30:00 GMT

The situation

The Nexus 5 that I've had for almost 5 years started bootlooping repeatedly.

I've replaced/repaired multiple parts of it over the years but lately it had been more flaky than usual. For example:

Sometimes when booting it would show a "Firmware update in progress, don't disconnect your computer" message, even though there was nothing connected to the USB port.
It would randomly freeze and soft reboot when using it about once a week.
When propping it up on a book or something to angle the screen towards me, the screen would sometimes turn off and on (as it turns out, this was a warning sign)

I've been ready to replace it for a while, but couldn't justify it while this one was perfectly (ok, mostly) functional. That changed today when I tried to turn it on and it started bootlooping.

So buy a new phone and move on, right? Right, except I committed the cardinal sin of living in the digital age - not having complete backups. Yeah, yeah, I know.

I got most of it. I'd been using Syncthing to automatically back up photos, videos, and other files to my home server every night so they were all good. The problem was that I had been using FreeOTP-export to back up my 2FA secrets manually and the latest backup was missing a bunch of logins.

So I needed to somehow get into my non-booting phone and pull the data off it. At least because I was replacing it, all destructive options were on the table.

Diagnosis

The first step was seeing if other people had this problem. Some searching revealed that it was most likely the power button gone bad and being stuck in the pressed position.

To confirm this, I rapidly hammered the power button on a hard surface while booting, hoping that the jarring would un-stick it and let it boot. This sort of worked in that it would boot, but stopping at any point would result it it powering off again. It was also extremely difficult to tap it consistently enough to keep the phone on for long enough to pull any data off it.

The issue I mentioned earlier where propping the phone up made the screen turn on and off now made sense. I'm guessing this was causing a slight flexing of the power button switch, causing it to close and trigger. Over time this must've permanently bent something so it stayed closed, causing the bootloop.

Now that I had pretty much confirmed that it was something physically wrong with the power button circuitry, I messaged my friend Mike. When it comes to electronics, he's definitely more experienced than me. Plus, he lets me borrow his tools :)

The "fix"

The next day Mike came over with his soldering iron and we started brainstorming.

The plan of attack was to just take the power button off the board entirely. It can't always be pressed down if it's not on the board right?

In preparation for this, I wanted the phone to automatically boot when it was plugged into a charger so I wouldn't have to manually short the pins on the board. Fortunately this can be done with a fastboot command: fastboot oem off-mode-charge 0 (basically: don't allow charging while off, therefore turn on when charging).

After desoldering the power button from the main board, it was still bootlooping. Not good. I went back and cleaned up the solder to make absolutely sure that the power button signal pins weren't connected in any way. Still bootlooping when powered on. It would get to the bootloader and stay there, meaning the power button wasn't being pressed anymore, but when launching to either recovery or the main OS, it would immediately restart.

Looked into dumping data from the bootloader - impossible.

Reflashed the boot and recovery partitions - same thing.

Tried booting into recovery using fastboot boot <recovery-image> - bootloop.

sounds like it's time to cry

-- Someone on Freenode's #lineageos-dev channel after I explained the situation

Thankfully, it was not. Right after that, someone else asked if I had reconnected the battery. I had not.

Turns out that a working battery is required to boot the phone, even when it's plugged in. I had taken the battery out to desolder the power button and never put it back in since the phone seemed to boot fine without it. Whoops.

After plugging the battery into the board and booting the phone, it launched the main OS without any issues. Huge relief.

Recovering the data

With the phone booted up normally I used adb (running as root) to pull the 2FA codes and other important things that I knew I needed over to my laptop.

I really don't trust myself to remember everything so I booted the phone into recovery mode and pulled a complete backup of /data/data, /data/app, and /storage/sdcard (not actually an sdcard) to my laptop as well.

To make absolutely sure I didn't miss anything, I also pulled a raw image of the entire flash memory (disk-based encryption was not enabled) using adb pull /dev/block/mmcblk0 mmcblk0.img. Everything I need should be in the normal backups, but if not, I can always go spelunking through that image.

Lessons learned

If it's important, back it up automatically. If it's not automatic, at some point it will be missed.
Make sure you have the ability to get root access on every device you own before you need it - in this case I wouldn't've been able to pull the 2FA codes or do full backups without it.

Future plans

For the Nexus 5, I'm planning on either buying a replacement power button or maybe just soldering some wires to the exposed pads and snaking them out of the phone to an external button if I can't find one. The phone can then continue to be used as an app development testbed, a Chromecast remote, or as a basic emulation console until it dies for real. In theory I could keep using it as a phone too, but at this point I just don't trust it enough.

For my next phone, I've decided on the Xperia XZ1 Compact. It's a smaller phone that's mostly waterproof (IP68), has a fast SoC (Snapdragon 835), a headphone jack, SD card support, and great battery life. I have high hopes for it. Also, as evidenced by this post, having root access to the OS is pretty critical at times so I'll be flashing LineageOS on it ASAP.

Introduction to Locality-Sensitive Hashing

tylerneylon.com — Tue, 18 Sep 2018 10:00:00 GMT

Locality-sensitive hashing (LSH) is a set of techniques that dramatically speed up search-for-neighbors or near-duplication detection on data. These techniques can be used, for example, to filter out duplicates of scraped web pages at an impressive speed, or to perform near-constant-time lookups of nearby points from a geospatial data set.

Authentication in ASP.NET Core Web API with Amazon Cognito

Serge Nevsky — Tue, 18 Sep 2018 03:00:00 GMT

Amazon Cognito is the user management and authentication product in AWS. It allows for unified sign-up and sign-in flows across web and mobile apps.

Time

taylor.town — Tue, 18 Sep 2018 03:00:00 GMT

I think we lost something when we moved from analog to digital clocks.

Summer Haute Route - Part 5: La Fouly, Courmayeur

Wilderness Prime — Tue, 18 Sep 2018 03:00:00 GMT

Over the pass to Italy and down the valley to Courmayeur.

Onwards and upwards into the foothills of Mont Blanc from Lac Champex towards Italy. La Fouly is a cute little town with an amazing view, and I'd highly recommend staying at the excellent Maya Joie hostel.

As we climbed further into the hills, the buildings became less frequent and the landscape more barren. The view down the valley towards Courmayeur from the Swiss Itallian border was one of the best of the trip.

Refugio Bonetti is always a pleasure to stay at, winter or summer and this time it didn't disappoint. The Mont Blanc massive growing larger as we make our way down the valley.

What is school for? (And other unconventional opinions on school)

Yasha Solutions — Mon, 17 Sep 2018 19:36:59 GMT

I am not bashing school – but I think we need more conversations on this topic. Here are a couple of videos which I think raise some interesting points to think about. Don’t Stay in School A rap video that went viral a few years ago, from And the follow up video, addressing comments:What is school for? A more recent video from Prince EA Teachers React to original video on school from Prince EA

Automating MySQL EXPLAIN checks

Phil Booth's Blog — Mon, 17 Sep 2018 18:52:30 GMT

About a month ago, we had an outage caused by a slow-running query in MySQL. This particular slow query wasn’t spotted when it was deployed because it depended on data inserted by client browsers and the related preference in Firefox was not enabled at that point. A few weeks after it shipped, the client pref was flipped on and as the table grew, it slowed down MySQL increasingly until the whole of Firefox Accounts became unresponsive. And of course, because Sod’s Law is one of the fundamental forces of nature, this happened late on a Friday night. There were some complicating factors that slowed down diagnosis, but it’s also fair to say we could have caught it at source with an EXPLAIN of the offending query during code review. Because of that, I decided to try and automate EXPLAIN checks for our MySQL queries.

My broad objective was to write a script that extracts individual queries from stored procedures, turns them into EXPLAIN statements and then feeds those into MySQL. The results would then be used to emit warnings and fail pull requests when bad smells like filesort or a full table scan are detected.

Identifying procedure names

Our stored procedures are versioned and the old versions continue to live in the tree so that reverse migrations can be applied if necessary. The first hurdle, then, is to identify which stored procedure versions are currently active in the codebase.

This proves to be as simple as grepping for CALL statements in our JavaScript code, then using awk and cut to pull out the name of the called procedure:

grep 'CALL ' lib/db/mysql.js | awk -F 'CALL +' '{print $2}' | cut -d '(' -f 1

Similarly, the relevant source file for each procedure can be found by grepping for CREATE PROCEDURE statements. Here git grep is used because it includes the file name in the output:

git grep "CREATE PROCEDURE `*$PROCEDURE" | cut -d ':' -f 1

Extracting SELECTs

The next job is to extract queries from those stored procedures and turn them into valid EXPLAIN statements. For the first cut, I am ignoring INSERT, UPDATE and DELETE, and just focusing on SELECT instead. If that proves valuable, it should be pretty straightforward for a future pull request to transform other query types to an equivalent SELECT and build an EXPLAIN from that.

For this step I wanted to parse the SQL but none of the off-the-shelf parsers in npm are able to process our stored procedures. That’s okay though because we don’t actually need a complete parser, we just need a solution that’s good enough to extract individual queries from our codebase.

Looking at the code, the majority of our stored procedures conform to some basic assumptions:

There are never multiple queries on a single line.
CREATE PROCEDURE and its matching END; start at column 1.
Arguments to procedures are named either inXxx or xxxArg.
SQL comment delimiters never appear inside string literals.

That means we can process our SQL files line by line and use simple regular expressions to pull out individual SELECT queries:

const COMMENT = /--.+$/
const CREATE_PROCEDURE = /^CREATE PROCEDURE `?([A-Z]+_[0-9]+)/i
const END_PROCEDURE = /^END;$/i
const SELECT = /^\s*SELECT/i

function extractSelects (path, procedure) {
  let isProcedure = false, isSelect = false

  const src = fs.readFileSync(path, { encoding: 'utf8' })
  const lines = src.split('\n')

  return lines
    .reduce((selects, line) => {
      line = line.replace(COMMENT, '')

      if (isProcedure) {
        if (END_PROCEDURE.test(line)) {
          isProcedure = isSelect = false
        } else {
          if (isSelect) {
            selects[selects.length - 1] += ` ${line.trim()}`
          } else if (SELECT.test(line)) {
            selects.push(line.trim())
            isSelect = true
          }

          if (line.indexOf(';') !== -1) {
            isSelect = false
          }
        }
      } else if (procedure) {
        const match = CREATE_PROCEDURE.exec(line)

        if (match && match.length === 2 && match[1] === procedure) {
          isProcedure = true
        }
      } else {
        isProcedure = CREATE_PROCEDURE.test(line)
      }

      return selects
    }, [])
    .map(select => purgeUnbalancedParentheses(select))
}

function purgeUnbalancedParentheses (select) {
  const openingCount = select.split('(').length
  const closingCount = select.split(')').length

  if (openingCount < closingCount) {
    for (let i = 0; i < closingCount - openingCount; ++i) {
      const index = select.lastIndexOf(')')
      select = select.substr(0, index) + select.substr(index + 1)
    }
  } else if (openingCount > closingCount) {
    for (let i = 0; i < openingCount - closingCount; ++i) {
      const index = select.indexOf('(')
      select = select.substr(0, index) + select.substr(index + 1)
    }
  }

  return select
}

This is good enough to work for 90% of our queries. That’s fine because currently we automate EXPLAIN checks for 0% of our queries, so it’s still a significant improvement even though it’s not perfect.

Replacing arguments

In many cases, the code above returns a string that looks like this:

SELECT * FROM table WHERE column = columnArg;

Here columnArg is the name of an argument to the stored procedure the query comes from. Running it through MySQL outside that context will fail, because it doesn’t know what columnArg is. So for the EXPLAIN to work, we must replace the argument name with a literal value. And the literal value needs to be of the correct type in order for MySQL to perform valid analysis.

To fix this, we can rewind to the beginning of the script and add a step that inserts a bunch of known values into the database. Then we can write a function that replaces arguments by looking up the argument name in a dictionary of known values and making the appropriate replacement:

function replaceArgs (query) {
  return query
    .replace(/([ \(])`?in((?:[A-Z][A-Za-z]+)+)`?/g, replaceArg)
    .replace(/([ \(])`?([a-z]+(?:[A-Z][A-Za-z]+)*)Arg`?/g, replaceArg)
}

function replaceArg (match, delimiter, arg) {
  arg = arg.toLowerCase()

  let value = KNOWN_ARGS.has(arg) ? KNOWN_ARGS.get(arg) : ''

  if (Buffer.isBuffer(value)) {
    value = `UNHEX("${value.toString('hex')}")`
  } else if (typeof value === 'string') {
    value = `"${value}"`
  }

  return `${delimiter}${value}`
}

Interpreting results

All that’s left now is to prefix each SELECT with EXPLAIN, run it through MySQL and interpret the results. There are two columns of interest in the EXPLAIN result set, type and Extra. These can be set to many values, but I’m mostly interested in four particular bad smells:

If type is ALL, it means the query is performing a full table scan.
If type is index, it means the query is doing a full index scan.
If Extra includes using filesort, it means an expensive sort operation is involved.
If Extra includes using temporary table, it means a temporary table is being created.

With those in mind, the following function can be used to turn the EXPLAIN results into an array of warnings:

const TYPE_FULL_TABLE_SCAN = /^all$/i
const TYPE_FULL_INDEX_SCAN = /^index$/i
const EXTRA_FILESORT = /filesort/i
const EXTRA_TEMPORARY_TABLE = /temporary/i

function warn (explainRows) {
  return explainRows.reduce((warnings, row) => {
    if (TYPE_FULL_TABLE_SCAN.test(row.type)) {
      warnings.push('full table scan')
    } else if (TYPE_FULL_INDEX_SCAN.test(row.type)) {
      warnings.push('full index scan')
    }

    if (EXTRA_FILESORT.test(row.Extra)) {
      warnings.push('filesort')
    }

    if (EXTRA_TEMPORARY_TABLE.test(row.Extra)) {
      warnings.push('temporary table')
    }

    return warnings
  }, [])
}

Ignoring procedures

The last thing I want to add is a mechanism for ignoring specific stored procedures. This is so we can turn it on in CI without it failing for queries that we know can’t be parsed yet. Then, any new query that causes the script to fail will get a big red ❌ against its respective pull request.

Assuming the existence of a file called .explain-ignore, containing one procedure name per line, we can parse it into a JS Set instance like so:

function parseIgnoreFile () {
  return new Set(
    fs.readFileSync('.explain-ignore', { encoding: 'utf8' })
      .split('\n')
      .map(procedure => procedure.trim())
      .filter(procedure => !! procedure)
  )
}

Now we can filter our list of procedures and skip processing the ones that we want to ignore.

The final script

The end result of all this is a script that I merged into our database repo today. You can see the final script in the pull request. It’s also running in CI and the output looks like this:

$ npm run explain
> fxa-auth-db-mysql@1.120.0 explain /home/travis/build/mozilla/fxa-auth-db-mysql
> node scripts/explain-warn
Found 0 warnings and failed to explain 0 queries.
The command "npm run explain" exited with 0.

Wowen Wilson

The Curious Creations of Tim Holman — Mon, 17 Sep 2018 07:00:00 GMT

Perhaps the most important quiz thats ever existed. Hear the ‘wow’ and name the movie.

tealeaves gets rust and docker updates

Pete's Points — Mon, 17 Sep 2018 04:00:51 GMT

So I've been trying to do as much of my local development from within docker containers. Each project I tweak things to be a little bit nicer than the last one. Most recently I went to update my tealeaves ssh key parser utility (which is coded in rust). Initially I was getting all manner of frustrating errors trying to get a reproducible version of rust installed with 1.30, clippy, and rustfmt. I bailed one night in frustration only to later learn that I had left my Cargo.toml modified from an attempt to use a local clone of one of my dependencies, and I had never cloned that repo on my new laptop, so rustc couldn't find the files it needed. Once I fixed that silly mistake things mostly started to make more sense. I was able to get a configuration where:

I can do all my local development in the docker container
Filesystem user is the same in the container and the host, no annoying permission denied errors with root-owned files
Cargo properly caches stuff so every time you stop and start the container you don't have to redownload and rebuild the universe.

So after I had my dockerized development setup working, I went to continue my long-postponed effort to upgrade to the nom v4 crate. I was delighted to discover that in the lengthy interim, all my direct dependencies had already upgraded to nom v4. So I spent a while chasing compilers errors and making the necessary code adjustments. I almost gave up a few times, but eventually the thing compiled! I almost didn't believe the terminal. But it compiles now and all the unit tests still pass and it seems to work, so I'm pleased with that.

Always Waiting

taylor.town — Mon, 17 Sep 2018 03:00:00 GMT

I find myself waiting for things a lot.

I'd like to become a person who doesn't wait for things.

But I guess I'm waiting for that to happen too.

Be Unreasonable

taylor.town — Mon, 17 Sep 2018 03:00:00 GMT

It's really useful to be unreasonable.

When you have a large task ahead of you, try asking yourself some odd questions. Like, "what if I finished it in 1/10th of the time?" or "what it got 100x more users than anticipated?".

Start with "crazy ideas" and "unrealistic expectations", and then play around for a bit! By working backwards from unreasonable demands, you'll devise bolder schemes!

Plausible Deniability

taylor.town — Mon, 17 Sep 2018 03:00:00 GMT

Three people in a public restroom is a chorus. Two people in a public restroom is a standoff.

You're browsing emails atop your porcelain throne. You chose the stall furthest from the door, because -- we never know -- it's probably cleanest.

And then you hear the dreaded sound: the door opens. You double check that your stall is locked (just in case The Stranger tugs on your door, leaving you completely exposed in this all-too-human position). And just in case, you shuffle your feet forward to signal that "this stall is occupied". The Stranger accepts your message and chooses the furthest stall away from you.

At this point, all sounds are sacred. You may not particularly mind hearing the other person's lunch, but by golly The Stranger is not going to hear yours.

There's one major defensive strategy during potty standoffs: create masking noise.

Coordinating flushes/coughs/sneezes with flatulence/ploops is a classic defensive move.
If you or The Stranger were particularly averse to standoffs, he/she could've turned on some faucets from the sink to create some noise -- it's an uncommon, albeit completely legitimate defensive move.

Of course, The Stranger is in on the ruse too, because he/she uses the same tactics. But the goal is not to not be heard, but to create plausible deniability for yourself. Because you can't stand the idea of knowing that The Stranger knows that you know that they know the timbre of your anal sphincter.

Brainstorming the SnailLife gene system

Liza Shulyayeva — Sun, 16 Sep 2018 17:54:06 GMT

In the original PHP prototype of SnailLife there exists a very rudimentary genetics system. Snails had what I called “visual traits” and “functional traits”. Visual traits had genes associated with them. The genes were all stored in the snail’s table and each gene had two “alleles”. During breeding I’d make virtual Punnett Squares of sorts to pass on genes from parents to offspring. But the system was inflexible - each allele for each gene and each gene type were hard coded in the snails table. This left little room for surprise emergent behaviour (ie is it possible for a snail to accidentally get more or fewer variations of a gene than expected? I’m not sure, but I want a system to support that) without having to modify db columns).

Tutorial Markdown

The Curious Creations of Tim Holman — Sun, 16 Sep 2018 07:00:00 GMT

I’ve open sourced the magic behind the generative artistry website, aptly named tutorial markdown

Searching for Sanity

bpev.me — Sun, 16 Sep 2018 03:00:00 GMT

New Job, New City

Playground — Sun, 16 Sep 2018 03:00:00 GMT

In my adult life, I have never lived outside Delhi for more than a month. I came here at the start of 6th Standard (or Grade). During my 12 Standard, I used to daydream about going to a college outside Delhi. I’ll live in a hostel and enjoy the broke but independent life of a hosteler. Little did I know that I’ll be doing my undergrad from a college in Delhi itself, as a day scholar. Then came the placement season. This time again, I thought I’ll get to go outside my home city. And, this time I wont be (too) broke either. Yet again, I got a job in Noida, very close to Delhi. Now, after working for 2 years in Delhi, I have finally moved to a different city - Bangalore - also known as City of Gardens and City of Lakes and in tech industry, the Silicon Valley of India. I have joined as a Data Scientist in Reliance Jio.

Solving problem-finding

Steph Ango — Sun, 16 Sep 2018 03:00:00 GMT

Methodologies for problem-solving are fairly well established. The scientific method is perhaps the best problem-solving template we have. However, finding good problems to solve is a different skill altogether, one that we don’t teach.

Problem-finding is about looking for an area that you can invest your problem-solving skills into. It’s the intersection between problems worth solving, and problems you will be good at solving.

Problem-finding is harder than problem-solving because there is no established methodology around it. It’s a hole in our educational system because it leads to people dedicating valuable years of their lives to problems that aren’t particularly important.

Common wisdom encourages us to solve the problems we see in our immediate surroundings — in writing it’s distilled as “write what you know”, but it can be generalized as “do what you know”.

Without deep experience in a specific field, this approach rarely yields good problems to solve, usually for one or both of these reasons:

The problem is very personal, it doesn’t help very many people
The gains to be made are small

Of course there are many big problems in the world that are well-documented.

What I’ve found is that most problems that are worth solving aren’t immediately obvious, even if they turn out to be big problems. Often it’s because everyone dealing with the problem has given up on solving it, or they’ve gotten so used to the problem that they don’t notice it anymore.

Every complex system has problems. Most industries are complex and ripe for problem-finding. The challenge is knowing enough about the domain, yet retaining the beginner’s mind necessary to actually see the problem.

If you are coming out of school, or embarking on an entrepreneurial path, it really helps to put in a few years in an industry. Anything you are interested in will do. Try to understand it from the inside out, and study the inefficiencies. Do the work yourself, feel the pain. Stay curious, and keep branching out until you can see the whole picture.

The problems you will be solving in your work are probably not the important ones to solve. The real problem will be more meta, it will be that you have to spend time solving these smaller problems in the first place.

The problem you should solve is the biggest problem you are capable of solving. Perhaps that is why problem finding is such a hard problem in itself — someone needs to care about the problem before it can be found.

Growth Accounting and Backtraced Growth Accounting

Bytepawn - Marton Trencseni — Sun, 16 Sep 2018 01:00:00 GMT

Previously I wrote two articles about data infra and data engineering at Fetchr. This time I want to move up the stack and talk about a simple piece of metrics engineering that proved to be very impactful: Growth Accounting and Backtraced Growth Accounting.

Comments to "Whither Peer Production"

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 15 Sep 2018 11:00:00 GMT

“Then Peer Production sat down and wept, because there were not other worlds for her to conquer”

Networking Essentials: Routing

swyx's site RSS Feed — Sat, 15 Sep 2018 10:25:35 GMT

How the Internet cobbles together thousands of Autonomous Systems with the Border Gateway Protocol

Becoming

taylor.town — Sat, 15 Sep 2018 03:00:00 GMT

I often think that I'd like to do everything. Of course that'd be nice, but I think what I really want is to become the type of person that can do everything.

It's not about experience -- it's about ability. Some people derive their self-worth from the places they've been. But narcissists like me measure themselves by where they could've been if they tried.

In this way, people like me never actually get anywhere. We end our journeys prematurely, right after we purchase the tickets. We say, "Oh, I'm sure it's just like the pictures" when we really mean "I could travel, but it's too much work".

There isn't any problem with not-travelling. The problem is that all the money spent on tickets-to-nowhere add up over time, and the inescapable reality sets in once you realize that you can't fill scrap books of things that never took place.

To reiterate, the problem is not being stagnant. And the problem is not planning, if you enjoy planning. The problem is planning for a journey that you'll never attempt. Why not plan for an adventure at-home? Or somewhere that you actually want to go?

Stop pretending to be a person that could travel, and just stay at home. And you'll be surprised at the places you'll go!

JSON Considered Harmful

taylor.town — Sat, 15 Sep 2018 03:00:00 GMT

So you want to powerful systems to talk to each other over a network with limited capacity and unreliable connections?

Let's try to build the worst "universal" notation possible:

Ensure that the common notation that all the systems speak is extremely low-level, so that you have to encode/decode all the complexity of modern programming languages back-and-forth from lists of strings and numbers and booleans. Please don't add any data-types for NaN, Infinity, undefined, time, dates, currency, rationals, enumerable types, sets, symbols, tags, regular expressions, errors, functions, ADTs, IDs, paths, addresses, characters, or objects. Don't include these, especially if its parent language understands them completely. It's completely reasonable to expect that systems on both sides will always perfectly understand what the value is supposed to mean, and that we can encode/decode them with no errors.
Make absolutely no formalized system of discovery. No documentation standards, no schemas, nothing. The developers on both ends of communication will surely double-check that the id that they're passing back-and-forth refers to the same data on both systems. And because it's encoded in a low-level notation (see #1), they'll absolutely make sure to document things in such a way that other developers can refer back to exactly what each key means in an associative array.
We don't want computers that are able to figure out how to communicate with each other, either! Don't give a common repository of objects/schemas for both systems to agree upon, so that they can ease the programmer's burden. Otherwise, how will engineers get paid?
Under no circumstances should this notation have any capacity to talk about functions. It would be ridiculous to enable computers to talk about the most fundamental part of programming. It's much better if programmers have to encode/decode data into code, and check for vulnerabilites at the data level.
If you make API generators, make sure that they're only in languages that nobody uses.
Make sure that everything can be optional! Every system will equivalently handle missing keys with those keys set to null.
Even though we only have numbers, strings, and arrays, make sure that every time we make an API, we have to manually program the constraints of each range. Under no circumstances should we make libraries to check that something is say, not negative. It doesn't make sense, in any language, to make libraries that generate interfaces based on schemas. Because obviously we want to re-write the interface in every system that touches the data.
Please require that the top-level data CANNOT be a single value. Even though the language is just arrays, strings, and numbers, you definitely shouldn't be able to send strings or numbers without wrapping them.
Give us way of executing programs on other machines (JSON-RPC), and make sure that nobody uses.
Make sure that the system is not extensible! Nobody will ever want to change the protocol to fit their needs.
Make sure that unescaped strings break the languages that it's designed for!
Make inconsistent standards on parsing scientific numbers (e.g. 4e-21). Of course computers won't want to ever talk about very large or very small numbers.
Even if processing is cheap and bandwidth is expensive, make sure it takes up as much space as possible in the network. Oh, and don't make it easy to encode/decode either. Unique data-types would help reduce the size of messages, but we don't want that.
Never, ever, send programs to computers! Especially if it's just talking to a database and it makes sense to send a query. Don't put any work into formalizing database security and sanitization. It makes much more sense to extend our API every time we want a slightly different query.
Even if two computers speak the same language, don't serialize native data-types in a common way. Make the programmer do it by hand.
Use different protocols for everything! If it's good enough for remote connections, it's probably not good-enough for inter-process communication. We want to make as many different interfaces as possible, which won't affect security or development time.
Make it hard to stream! Computers don't need to stream things.
Don't make any formalized system of authentication using this notation. If you do, make JSON Web-Tokens, which are hard to use and have serious vulnerabilities.
Actually, put vulnerabilities in all the major parsers that people use. Nobody will ever try to make the parsers work extra hard in order to crash remote systems.
Don't make any easy ways to convert one notation to another. And if you do, reimplement it from scratch in each language rather than making a common meta-schema that each language can implement just once and be done.
Don't standardize any objects or APIs. Everybody is unique!
Make support for references, but then break every parser that tries to use them.
Don't make it "zero-copy". We want every computer to use its memory inefficiently when trying to read it from the socket.
Don't package any recommendations for how to read the data with the data object itself. It's too confusing to send encoders with the data. Let's just have different programmers make encoders/decoders for every system.
Under no circumstances should encoders/decoders work both ways! Encoders and decoders are not mirror-images of each other, and they shouldn't be. Systems won't break if we interpret in different ways on different systems. Plus, we want to write everything twice! After all, our notation is very complicated, despite the fact it has no complex data-types and is pretty much reversible in every mapping we can imagine.
Don't package any indication of the serialization format with the data. That way, we have to use JSON.parse, XML.parse, etc. rather than having a singular parse function.
Make sure that all parsers interpret common values like undefined, NaN, and Infinity as strings, with no warnings or errors. It's much more exciting to hunt down the bug when it happens in production.
The alternatives to the "universal notation" should be orders-of-magnitude worse than the most popular option. Stockholm Syndrome forever!
Most parsers shouldn't give you the line and character where the mistake occurred. Programmers love trying to find single-character syntactical errors in large bodies of text!
Make a universally-accepted text-encoding (Unicode) and don't properly support it.
Don't allow comments!
Even though the types are simple enough for most databases to read the notation, force programmers to encode the data before putting it into the database. You definitely need to have three different layers of encoding for simple insert/update operations.
Make the universal language unconnected to byte streams! There's no reason to ever connect packets of arbitrary data (like video) to higher-level data types. Make them do extra network calls for images! There's no reason to allow images inside of this format, for any reason.
Make the universal language resistant to skipping! The parser must read everything, even if it doesn't need to!
When you make an API, only define the outer interface for it, in a non-standardized format. Make sure that every person that wants to interact with your system has to read your documentation and re-implement the same thing. Of course they don't want to drop in a schema and automatically generate a type-safe boundary layer! They really want to do all the dirty work and figure out the quirks of your poorly-implemented API.

Summer Haute Route - Part 4: Verbier, Champex

Wilderness Prime — Sat, 15 Sep 2018 03:00:00 GMT

Onwards down to La Chable and Champex.

Arguably the most beautiful town in the European Alps, Verbier overlooks the most picturesque valley. After the last few days of rocks, scrambling and mountains, this all seems rather civilised.

After a day hiking along the valley floor, again I'm ascending into the hills towards Lac Champex.

Again this doesn't disappoint.

Aadhaar Vulnerability Public Disclosure

Nemo's Home — Sat, 15 Sep 2018 03:00:00 GMT

The Vulnerability

The UIDAI Resident Portal (with read access to entire Aadhaar Demographic data) is runing a vulnerable version of LifeRay software. It is running LifeRay 6.1, which was declared End-of-Life in Febrary 2016.

This release includes multiple known vulnerabilities, including:

A XSS issue, for which a PoC can be found at resident.uidai.gov.in (Picture Credits: @sanitarypanels)
Multiple RCEs: See issue-62 for eg.

In fact the release is so old it does not even appear on the “Known Vulnerabilities” page on the LifeRay website; you have to go look at their Archived Vulnerabilities.

The PoC

You can find a simple Proof of Concept for the XSS issue at resident.uidai.gov.in.

The cdn_host parameter injects javascript from $CDN_HOST/Resident-theme/js/custom.js, in this case https://scan.bb8.fun/Resident-theme/js/custom.js which hosts a small snippet to overwrite the HTML of the page.

It shows up like:

Fun

The current script allows for embeding any tweet using a tweet parameter. To embed:

Go to any tweet, copy the part after twitter.com and pass it as the tweet parameter. For eg, to embed this tweet:

Breaking: Exclusive footage from inside @UIDAI's IT department after media reports of Aadhaar data leaks. pic.twitter.com/W7m9L0HvEX
— Aadhaar Compound Wall (@13footwall) March 29, 2018

Look at the URL: https://twitter.com/13footwall/status/979301578686345216
Copy 13footwall/status/979301578686345216 and pass it as the tweet parameter:
The URL becomeshttps://resident.uidai.gov.in/?cdn_host=https://scan.bb8.fun&tweet=13footwall/status/979301578686345216
SHARE IT

The Report

I initially reported this to help@uidai.gov.in in Jan 2017:

Forgot all about it till Jan 2018, when someone mentioned I should try my luck with CERT-IN instead:

Update

There is some confusion regarding which version of LifeRay is UIDAI running. They seem to be running 6.1.1, released in 2013-02-26.

The exact version is not relevant to the fact that UIDAI is:

running an unsupported release
which is 5 year old
not updating it despite being notified multiple times

0800 16-Sep: UIDAI seems to have patched the issue by putting a block on the cdn_host parameter. This still leaves them vulnerable to multiple vulnerabilities until they update to a supported release.

Timeline

The vulnerability is still not fixed. Here is a complete timeline:

Date	What?
16 Jan 2017	Initially reported to `help@uidai.gov.in`. No response
21 Jan 2018	Reported to `ceo@uidai.gov.in` and `info@cert-in.org.in`. No response
19 Feb 2018	Reminder sent to `ceo@uidai.gov.in` and `info@cert-in.org.in`
19 Feb 2018	Acknowledgement from CERT
15 Mar 2018	Reminder sent. No response
17 Mar 2018	Notified NCIIPC
18 Mar 2018	Confirmation from NCIIPC asking for more details. I replied back with a quote of previous exchange
19 Mar 2018	Confirmation from NCIIPC thanking me for the report.
19 Apr 2018	Reminder sent to UIDAI asking for acknowledgement
30 May 2018	Reminder sent to NCIIPC and CERT asking for updates

The only change that I’m aware of since my initial report is that the website stopped declaring the LifeRay version in a HTTP response Header.

Westworld: Season 2

🦉 olshansky 🦁 — Fri, 14 Sep 2018 15:03:59 GMT

Olshansky's review of Westworld: Season 2

Why I share OLD bookmarks

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 14 Sep 2018 11:00:00 GMT

This month, I started sharing lots of OLD (5 or more years old) web pages. Yesterday, I discovered why.

Case study: R (tidyverse) vs Python (pandas)

Negative Feedback — Fri, 14 Sep 2018 09:52:04 GMT

The problem

An old friend e-mailed me a challenge today. He was working with a data file which contains measurements of compounds from groundwater taps. The file looks like this (when opened in a spreadsheet program):

In other words, it contains measurements for a particular date for a particular tap point and compound.

He was using the following R code to plot how the Iron levels varied over time at different sampling points:

library(tidyverse)

ggplot(data) +
  geom_line(mapping = aes(x=Date,y=Tap4.Fe,color="red")) +
  geom_line(mapping = aes(x=Date,y=Tap5.Fe,color="blue")) +
  geom_line(mapping = aes(x=Date,y=Tap6.Fe,color="purple")) +
  geom_line(mapping = aes(x=Date,y=Tap7.Fe,color="green")) +
  geom_line(mapping = aes(x=Date,y=Tap8.Fe,color="orange")) +
  geom_line(mapping = aes(x=Date,y=Tap9.Fe,color="brown"))

But he could sense that this was not an optimal solution to the problem and wanted some advice on getting this to work better.

Tidy data

My first comment to him was that this data is not in the format which the tidyverse wants. Hadley Wickham appears to have revitalised the R language with his clever packages and wrote a manifesto of sorts to the benefits of tidy data, which I feel is really just a no-nonsense approach to data normalisation. There are many things wrong with the data in its current form, but I see spreadsheets like this quite often.

The problem is that people want an easy way of entering data and often end up focusing on ease of entry instead of ease of analysis. It is easy to understand how this format grows by adding more readings as extra columns on the spreadsheet and having one date per row.

From a normalisation perspective, the problems with this format are that there is data in the column name: Tap4.Fe is actually encoding two fields - a position and a compound. Our first job is therefore to tidy up this data.

The tidyverse supplies a lot of useful functions to do just that. Here is how I tidied the data:

measurements <- data.csv="" read_csv="">%

gather(measurement, value, -Date) %>%

separate(measurement, c("position", "compound"), sep="\\.") %>%

drop_na()

Now measurements is much "taller" and less wide:

The key points here are that we have a single row for a single observation. This is the essence of the tidy data philosophy and it makes downstream processing very easy.

Here is how I recreated his plots:

filter(measurements, compound == "Fe") %>%
ggplot +
aes(x = Date, y = value, color = position) + geom_line()

That's really nice. I can filter to find the Iron measurements and I can have ggplot handle the colours automatically by looking at the position.

The resulting plot looks like this (this is just with all default settings):

Python

I was very happy with how succinct the R version of the solution was, but I use Python more than R, so I was interested to see how Pandas handles this.

The "tidyfication" of the data requires a lot more ceremony in Pandas:

df = (pandas.read_csv('data.csv', parse_dates=['Date'])

.melt(id_vars='Date')

.dropna())

df[['position', 'compound']] = df.variable.str.split('.', expand=True)

del df['variable']

I'm especially critical of how hard it is to replace a compound column with its split version (the last two lines). Wickham clearly understands that this is a common problem in data files, where the Pandas version feels "hacky".

Now the code to recreate the graphics:

df.query('compound=="Fe"').pivot(index='Date', columns='position', values='value').plot()

Again, I am using the default settings. Although that might look a bit simpler than the R code, it took me ages to write. There is a bit of counter-intuitive (to my mind) pivoting so that I can use the built-in Pandas plot command and I have much less flexibility about the mapping.

I suppose the take-home here is that tidying your data is a win regardless of your platform, but that R and the tidyverse is really a force to be reckoned with. I'm hoping the Pandas API will end up providing some of the powerful verbs from tidyr et al eventually.

Distractions Cause Bad Code

ntietz.com blog - technically a blog — Fri, 14 Sep 2018 03:00:00 GMT

We are barraged by constant distractions, and they are degrading the quality of our work. Our digital society now is set up to allow us to focus for mere minutes at a time, since we are in an attention economy and the sole objective of companies is to capture more of our time. Facebook, Google, and Snapchat are all incentivized to get us to look at our phones many times a day.

Distractions permeate everything, even at work. GitHub has notifications for so many things that if you have work and personal projects on the same account, you will get unrelated notifications all the time. Our employers set us up with ping-pong tables, open offices, and Slack, the open-office of chat tools.

With all these distractions surrounding us and with all these notifications, we are expected to get deep work done. Personally, I cannot. And you cannot, either.

When I'm highly distracted, I'm prevented from entering flow. To my core, I'm a maker. I get such a thrill from making things that are usable and useful, and these distractions cut through that in a way that makes it impossible to have a productive, fulfilling day.

Flow is important if you want to get anything meaningful done. Context switching takes a lot of effort and time and you can only do it so many times in a day. If you are constantly distracted, you will never enter flow and you will never have great, innovative ideas.

If you never concentrate and go deep, you will produce bad work: you will produce bugs, and fail to debug them; you will create security issues; you will cause performance problems; and you will architect things poorly.

You cannot live in a vacuum: you have to talk to users and stakeholders and your teammates and your manager. But that should be done on your schedule, not on theirs (most of the time) so that when you are done talking to them and you have a good idea of what to build, you can go crank out a high quality first version. This version will be on the right track, technically: good architecture, usable performance, well-tested, with minimal bugs. This is a first iteration you can go take to users to get concrete feedback and keep iterating.

Our attention is being squandered and we have an opportunity now to reclaim it. Fight back. Get rid of the ping-pong table; delete Facebook and Snapchat; disable push notifications for emails; build some walls to establish real offices. Setup processes on your teams to give people large chunks of time where they can go deep, for days at a time. Put walls or even hundreds of miles between your employees. Embrace flow, and get some work done. You'll feel better, I promise, and what you produce will be better as well.

Minard / Morse / Tufte and Authenticity on the Web

mass:werk – Now Go Bang! — Thu, 13 Sep 2018 15:00:00 GMT

Observations on information spread and authenticity of well-known objects on the Internet.

Summer Haute Route - Part 3: Dix, Prafleuri

Wilderness Prime — Wed, 12 Sep 2018 19:47:36 GMT

Over the Pas de Chèvres, Mont Blanc de Cheilon, La Cabane des Dix

The next section led deep into the mountains. Up the valley to the ski resort of Arolla, and over the Pas de Chèvres to the imposing view of the Mont Blanc de Cheilon with La Cabane des Dix overlooking. I've visited the Dix hut several times on the winter version of this route, so I pushed on towards the Pas du Chat and the Lac Dix beyond.

Cabane de Prafleuri isn't in the most picturesque setting - crouching in the barren quarry carved out of the landscape during the construction of the Dix dam. I was running low on water for the last couple of hours hiking, so arriving at the cabin was very welcome.

The journey onwards to Mont Fort was perhaps the most perilous of the trip - an intense scramble up and over the Col de la Chaux. Very happy to arrive at Cabane du Mont Fort in one piece.

209 Netfix Categories You Dont Know About

MattCrampton.com — Wed, 12 Sep 2018 08:00:00 GMT

209 Netfix Categories You Dont Know About Netflix is available just about everywhere and in every country. But they change which movie and TV categories they promote in each market. That means there are some hidden Netflix categories you haven't seen before. So here's a set of links to 209 of the hidden channels available, listed below. There's a ridiculous number of very specific categories like Alien Sci-Fi, Boxing Movies or Satanic Stories, see the list below... The Netflix Hidden Category List... Category # Netflix Category Name 1365Action & Adventure 43040Action Comedies 1568Action Sci-Fi & Fantasy 43048Action Thrillers 11881Adult Animation 7442Adventures 3761African Movies 3327Alien Sci-Fi 5507Animal Tales 7424Anime 2653Anime Action 9302Anime Comedies 452Anime Dramas 11146Anime Fantasy 3063Anime Features 10695Anime Horror 2729Anime Sci-Fi 6721Anime Series 29764Art House Movies 77232Asian Action Movies 5230Australian Movies 8195B-Horror Movies 12339Baseball Movies 12762...

Password recovery from an Alcatel-Lucent Cellpipe 7130

Carey Metcalfe — Wed, 12 Sep 2018 05:30:00 GMT

Background

In my current rental situation, the landlord provides internet included with the rental. Since it's fast enough and has enough bandwidth this isn't an issue at all. The only problem is that the modem provided by the telco is an ancient Alcatel-Lucent Cellpipe 7130 5VzA2001 modem/router combo. It's running firmware version 1.0.4.4R8-wh, released on 2012-05-08 16:30 (there are no updates, I checked).

Since the Cellpipe only has 4 x 10/100 LAN ports and terrible WiFi performance, I would much prefer to use my own equipment to perform the routing and just use the Cellpipe 7130 as a WAN gateway. Unfortunately, for some reason, there is no way to turn off the routing aspect of the Cellpipe and just use it as a modem (aka bridge mode). There is also no way to turn off DHCP functionality on it.

My current hacky solution to these problems is to let my router get an IP from the modem/router via DHCP, then set that IP as the DMZ host on the Cellpipe. This effectively forwards all TCP and UDP ports to the router, making it seem like it's the boundary device for the network. Luckily, my router always seems to request the same IP no matter what so this continues to work even if both devices are power cycled.

Since this is not an ideal situation, I'd like to replace the Cellpipe with a more performant modem without any of the routing overhead. To do this, I just need to port the configuration details from the Cellpipe (like the username and password for the upstream PPPoE connection) over to a new modem. Unfortunately, there's nothing in the web interface that will allow me to see the password and there's no option to do a configuration backup. I also can't just call up the telco since it's the landlord's account.

This means we're stuck, right?

Digging into the Cellpipe

Doing a quick CVE search for the Cellpipe only turns up CVE-2015-4586 (a CSRF vulnerability) and CVE-2015-4587 (an XSS vulnerability), both of which aren't useful in this case. Oh well.

Seeing as the Cellpipe 7130 manual(PDF) mentions various functionality that my device doesn't seem have, my hypothesis is that there might be a configuration backup/restore function somewhere, just not exposed in the UI. With that in mind, I went trawling through the HTML and JavaScript files the web UI served up. I found:

The color 'red' is commented out and redefined as white. This suggests that they could be hiding error messages by just making them the same color as the background:

//var red    ='#FF0000';
var red    ='#FFFFFF';

Debugging in production builds with commented out alert statements:

//alert(isRouter);
//alert(isAPmode);

Various comments indicating lack of source control:

//Jamie hide 20111109
/*if(menuItem=='HPNA')
{
    printMenuItem('hpna.html', 'HPNA', white, darkBlue);
}else{
    printMenuItem('hpna.html', 'HPNA', black, blue);
}
*/

//fanny add 2011/02/22
printMenuItem('wifi_statistics.html', 'WLAN Statistics ', black, white);

<!-- jonathan 2004.04.07    Begin  -->
<form name="logoutForm" method="post">
    <input type="hidden" name="logoutMsg">
</form>
<!-- jonathan 2004.04.07    End  -->

These all seem to allude to shoddy release processes, meaning that it's highly likely that per-model customizations were rushed UI-level hack jobs on top of the current release and not maintained branches where actual functionality was changed.

Since a configuration backup will include the PPPoE username and password, let's do a search for "config". Sure enough:

printMenuSection('util_main.html', 'Utilities', white);
printMenuItem('lang_set.html', 'Language Setting', black,white);
printMenuItem('util_reboot.html', 'Reboot Gateway', black,white);
//printMenuItem('util_reservice.html', 'Restart Service', black,white);
printMenuItem('util_factory.html', 'Restore Factory Defaults',black, white);
//printMenuItem('util_cfgstore.html', 'Configuration Store', black,white);
//printMenuItem('util_cfgrestore.html', 'Configuration Restore', black,white);
printMenuItem('util_webfirmware.html', 'Web Firmware Upload', black,white);//Jamie add back 20111109

Visiting http://<router_ip>/util_cfgstore.html shows a nice "Store" button that downloads a text file containing 771(!) key/value pairs, including the username and password for the PPPoE connection. Success!

Future plans

There are a ton of other options in the downloaded config file, some of which look like they enable bridge mode. However, flipping those settings on and applying the new config using http://<router_ip>/util_cfgrestore.html didn't seem to change anything. I'm going to keep messing with it while I look into sourcing a better modem to use my newfound credentials with.

Longer-term, the plan is to transition to a dedicated modem that provides WAN access to a low-power computer running pfSense that handles firewall and routing duties. From there, an unmanaged switch (possibly injecting PoE) can provide access for enough dumb wireless APs to bathe the house in WiFi, as well as wired hookups for the devices that don't move around too much or need the throughput. For now though, I'll settle for replacing the modem.

The Rise of Immer in React

swyx's site RSS Feed — Wed, 12 Sep 2018 03:00:00 GMT

2018: Published on the Netlify Blog as The Rise of Immer in React

WooCommerce and Russia

Yasha Solutions — Tue, 11 Sep 2018 12:33:53 GMT

Here is a little research project I did at request to figure out all the gateways, if you want to sell stuff in russian speaking countries – turns out Stripe or Paypal are not exactly going to cover all your needs. So here is the results of my research on the topic. Feel free to add in the comments if you have some extra input on the topic. Yandex Money / Kassa The first plugin is from Dayes Design :

London bar backlog

xenodium.com @alvaro — Tue, 11 Sep 2018 03:00:00 GMT

Soho

68 and Boston (cocktails and wine bar).

Building a snail one organ at a time

Liza Shulyayeva — Sun, 09 Sep 2018 20:24:50 GMT

I think enough of the SnailLife Go infrastructure is ready now for me to start digging into the snail itself. In the PHP draft of SnailLife, I started with a basic concept of a snail “Here’s a snail. All it has is a name, size, and speed” and then made it more and more complex. By the end I had a model that had organs, genes, a reproductive system, macro/micro nutrient requirements, locomotion, a brain with memories, mood, etc. With the Go rewrite I have the luxury of hindsight and a vision based on the draft. I know roughly where I want to end up - “organs, genes, a reproductive system, macro/micro nutrient requirements, locomotion, a brain with memories, mood, etc.”

Stream Programming

Anardil — Sun, 09 Sep 2018 10:00:00 GMT

Defining data in terms of streams to simplify programming

Summer Haute Route - Part 2: Moiry

Wilderness Prime — Sun, 09 Sep 2018 03:00:00 GMT

Lac de Moiry and the Cabane du Moiry

The hike over the mountains to Gruben was a long way, but well worth the effort. The next day came a long traverse past the Lac de Moiry and up to the snout of the Moiry Glacier

The glacier was spectacular, and the cabin was the most amazing place, perched on the side of the mountain overlooking the river of ice.

Lean Impact – NGO business model the smart way

Yasha Solutions — Sat, 08 Sep 2018 22:30:10 GMT

Couple of notes : Impact is a the third parameter to be added to the traditional Growth / Value focus Separate end users and funders. Change the funding models to facilitate experimentatino on the early stage Scaling to early also happen to the non-profit – with similar issues are the in the traditional business Separate end users and funders.

Origin Stories

Avy Faingezicht — Sat, 08 Sep 2018 03:00:00 GMT

Boston holds a special place in my life. When I was starting my senior year of high school, and my brother Max was applying for business school, he was invited to MIT for an interview at Sloan and thought it was a good idea for me to tag along on his visit. I did, and that trip was the trigger that made me consider going to college in the US. I explicitly remember walking through security at the San José airport with Max, discussing why I should look beyond the University of Costa Rica (UCR), where he, my older brother Danny, and my mom had all gone to school. Until then, I was pretty set on studying engineering at the UCR, just like they had. His goal bringing me along was to open my eyes to what else was out there. He did. This past weekend I went to visit Max, who ended up going to Sloan and is still in Boston almost ten years later. I had gone back to Boston a few other times, but on this trip, more than any other, I turned pretty nostalgic about how I ended up where I am. While Max interviewed at Sloan, I toured the campus, and went on to the MIT Museum, where I was amazed by the Arthur Ganson exhibit. Last week I did pretty much the same walk, went to see the exhibit again, and walked past a bunch of places that Max's friend Rodolfo, who had also studied at the UCR and gone on to MIT for his masters, had recommended on my first trip. It's odd to think that if Rodolfo had not gone to MIT, Max probably wouldn't have gone to MIT, I probably wouldn't have gone to Northwestern, and I wouldn't be living my current life. Many worlds. Forking paths. Origin stories are strange.

Boston, 2018

Avy Faingezicht — Sat, 08 Sep 2018 03:00:00 GMT

After a long time without visiting, I finally got to spend some time in Boston this past weekend with Max and Einat. We decided not to plan much, which turned out to be a great decision.

r2con 2018 and r2wars

Linux Engineer's random thoughts — Sat, 08 Sep 2018 01:00:00 GMT

I tried something new this year, by going to r2con, a conference dedicated to radare2, a reverse engineering toolkit.

r2con

This conference is one of the most affordable security conference out there. I've used radare2 (r2) in the past, but I don't think I fully understood its philosophy until going …

Flex justify content equally distanced items

Muffin Man — Fri, 07 Sep 2018 03:00:00 GMT

I want to share a hack (or a trick if you prefer) my colleague Marko showed me today.

What is the problem? We want our flex items to be equally distanced between each other and from the parent's edge. Unforunately space-evenlyI wasn't aware of this property, thank you Marc for mentioning it in the comment below. is not supported in IE and Edge. And space-around or space-between won't work.

You can see what we are trying to achieve in the first row, and how the latter two are behaving.

Trick is really simple, use space-between and add empty pseudo elements (::before and ::after) to the parent element. Pseudo elements have no width, but they are still "pushing" real elements from themselves.

.wrapper {
  display: flex;
  justify-content: space-between;
}

.wrapper::before,
.wrapper::after {
  content: '';
}

This hack works with dynamic number of elements, dynamic widths and for any flex direction.

You can see it in the action below:

Don't Disrupt Things; Fix Them

ntietz.com blog - technically a blog — Fri, 07 Sep 2018 03:00:00 GMT

People talk about disrupting industries when those industries appear to be in a stable but inefficient state. For example, the taxicab industry: there was little innovation going on in it, and it was stable, but it seemed like it was far from ideal. Along came Uber, intent to disrupt the industry - and disrupt it they did.

Uber has a culture of ignoring laws around the world when it's convenient to do so, when it helps them earn a buck and disrupt the existing industry. As a result, we have an app that millions of people love and use regularly, and which provides income to millions of drivers. This seems pretty good, in this framing: we got something we all like using, and people are earning money from it.

However, there is a darker framing to it. If Uber disrupted something, what did it disrupt? It disrupted the livelihoods of the millions of taxicab drivers.

The drivers who existed in the old, stable-state system were obeying the laws, in general. They played by the rules, even when it was expensive for them to do so. Taxi medallions in New York City cost over $1 million at their peak, and drivers or taxi company owners had to invest lots of money in this resource which appeared scarce. Now, those medallions are often valued under $200,000, having lost over 80% of their value. Uber has gained value by breaking the law (creating unregulated taxis) and law-abiding businesses have lost out by playing by the rules. The old rules did not make sense in many instances, as there was clearly artificial scarcity at play here, but that does not change the fact that the ethical businesses lost by being ethical.

Similarly, Airbnb has disrupted the hotel industry. The loser here? Big hotel chains which abide by regulations and pay their local hospitality taxes. And consumers, who are now staying in unregulated, potentially unsafe hotels rather than staying in hotels which are regulated by their local governments. Again, you can argue against paying the taxes and against the regulation, but what you can't argue with is this: the businesses who played by the rules lost, and the players who ignored laws and rules came out ahead financially.

Clearly, there is something broken here. It should not be a viable business model to ignore and violate local rules and regulations and then just pay fines down the road, because the economic impact to many is so great. The focus of these businesses should not be disrupting, but fixing. If the taxi industry is broken, let's fix it! If the hotel/hospitality industry is broken, let's fix it! But consider the side effects in the process. Consider who you're putting out of business and what will happen to their livelihood if you do disrupt their life.

Disrupting things is not inherently valuable for society. In fact, while a disruption will push you out of a steady state, you have absolutely no guarantee that you will be in a better position when you get to the new steady state. You could leave society a better place - but you could also make it actively worse.

Think about that next time you set out to solve a problem: instead of disrupting an industry, let's solve a problem and consider the wider impacts.

The Original Task Runner

Vimist — Fri, 07 Sep 2018 03:00:00 GMT

It seems to me that the tools we use are changing so quickly that sometimes we forget to look around at what we already have before we invent something new. Don't get me wrong, specialised build tools and task runners have their place and can make things easier, but they can sometimes unnecessarily overcomplicate things. Simplicity is prerequisite for reliability. Edsger Dijkstra Enter Bash; installed on most Linux distributions by default, available on Mac and even Windows. Cross platform! People have been using Bash scripts to do this sort of thing for quite a while now, but hopefully this will put a slightly different spin on it.

Douglas Adams 3 rules of innovation

Yasha Solutions — Thu, 06 Sep 2018 14:07:21 GMT

Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works. Anything that’s invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it. Anything invented after you’re thirty-five is against the natural order of things.

Showing code coverage for a React app using Code Climate and Travis CI

Jason Raimondi on Software Engineer — Thu, 06 Sep 2018 10:00:00 GMT

Test coverage badges in GitHub README

The other day I was trying to figure out how to use Code Climate to generate Code Coverage for a Electron/React app I am building in TypeScript. I’ve used Code Climate in the past for generating code coverage with PHP projects with success and remember it being really easy to set up.

On the Code Climate documentation, all that it says for JavaScript is.. lcov (generated by Istanbul.

Ink Pots & Johnston Canyon

Dustin.Lammiman — Thu, 06 Sep 2018 05:05:06 GMT

Length: 11.7km (Round Trip)
Elevation Gain: 450m
Time: 4 Hours
Trailhead: Johnston Canyon Parking Lot (map)
Type: Out & Back
Rating: Moderate

Johnston canyon is extermely popular. This isn't without reason as the canyon is beautiful. But if you keep hiking beyond the upper falls you will eventually reach the Ink Pots which are the real jewel of this hike. The portion of the trail after the upper falls is the most intense, with significantly more elevation gain. But you will also be glad to be away from the crowds for a while. After a long, steady incline you'll eventually head back down hill and reach the Ink Pots, which are spring fed pools of water with brilliant color in the midst of a mountain meadow. Be sure to heed the signs and don't disturb the pools. Pack a lunch and take some time to picnic while you enjoy the natural beauty of your destination.

Mussum Ipsum

Heitor's log — Thu, 06 Sep 2018 03:00:00 GMT

Mussum Impsum.

3 interesting read on innovation

Yasha Solutions — Wed, 05 Sep 2018 20:11:27 GMT

[Why You Don’t Understand “Disruption” ]1 by Bill Barnett where you learn that Kodak invented Instagram in the 90’s. What Your Innovation Process Should Look Like where Steve Blank share some knowledge on innovation processes inside an organisation Optimize when you can, standardize if you must. Not 100% on innovation but still important to understand how to approach these two parameters of every technological project

test

Heitor's log — Wed, 05 Sep 2018 03:00:00 GMT

Test description.

Summer Haute Route - Part 1: Zermatt

Wilderness Prime — Wed, 05 Sep 2018 03:00:00 GMT

The first part of my journey from Zermatt along the Europaweg trail.

I hiked the Haute Route in the opposite direction to most people, starting in Zermatt under the majestic Matterhorn and heading down the valley towards Grächen along the Europaweg trail.

The Charles Kuonen Suspension Bridge is the longest in the world, spanning almost 500m (1620ft). A recent landslide blocked the trail shortly after the bridge, forcing a re-route down into the valley to Randa.

Networking Essentials: Switching

swyx's site RSS Feed — Wed, 05 Sep 2018 00:01:50 GMT

Discussing why we need Switches in a network and how they work

Marketing 101 – 1 – Markets are conversations

Yasha Solutions — Tue, 04 Sep 2018 14:14:27 GMT

Ok so you want to up your marketing game and you are looking for recommendations where to start. Here is the first post of a serie where I share with you what I think are the best references and guiding principles in marketing. This is stuff I have collected, read and applied over the course of more than 10 years and I will probably keep updating it as time pass.

Networking Essentials: Architecture and Principles

swyx's site RSS Feed — Tue, 04 Sep 2018 12:18:56 GMT

Discussing the architecture of the Internet and its the principles that guided its original design.

Data Warehouse vs Data Lake

Yasha Solutions — Tue, 04 Sep 2018 11:30:24 GMT

The Principal Agent Problem & Hayek’s Knowledge problem

Yasha Solutions — Tue, 04 Sep 2018 11:29:52 GMT

The Principal Agent Problem The Principal Agent Problem occurs when one person (the agent) is allowed to make decisions on behalf of another person (the principal). In this situation, there are issues of moral hazard and conflicts of interest. The Principal Agent ProblemHayek’s Knowledge problem Frequently mentioned as the most important essays in the history of economic thought, Hayek’s essay “The Use of Knowledge in Society” question the fundamental problem that economics exists to solve?

Exploring Iceland's Vegetation Anomalies

Eric Nussbaumer — Tue, 04 Sep 2018 09:00:00 GMT

Tracking changes in Iceland's vegetation through time. This work was done in collaboration with Kevin James and links out to our Visual Perspective website -

¿Se puede hacer eso con JSX?

Rodrigo Pombo — Tue, 04 Sep 2018 03:00:00 GMT

— at Meetup.js Argentina

MEGA Chrome Extension Hacked - Detailed Timeline of Events

SerHack - Security Researcher — Tue, 04 Sep 2018 03:00:00 GMT

On 4 September at 14:30 UTC, an unknown attacker managed to hack into MEGA’s Google Chrome web store account and upload a malicious version 3.39.4 of an extension to the web store, according to a blog post published by the company. Upon installation or auto-update, the malicious extension asked for elevated permissions to access personal information, allowing it to steal login/register credentials from ANY websites like Amazon, Github, and Google, along with online wallets such as MyEtherWallet and MyMonero, and Idex.market cryptocurrency trading platform. The trojanized Mega extension then sent all the stolen information back to an attacker’s server located at megaopac[.]host in Ukraine, which is then used by the attackers to log in to the victims’ accounts, and also extract the cryptocurrency private keys to steal users’ digital currencies. -Continue to read the article on SerHack.me

Getting lost

D13V — Tue, 04 Sep 2018 00:21:27 GMT

There I was some library in DC, checking out binder folders of reference material about hydrants. That’s where I felt lost in precisely the same way. Browsing through and not finding anything to bite on.

Why was I searching? I was convicted by the judge, who was a friend of mine because I used a fire extinguisher to extinguish a fire even though there was a hydrant nearby. The law apparently said that a hydrant must be used if nearby. There were precedents aplenty of other people convicted, and the judge couldn’t give me a pass. I was not angry at him. I was mad at the hydrant lobby creating such a stupid law.

Mindlessly I continued to look at different binders of law and precedents, searching for a way to fight that damn hydrant lobby. It didn’t feel like I was making any progress. Lost. Helpless. Mindless.

Let’s pause here. If this doesn’t make sense, it’s because it should not. It was a dream I had several days ago, and like any dream, the story is not supposed to make sense. It is just a bunch of thoughts and emotions associated. That there is a little bit of story is quite surprising itself.

As with fiction stories, it is not the funny details that matter- it is the emotion of the characters. The situation is made up, but the emotions are real. Even outside the dream, I was experiencing a sense of confusion of having lost my path. The story was made up, but the emotions are real.

We all live our life on a path. By definition, it is the path of conformity. It often leads to a boring life and death. Some call it nine-to-five, some call it the rat race. The path of conformity is wide, beaten and solid. It is ancient, like a road from the Roman Empire. It has been there for a long time, and it will be around for much longer. It prescribes our action, from the major decisions of religion, diet, holidays, to seemingly minor everyday choices as to drinking coffee or tea in the morning.

It takes courage to stray off the path. It also takes awareness and awakening. Sleepwalking through life is when you don’t question your own decision making. You can be very awake on the fact that Priuses are better for the environment than trucks, and condemning anyone who uses a gas guzzler. And yet, at the very same time, you might be asleep about all the other ways we humans are changing the earth and the ecosystem. And you might be extremely asleep about why you’re caring about the environment. You might be thinking that it’s for the betterment of the “environment” or “nature” itself, and miss the point that it is only for the betterment and survival of humans. Because the Earth and Nature and Life are extremely resilient, but the human species is not so much.

And yet being able to see through the lies if the Jedi, one starts seeing schemes within schemes within schemes, as they say in Dune. For every motivation and incentive that we feel there’s ofta strategically larger scheme, which motivates it. And our actions. Even though I like to pretend that I’m and independent thinker, I realize, that even “independent thinking” is a conformist thing, and expected part of society, bound to cover a spectrum of opinions.

Why am I talking about this meta bullshit, Bene Gesserit, and “saving the world”? How is it related to the title of this essay about being lost? Let’s discuss.

In my life, I tried to escape the beaten roman road, the hamster wheel, but starting to veer off, I am feeling like I might be getting lost.

Up until nine months ago, I was marching along, stomping my boots on the paved shiny stones. All roads lead to Rome. I had graduated from university, got a good job, and had recently gotten a promotion. I was pretty good at marching, and I had a good speed.

Then I decided to take a side path, a fork in the road. In one of my favorite coffee shop “Crossroads Cafe” in San Francisco, the personal all wear t-shirts with the saying “When you come to a fork in the road, take it”. I had seen that phrase many times and yet heeded it only a few. Eight months ago, I took what it looked like a very promising fork. I quit my job.

The new path was and still is, exciting. Instead of staying in the valley, it started going up in the forest, metaphorically speaking. And in this forest is where I’m starting to feel lost. To go through the woods and up the mountains, it is sometimes possible to use an existing path and follow the signs. But is not necessary- one can take a different route making way through the thick vegetation.

Yes! It is possible to get lost and miss the path. But this is not the type of confusion that I’m talking about. The reason for my feeling lost is not for the lack of paths and the forest thickness. It is for having too many directions to choose from, and not deciding which one to go through.

Some paths lead right back to the Roman road. Others offer arduous and perilous shortcuts ahead. And others go further into the wild. I am still far from the genuinely unexplored wilderness and doubtfully equipped to handle it. But even that path is still there, in consideration. What that path promises, is not getting far, but getting high.

A few days ago, hiking in the Pyrenees, I got off the main path. Coming down, I was following the markings- small stone pyramids, when I realized that up ahead was a snow section I was not equipped to cross. It turned out that this was an alternative, harder path. I had to backtrack up to the main route.

It that case, you can say I got lost. But being lost is subjective. In the same situation, I could have decided to continue on the alternative path and explore. Then I would NOT have been lost. I would have been wandering and exploring.

The change of perspective is all it takes to stop being lost. I realize that I was worried and anxious and feeling lost, in life, because I had the attitude that I must stay on the path. Even if it is the new path I am creating.

Instead of getting stressed over it, I can embrace the wandering. I am in a new territory for me, and I am only going to find my way by doing enough exploration. By discovering the dead ends and the passages to the hidden lakes and vista points. Deciding how high, and how far I want to go, and my skills and capabilities would take me.

Richard Feynman once got so burned out by the BS of academia, that decided to stop doing “serious” physics, and instead played with “silly” things such as the wobbliness of plates spinning in the air. The equations he discovered on that silly playtime, eventually blazed the trail for him to get a Nobel Prize, and more importantly, rekindled his love for doing “serious” physics. The moral I take from that story is that a little silliness doesn’t always hurt.

I no longer feel lost and anxious. I am starting to find joy in the discomfort of being out there, in the open. I am going to look at the maps, I am going to lighten up my load, and I am going to level up by doing side quests. I give myself a permission to be silly.

And then I will continue walking and my path will take me somewhere.

No error left behind

Blog on blog.gnoack.org — Mon, 03 Sep 2018 23:36:41 GMT

Which design guidelines should software follow in order to run reliably? Which guidelines do you use in your day-to-day work?

Error handling is a significant complexity in real-world projects, but programming textbooks are often giving very little advice on how to think about it at the higher level. This article tries to fill that gap:

We will define what an error is.
We will find out that each error has a human stakeholder.
We will discuss technical mechanisms to bring errors to human attention.
The final section contains various examples of how to apply this.

To handle the error cases in your software, you should ask yourself who is the human stakeholder who would be able to address it, and then find a suitable mechanism to escalate the error in the direction of that stakeholder.

I’ve cross-checked this approach with multiple people from various software backgrounds, but I’m still happy to receive your feedback.

What is an error?

Definition: An error is when the program is operating outside the intended path of execution.

Running programs within the intended path of execution is a goal by definition. To track and reach this goal, we need to make it observable whether the program is running healthily:

Rule: All errors should be observable from the outside.

Observable in this context means that it’s possible to bring the error to a human’s attention (e.g. end user, system administrator, developer) in an automated fashion.

The outside is defined by the bounds of your program. The error reports need to be made accessible to the surrounding environment, so that stakeholders can access them.

Error stakeholders

The stakeholders for an error are the people involved in the software lifecycle, such as: End users, developers, system administrators, network administrators etc.

Rule: For each error, there is a corresponding stakeholder.

Different errors have different stakeholders who can address the issue.

Is the error cause within the program itself, then the developer is the stakeholder.

Is the error cause in the user input or interaction (e.g. invalid input), then the user is the stakeholder.

Is the error cause in the execution environment, then the operator of that environment is the stakeholder (e.g. sysadmin, network admin, SRE).

Mechanisms

Routing errors to the stakeholders

Routing error indicators to the right stakeholders is partially done in code, and partially within the deployment. Particularly, displaying errors to end users is usually done in the program code itself. On the other hand, metrics collection set-ups can be monitored by developers and administrators alike.

There are two main strategies to make errors observable:

Propagate the error upwards towards the initiator of the erroring operation.
Propagate the error to the side (monitoring and logging).

At least one of these should be used for any error.

A function can propagate errors upwards or to the side,
at least one of these should be used.

Propagating an error upwards

Propagating “upwards” moves the responsibility of error handling to a higher level. In general, higher level software has more context for the failing operation, so they are often in a better position to route the error in the right direction.

Propagating errors upwards can take many forms.

Function level: Propagate to the calling procedure
- Return an error code
- Raise an exception
- Propagate a lower-level exception
Process level: Crash the process
- Exit the program with an error status (e.g. Unix exit())
- Abort the program (e.g. Unix abort())
Network level: Return the error in a network response
User level: Show the error to the user
- e.g. in a UI dialog

Note: Remember to convert the error into a representation which fits the level of abstraction that the caller expects. Many languages support nesting (wrapping) errors so that the full context is retained.

Propagating an error to the side

Sometimes passing the error upwards is not an option or not sufficient, for example because you want to transparently recover or hide a subsystem failure from the user.

Propagating errors to the side can be one of the following:

Increment a counter which can be monitored from the outside. (e.g. using Prometheus or Linux kernel stats counters)
Write the error to an error collection service.
- For server side software: Record stack traces and notify developers about them.
- For client side software: Ask the user to send bug reports when errors are encountered.

Make sure that all the relevant error propagation sinks are monitored by the right stakeholders during the program’s operation.

Systems like Prometheus are built with monitoring and alerting in mind. It’s often a good idea to alert on symptoms close to your business needs and then use collected metrics on other errors for further analysis¹.

Warning: Textual logging is not an error handling strategy. Logs are not meant for machine consumption, so it’s hard to have automated monitoring based on them.

Warning: Ignoring errors is not an error handling strategy either.

What if my case is not a fit?

If your case it not a fit, it’s possible that the condition at hand might not be an error to begin with, but is maybe only a “corner case” which may happen in normal operation (e.g. a lookup key was not found).

Consider switching the way that the condition is propagated. The following options are alternatives where otherwise errors may be used.

Null/empty object: Returning empty lists, sets, Optional<T> types, or other Null Objects. Note that this is not the same as null.
Separate channel: Return the non-error conditions through a separate channel (e.g. don’t use an exception, but return specially-built objects for these conditions). Example: A linter tool’s output is produced as part of the expected operation and is therefore not an error within that context.
Null: Returning null, nil or similar is idiomatic in some languages too, but may lead to follow-up errors² when the value is dereferenced.

If none of these worked for you, I’d love to hear from you, so I can correct my understanding.

Sometimes the most elegant solution is to change the API to make the error impossible³. For example, replace a dynamic check with one that’s statically guaranteed by the type system, or change to idempotent operation semantics, so that multiple invocations do not conflict with each other.

Examples

These are all written from the perspective of a piece of code detecting an error.

Recovery through redundancy

Example

A disk in a software RAID is failing.

Stakeholder

The system administrator, as soon as the percentage of recovered executions > threshold. They can then investigate the cause of network issues, exchange hard drives or similar.

Mechanism

Recover by retrying the operation on a different disk.
Increment an error counter indicating the failure cause. When incremented in a monitorable system, this counter can trigger an alert.

Similar cases

A TCP network packet is lost. (Difference: Retry has less control over the network routing)

Recovery through omission

Example

A web server uses multiple database backends. One of the non-essential backends starts to return errors.

Stakeholder

The operator of the failing subsystem.

Mechanism

Omit the backend’s response. (Treat it as if it didn’t exist.)
Increment an error counter indicating the failure cause, so that the subsystem operator can be alerted.

Delegating the decision to the calling procedure

Example

The Unix open() function is asked to open a given filename for reading, but the file doesn’t exist.

Stakeholder

We don’t know who is responsible for passing the wrong filename, but somewhere in the call chain, someone is going to know where that filename comes from.

Mechanism

Report the error to the caller with the documented ENOENT error code.

Similar cases

Mistakes in passed input values are best handled by the code which passed them, and which can judge why they happened.

Reporting upwards and sideways at the same time

Example

A web server’s servlets are returning errors in the form of HTTP status codes.

Stakeholders

The requester is a stakeholder because they care about the request.
The web server’s operator is a stakeholder because they care about keeping overall error fractions within reasonable bounds.

Mechanisms

Propagate status via HTTP, upwards to the caller who has more context.
Count HTTP statuses keyed by relevant criteria (such as servlet), e.g. in Prometheus, for the web server operator.

A good background read is the Site Reliability Engineering Book, section “Symptoms vs. Causes” ↩︎
Wikipedia: Tony Hoare, section “Apologies and retractions” ↩︎
The book Philosophy of Software Design by John Ousterhout has an entire chapter on the idea of defining errors away. The author has also given a tech talk. ↩︎

Go's Memory Allocator - Overview

andrestc.com — Mon, 03 Sep 2018 21:03:34 GMT

I’ve decided to learn how the Go runtime allocates memory and this is the first in a series of posts that are going to cover what I learned (and also the process itself: how I learned). These posts are based of my notes for the talk I gave at Gophercon UK in 2018:

Slides are available here.

In this first post we are going to cover:

Memory allocation 101
How the allocator is invoked
An overview of the tcmalloc algorithm
An overview of the Go’s runtime allocator

The objective of this post is to give a broad idea of how allocation works on the runtime, without actually showing any of it’s code. The following posts on the series will be more dense and with lots of code samples extracted from the runtime. Each of the future posts will cover an specific detail of the allocator.

CMake bookmarks

xenodium.com @alvaro — Mon, 03 Sep 2018 03:00:00 GMT

silent spring

one mikro2nd — Sun, 02 Sep 2018 15:24:25 GMT

Spring is Sprung Da grass is riz I wonder where Da Boidies iz... Dey gone becos De Bugs dey ate Has all gone die From Glyphosate. An' where de Frogs An' Bees an' Ticks? Dey all die From Neonics!

Marbling It Up

Ben Garney — Sun, 02 Sep 2018 09:47:10 GMT

@MarbleItUp has gone public. We will be launching soon on Switch. I wanted to talk a little bit about the game and how it came to be… and share some sweet GIFs (in high framerate black and white for download size reasons). Just over a year ago, Mark Frohnmayer came to me with a proposal – […]

To Be Unboreable by David Foster Wallace

ho.dges.online — Sun, 02 Sep 2018 03:00:00 GMT

To be, in a word, unboreable… It is the key to modern life. If you are immune to boredom, there is literally nothing you cannot accomplish."

A little bit every day

Steph Ango — Sun, 02 Sep 2018 03:00:00 GMT

When trying to jumpstart a new habit, the only thing that matters is making consistent progress.

It’s easy to get in your own way by setting the bar too high. Instead, set the bar as low as possible. Any progress at all is a good thing. What is the smallest unit of progress you can make?

Do one push up
Read one page
Write one sentence
Take one photo
Draw one doodle

Just do one every day.

If you can only do one per day for weeks or months, it’s more progress than doing nothing. Soon enough, you’ll have done more than you could have imagined.

Compiling dynamic programming languages

Notes on software development — Sun, 02 Sep 2018 03:00:00 GMT

It can be difficult to disassociate the idea that dynamically typed programming languages are tied to byte-code interpreters (e.g. YARV Ruby, CPython, V8, Zend Engine, etc.). But for many languages, a compiled implementation also exists. Cython, Chicken Scheme and SBCL are good examples.

In this post I will briefly describe how I built a compiler for my Scheme implementation using artifacts from the interpreter. In doing this, I learned a simple (not novel) technique for compiling dynamic languages. I'll introduce the Javascript to C++/V8 compiler I am developing using this technique.

BSDScheme

For the past year I've developed a Scheme implementation, BSDScheme. I started with an AST-interpreter (as opposed to a byte-code compiler and VM). A more detailed blog post on the first few steps writing BSDScheme can be found here.

As I built up support for the various objects and operations in the language, I had a sizeable base of D code for the BSDScheme runtime. This included an object representation for primitive types (and support for converting to and from types in D) as well as basic Scheme operations (+, -, car, cdr, etc.).

When the time came to implement a compiler backend, I only needed to do codegen since the parser already existed. Furthermore, the fundamental bits had already been written: object representation and much of the standard library. So I wrote the simplest compiler I could think of by targeting D and the objects/functions I had already written to support the interpreter.

Take, for example, the equals function in the standard library:

Value equals(Value arguments, void** rest) {
  auto tuple = valueToList(arguments);
  auto left = tuple[0];
  auto right = car(tuple[1]);

  bool b;

  switch (tagOfValue(left)) {
  case ValueTag.Integer:
    b = valueIsInteger(right) && valueToInteger(left) == valueToInteger(right);
    break;
  case ValueTag.Char:
    b = valueIsChar(right) && valueToChar(left) == valueToChar(right);
    break;
  case ValueTag.String:
    b = valueIsString(right) && valueToString(left) == valueToString(right);
    break;
  case ValueTag.Symbol:
    b = valueIsSymbol(right) && valueToSymbol(left) == valueToSymbol(right);
    break;
  case ValueTag.Function:
    b = valueIsFunction(right) && valueToFunction(left)[1] == valueToFunction(right)[1];
    break;
  case ValueTag.Bool:
    b = valueIsBool(right) && valueToBool(left) == valueToBool(right);
    break;
  default:
    b = false;
  }

  return makeBoolValue(b);
}

So long as my compiler generated code that used the Value object to represent Scheme data, I already had an equals function and large swaths of a Scheme standard library that I could share between the compiler and interpreter.

Ultimately I only needed to implement a few control structures to support compiling a large subset of what I supported in the interpreter. The key aspects here include: function definitions (in D), function calls (D function calls), if/else (if/else in D) and so on.

To give a concrete example of a whole program compiled, this Scheme program:

(define (exp base pow)
  (if (= pow 0)
      1
      (* base (exp base (- pow 1)))))

(define (main)
  (display (exp 2 16))
(newline))

when run through the BSDScheme compiler would become:

import std.stdio;
import lex;
import common;
import parse;
import utility;
import value;
import buffer;

Value exp(Value arguments, void** ctx) {
    Value[] tmp = listToVector(arguments);
    Value base = tmp[0];
    Value pow = tmp[1];

    Value equals_result = equals(vectorToList([pow, makeIntegerValue(0)]), null);
    Value if_result;
    if (truthy(equals_result)) {
    makeIntegerValue(1);
    if_result = makeIntegerValue(1);
    } else {

    Value minus_result = minus(vectorToList([pow, makeIntegerValue(1)]), null);

    Value exp_result = exp(vectorToList([base, minus_result]), null);

    Value times_result = times(vectorToList([base, exp_result]), null);
    if_result = times_result;
    };
    return if_result;
}

Value BSDScheme_main(Value arguments, void** ctx) {

    Value exp_result = exp(vectorToList([makeIntegerValue(2), makeIntegerValue(16)]), null);

    Value display_result = display(vectorToList([exp_result]), null);

    Value newline_result = newline(vectorToList([]), null);
    return newline_result;
}

void main() { BSDScheme_main(nilValue, cast(void**)0); }

Where every imported function had already been written for the interpreter. I had only to translate a few lines to D and import/call these existing libraries. Now I had a small binary of compiled Scheme.

It was at this point I realized I was using the same technique used by Cython to compile Python code.

...the Cython project has approached this problem by means of a source code compiler that translates Python code to equivalent C code. This code is executed within the CPython runtime environment, but at the speed of compiled C and with the ability to call directly into C libraries. http://docs.cython.org/en/latest/src/quickstart/overview.html

jsc

I played with many PL-research-y languages over the years and wanted to do build something a little more practical. So I took what I learned writing the BSDScheme compiler and decided to write a Javascript compiler. Specifically, it would target the easiest backend I could imagine: C++ using the V8 C++ library and generating a Node addon.

There already existed well-trodden guides/means of writing Node addons in C++ so I spent some time trying to hand-compile simple Javascript programs to C++ and V8. A string in Javascript would become a v8::String type in C++. A number in Javascript would become v8::Number in C++ and so forth.

I decided to write this compiler in Rust given its roots in (and my familiarity with) ML and Python. I found a Javascript parser by Dave Herman and after a few lazy weeks finally got a "Hello world!" program compiling. Getting my first program to compile has by far been the hardest part of building jsc.

Let's look at a concrete example of a recursive fibonacci program (example/recursion.js in the repo):

function fib(i) {
  if (i <= 1) {
    return i;
  }
  return fib(i - 1) + fib(i - 2);
}

function main() {
  console.log(fib(20));
}

Let's add a call to main() at the end and time this with Node to get a baseline:

$ time node example/recursion.js
6765
node example/recursion.js  0.06s user 0.02s system 97% cpu 0.083 total

Now let's install jsc to compare. We'll need Rust, Cargo, Node and Node-GYP.

$ git clone https:/github.com/eatonphil/jsc
$ cd jsc
$ make && make install
$ jsc example/recursion.js

jsc produces a Javascript entrypoint that imports our addon (build/recursion.js):

require("./build/Release/recursion").jsc_main();

And it produces a C++ file that represents the entire program (build/recursion.cc):

#include <string>

#include <node.h>

using v8::Boolean;
using v8::Context;
using v8::Exception;
using v8::Function;
using v8::FunctionTemplate;
using v8::FunctionCallbackInfo;
using v8::Isolate;
using v8::Local;
using v8::Null;
using v8::Number;
using v8::Object;
using v8::String;
using v8::False;
using v8::True;
using v8::Value;

void fib(const FunctionCallbackInfo<Value>& args) {
  Isolate* isolate = args.GetIsolate();
  Local<Value> i = args[0];
tail_recurse_1:
  Local<Context> ctx_2 = isolate->GetCurrentContext();
  Local<Object> global_3 = ctx_2->Global();
  Local<Function> Boolean_4 = Local<Function>::Cast(global_3->Get(String::NewFromUtf8(isolate, "Boolean")));
  String::Utf8Value utf8value_tmp_5(i);
  std::string string_tmp_6(*utf8value_tmp_5);
  String::Utf8Value utf8value_tmp_7(Number::New(isolate, 1));
  std::string string_tmp_8(*utf8value_tmp_7);
  Local<Value> argv_9[] = { (i->IsNumber() || Number::New(isolate, 1)->IsNumber()) ? Boolean::New(isolate, i->ToNumber(isolate)->Value() <= Number::New(isolate, 1)->ToNumber(isolate)->Value()) : ((i->IsString() || Number::New(isolate, 1)->IsString()) ? Boolean::New(isolate, string_tmp_6 <= string_tmp_8) : (False(isolate))) };
  Local<Value> result_10 = Boolean_4->Call(Null(isolate), 1, argv_9);
  if (result_10->ToBoolean()->Value()) {
    args.GetReturnValue().Set(i);
    return;
    return;
  }
  Local<Value> arg_11 = (i->IsNumber() || Number::New(isolate, 1)->IsNumber()) ? (Number::New(isolate, i->ToNumber(isolate)->Value() - Number::New(isolate, 1)->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate));
  Local<FunctionTemplate> ftpl_13 = FunctionTemplate::New(isolate, fib);
  Local<Function> fn_12 = ftpl_13->GetFunction();
  fn_12->SetName(String::NewFromUtf8(isolate, "fib"));
  Local<Value> argv_14[] = { arg_11 };
  Local<Value> result_15 = fn_12->Call(Null(isolate), 1, argv_14);
  Local<Value> arg_16 = (i->IsNumber() || Number::New(isolate, 2)->IsNumber()) ? (Number::New(isolate, i->ToNumber(isolate)->Value() - Number::New(isolate, 2)->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate));
  Local<FunctionTemplate> ftpl_18 = FunctionTemplate::New(isolate, fib);
  Local<Function> fn_17 = ftpl_18->GetFunction();
  fn_17->SetName(String::NewFromUtf8(isolate, "fib"));
  Local<Value> argv_19[] = { arg_16 };
  Local<Value> result_20 = fn_17->Call(Null(isolate), 1, argv_19);
  args.GetReturnValue().Set((result_15->IsString() || result_20->IsString()) ? Local<Value>::Cast(String::Concat(result_15->ToString(), result_20->ToString())) : Local<Value>::Cast((result_15->IsNumber() || result_20->IsNumber()) ? (Number::New(isolate, result_15->ToNumber(isolate)->Value() + result_20->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate))));
  return;
}

void jsc_main(const FunctionCallbackInfo<Value>& args) {
  Isolate* isolate = args.GetIsolate();
tail_recurse_21:
  Local<Value> arg_22 = Number::New(isolate, 20);
  Local<FunctionTemplate> ftpl_24 = FunctionTemplate::New(isolate, fib);
  Local<Function> fn_23 = ftpl_24->GetFunction();
  fn_23->SetName(String::NewFromUtf8(isolate, "fib"));
  Local<Value> argv_25[] = { arg_22 };
  Local<Value> result_26 = fn_23->Call(Null(isolate), 1, argv_25);
  Local<Value> arg_27 = result_26;
  Local<Function> fn_28 = Local<Function>::Cast(Local<Object>::Cast(isolate->GetCurrentContext()->Global()->Get(String::NewFromUtf8(isolate, "console")))->Get(String::NewFromUtf8(isolate, "log")));
  Local<Value> argv_29[] = { arg_27 };
  Local<Value> result_30 = fn_28->Call(Null(isolate), 1, argv_29);
  result_30;
}

void Init(Local<Object> exports) {
  NODE_SET_METHOD(exports, "jsc_main", jsc_main);
}

NODE_MODULE(NODE_GYP_MODULE_NAME, Init)

Let's time this version:

$ time node build/recursion.js
6765
node build/recursion.js  0.16s user 0.03s system 107% cpu 0.175 total

jsc, over twice as slow, is already falling behind Node. :)

As I incremented the number passed to my fibonacci function the compiled program time to completion get exponentially worse. Node stayed the same. I decided to try tail-call optimization to decrease the performance distance between Node and jsc.

I implemented tail-call optimization for the interpreter in BSDScheme by putting all functions in a loop that would break if tail-call elimination was not to happen. It took me a week to implement this and I never put it in place for the compiler. This time around I was able to add basic tail call elimination to jsc in two hours. It is done by labels and gotos instead of a tail call when applicable.

Here is a tail-call optimized version of the same program (example/tco.js):

function fib(n, a, b) {
    if (n == 0) {
        return a;
    }

    if (n == 1) {
        return b;
    }

    return fib(n - 1, b, a + b);
}

function main() {
  console.log(fib(50, 0, 1));
}

We add a call to main() again for Node and time it:

$ time node example/tco.js
12586269025
node example/tco.js  0.06s user 0.02s system 96% cpu 0.080 total

And compile it with jsc and time it:

$ jsc example/tco.js
$ time node build/tco.js
12586269025
node build/tco.js  0.07s user 0.02s system 95% cpu 0.087 total

Well that's not bad at all. :)

Next steps with jsc

jsc has very limited support for... everything. Today I added almost all primitive numeric operations + equality/inequality operations + unit tests. jsc does not yet support nested functions, callbacks, or closures. It supports while loops but not yet for loops. And I'm not sure if it supports else if. It does not support arrays or objects let alone constructors and prototypes. Adding support for these is low-hanging fruit.

After the low-hanging fruit, more interesting projects for jsc include:

generating C++ with embedded V8 rather than only targeting Node addons
type inference or type hinting for generating unboxed functions a la Cython and SBCL

Xft but for XCB

Venam's Blog — Patrick Louis (Lebanon) — Sat, 01 Sep 2018 20:00:00 GMT

In this post I'm going to go over "fonts for xcb" a mini-project I've been working on recently and I'll document the parts that are not usually found online.

Crazy Rich Asians

🦉 olshansky 🦁 — Sat, 01 Sep 2018 17:43:14 GMT

Olshansky's review of Crazy Rich Asians

How AI could destroy humanity.

mbutler — Fri, 31 Aug 2018 04:31:53 GMT

Governments and corporations realize personal data (consumer, creative, social, etc.) is the most valuable commodity. Tremendous wealth is generated from this, so a type of Universal Basic Income is established whereby people trade their data for a baseline standard of living. The poverty rate plummets and the move is heralded as a huge societal win. […]

Even bad estimates are valuable if you use them right

ntietz.com blog - technically a blog — Fri, 31 Aug 2018 03:00:00 GMT

Estimating software projects is hard, if not impossible. This seems likely to be fundamental to the work, because we're inventing new things and invention doesn't happen on a fixed schedule. And yet, many teams still estimate how long their tasks will take to finish. Why should you do this, if you can't do it accurately? You do it because it can help you reach your real goal of solving a problem as quickly as possible. But when you do it, you need to have really solid processes around estimating, or the estimates will be used and abused and can kill your team.

Let's establish a baseline first: what's an estimate? It's a measure of how long a piece of code is expected to take to complete. This includes the time you need to do non-code tasks, like reproduce a bug or model your data. This includes the time it takes to test your feature, to write automated tests, and to go through the code review and QA process, since those can lead to code changes. Simply put: it's the total amount of time that you expect any member of your team to invest in this change, in any way.

You can do these estimates a few different ways, such as with story points, t-shirt sizing, or time buckets. One important thing to do, regardless of which metric you use, is to think about and quantify your uncertainty: if you're highly uncertain of an issue's size, then you might want to timebox some investigation into the issue to reduce the uncertainty and de-risk it. These estimates, of any type, are useful to let you know when things are going off the rails^[1]. Each sprint, you decide on what your team is trying to accomplish. During the sprint, you let everyone know what you're working on and what you're blocked by at a daily standup. That standup is generally the place for you to say "Hey, I'm working on feature X, but it's turning out to be a lot more complex than we thought; could anyone see if I'm missing something, or should we reduce scope on this?" Then your team can make an informed decision and you can either change course to reduce scope, remove some blockers, or charge ahead as planned and accept that this task is more complex than you anticipated (it happens!). But without these estimates, you're flying blind, and you'll just always charge ahead, missing opportunities to reduce scope or collaborate more with your team members.

With estimates, you also are forced to think through things at the beginning. You switch from fast, instinctive thinking into slow, deliberate thinking so you find the true complexity of issues rather than assuming their surface level simplicity is accurate. This is incredibly helpful in reaching where you want to go, because it leads you to focus on creating the shortest path to a solution which you can test with users. If creating a login page is super complicated, well, do you need the login page to test your app with real humans? Or can you hack it, using an identity-as-a-service provider or even using no login for hands-on user trials?

Doing estimates does have drawbacks, however. You need to have buy-in from everyone your team interfaces with, as well, or you risk Deadline Driven Development. If you have solid estimates and the business team gets their hands on them - without explanations from you - you can expect that these features will be promised on some form of timeline. So, you must explain to your stakeholders beforehand that these estimates are only for course correction during the development process, and they're separate from estimates you will give of when features will be done overall. If this isn't done, you can lose trust on your team, you will lose trust of the people outside of your team, and morale can drop precipitously.

The other main drawback is simply that providing estimates takes time, which is time you could spend just writing code instead. If you never use the estimates to adjust what you are working on, then putting in the time to do estimates is a pure waste. However, if you do put in the time to do estimates, you will spend less time coding - but because your team will be able to respond to things immediately, you will still reach your objective more quickly.

An ideal scenario for estimating and using them well looks like this:

You are using two-week sprints, within the context of a larger goal (solve problem X)
You have daily standups which everyone on your team attends
At the beginning of each sprint, you plan what everyone is working on and estimate it to ensure that it's an appropriate amount of work for one sprint (you may also add "background tasks" to fill time when people are blocked)
Every day, you run standups to see what's at risk of going off the rails and what's blocking progress so the team can get out in front of it
Whenever things look like they might go off the rails, you reassess and adjust course: shrink scope, expand estimate, or remove blockers
Throughout the process, everyone outside of the team either cannot see your estimates or understands that they are not deadlines or promises

So go forth and try doing estimates, and see how it goes! It's challenging, but you can improve at it quickly, and the benefits are really great for doing them, especially in a team environment. You will quickly find that you can anticipate issues more quickly and that you think about risks earlier in the project. Just don't let your business team make promises based on them!

This is why I use clock time for my estimates, rather than story points or t-shirt sizes. When you're using them to adjust course mid-sprint, you need to be able to quickly tell if you're going off the rails. That's much harder with t-shirt sizes, since you need to convert from the size to clock time and then compare your progress - and the sizes don't correspond to exact clock times, anyway! ↩

3D Printed Thru-Axle Adapter

huphtur — Fri, 31 Aug 2018 03:00:00 GMT

Recently I got me one of those bikes with disc brakes. However my Tacx Spider Team Repair Stand (T3050) only came with a quick release fork mount, which doesn’t work with the new disc brake fork (12mm × 100mm).

Sure, one could go and buy the updated stand (which comes with all kinds of handy tools), or other weird looking adapters, but since I’m a frugal Dutch guy, and already spent all my money on a new bike, I decided to 3D print an adapter.

First up, I grabbed a hammer and bashed out the aluminum quick release axle from the front frame holder. It sat in pretty pretty tight so had to use some force (which is not easy for scrawny cyclists), but eventually it came out.

Next, I measured all the things and created a 3D model, visited my local 3D printer and got the adapter printed.

Fresh off the press, this print is solid. Only took a couple hours to print. Cost was $60 (you can probably find better deals online), cheaper than buying a new work stand.

Note: I have no idea what type of filament was used. Probably some basic material. The adapter doesn’t have to be that strong as not too many forces are put on it, just enough to hold the skewer and fork in place.

All that was left to do is insert it in the holder and presto! My fancy bike now works with the old stand.

How to Print Your Own

I’ve uploaded the 3D file to Thingiverse so you can go and print your own.

The dimensions are for 12mm × 100mm thru-axle forks, but you can easily adapt the 3D file for other sizes.

Use 3D Hubs, Google Maps, or Yelp to find a 3D printer.

Hit me up on the socials @huphtur if you printed one.

Registering for affordable French classes at the Commission scolaire de Montréal

David Schlachter — Thu, 30 Aug 2018 20:00:00 GMT

Full- and part-time French classes in Montreal for about $50 per semester? Information about these classes is available only in French: here's an overview if you're not yet fluent enough to get the details in French.

Helping - How to Offer, Give, and Receive Help

june.kim — Thu, 30 Aug 2018 03:00:00 GMT

Oracle SQL Monitoring and Advanced ASH Usage Hacking Sessions!

Tanel Poder Blog — Wed, 29 Aug 2018 21:25:38 GMT

It’s almost the beginning of September, time to get back to school, everyone!

I recently decided to do my Oracle SQL Tuning and Performance & Troubleshooting training classes again. So I will also run some free hacking sessions for promoting these classes, but also for sharing cool new stuff with people who can’t attend the training and of course for fun too!

The hacking sessions do not follow a formal training structure, have very few or no slides at all, just live demos of putting performance tools & techniques into use!

Unexpected Benefits From Shrinking My Pull Requests

denvaar's website — Wed, 29 Aug 2018 08:40:09 GMT

Small pull requests help improve the code review process, but making it a priority has helped me improve in several ot her ways.

self-publishing vs publisher

Yasha Solutions — Wed, 29 Aug 2018 08:25:50 GMT

Also read this piece by tim ferris : How to Write a Bestselling Book This Year — The Definitive Resource List and How-To Guide

designers eyes problem

Yasha Solutions — Wed, 29 Aug 2018 06:30:14 GMT

Waspr Project - Inception

CraftyDuck.Rocks — Wed, 29 Aug 2018 03:00:00 GMT

How the whole thing started

Reason N. 999... why Sustainable Development needs good Open Data

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 28 Aug 2018 21:00:00 GMT

is right behind, or above, urban population growth.

GTD/Get things done bookmarks

xenodium.com @alvaro — Tue, 28 Aug 2018 03:00:00 GMT

Pandoc bookmarks

xenodium.com @alvaro — Tue, 28 Aug 2018 03:00:00 GMT

JavaScript Object Type Coercion

Thomas Hunter II — Tue, 28 Aug 2018 03:00:00 GMT

Sloc Cloc and Code Revisited - A focus on accuracy

Ben E. C. Boyter — Tue, 28 Aug 2018 03:00:00 GMT

Update 2019-03-13

Two things prompted me to start looking at my code counter scc again. The first being the release of Go 1.11. New releases of compilers, libraries and tool-chains have a wonderful habit of making things go faster without you having to do anything other than recompile. In addition they often provide new methods which assist with this and are worth exploring.

A Universal Lemma For Compliance

Privacy, Power, & Protection In The Cyber Century — Mon, 27 Aug 2018 21:38:04 GMT

2500 words, 9 1/2 minutes. Matthew Hopkins, Witchfinder General. 1647 Engraving. Here I describe a lemma1 or helping theorem for technical compliance of IT with a focus on Information Security. It’s an approach for all compliance regimes whether regulatory or corporate. It doesn’t date, nor is it predicated on a technology or platform. It isn’t a trick. It doesn’t provide cover for inadequate security or incompetent staff. If you’re looking to evade compliance, disguise incompetence, or shirk accountability then you’re in the wrong article.

Wasm with Go to build an S2 cover map viewer

Fabrice Aneche — Mon, 27 Aug 2018 05:19:57 GMT

I needed a reason to use the new Go 1.11 Wasm port for “real”. To make it short, it compiles Go code to Wasm binary format for a virtual machine running in web browsers. I’ve always needed a debug tool to display S2 Cells on a map for different shapes, some online tools already exist: s2map.com unfortunately the backend is often/currently dead. regioncoverer which doesn’t allow polygons. I’ve planned for a Qt Go app or a QGIS plugin with C++ bindings to Python but to ship those modules would be a nightmare.

Mauritius travel bookmarks

xenodium.com @alvaro — Mon, 27 Aug 2018 03:00:00 GMT

Scala bookmarks

xenodium.com @alvaro — Mon, 27 Aug 2018 03:00:00 GMT

Functional Programming for Mortals (Leanpub).

Using validate.js instead of Vuelidate in Vue.js applications

Posts by Petr Stribny — Sun, 26 Aug 2018 19:53:46 GMT

Are YOU experiencing shady output from your Go debugger? Try these things!

Liza Shulyayeva — Sat, 25 Aug 2018 22:50:19 GMT

A couple of days ago I noticed some unreliable output when stepping through the SnailLife server. I use GoLand and Delve debugger, but in the course of looking into this I picked up some tips from trial and error plus from some very helpful people which I think might be useful regardless of the debugger you use.

Dripform - No Bullshit Forms

ferrucc.io — Sat, 25 Aug 2018 19:46:03 GMT

A little more than two months ago Konstantin asked me how long would it take for me to develop something that would let him receive forms on his website as Telegram messages.

I wasn’t sure how long it would take, but finding the task interesting I decided to just do it.

In an afternoon of work I made the first version of “Dripform”.

After making it I started exchanging ideas via email with Marc-Antoine who is working on a similar product. (btw Marc thanks for correcting my messages in french)

Page Builders and WYSIWYG in JS / React / Vue

Yasha Solutions — Fri, 24 Aug 2018 14:34:03 GMT

Gramateria https://www.gramateria.com/GrapeJS https://grapesjs.com/GrapesJS is a multi-purpose, Web Builder Framework. It can be used to create web pages, Newsletters, Native Mobile Applications, Native Desktop Applications (eg. Vuido), PDFs (eg. React PDF), etc. DanteII https://michelson.github.io/dante2/“Just” another medium clone built on top of DraftJs – in React SlateJs https://github.com/ianstormtaylor/slateReact Draft WYSIWYG Just a react editor https://github.com/jpuri/react-draft-wysiwygFroala A multi framework editor Warning : not free. Cost minimum 144 dollars and up to 1800 / year.

Doubts on Software Licenses, 2018 version

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 24 Aug 2018 11:00:00 GMT

Last month I found, once more, that this fundamental topic of our times keeps going through the same, decades-long doubts, and maybe errors.

The digital transformation done right – a conceptual case study

Yasha Solutions — Thu, 23 Aug 2018 13:22:53 GMT

Source: Podcast with Adams Jacob from ChefRight now, the most common line of thinking – when it comes to digital transformation – is we will take some large enterprise, like Procter and Gamble, and we’re going to rewrite Procter and Gamble so that it works the way Google works. So, if Google had built Procter and Gamble, what would Google have done? And that’s our strategy. The big plot is to take the entirety of your business, burn it to the ground, and then rebuild it as if they were another organisation they have never seen before, in a pattern they’ve never seen?

Testing a TypeScript React app using TS-Jest without using Create React App

Jason Raimondi on Software Engineer — Thu, 23 Aug 2018 10:00:00 GMT

There is very little documentation on setting up a TypeScript React project that is not using the create-react-app magical nonsense.

The [Create React App](GitHub - facebook/create-react-app: Create React apps with no build configuration.) helper works great for greenfield projects, or projects that are not going to deviate at all from the structure and tools the Create React App helper gives you.

My goal is to give you a fairly slim (while remaining production ready) tested, TypeScript React app, WITH code coverage.

Forkbox

Rodrigo Pombo — Thu, 23 Aug 2018 03:00:00 GMT

Fork, dev, test, and merge PRs online

Topologies of Remote Teams

ntietz.com blog - technically a blog — Thu, 23 Aug 2018 03:00:00 GMT

When you're building or scaling a software engineering team, you naturally run into a choice at some point: will we all be in the same office, or will we do this "remote work" thing? There are a lot of factors that go into whether or not remote work will work for your team, like if you know how to work remote. Another consideration, to make it more complicated, is which form of remote work you want to consider.

There are four different "topologies" of remote work that I've observed:

The Linux model: fully remote, fully asynchronous
The Basecamp model: fully remote, somewhat synchronous
The hybrid model: half remote, half colocated, fully synchronous
The traditional model: colocated team, possibly with some remote team members

I've been on three of these types of teams, and I've seen the other quite a bit. Let's take a deeper dive into each of them, and then talk about how to make a decision at the end.

The Linux model: fully remote, fully asynchronous

This model is common in open source software projects, due to practical concerns: people work on it at odd hours and cannot be expected to be on chat all at the same time.

You can get a lot of great work done this way. This blog post was written using software that was mostly created this way. The Linux kernel certainly was, at any rate: Linus Torvalds uploaded the source code and sent out some emails on a mailing list, and then other programmers were able to send patches in. As far as I know, the Linux kernel developers don't hop onto Slack to talk all day and to have video calls, so most of their communication is through asynchronous means like email.

This model will work for you when the people on your team work very well with high degrees of autonomy. Since it's asynchronous, they have to be able to do this, or they will run into periods of indecision and stall out.

This is the model which I haven't lived first-hand. My evenings aren't filled with open source contributions (I'd rather spend the time cooking or reading a good book or writing English). I haven't seen this at a lot of companies, although there are some companies where it's debatable if they fall in this category or the next one.

Personally, I find this one suboptimal; it's really nice to have a couple of hours each day where you overlap with the coworkers you're working closely with so you can bounce ideas off them directly, whether it's for debugging or for designing a new feature or for solving a gnarly architecture problem. But it can work and it emphasizes deep work, so there's a big benefit there.

The Basecamp model: fully remote, somewhat synchronous

This model is probably the prototypical commercial fully remote model. This is how I'd categorize companies like Basecamp, GitHub, and others. They're almost entirely remote but they tend to have at least a few hours of overlap of timezones between people who are working closely together to allow for those immensely valuable interactions where you put two people together but get more than two times better solutions as a result.

This model works well when people on your team can work with high degrees of autonomy but they don't have to be quite as autonomous as when it's fully asynchronous, since you have some overlap to bounce ideas off of people and get input from them and talk about where you're going next.

The pitfall in this model is that some activities are simply harder. To the best of my knowledge, there is no great way to whiteboard together with remote employees, and that's a great technique for designing software. It's also difficult to pair program, and mentoring junior engineers just has higher friction when remote.

That said, this is an incredibly fun and productive way to work. I did this on a contracting project, and it was really great for the freedom of it, since you could be offline at any time as long as you were getting your work done. The biggest drawback, on our project, was that it was difficult to get a rhythm going due to all of us having different schedules (despite all living in the same timezone, we actually had few overlapping work hours), which just emphasizes the importance of overlapping work hours and also how hard schedules are, even for a handful of people.

The hybrid model: half remote, half colocated, fully synchronous

For pragmatic reasons, more and more companies are adopting this pattern. The company I work for, Remesh, does this: we have engineers at our HQ in NYC, but we actually have a slim majority of our engineers spread out across the US. We got into this model because the team had two engineers and needed to staff up, so they brought me on as the first remote engineer; it went well, so we gradually hired more remote engineers.

In general, this model is charcaterized by a very strong geographic presence in one location but with a large number of remote engineers. Because these teams have a lot of colocated engineers, they tend to emphasize having a lot of overlap in their days, prioritize synchronous communication, and have high team cohesion.

This configuration has a lot of benefits. Having a lot of colocated engineers makes it easier to build team cohesion. But by having so many engineers remote, you are forced to adopt remote-work patterns. The whole taem benefits from better documentation and more location indepependence. Not to mention, you also reap one of the biggest benefits of remote work: the gigantic pool of talent out there, since most of the talented engineers don't live where you live.

The biggest difficulty here is keeping cohesion between your colocated and remote team members. The colocated engineers will tend to form tighter bonds because they see each other every day. There are some ways around that, like having frequent meetups for the remote team members, but it's a risk factor you just have to be aware of and have to work to mitigate.

If you go this route, make sure that you actively engage the remote engineers, and consider forcing all the engineers to spend some time remote to build empathy and stronger habits on the team as a whole. Think of it as chaos engineering for your team: if you randomly prevent people from working inside the office, you will force your team to document better, be remote friendly, and be more independent and autonomous.

The traditional model: colocated team with a few remote team members

This one is easy to identify from a distance: everyone is in one location except for a few loners who are remote. This usually happens in traditional work environments when something major changes: either an employee is going to move to another city and the company makes this work to keep them on; or they need to bring on talent for a specific skillset and they cannot find it locally.

I don't recommend this model. The benefits are minimal, and are just centered around a specific person that you want to be able to work with. But the drawbacks are huge. You will have a lot of difficulty integrating the remote person in, since your work patterns are all set up for colocated engineers. You will struggle to retain this remote employee for a long time, since you will in all likelihood alienate them or they will simply feel left out by being unable to participate in local team events. It can work with people who you have a really good rapport with or who are already very, very good at remote work, but it usually does not work well.

The only situation I'd recommend this for, as the employer or the employee, is for short-term contracts. For anything beyond that, if you want to embrace remote work, go all in and at least embrace the patterns that will make it work well, since those will benefit your colocated employees and you will broaden your talent pool.

Recommendations

Of these, the "traditional" model is to be avoided at nearly any cost since it is painful with few benefits, and the Linux model is ill-suited for most businesses since you sacrifice agility of decision making which is critical to launching good, fresh products.

So we're left with two realistic models: fully remote but mostly synchronous; or half remote and half colocated. If you execute either of these well, you reap tremendous benefits, so the difference comes down to how much you value colocation. I personally don't find it intensely valuable, so I'll always vote remote; others do find it valuable. One way to make this decision, to find out how much colocation is important for you, is to simply try to be fully remote. With your existing team, close the office for a month and see how you all get on; or if you're scaling up, require half the team to work from outside the office half the time, so you don't have to get that bigger office space just yet. It will be difficult at first, but it will tell you whether or not you truly need colocation and it will expose what you were getting from it - and what it was costing you. And then you will know which way you should go.

Personally, when I start a company, I'm going to go all-in on remote work from day one.

HR mumbo jumbo in ads – Sorry.

Yasha Solutions — Wed, 22 Aug 2018 15:35:28 GMT

The ad At LinkedIn, we are transforming the way organisations attract talent. Join us in changing the way the world works. The SMB Solutions team is looking for a French-speaking Media Solutions Manager to drive success with our campaign media customers. Using your analytical, influencing and relationship building skills, you will act as a business partner to the French and German SMB market, partnering with the sales team and our clients to ensure talent attraction and business development campaigns are utilised and optimised, and to ensure our customers come back for more!

Quantifying Qualitative Research by Leisa Reichelt at Mind the Product San Francisco 2018

Yasha Solutions — Wed, 22 Aug 2018 13:51:06 GMT

Must watch : About surveys : 18min47 Note to self : must do a summary of this awesome talk

Thermomorph Postmortem

Varun Ramesh's Blog — Wed, 22 Aug 2018 13:14:11 GMT

A postmortem for Thermomorph, an Alien-inspired horror game.

UX heuristics

Yasha Solutions — Wed, 22 Aug 2018 13:06:31 GMT

Heuristic Evaluation A heuristic evaluation is a usability inspection method for computer software that helps to identify usability problems in the user interface (UI) design. It specifically involves evaluators examining the interface and judging its compliance with recognized usability principles (the “heuristics”). These evaluation methods are now widely taught and practiced in the new media sector, where UIs are often designed in a short space of time on a budget that may restrict the amount of money available to provide for other types of interface testing.

Actionable URLs in Emacs buffers

xenodium.com @alvaro — Wed, 22 Aug 2018 03:00:00 GMT

Should have enabled actionable URLs in my Emacs buffers long ago. Can now click or press return to follow links. It's great on eshell, compilation buffers, async shell commands, code, etc.

(use-package goto-addr
  :hook ((compilation-mode . goto-address-mode)
         (prog-mode . goto-address-prog-mode)
         (eshell-mode . goto-address-mode)
         (shell-mode . goto-address-mode))
  :bind (:map goto-address-highlight-keymap
              ("<RET>" . goto-address-at-point)
              ("M-<RET>" . newline))
  :commands (goto-address-prog-mode
             goto-address-mode))

Bazel bookmarks

xenodium.com @alvaro — Wed, 22 Aug 2018 03:00:00 GMT

CSS and SVG input animations

Muffin Man — Wed, 22 Aug 2018 03:00:00 GMT

Yesterday, one of my colleagues shared UI Movement website. It resulted in me playing with random animations trying to replicate mockups I liked.

Here are the results:

Docker Compose, Traefik 1.7 + Lets Encrypt

Jason Raimondi on Software Engineer — Tue, 21 Aug 2018 10:00:00 GMT

If you’ve ever configured Docker + Nginx + Lets Encrypt before, it takes some time. Tools like Certbot have made this process easier, but they are not without flaws, especially when deploying to a docker environment. Traefik makes it a cinch.

Maintaining nginx configurations as well as SSL certificates is doable. Deploying a new container to a domain and getting that configured with SSL can take a bit of time but it is by no means impossible. For each deployed site, you need to have a site.conf nginx file and several more files for nginx certificates. All of these files need to be deployed in the nginx container and mounted to the host machine so when the container is rebuilt you do not lose your certificates. Check out my post on configuring lets encrypt with nginx using docker if you really want to find out.

Oracle SQL Tuning training & Oracle Troubleshooting training and some Linux stuff too!

Tanel Poder Blog — Tue, 21 Aug 2018 04:53:00 GMT

Update: Starting from 2022, I am delivering all my classes in a new format, via my new Learning Platform. Check out what are Learning Weeks and the new Continuous Learning method here:

https://learn.tanelpoder.com

More details about the class contents here:

Here are some example slides from the AST class for eye-candy :)

Execution plan data flow	Reading SQL Monitoring reports

I hope you’ll find some of this stuff interesting and “See” you soon :-)

Exploring user offboarding – UX reading list

Yasha Solutions — Mon, 20 Aug 2018 14:50:25 GMT

Designing Friction For A Better User Experience Offboarding — The Way to Say Goodbye to Your Users Design Pleasant Cancellation Experience and Improve Retention Offboard as a designer Offboarding: How and How Not to End Relationships with Users

Binary Tree in Haskell

Anardil — Mon, 20 Aug 2018 10:00:00 GMT

Binary Tree in Haskell

Reading files quickly in Rust

Ben E. C. Boyter — Mon, 20 Aug 2018 03:00:00 GMT

I have been wanting to learn Rust for a while now. I did play with it some time back https://boyter.org/2017/09/working-rust/ for solving some of the project Euler problems and was reasonably impressed with how it turned out. However as I had no practical use for it at the time I ended up investing more time in Go.

Go as it turns out is a pretty decent language, and somewhat akin to Python in terms of getting things done. There is usually one obvious way to solve any problem. I was even very happy with the performance I was getting out of it. As such I took on building the fastest version of a count lines of code program I could using Go scc which you can read about at sloc cloc and code.

Roee: Self Modifying Go Simulation Experiment - Part 2

Liza Shulyayeva — Mon, 20 Aug 2018 00:15:09 GMT

<– PART ONE
Overview
ORM Intentionaliser
aggregationObserver
aggregationReifier
aggregationModifier
The new type
Qualifier
Recompiling and restarting
Visualizing the results
Conclusion

Overview

In Part 1 we talked about the general setup of the world/grid and what the Agent and Instructions metamodels do. We left off at the part where the grid sends a copy of itself to whatever observer channels it has (currently just the aggregationObserver). So in this part we can go over my very rudimentary version of the “ORM Intentionaliser”.

Make Your Source Code Developer-Friendly

Mike Kasberg's Blog — Mon, 20 Aug 2018 00:00:00 GMT

Is your source code developer-friendly? Developers talk about user-friendliness of applications all the time, but don’t often think about whether their own source code is user-friendly. Code tends to rot over time. The documentation gets out of date, configuration files change, little “hacks” are put into place… Eventually, it gets so bad it takes days for a new developer to get his environment set up. And along the way, he has to talk to two or three gurus that know all the little tricks to make things work right. As code ages, it becomes less developer-friendly. As developers, it’s our job to counteract that tendency.

It’s pretty easy to assess the developer-friendliness of a code base. A developer who’s never seen your code before should be able to clone your Git repository, run your tests, run your application, and produce a build without any extra help. And the time it takes him to do all of that should be measured in minutes, not hours. Most enterprise code bases probably fail miserably by these standards but I don’t think it’s impossible to live up to them, even with a large code base. And at least they provide a goal to shoot for.

Software tends to change, and it tends to change quickly. Most code bases are probably pretty friendly when they’re first created. Young code bases are simple, so it’s easy to meet all the criteria listed above. And everyone has every intention of keeping it that way. But as code bases grow, they become very complex very fast. In what seems like an instant, the code isn’t developer-friendly anymore. Developers need to be proactive about fighting this trend.

Why is this important?

Developer-friendly code has business value. When your code isn’t developer-friendly, it takes new developers longer to get their environment set up. It also takes seasoned developers longer to keep their environment maintained and up-to-date. Suppose a mid-level engineer drops and breaks his laptop on the way to work and needs to set up his environment from scratch. Will that take him a couple hours or a couple days? Alternatively, suppose a team needs to add some features to a project that hasn’t been touched in a few years. What kind of state was that project left in, and how long will it take the team to create environments to build, test, and deploy the code? The long-term, repetitive costs of code bases that aren’t developer-friendly add up quickly.

How do you fix it?

So, what do you do if you have an older code base that needs a little love and attention?

Write a README

Github has influenced the software development industry in a lot of ways. One subtle influence is that Github really encourages the use of a README.md file at the root of your repository by rendering it on your repository’s Github page. Even if you’re not using Github, including a README.md file in your repository is a great practice. When you have a readme file, developers don’t have to go searching for the documentation - it’s right there next to the code. Because the readme is checked into the repository, it’s version controlled, and can be updated as the code changes. Even if it gets out of date, it’s easy for any developer to fix it, since they already have permissions and know how to make source file changes in the repository. When you’re writing a readme, keep your developer-friendliness goals in mind. A good readme should explain how to run the tests, run the application, and produce a build.

Simplify Your Toolchain

If you create a readme file and it seems way too long, it probably is. The instructions in your readme file should be simple. And if they’re not, it’s a good indication that your process is too complicated. Ideally, there would only be a single command to test your application, a single command to run it, and a single command to deploy it. If any of those steps currently require more than one or two commands, consider creating a shell script (like test.sh). This makes it easy to run the tests without worrying about the complicated bits - and a developer can still inspect the script if he wants to understand what’s actually going on. Short, simple commands are easier for new developers to grasp and faster for experienced developers to use.

Simplify Your Process

If you have a readme file and it’s easy for developers to get things running, you’re already well on your way to improving the developer-friendliness of your code base. But there’s probably still more you can do. Do developers know what they have to do to get their code merged and approved? Many open source projects document that process in a CONTRIBUTING.md file. What about continuous integration? Do you have an automated process to ensure that code passes quality checks before it gets merged into master? Automated test, build, and deployment processes prevent human mistakes while also making your repositories easier to interact with.

Roee: Self Modifying Go Simulation Experiment - Part 1

Liza Shulyayeva — Sun, 19 Aug 2018 19:10:14 GMT

Intro
ORM Intentionaliser
Questions
Self-modifying simulation in Go?
What are we bootstrapping?
Vague definitions and bad practices
The world
Instructions
Running the World
PART TWO –>

Intro

I took up a small pet project over summer that I called Roee. It was very loosely based on a paper I’d read by Susan Stepney and Tim Hoverd, Reflecting on Open Ended Evolution.

Helping as Learning

Liza Shulyayeva — Sun, 19 Aug 2018 15:07:05 GMT

Answering questions or helping other people seems to be a good way for me to learn more about a language. I’ve been trying to be more active on the Go forums for this reason, answering questions as they pop up (especially focusing on the ones that may not have gotten much attention). Even a very simple question can get you to go down a rabbit hole and learn more as you’re formulating an explanation as part of the answer.

How to Root Moto X (2nd Gen) XT1093-Victara

XOSH.ORG — Sun, 19 Aug 2018 03:00:00 GMT

Steps to root and unlock Moto X (2nd Gen) boot loader.

Are all photos the same on Instagram?

Yasha Solutions — Sat, 18 Aug 2018 17:06:11 GMT

InstaRepeat the account that gather insta gram photos who look alike. Very surprising. Check out the account InstaRepeat

Palestine travel bookmarks

xenodium.com @alvaro — Sat, 18 Aug 2018 03:00:00 GMT

Rukab's Ice Cream, Ramallah (Trip advisor).

Enabling Control-Meta(⌘)-D on macOS

xenodium.com @alvaro — Sat, 18 Aug 2018 03:00:00 GMT

I use command (⌘) as my Emacs Meta key. Recently discovered C-M-d is not available to Emacs for binding keys on macOS. Stack Exchange had the workaround:

defaults write com.apple.symbolichotkeys AppleSymbolicHotKeys -dict-add 70 '<dict><key>enabled</key><false/></dict>'

A records on top level domains

Nemo's Home — Sat, 18 Aug 2018 03:00:00 GMT

A few more changes since the last time I ran this.

Update: An automatically updated version of this is available at https://captnemo.in/tld-a-record/

TLD	IP	Web
ai	209.59.119.34	[http] [https]
arab	127.0.53.53	[http] [https]
bh	88.201.27.211	[http] [https]
charity	127.0.53.53	[http] [https]
cm	195.24.205.60	[http] [https]
dk	193.163.102.58	[http] [https]
gg	87.117.196.80	[http] [https]
inc	127.0.53.53	[http] [https]
je	87.117.196.80	[http] [https]
pa	168.77.8.43	[http] [https]
pn	80.68.93.100	[http] [https]
politie	127.0.53.53	[http] [https]
tk	217.119.57.22	[http] [https]
uz	91.212.89.8	[http] [https]
ws	64.70.19.33	[http] [https]
мон	202.170.80.40	[http] [https]
мон	218.100.84.27	[http] [https]
мон	180.149.98.78	[http] [https]
政府	127.0.53.53	[http] [https]
عرب	127.0.53.53	[http] [https]

Diff:

+bh
+charity
-etisalat
+inc
-اتصالات
-招聘
 政府

SSH configuration tips for basic server security

Yasha Solutions — Fri, 17 Aug 2018 23:30:07 GMT

Don’t run the internet facing SSH on an IP used by a key application. Use a dedicated host acting as a jump box. Use port knocking. Use key pairs. Use fail2ban. Don’t allow root logins. Restrict access to a named group. Don’t use shared logins. Don’t allow direct access below the presentation tier. Have more ? Ping me in the comment or in twitter and I’ll add them here.

Observing Minard Observing Napoléon

mass:werk – Now Go Bang! — Fri, 17 Aug 2018 15:00:00 GMT

Observations on textual strategy in infographics by the example of the "Greatest Infographic of All Times".

A brief guide to OIAs in New Zealand

utf9k — Fri, 17 Aug 2018 14:00:00 GMT

My extremely brief crash course in the Official Information Act and how it can be used for some interesting civic side projects

Short Crypto Rant

Zefram's Blog — Fri, 17 Aug 2018 07:05:00 GMT

Disclaimer: No one in the world wants what I said in this post to be false more than I do.

One

ConsenSys is not all rainbows, sunshine, and unicorns. (to the surprise of no one)

The flat governance structure is something that definitely makes ConsenSys unique and enjoyable to work at: everyone treats you as equal, snobby dicks are rare, and no one tells you what to do at work everyday or when you go to work. It would be hard to go back to the traditional hierarchical world.

Transactions on Graphics Paper- The Design and Evolution of Disney's Hyperion Renderer

Code & Visuals — Fri, 17 Aug 2018 03:00:00 GMT

The August 2018 issue of ACM Transactions on Graphics (Volume 37 Issue 3) is partially a special issue on production rendering, featuring five systems papers describing notable, major production renderers in use today. I got to contribute to one of these papers as part of the Hyperion team at Walt Disney Animation Studios! Our paper, titled “The Design and Evolution of Disney’s Hyperion Renderer”, discusses exactly what the title suggests. We present a detailed look inside how Hyperion is designed today, discuss the decisions that went into its current design, and examine how Hyperion has evolved since the original EGSR 2013 “Sorted Deferred Shading for Production Path Tracing” paper that was the start of Hyperion. A number of Hyperion developers contributed to this paper as co-authors, along with Hank Driskill, who was the technical supervisor on Big Hero 6 and Moana and was one of the key supporters of Hyperion’s early development and deployment.

Here is the paper abstract:

Walt Disney Animation Studios has transitioned to path-traced global illumination as part of a progression of brute-force physically based rendering in the name of artist efficiency. To achieve this without compromising our geometric or shading complexity, we built our Hyperion renderer based on a novel architecture that extracts traversal and shading coherence from large, sorted ray batches. In this article, we describe our architecture and discuss our design decisions. We also explain how we are able to provide artistic control in a physically based renderer, and we demonstrate through case studies how we have benefited from having a proprietary renderer that can evolve with production needs.

The paper and related materials can be found at:

We owe a huge thanks to Matt Pharr, who came up with the idea for a TOG special issue on production rendering and coordinated the writing of all of the papers, and Kavita Bala, who as editor-in-chief of TOG supported all of the special issue papers. This issue has actually been in the works for some time; Matt Pharr contacted us over a year ago about putting together a special issue, and we began work on our paper in May 2017. Matt and Kavita generously gave all of the contributors to the special issue a significant amount of time to write, and Matt provided a lot of valuable feedback and suggestions to all five of the final papers. The end result is, in my opinion, something special indeed. The five rendering teams that contributed papers in the end were Solid Angle’s Arnold, Sony Imageworks’ Arnold, Weta Digital’s Manuka, Pixar’s Renderman, and ourselves. All five of the papers in the special issue are fascinating, well-written, highly technical rendering systems papers (as opposed to just marketing fluff), and absolutely worth a read!

Something important that I want to emphasize here is that the author lists for all five papers are somewhat deceptive. One might think that the author lists represent all of the people responsible for each renderers’ success; this idea is, of course, inaccurate. For Hyperion, the authors on this paper represent just a small fraction of all of the people responsible for Hyperion’s success. Numerous engineers not on the author list have made significant contributions to Hyperion in the past, and the project relies enormously on all of the QA engineers, managers/leaders, TDs, artists, and production partners that test, lead, deploy, and use Hyperion every day. We also owe an enormous amount to all of the researchers that we have collaborated directly with, or who we haven’t collaborated directly with but have used their work. The success of every production renderer comes not just from the core development team, but instead from the entire community of folks that surround a production renderer; this is just as true for Hyperion as it is for Renderman, Arnold, Manuka, etc. The following is often said in our field but nonetheless true: building an advanced production renderer in a reasonable timeframe really is only possible through a massive team effort.

This summer, in addition to publishing this paper, members of the Hyperion team also presented the following at SIGGRAPH 2018:

Peter Kutz was on the “Design and Implementation of Modern Production Renderers” panel put together by Matt Pharr to discuss the five TOG production rendering papers. Originally Brent Burley was supposed to represent the Hyperion team, but due to some outside circumstances, Brent wasn’t able to make it to SIGGRAPH this year, so Peter went in Brent’s place.
Matt Jen-Yuan Chiang presented a talk on rendering eyes, titled “Plausible Iris Caustics and Limbal Arc Rendering”, in the “It’s a Material World” talks session.

Snippets for Markdown to LaTeX with Pandoc

Posts on Sean Lane — Fri, 17 Aug 2018 03:00:00 GMT

This is another living post that serves more for my personal benefit than anything else. By living, I mean that I will continue to update it over time as I find things I want to keep easily accessible. This post is a place to paste snippets and boiler plate of Markdown/Pandoc/LaTeX material that I find useful but often forget. There’s probably nothing very genius or awe-inspiring here, but hopefully I won’t have to grep through by bash history as often, trying to remember what I did that one time to do that one thing.

I love using Markdown¹ for a variety of tasks, and if I need to present my work elsewhere (like homework assignments, distribution of meeting notes, etc.) I use Pandoc to convert the Markdown into $\LaTeX$, which makes some lazy writing look much more professional, in my opinion.

Switch back and forth between multiple columns in Pandoc markdown

Include the following in the YAML header:

header-includes:
  - \usepackage{multicol}
  - \newcommand{\hideFromPandoc}[1]{#1}\hideFromPandoc{\let\Begin\begin\let\End\end}

Then begin and end sections of multiple columns like so:

\Begin{multicols}{2} 

#### Sample Equations

\begin{equation}
\frac{dx_1}{dt} = x_2^2 + x_1^3
\end{equation}

\begin{equation}
\frac{dx_2}{dt} = \frac{1}{x_1} \cdot x_2 
\end{equation}

\End{multicols}

Options commonly specified in header

Include in the YAML header as needed:

title: My awesome title
subtitle: and cool subtitle
date: \today # Note: Only works on Linux/OSX going from Markdown -> PDF
geometry: margin=1in
classoption=twocolumn

Output complete LaTeX source for a document

Running the following only outputs the LaTeX source for the content in the document:

$ pandoc source.md -o output.tex

Running this inserts the LaTeX content into your default (or other specified template) and then outputs the complete LaTeX source for generating the document:

$ pandoc source.md -o output.tex  --template default.latex

Adjust space around title in Pandoc

Use the \vspace command in the title option of the YAML header to adjust space as needed around the title. You can use it before, after, or before and after the desire text. In this example, I’m reducing the amount of space before and after the title:

title: \vspace{-5ex}Sample title\vspace{-5ex}

Inspired by this Stack Overflow answer: https://tex.stackexchange.com/a/593

https://daringfireball.net/projects/markdown/ https://en.wikipedia.org/wiki/Markdown ↩︎

Epictetus: Decide, Do, Become

Jayson Salazar Rodriguez | @jdsalaro | Blog — Fri, 17 Aug 2018 03:00:00 GMT

In the third volume, chapter XXIII §1, of his discourses compiled by Arrian1, Epictetus2 gives his advice “to those who read and discuss for the purpose of display”.

Improve code readability by putting spaces around the "not" symbol

Databases & Life — Thu, 16 Aug 2018 17:13:25 GMT

When I first started working, I noticed my boss at the time had a coding convention that I’d not seen elsewhere. As I was new to work, and the convention seemed logical, I adopted it, thinking it might be standard in the world of work. Alas I was wrong, I’ve never seen anyone else use it. But I still think it’s a good convention.

And that’s based around the following fact. The C family of languages use the exclamation mark “!” as the symbol for “not”, and that’s actually a visually small and easy-to-oversee symbol.

What is Resource Leveling and Why Do You Need it?

Databases & Life — Thu, 16 Aug 2018 13:31:40 GMT

A Guide to the Project Management Body of Knowledge (PMBOK Guide) defines Resource Leveling (also known as Predictive Planning or Priority-Based Scheduling) in the context of project management tools as:

A technique in which start and finish dates are adjusted based on resource constraints with the goal of balancing demand for resources with the available supply.

Many project management and project planning tools force you, the user, to specify when you think tasks will start and end. For example drawing bars on a Gantt chart. “Bob will do the web design full-time from Monday to Friday next week”.

Start your own app with React, GraphQL and Serverless architecture: Part 3 - Let's build a GraphQL backend!

Max Rozen — Thu, 16 Aug 2018 08:52:00 GMT

Start your own app with React, GraphQL and Serverless architecture: Part 3 - Let's build a GraphQL backend!

JavaScript Object Property Descriptors, Proxies, and Preventing Extension

Thomas Hunter II — Thu, 16 Aug 2018 03:00:00 GMT

Quick and Dirty Function Timing in Go

Liza Shulyayeva — Wed, 15 Aug 2018 21:17:09 GMT

When I want to do some rough timing comparisons or get a quick idea of what part of Thing X is taking so long I sometimes just time a few functions. I used to do this by getting st := time.Now() at the beginning of the function and then time.Since(st) at the end. It recently hit me that the same thing could be achieved with less duplication and messiness to clean up (when removing the timings) with something like this:

Coordination

Liza Shulyayeva — Wed, 15 Aug 2018 16:40:07 GMT

When you’re helping coordinate a workflow switch between half a dozen different studios, making sure all tools support new workflow, and shipping at the same time:

Dots and Dashes

Code Explainer — Wed, 15 Aug 2018 14:08:11 GMT

Perl is (in)famous for the ability to write programs that look like line noise, and today I will take apart this beautiful piece of code. Written by tybalt89, it can decode sequences of dots and dashes to Latin letters in … Continue reading →

Project Management tools that offer "Resource Leveling"

Databases & Life — Wed, 15 Aug 2018 10:17:41 GMT

Tool	Cost	What it does	What it doesn't do
Microsoft Project https://products.office.com/en-us/project/project-online-professional	$30 per user per month (if you pay monthly)	Has a lightweight online version to show workers their tasks, but for resource leveling you need the full program which requires Windows Has a button "Resource > Level All" which you click to perform resource leveling "What-if" scenarios can be tested by coping the file	No Mac/Linux version No range estimates to capture uncertainty
LiquidPlanner https://www.liquidplanner.com/	$45 per project manager per month (minimum 5 users). Free for workers to see their tasks.	Web-based tool Has a view (costing nothing) for workers to see their tasks Workers still see tasks even if they are blocked by other dependencies Range estimates to capture uncertainty ("we think this will take min 1 day, max 2 days") Shows resource utilization ("when will each of my employees run out of work?")	No view of estimated predicted costs No "what-if" scenarios
Calligra Plan https://www.calligra.org/plan/	Free and open source	Desktop application, mainly for Linux (also has Windows and Mac versions) Looks actively maintained (last release 6 months ago at the time of writing)	Not obvious how to install it on Mac or Windows No view for workers to see what their tasks are No range estimates to capture uncertainty No view of estimated predicted costs No view to see how long workers have work for (resource utilization) No "what-if" scenarios
Gantter https://www.gantter.com/	$5 per user per month	Web-based tool Use the menu option "auto-level resources" and instantly see the new plan (no wait for server-side calculations) Tasks can be marked as "pinned" (you specify start/end date) or not (auto-leveling algorithm predicts start/end dates) Designed to look visually similar to MS Project	No view for workers to see what their tasks are No range estimates to capture uncertainty No view of estimated predicted costs No view to see how long workers have work for (resource utilization) No "what-if" scenarios
Gantt Chart Project for Jira http://www.jiraproject.com/confluence/display/JRAGANTT/Gantt+Chart+Project+for+Jira	$10 per month for up to 10 users, $6 per user per month thereafter	A module within Jira; good if you're already using Jira Can import MS Project files	No range estimates to capture uncertainty No view of estimated predicted costs No view to see how long workers have work for (resource utilization) No "what-if" scenarios
Kelloo https://www.kelloo.com/	$29 per user per month for the basic version (12 month plans), $49 per user per month for 3x "what if" scenarios	After you make a change, click a button to recalculate, then click again to refresh the schedule once it's recalculated. Allows you to ask questions like "what if I add another developer?" without actually changing the live plan.	Very slow and clunky! No view for workers to see what their tasks are No range estimates to capture uncertainty No view of estimated predicted costs No view to see how long workers have work for (resource utilization)
Merlin https://www.projectwizards.net/en/products/merlin-project/what-is	$13 per user per month	Mac-only desktop application Resource utilization (view to see how long workers have work for) Calculate expected costs of a project in addition to time	No Windows or Linux versions No range estimates to capture uncertainty No view for workers to see what their tasks are No "what-if" scenarios
OmniPlan https://www.omnigroup.com/omniplan/	$300 one-time payment	Mac desktop application, or iOS app Offers advanced Monte-Carlo simulation for predicting schedules Click Shift-Command-L to level resources, or enable "automatic leveling' in the Project menu	No Windows or Linux versions No view for workers to see what their tasks are No range estimates to capture uncertainty No view of estimated predicted costs No view to see how long workers have work for (resource utilization) No "what-if" scenarios
TaskJuggler http://taskjuggler.org/	Free and open source	Command-line application, no GUI (neither web nor desktop) Store your plan in text files and edit them with a text editor	No GUI! No view to see how long workers have work for (resource utilization) Looks unmaintained. Most recent release was in 2016, before that 2013, lots of docs are from 2006

Please note these links here are not sponsored links. and I have not been paid in any way for this post. The opinions are my own. This page exists solely to help you, the reader of my website. They are correct as of August 2018. If you think you’ve found any errors, please write to me.

Active worms

Liza Shulyayeva — Wed, 15 Aug 2018 10:00:07 GMT

My worm bin is about a month old now, which is still very early by worm bin standards. Today I dug around in there and look at the nice active worms I found! I was worried about it before but now I’m hoping it might just be ok.

Visualizing Iceland's Vegetation

Eric Nussbaumer — Wed, 15 Aug 2018 09:00:00 GMT

Using satellite data to observe Iceland's seasonal vegetation. This work was done in collaboration with Kevin James and links out to our Visual Perspective website -

Start your own app with React, GraphQL and Serverless architecture: Part 2 - Deploying to a real domain name

Max Rozen — Wed, 15 Aug 2018 08:52:00 GMT

Start your own app with React, GraphQL and Serverless architecture: Part 2 - Let's get this running in production

Links - August 15, 2018

Avy Faingezicht — Wed, 15 Aug 2018 03:00:00 GMT

Outside Lands, 2018

Avy Faingezicht — Wed, 15 Aug 2018 03:00:00 GMT

Another year in San Francisco, another Outside Lands. After a few years in SF, I've been to several of these festivals, and honestly this year I was less excited than for previous ones. We had purchased the early bird tickets, and were a bit disappointed when the lineup came out. However, being less excited about the concerts meant that we'd spend less time going back and forth in the park, and that we'd be less stressed about what concerts we'd miss because something else had overlapped. As the day got closer, I started looking forward to it more, and once we were at the festival I was glad we had bought the tickets. I knew I'd spend the weekend taking photos of happy people, just like I have in previous years. This time I again took 1000+ photos, so the editing process was tough. If you know I took your photo, and you can’t see it here, reach out and I’ll send you a copy. If you were at previous Outside Lands, maybe I took your photo there? Check out my sets from 2015, 2016 and 2017. Anyway.... on to the 2018 pics:

Homo Deus: A History of Tomorrow

🦉 olshansky 🦁 — Wed, 15 Aug 2018 03:00:00 GMT

Olshansky's review of Homo Deus: A History of Tomorrow by Yuval Noah Harari

I Published My AWS Secret Key to GitHub

Danny Guo — Wed, 15 Aug 2018 03:00:00 GMT

Vermicomposting

Liza Shulyayeva — Tue, 14 Aug 2018 14:27:55 GMT

Letstalkaboutp4

Liza Shulyayeva — Tue, 14 Aug 2018 14:25:56 GMT

Let’s talk about Perforce.

10 to 1 Rule

Liza Shulyayeva — Tue, 14 Aug 2018 14:19:07 GMT

Similarly, the ratio of “code churn” to “lines of code” in mature and non-trivial software is also at least 10:1. Keep this in mind the next time a manager or customer asks you for a time estimate! To build a 10,000 line app, expect to write roughly 100,000 lines.

UAC Notes

Liza Shulyayeva — Tue, 14 Aug 2018 14:18:19 GMT

Random UAC permissions for C# applications notes:

Ask for elevated permissions via app.manifest - see <requestedExecutionLevel level="highestAvailable"/> or requireAdministrator - https://msdn.microsoft.com/en-us/library/6ad1fshk.aspx
Can’t change elevation of a running process, need to start a new process.
To run an elevated cmd.exe and make it look like you’re request permissions mid-run for things like file operations (when you’re really starting new elevated process), set Verbs property of ProcessStartInfo to "runas"

Staple Remover Obsession

Liza Shulyayeva — Tue, 14 Aug 2018 09:53:52 GMT

Do we really need to know about Randy’s young-age staple remover obsession?

Créer un blog avec Hugo, Emacs et ox-hugo

Accueil on dbeley — Tue, 14 Aug 2018 03:00:00 GMT

Pré-requis

Emacs, avec les plugins org-mode et use-package
une instance hugo

ox-hugo

ox-hugo est un plugin permettant l’export d’item org-mode vers un blog hugo.

Installation d’ox-hugo

Ajoutez à votre fichier de configuration emacs (.emacs.d/init.el par exemple) l’instruction use-package suivante :

 (use-package ox-hugo
 :ensure t
 :after ox)

Une fois ox-hugo installé, les fonctions d’exports vers une instance d’hugo seront proposées dans le menu d’exports d’org-mode.

Utilisation

Deux modes de publication sont proposés, l’un utilisant un fichier org-mode pour chaque post, l’autre utilisant un fichier global.

Fetchr Data Science Infra at 1 year

Bytepawn - Marton Trencseni — Tue, 14 Aug 2018 01:00:00 GMT

A description of our Analytics+ML cluster running on AWS, using Presto, Airflow and Superset.

Going Completely Overboard with a Clustered Homelab

Tyblog — Mon, 13 Aug 2018 09:00:00 GMT

A few months ago I rebuilt my router on an espressobin and got the itch to overhaul the rest of my homelab. While I could pick up some post-market AmaFaceGooSoft equipment for a typical high-power x86 lab, I decided to put the devops mantra of a distributed, fault-tolerant architecture to work and see how far I could get with lots of small, cheap, low-power machines instead.

In a nutshell, I'm running ~20 ARM-based single-board computer cluster that drives a container-scheduled application runtime (Nomad) backed by distributed storage (GlusterFS) with service discovery in place (Consul) to provide me with a platform to run all my applications and services with a mostly self-configured HTTPS (Let's Encrypt) front end (Traefik). Vault, Prometheus, and a bunch of supporting applications are also deployed in order to make operating this setup secure, easy, and eminently scalable.

I've diverged from this architecture since time of writing (2018). See this post with some updates; which are mostly replacing Traefik with consul-template, adding a wireguard mesh, and migrating to ODroid HC4 machines for 64-bit support.

Figure 1: Do you like my homemade diagram?

I kept meaning to blog about this when I finished each piece but ended up with the novel you see before you, so while I've written up my journey in order of each system's dependencies, there's a lot to digest here - but don't let that dissuade you. It's a really fun setup - who doesn't want to config-manage a dozen machines and services in their spare time??? (probably quite a few people)

Note: Some have caught that the Gluster project has planned to drop support for 32-bit platforms. If you're planning on building something similar to what I describe in this post, tread carefully - I've already had numerous problems with GlusterFS 7.0, so test and verify your setup. Thank you to /u/legogris for catching that.

Preface
Part 1 - Backing Services
Part 2 - The Runtime
- The Scheduler
- Exposing Applications
Part 3 - Using It
- Prelude - A Private Registry
Deploying Bitwarden
- Summary
Synopsis and Future Work

Preface

Before I dig in, for any fellow SREs/DevOps folks: I wouldn't consider this a reference architecture or even a recommended approach to achieve what I've built. For example, the fact I have four hosts backing my network attached storage is probably more risky than relying on a single host's inventory of hardware when I could ostensibly fit everything I have on a RAIDZ pool with 8TB disks - Gluster is probably strictly more "correct" for a vastly larger infrastructure. Despite that fact, there's some worthwhile tidbits in here, and maybe I can steer you towards some solutions that may work elsewhere. Many aspects of what I've built actually work quite well, and would probably scale to larger deployments.

In addition, some notes here to keep in mind while reading:

I use example.com in place of my real personal domain I like to use for my LAN. In practice, I suggest delegating a subdomain to your LAN for things like this so that it's easy to use real domain names both inside and outside your network if necessary using some different DNS rules for resolving hosts in that domain inside and outside the network (I use the lan subdomain, personally).
I don't dive into security too much in this post. If this inspires you to build a similar stack, pay careful attention to how you're managing authorization in many of these cases (Consul access, using Vault for secrets, PKI with GlusterFS, etc.)

Part 1 - Backing Services

First, I need to cover some of the infrastructure I built to support the higher-level services I'm running in the homelab.

Persistent Storage

This piece was originally meant to be a section of its own. After many years of faithful service, my venerable HP N40L started to creak under the load of dozens of services and applications I had loaded on it.

The primary storage mechanism on that NAS is a ZFS RAIDZ, and to be clear: my perspective is that ZFS on Linux is still the king of reliable, stable storage on Linux at the moment. Scrubbing, snapshots, and send/receive have been massively useful in a very real, applicable way. However, I wanted to try a distributed filesystem this time for a few reasons:

Experimentation, purely to try something new out (it's homelab, that's what it's for)
When expanding my storage pool, I'd like to avoid rebuilding it. Adding a disk to a RAIDZ pool is a little inflexible, so if I can add storage incrementally as necessary, that'd be a big win
I'd like to be able to perform maintenance (kernel updates, package upgrades, etc.) on nodes without downtime. Running a distributed filesystem means I can perform rolling restarts without losing access to my storage volume(s)

The two main contenders in this space are Ceph and GlusterFS. This article isn't about a Ceph vs. GlusterFS comparison, so here's a slimmed-down list of why I opted for GlusterFS:

The underlying file objects are plain, recognizable files. That is, you can log in to a cluster peer, navigate to a brick for the volume you're interested in, and the files allocated to that peer aren't encoded in any special way and could potentially be recovered in a seriously bad disaster recovery scenario. Note that this doesn't hold if you're using disperse volumes (files will be erasure-encoded)
GlusterFS is dramatically more simple than Ceph. Ceph has improved their story with ceph-deploy, which is great addition, but for a relatively straightforward environment like a small-scale personal network, I just want a POSIX-like filesystem I can mount somewhere, and GlusterFS achieves that in a couple of stock, vanilla commands
My desired capability to expand volumes incrementally without rebuilding the storage group is possible with distributed replicated volumes

I'd note here that shortly after completing my setup, an intrepid Redditor blogged about a similar Gluster+ODroid build. If this writeup interests you, I'd recommend reading his summary as well.

Hardware

This was the fun part: when I was looking into a NAS rebuild initially, I checked out some hardware suited for a large, single-host storage machine. In a distributed storage setup, I could instead buy lots of little hosts to bind together cheap storage and cheap hosts so that expansion is affordable and hardware failure isn't a big deal.

Figure 2: Close-up of the ODroid HC2

Let me introduce you to the ODroid HC2. The hardware is purpose-built for low-cost storage clustering: the entire machine is one big heatsink and essentially just a way to make a spinning disk available over a network (the aluminum case is also stackable, so they cluster easily physically as well). In essence the board is a modified ODroid XU4, so you need to run the machine with a Linux distribution built for the armv7 architecture.

Over the course of the months I've used these boards, they've been very reliable - their NICs are performant, no crashes, and they pack more than enough power to drive Gluster with the specs of the XU4 (in fact, they're so powerful I run fully-loaded Elasticsearch data nodes alongside Gluster without any problems, but that's another blog post).

Installation

I built a 4-node cluster on Arch Linux ARM. I use Arch since I'm most familiar with it and know how to package for it easily, but in this and other ARM SBC cases, I think Armbian is a great choice as well.

I mentioned simplicity with Gluster, and this is how to peer each node. This is what I ran from my codex01 host, and used similar commands from the other nodes in the storage cluster.

shell

gluster peer probe codex{02..03}

Then you can check the clustered state:

shell

gluster peer status

Before creating any volumes, mount your SATA-connected drive somewhere that you can consume in a Gluster brick.

Configuration

So here's some experience that bears repeating with GlusterFS: although disperse volumes are a slightly more sophisticated method of pooling storage resources, they have problems. The aforementioned other Reddit user confirmed similar problems I observed with disperse volume behavior. Despite reporting this bug upstream, it was closed as apparently the Gluster policy is to aggressively close bugs in the event of new releases, so I'm mildly concerned about the sustainability and stability of the Gluster ecosystem in general.

Despite my bad experience with disperse volumes, distributed replicated volumes have been a better experience. I went with a 2+1 setup, so I can expand my storage nodes in increments of 2 (plus one arbiter). My arbiters are other random ARM SBCs I have on my LAN that serve miscellaneous purposes but aren't pure storage nodes, if you don't have that luxury you can always go with a different distributed replicated topology with more storage nodes.

Again, using Gluster for its simplicity pays off here:

shell

gluster volume create <volume-name> replica 3 arbiter 1 \
    brick01:/srv/storage/<volume-brick> \
    brick02:/srv/storage/<volume-brick> \
    arbiter01:/srv/storage/<volume-brick> \
    brick03:/srv/storage/<volume-brick> \
    brick04:/srv/storage/<volume-brick> \
    arbiter02:/srv/storage/<volume-brick>

This allocates the actual storage onto the brick nodes and purely cluster-consensus metadata to the arbiter nodes. In this example cluster, brick01 and brick03 can become unavailable with no downtime, but brick01 and brick02 simultaneous faults will cause the files in their charge to disappear. This means that I sort of have tolerance for 50% failure, but from what I've seen, you don't want to gamble with disperse volumes - it's an even worse experience.

The other step you should take in this setup is to enable bitrot scrubbing, which will help keep the volume healthy for you (this is enabled on a per-volume basis):

shell

gluster volume bitrot <volume-name> enable
gluster volume status <volume-name>

That last command returns some important information you should keep an eye on like the status of the auto self-heal daemon, which will ensure that your files are in a consistent, healthy state.

So! At this point, I've got a network volume I can mount on multiple hosts, backed by multiple machines with a degree of fault tolerance. Distributed and replicated persistent storage in the homelab: check.

Compute Hardware

As I mentioned, a collection of ODroid HC2s are running my Gluster storage cluster. What about the compute nodes that'll run the compute-heavy Nomad workloads?

Whereas the HC2 is designed as a mass storage node, the ODroid MC1 is the compute counterpart: with just an SD slot to drive the OS and no SATA interface, it isn't built to house gobs of data, but is moreso a compact compute cluster. A built-in fan fastened to the aluminum housing means that you can drive the boards relatively intensely, and like the HC2, they're completely headless (no HDMI ports here, though you can use a USB serial cable to check the hardware console).

There actually isn't much to say here: install your ARM distro of choice onto an SD card, repeat for each node, and load it up. At that point you're ready for the next steps for each compute node (Consul, Nomad, and so on). Some additional packages are necessary to support executing the workloads as well, like Docker, and you'll need your distribution's Gluster package to mount volumes that your workloads might need.

One item of note is that, if you're like me, you like to keep the metrics for your machines highly visible, and that means pulling in system stats for collection and monitoring. The systems I experimented with (ganglia, Telegraf, a few others) get most of the story, but my MC1 nodes expose some additional data for CPU temperatures under /sys, so I use the following script as an exec input to Telegraf to watch each CPU core's temperature under load.

Shell-script

Font used to highlight builtins.
Font used to highlight strings.
Font used to highlight variable names.
Font used to highlight keywords.
Font used to highlight comments.

#!/usr/bin/env sh
for thermal_zone in /sys/class/thermal/thermal_zone*
do
    cpu_id=${thermal_zone##*thermal_zone}
    temp="$(echo $(cat $thermal_zone/temp)/1000 | bc -l | xargs printf '%.2f\n')"
    echo "cpu_temperature,cpu=cpu${cpu_id} celsius=${temp}"
done

I use these metrics as part of a greater overall monitoring strategy with Telegraf, Prometheus, and Grafana. Metrics like this that are directly impacted by load are useful to help tune Nomad allocation resources later on when you need to gauge exactly how much CPU and memory to carve out for an application container.

Network Consensus and Service Discovery

In order to operate a system that can dynamically scale and run workloads on arbitrary hosts, you're gonna need both a) a way to discover and hit those services, and b) almost certainly some sort of consensus algorithm in the mix. My day job has healthy dose of Kubernetes, which achieves this with a) Kubernetes' service abstraction, and b) etcd. Because I wanted to build my setup atop Nomad, Consul seemed like an ideal choice for this, as it handles both and fits very closely with Nomad.

If you haven't tinkered with Consul lately, the landscape has become very operator-friendly. My distro of choice, Arch Linux ARM, ships an ARMv7 package, and setup isn't bad. If you go for a typical three-server setup for quorum and deploy clients everywhere else, it's a pretty resilient topology, which is what I did.

For the sake of completeness, here's my consul server configuration json for an example host (i.e., the host that isn't farnsworth or curie):

JSON

Font used to highlight constants and labels.
Font used to highlight strings.
Font used to highlight keywords.

{
    "advertise_addr": "{{ GetInterfaceIP }} \"eth0\"",
    "bind_addr": "0.0.0.0",
    "bootstrap_expect": 3,
    "client_addr": "127.0.0.1 {{ GetInterfaceIP }} \"docker0\"",
    "data_dir": "/var/lib/consul",
    "datacenter": "lan",
    "disable_update_check": true,
    "log_level": "INFO",
    "retry_join": [
        "farnsworth",
        "curie"
    ],
    "server": true,
    "ui": true
}

First of all, serious love to Hashicorp for supporting golang template interpolation to dynamically set IP address, it's a really wonderful idea. Second, note that I'm telling the client to listen on both localhost and the docker0 interface. This is crucial, because when I end up running services in Docker containers, I need to have a way to hook those containers into the local consul daemon (and I don't feel like running consul in a container). This permits me to instruct containers to talk to their host's IP to get at consul, which is pretty important later on.

Lastly, because consul is listening on whatever-the-ip-is for docker0, it's now dependent on the docker service being up so that interface is available to listen on, so I ended up dropping a systemd service override in for consul:

Systemd

Font used to highlight keywords.
Font used to highlight type and class names.
Font used to highlight comments.
Font used to highlight comment delimiters.

# /etc/systemd/system/consul.service.d/depend-docker.conf
[Unit]
After=docker.service
Requires=docker.service

After a quick

shell

sudo systemctl enable --now docker
sudo systemctl enable --now consul

On the consul servers/leaders, I've got a nice cluster goin'. The clients don't need quite as much in their configs:

JSON

Font used to highlight constants and labels.
Font used to highlight strings.
Font used to highlight keywords.

{
    "data_dir": "/var/lib/consul",
    "datacenter": "lan",
    "disable_update_check": true,
    "log_level": "INFO",
    "retry_join": [
        "farnsworth",
        "curie",
        "planck"
    ],
    "server": false
}

That's enough to get the LAN clustered. While it isn't flashy now, consul will be critical later. At the very least, you've got some rudimentary uptime checks when you look at live cluster members:

shell

consul members

With consul operational, I took one additional step to make checking in on services that register with consul even easier from non-cluster machines (like my laptop or phone) a little easier. The following line in my router's /etc/dnsmasq.conf instructs it to forward requests for consul domains to the local consul listener:

Conf[Unix]

Font used to highlight comments.
Font used to highlight variable names.

server=/consul/127.0.0.1#8600

This means that, for example, I can curl nomad.service.consul:4646 from anywhere on my network to check in on Nomad, and makes some other operations easier as well that I'll go into later on.

Secret Management

Although you don't strictly need this, if you run Nomad workloads with credentials of any sort (such as database connection credentials, API keys, etc), this will enable you to avoid committing credentials when you manage Nomad tasks. In addition, big chunks of infra like to use certs for authentication (Gluster, for example), and trust me: you will want to use Vault for PKI management and not horrendous openssl commands. I've done this for journald-remote authentication and it works swimmingly.

For self-managed secret management, Vault is pretty much the gold standard. Good news: with consul up, you can pop Vault atop it to serve as both the high-availability and storage backend. Truly, we live in the future.

I actually opt to run Vault in a Docker container, primarily because with Consul as the state backend, I have literally no persistence and/or confguration requirements for Vault other than "talk to consul", so running as a container is just a little easier than managing an operating system package.

If you're really curious, here's the Ansible stanza for my Vault config:

YAML

Font used to highlight constants and labels.
Font used to highlight strings.
Font used to highlight variable names.

- name: Run Vault
  docker_container:
    name: vault
    capabilities:
    - IPC_LOCK
  command:
  - server
  env:
    VAULT_LOCAL_CONFIG: |
      {
        "backend": {
          "consul": {
            "address": "172.17.0.1:8500",
            "service_tags": "traefik.enable=true"
          }
        },
        "default_lease_ttl": "168h",
        "listener": {
          "tcp": {
            "address": "0.0.0.0:8200",
            "tls_disable": true
          }
        },
        "max_lease_ttl": "720h",
        "ui": true
      }

  image: vault:0.10.3
  published_ports:
  - 8200:8200
  - 8201:8201
  restart_policy: always
  state: started

I know, it needs real TLS. That's in the works; we're securing stuff like my Postgres password here, not nuclear launch codes.

Once those containers are in place, they'll cluster and you can follow the typical Vault bootstrapping steps (initializing Vault, generating unseal keys, delegating to a non-root token, and so on). That Traefik tag is for later; it'll expose Vault over TLS for us when the time comes for everyday use.

Again, I've had nothing but good experiences running this on ARM single-board computers, and Hashicorp's official images include builds for ARM, which is a rare treat. My cluster is spread among an ODroid HC1, a Raspberry Pi B+, and an ODroid XU4 on different network paths for good measure.

Part 2 - The Runtime

Okay, at this point we have reliable persistence for workloads, service discovery and distributed consensus, as well as secret management. Let's talk about actually running services and applications!

The Scheduler

It's no secret that Kubernetes is massively popular, and it's mostly deserved – it's extremely powerful, and can handle pretty much anything you throw at it. But that power comes at a cost, and you pay for it with the inherent complexity with everything it can do. I've set up Kubernetes from scratch and wow, there are a lot of moving parts, and many of those pieces are still somewhat of a mystery to me (I still don't fully grok networking between pods). I have very little confidence I could fix a problem with a broken cluster, let alone debug something occurring at the etcd consensus layer or lower.

I don't deny the irony in observing that I've had to set up many other systems to support Nomad which creates complexity anyway, but the difference is that if I experience, for example, problems with request routing in Traefik, the inputs, outputs, and configuration for that system are clearly defined and visible.

Anyway: though there are other container schedulers out there like Mesos, Rancher, and more, I thought I'd put some faith in Nomad given that I have a high degree of confidence in its component parts (Consul, Vault, and so on). It's got the typical convenient trappings of most distributed runtimes (min/max [un]available enforcement, secret injection, service advertisement, etc.) plus some that are nicer than Kubernetes (like executors other than Docker). Some features are missing - most notably for me, truly native persistent storage support and scheduling algorithms other than bin packing - but those haven't been dealbreakers.

Like most modern Hashicorp products, Nomad is a go binary, and by hooking it into Consul, we get clustering operational a little more easily. Here's a server config (Nomad also operates under the quorum of servers + rest are clients paradigm as Consul):

JSON

Font used to highlight constants and labels.
Font used to highlight strings.
Font used to highlight keywords.

{
  "bind_addr": "0.0.0.0",
  "data_dir": "/var/lib/nomad",
  "disable_update_check": true,
  "leave_on_interrupt": true,
  "leave_on_terminate": true,
  "server": {
    "bootstrap_expect": 3,
    "enabled": true
  },
  "client": {
    "enabled": false
  },
  "datacenter": "lan",
  "telemetry": {
    "prometheus_metrics": true,
    "publish_allocation_metrics": true,
    "publish_node_metrics": true
  },
  "vault": {
    "address": "http://vault.service.consul:8200",
    "create_from_role": "nomad-cluster",
    "enabled": true
  }
}

As you can see, I've got Nomad prepped for metrics collection via Prometheus and secret injection from Vault. My client configs are similar:

JSON

Font used to highlight constants and labels.
Font used to highlight strings.
Font used to highlight keywords.

{
  "bind_addr": "0.0.0.0",
  "client": {
    "meta": [
      {
        "cluster": "compute"
      }
    ]
  },
  "data_dir": "/var/lib/nomad",
  "disable_update_check": true,
  "leave_on_interrupt": true,
  "leave_on_terminate": true,
  "datacenter": "lan",
  "server": {
    "enabled": false
  },
  "telemetry": {
    "prometheus_metrics": true,
    "publish_allocation_metrics": true,
    "publish_node_metrics": true
  },
  "vault": {
    "address": "http://vault.service.consul:8200",
    "create_from_role": "nomad-cluster",
    "enabled": true
  }
}

Metadata for scheduling tweaks is very useful - in addition to adding metadata for homogeneous nodes, I also add some node index/count data for more flexibility. With the previously-mentioned changes to DNS resolution, I can hit a simple DNS name to run typical nomad commands from my workstation (with or without a local consul agent):

shell

NOMAD_ADDR=http://nomad.service.consul:4646 nomad status

In addition to some lovely dashboards available at the same address:

Figure 3: Nomad Web Interface

Exposing Applications

At this point we nearly have a ready-made system to submit workloads and run them in a highly-available, clustered setup. However, one more piece will make webapps infinitely more easy to use once we deploy them: Traefik.

In a nutshell, Traefik looks at the metadata for schedulers like Kubernetes, Mesos, Nomad, and others, and acts as the routing glue so that you can talk to a simple frontend like https://app.mydomain.com and Traefik will send traffic to the right workload (container), no matter where it's running - which is important when your applications can be running on any given host in a cluster at any time (potentially even multiple hosts).

For an introductory tour of Traefik, I do recommend checking out the docs to get started, since there are a lot of concepts in there to get familiar with. This post is already pretty excessive, so I'm not going to give a blow-by-blow of how it works, but rather the highlights to get it up and running.

First: the configuration backend. You can configure Traefik with a simple config file, but storing your configuration in Consul lets you 1) run more than one Traefik instance for high availability, 2) alleviate the need to manage a config file and related storage, and 3) lets you dynamically configure several aspects of Traefik just by writing Consul keys (which has been really nice in practice).

To get Traefik's configuration bootstrapped in consul, there's a helpful storeconfig command:

shell

cat traefik.toml

toml

loglevel = "INFO"
defaultEntryPoints = ["https","http"]

[entryPoints]
  [entrypoints.api]
  address = ":8080"
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]

[consul]
endpoint = "172.17.0.1:8500"
watch = true
prefix = "traefik"

[acme]
email = "you@example.com"
storage = "traefik/acme/account"
entryPoint = "https"
[[acme.domains]]
main = "*.example.com"
sans = ["example.com"]
[acme.dnsChallenge]
provider = "route53"

shell

traefik --consul.endpoint=127.0.0.1:8500 storeconfig traefik.toml

This (more or less) gets your Traefik config Consul-ized. Next you need to run Traefik somewhere. Sounds like a perfect fit for a clustered, dynamically scaling runtime like Nomad!

HCL

Font used to highlight constants and labels.
Font used to highlight comments.
Font used to highlight comment delimiters.
Font used to highlight type and class names.
Font used to highlight variable names.
Font used to highlight strings.

job "traefik" {
  region      = "global"
  datacenters = ["lan"]
  type        = "service"

  constraint {
    attribute = "${meta.cluster}"
    operator  = "=="
    value     = "compute"
  }

  task "traefik" {
    driver = "docker"

    config {
      image = "traefik:1.6.5"

      args = [
        "--accesslogsfile=/dev/stdout",
        "--web",
        "--consul.endpoint=172.17.0.1:8500",
        "--consulcatalog",
        "--consulcatalog.endpoint=172.17.0.1:8500",
      ]

      port_map {
        admin = 8080
        http  = 80
        https = 443
      }
    }

    resources {
      cpu    = 100 # Mhz
      memory = 64  # MB

      network {
        mbits = 10

        port "admin" {
          static = 8080
        }

        port "http" {
          static = 80
        }

        port "https" {
          static = 443
        }
      }
    }

    service {
      name = "traefik"
      port = "admin"
      tags = [
        "traefik.enable=true",
        "traefik.frontend.entryPoints=http,https"
      ]
    }

    template {
      change_mode = "restart"
      destination = "local/values.env"
      env         = true

      data = <<EOF
{{ '{{' }} with secret "aws-personal/creds/ddns" }}
AWS_ACCESS_KEY_ID = "{{ '{{' }} .Data.access_key }}"
AWS_SECRET_ACCESS_KEY = "{{ '{{' }} .Data.secret_key }}"{{ '{{' }} end }}
{{ '{{' }} with secret "secret/ddns" }}
AWS_HOSTED_ZONE_ID = "{{ '{{' }} .Data.hosted_zone_id }}"{{ '{{' }} end }}
EOF
    }

    vault {
      policies = ["ddns"]
    }
  }
}

Some notes about this configuration:

I'm using the Route53 ACME DNS provider backend coupled with Vault secrets for authentication. This gets me easy Let's Encrypt certs using purely ephemeral AWS creds that Vault creates then destroys just for this purpose.
Traefik (thankfully) builds and serves images for ARM architectures, so no special steps are necessary to run this on my ODroid MC1 cluster.
The constraint you see defined is necessary since I also run Nomad clients on my storage cluster. All nodes outside of my storage cluster get pooled into my compute cluster. I do this so that I can run some Nomad tasks on my storage machines if needed, though that's another use case that I may cover in another post.

That's it! But we're not quite done without some changes to the router on the network to make this even more easily accessible.

With this setup, the Traefik container will now happily serve domains under *.example.com. If you have a network infrastructure which you can CNAME something like foo.example.com to traefik.service.consul, great: when your browser asks for foo.example.com, your router will reply with a CNAME to Traefik, you'll snag the right IP (wherever in the cluster it's running), and you'll get routed correctly.

In my case, my router is on dnsmasq, and unfortunately, dnsmasq can't CNAME addresses to names outside of its scope (that is, names it has assigned). I had to get a little creative, and found a solution that isn't pretty, but is functional.

I listen on my router's LAN interface for HTTP/HTTPS and proxy all requests to Traefik by using nginx DNS lookups. Thus a request for foo.example.com resolves to my router's IP address, and my router forwards traffic on port 80 and port 443 to whatever IP traefik.service.consul currently has. The config was actually kind of hard to put together since reverse TCP proxying using DNS lookups is a pretty esoteric use case, but this is what it actually looks like in practice:

Nginx

Font used to highlight variable names.
Font used to highlight keywords.

stream {
    resolver 127.0.0.1:8600 valid=10s;
    map $remote_addr $traefik {
        default traefik.service.consul;
    }
    server {
        listen 80;
        listen 443;
        proxy_pass $traefik:$server_port;
    }
}

Note that this requires Consul on the same host to resolve DNS queries for *.consul domains - heavily firewall your router as necessary. Also note: I used HAProxy for this initially, which works great (and, incidentally, supports discovery of hosts over SRV records while nginx only does so in their paid version), but I needed UDP proxying as well for netflow collection, so I used nginx in the end.

In this setup, you need to have some DNS records set to point your apps at your router, but that's an easy task.

Part 3 - Using It

Alright, we've got a functional container/distributed execution runtime! Time to use it.

Prelude - A Private Registry

At this point, we can start hosting applications. If you're purely pulling public images from the Docker hub for your applications and everything you need is there, great! However, in my case, there are a lot of applications that don't ship images for ARMv7 CPUs. You can build and push these to the Docker hub, but I found it easier to just host a private registry on my LAN. As an added bonus, internet availability doesn't impact your ability to run and bring apps up and down, which is pretty important to me.

Of course, you need the registry hosting software present in the first place, so there's a dependency on the Docker hub to get your private registry running. Here's what my Nomad config looks like for my self-hosted registry:

HCL

Font used to highlight type and class names.
Font used to highlight variable names.
Font used to highlight strings.

job "registry" {
  datacenters = ["lan"]
  region = "global"
  type = "service"
  priority = 80
  constraint {
    attribute = "${meta.cluster}"
    operator  = "=="
    value     = "compute"
  }
  task "registry" {
    driver = "docker"
    config {
      image = "budry/registry-arm"
      port_map {
        registry = 5000
      }
      volumes = [
        "/srv/storage/registry:/var/lib/registry"
      ]
    }
    service {
      name = "registry"
      port = "registry"
      tags = [
        "traefik.enable=true",
        "traefik.frontend.entryPoints=http,https"
      ]
    }
    resources {
      cpu    = 100
      memory = 64
      network {
        port "registry" { }
      }
    }
  }
}

The config here isn't too bad, but the persistent storage is import to touch on. The mount point /srv/storage/registry is a Gluster volume mount present on each member of the cluster. Here's the mount snipped in fstab form:

Conf[Unix]

Font used to highlight variable names.

codex01:/registry /srv/storage/registry glusterfs defaults 0 0

With this mounted on each node that may potentially receive the registry job for scheduling, the container can always mount the data and persistence will "follow" the task since the shared storage volume is mounted everywhere. Since we include the helpful Traefik tags to indicate we want the service exposed, I can hit the registry via a plain DNS name to see my catalog of images:

shell

http -b https://registry.example.com/v2/_catalog

JSON

Font used to highlight strings.
Font used to highlight keywords.

{
    "repositories": [
        "bitwarden_rs",
        "elasticsearch",
        "ganglia",
        "kibana",
        "logstash",
        "prometheus",
        "tt-rss"
    ]
}

Now we can build and host Docker images locally with a highly-available, fault-tolerant, and distributed registry. Great!

Deploying Bitwarden

I tweeted a while back that I use Bitwarden now. It's a pretty nice self-hosted password manager that doesn't require your significant other to try and navigate the inscrutable maze that is GPG!

While the fact that you can host your own Bitwarden server is delightful, I am absolutely not about to deploy an application under my stewardship that requires a bunch of shell scripts to manage and stands up numerous behemoth containers with things like MS SQL server. Who would do that? You're much better served by deploying something will well-defined configurations and a simpler architecture that doesn't use a bespoke management procedure.

Fortunately, an enterprising OSS developer has released bitwarden_rs, a Bitwarden API server which runs as a simple, easy-to-deploy Rust API server. It persists data to a sqlite file, and that's it - the provided Docker container even bundles the web UI. I've been using it for a while, and I can vouch for the fact that it works great.

All of the preceding infrastructure I've outlined make this really pleasant to setup.

First, we need to build an image. If your cluster is on x86 CPUs, this isn't necessary, but bitwarden_rs doesn't ship an ARM container by default. That said, it isn't hard to make the Dockerfile for bitwarden_rs ARM-compatible. Here's the diff that made the Docker image buildable and runnable on an ARMv7 CPU:

Diff

‘diff-mode’ font used to highlight function names produced by "diff -p".
‘diff-mode’ font used to highlight added lines.
‘diff-mode’ font used to highlight removed lines.
‘diff-mode’ font used to highlight context and other side-information.
‘diff-mode’ font used to highlight hunk header lines.
‘diff-mode’ font used to highlight file header lines.
‘diff-mode’ font inherited by hunk and index header fonts.

diff --git a/Dockerfile b/Dockerfile
index 91e1d95..7a37a09 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -2,12 +2,12 @@
 #      https://docs.docker.com/develop/develop-images/multistage-build/
 #      https://whitfin.io/speeding-up-rust-docker-builds/
 ####################### VAULT BUILD IMAGE  #######################
-FROM node:9-alpine as vault
+FROM node:9 as vault

 ENV VAULT_VERSION "1.27.0"
 ENV URL "https://github.com/bitwarden/web/archive/v${VAULT_VERSION}.tar.gz"

-RUN apk add --update-cache --upgrade \
+RUN apt-get update && apt-get install -yq \
     curl \
     git \
     tar \
@@ -65,7 +65,7 @@ RUN cargo build --release
 ######################## RUNTIME IMAGE  ########################
 # Create a new stage with a minimal image
 # because we already have a binary built
-FROM debian:stretch-slim
+FROM rust

 ENV ROCKET_ENV "staging"
 ENV ROCKET_WORKERS=10

With that, you can build and push the container image to the private registry (or your Docker Hub account):

shell

docker build -t bitwarden_rs:master .
docker tag bitwarden_rs:master registry.example.com/bitwarden_rs:master
docker push registry.example.com/bitwarden_rs:master

Now that we have an image we can use, all that's left is the Nomad config. I mentioned previously that bitwarden_rs persists data to a SQLite file, so this is another candidate for our shared storage mount across cluster nodes. Aside from that, we'll be using the web app, so exposing a TLS-secured endpoint over Traefik will be important as well, so I include those tags in the Nomad service stanza.

HCL

Font used to highlight type and class names.
Font used to highlight variable names.
Font used to highlight strings.

job "bitwarden" {

  datacenters = ["lan"]
  region = "global"
  type = "service"

  constraint {
    attribute = "${meta.cluster}"
    operator  = "=="
    value     = "compute"
  }

  task "bitwarden" {
    driver = "docker"

    config {
      image = "registry.example.com/bitwarden_rs:master"

      port_map {
        http = 80
      }

      volumes = [
        "/srv/storage/apps/bitwarden:/data"
      ]
    }

    env {
      DOMAIN = "https://bitwarden.example.com"
    }

    service {
      name = "bitwarden"
      port = "http"
      tags = [
        "traefik.enable=true",
        "traefik.frontend.entryPoints=http,https"
      ]
    }

    resources {
      cpu    = 200
      memory = 64

      network {
        port "http" { }
      }
    }
  }
}

That's about it. The DOMAIN environment variable is specific to bitwarden_rs to enable FIDO 2FA with the application.

Figure 4: Bitwarden

Summary

After using a Docker registry and Bitwarden as an example, I hope they illustrate how awesome this setup is for deploying and running applications:

The runtime artifact is just a Docker image, and building/hosting those custom images is stupid easy. What's more, since your registry is backed by a GlusterFS cluster, I can expand the registry storage arbitrarily by adding nodes on the backend as necessary.
Traefik makes routing traffic to these containers - over TLS - almost effortless. It doesn't matter where the applications run (because my ODroid MC1 cluster has four nodes), Traefik will dynamically discover where the services run and balance requests between them.
I can show you almost the verbatim Nomad configs because my secrets get pulled in via Vault, so credentials like Route53 keys are totally handled by Vault for workloads that need them.

In the end, I've deployed maybe about a dozen services so far on my little ARM compute cluster. If there's any interest (leave a comment, send an email, etc.) I can go into more detail, but here's a sampling of what I'm hosting now:

Prometheus to collect metrics for everything - it sits atop a Gluster mount for storage, and works great with a small resource footprint. I used to use ganglia religiously, but I hit arcane error messages when I started using ganglia in my (admittedly edge-case-ish) scenarios on ARM boards and over Gluster network mounts.
Tiny Tiny RSS and a sibling Postgres deployment. This was fun to setup since I point tt-rss at the dynamic postgres.service.consul name on my network, and Consul acts as the service mesh glue to associate separate workloads on my Nomad cluster. I actually tried to implement the Vault Postgres secret backend to dynamically generate Postgres credentials on-the-fly for tt-rss, but I need to work on that a bit more. Once that's done, tt-rss will use purely ephemeral credentials to connect to my database, and I won't need to store connection credentials anywhere at all.
Logstash. Calling back to my router post and how I collect netflow data, I wanted to run Logstash on my cluster and used a similar approach to my router nginx reverse TCP proxy setup in order to send netflow data to my router's loopback address, then proxy that UDP stream out wherever logstash.service.consul is.
Elasticsearch. Yes, it's actually performant enough on ARMv7 to be a useful analytics tool. This is (another) blog post that needs to be written, but running Elasticsearch on ARM boards is non-trivial and takes some work to get reasonably performant. The tl;dr secret is to use a JVM that has native ARMv7 support - if you use, for example, a vanilla OpenJDK 8 build on Arch Linux ARM, performance will be so bad it's nigh-unusable. The eminently professional @shipilev set me straight on Twitter despite my snarky tweet, and he provided some great links to ARMv7 OpenJDK 10 builds that work fantastic with the latest ODroid boards.

Synopsis and Future Work

That's a (somewhat) high-level overview of how I've refactored my homelab from a very overloaded single server into a distributed, fault-tolerant, cheap-to-operate Nomad cluster based on ARMv7 boards (plus supporting services like Gluster and Vault). I think it's an improvement over what I had previously. Is it more complex? Sure. But when I (inevitably) hit capacity in the future - whether in compute power or storage space - expanding my homelab is almost trivially easy: add more ODroid HC2 bricks for space, or join a couple more nodes to my Nomad cluster when I need more horsepower.

This wouldn't be educational without some "lessons learned", so:

Don't try any ops projects with n servers without using configuration management for values of n greater than 1, seriously. I hand-managed my HP N40L, but after writing some very simple ansible this got a lot easier. As a sub-point, I would suggest not using config management roles you find online - just write them yourself, custom roles end up being very simple when you're not trying to support every distro version under the sun.
Run some disaster recover scenarios. My problems with Gluster disperse volumes would have been a huge pain if I hadn't pulled a few ethernet plugs to observe failure modes before putting my data on it. Contrast this with Nomad, which handled failures so well that last week one of my MC1 nodes died and I literally couldn't tell it happened for several hours because each failover piece - Nomad, Consul, Traefik, etc. - handled the member loss gracefully (loose network cable, if you were interested).
Modern software still isn't "optimize for concurrency" instead of "optimize for clock speed." ODroid XU4s have mostly-okay core clocks speeds, but they do pack eight cores into each board. Despite this degree of parallelism, many of the applications I've experimented with can struggle because they expect a fast CPU core and don't spread among multiple very well. In practice this hasn't limited me from running anything yet, and the ODroids do surprisingly well - but do not expect modern x86₆₄ clock speeds (though you will save a ton on electricity).

In terms of other interesting things I'm doing with my homelab, there are other projects in the works that may be of interest that are slated for more writeups in the future:

My monitoring story is a combination of Telegraf + Consul + Prometheus service discovery through creative use of Consul's catalog feature. It means I can monitor hosts dynamically when they register in Consul so I don't explicitly configure them for monitoring, an it works great.
One huge reason for using ARM single-board computers - lower power consumption - is something I haven't benchmarked rigorously yet. Although anectodal power measurements certainly imply it's hella cheaper than a big x86₆₄ rig, I have more plans to measure this more rigorously via my UPS and Prometheus.

Recycling bookmarks

xenodium.com @alvaro — Mon, 13 Aug 2018 03:00:00 GMT

My Plastic-free Life.

What Can You Do?

Sam Solomon — Mon, 13 Aug 2018 03:00:00 GMT

Last night a friend and I were up into the early hours of the morning talking about philosophy, life and mental health—as one does after several beers.

Extracting Entries from jrnl.com

Posts on Sean Lane — Mon, 13 Aug 2018 03:00:00 GMT

Relating to my previous post today about setting up a blog using AWS, Docker Compose, Caddy Server, and Ghost, I found the need to do some web scraping. A number of years ago, my wife began journaling her thoughts in an online service called LDSJournal.com (at least I believe that was the name). About 2 years ago, this service was acquired by a new site called jrnl.com. It seems to be a fairly neat service, but one thing we were concerned with is preserving the data should the account ever disappear.

Unfortunately, the only export option with jrnl.com seems to be the ability to download a PDF file that is created when you pay the service to have your journal entries printed physically.¹ According to that same source, there allegedly will be an option to backup the journal entries without having to purchase a physical copy, but at the moment it has been over a year since that helpdesk article promised that feature to be completed before the end of 2017. Aside from that, there could be loads of potential issues extracting my wife’s writings from the PDF file they produce, depending on how it’s put together. With that in mind, I used the following steps to retrieve her content.

In a similar manner to this article: Ian London: Web Scraping - Discovering Hidden APIs, I used the outgoing connections from the jrnl.com web application to identify their hidden API with which to access the entries. After logging into the service and navigating to the journal entries, you can view the request headers that your browser sends to jrnl.com to retrieve the entries and other content. The API key is one of these headers, with the first portion visible in the image below under Authorization.

Getting the API Key

Some more poking around showed that the base url for the API is https://jrnl.com/api/v1/, and the API endpoint for the entries is, unsurprisingly, https://jrnl.com/api/v1/entry. Using a REST API tool called Insomnia, we can plug in the API key and use the endpoint with the limit option set to allow more entries returned: https://jrnl.com/api/v1/entry?limit=250.

Then using something like the following Python script, you can convert the posts into a format for import elsewhere. This script is one I used to prepare the entries for import into the Ghost CMS platform which I set up following the instructions in my previous post. There is a little more post-processing to get it into Ghost, if you made it this far then following the Ghost documentation will get you the rest of the way. The script assumes that the entries from jrnl.com are isolated and saved as a JSON array in a file called posts.json.

#! /usr/bin/env python3

from datetime import datetime
import json, os

def millis(x):
  return int(datetime.fromisoformat(x).timestamp() * 1000)

with open('posts.json') as f:
  posts = json.load(f)

new_posts = []
for post in posts:
  temp = {
    'id': post['id'],
    'title': post['title'],
    'slug': post['title'],
    'html': post['content'],
    'image': None,
    'featured': 0,
    'page': 0,
    'status': 'published',
    'language': 'en_US',
    'meta_title': None,
    'meta_description': None,
    'author_id': 1,
    'created_at': millis(post['created']),
    'created_by': 1,
    'updated_at': millis(post['modified']),
    'updated_by': 1,
    'published_at': millis(post['entry_date']),
    'published_by': 1
  }
  new_posts.append(temp)

with open('new_posts.json', 'w') as f:
  json.dump(new_posts, f, indent=2)

http://helpdesk.jrnl.com/kb/article/150-can-i-backup-my-jrnl/ ↩︎

A simple guide to git

Jason Eckert's Website and Blog — Mon, 13 Aug 2018 03:00:00 GMT

Simply put, Git is a version control system that can be used to keep track of the changes that you make to files over time - it allows you to revert files to a previous state, or see who made changes to a file at different times if several people are working the those files collaboratively.

Git was original developed back in 2005 by Linus Torvalds (the creator of Linux) to aid in making changes to the Linux kernel, but has since become the most common open source version control system used today. You can run Git on most platforms, including Windows, Linux and macOS. Although people primarily use it today for version controlling software projects, you can use it to provide version control for any files on your system. In this blog post, I’ll use it to provide version control and central access for Linux shell scripts (something that is very useful for Linux administrators).

Refactoring with Rust macros

Phil Booth's Blog — Sun, 12 Aug 2018 15:54:12 GMT

Refactoring boilerplate code is always easy in dynamically-typed languages, but sometimes takes a bit more effort when constrained by strong typing. This is something I was puzzling over recently, when the penny dropped for me about how Rust’s macros can be used to bridge the gap.

If you have control over all of the code in question, macros probably aren’t needed of course. Some combination of generics, traits and enums would typically provide a better (and more readable) solution. But there are times when the types involved are out of your control and that is a niche which macros can thrive in.

Here is a basic example I encountered last week. Actix-web exports a TestServer struct that helps you test your endpoints. TestServer::new expects a configuration function that takes one argument of type TestApp:

TestServer::new(|app| {
    // Set up routes, resources and middleware
    // by calling methods on `app`...
})

For the production code there is a similar App type, which has methods with the same signatures as the ones on TestApp. Since the routes you want to test are likely the same as the routes on your production server, a natural next step might be to try and write a common route-setup function that works in both contexts. Unfortunately though, TestApp and App are not related. Those “common” methods aren’t inherited from some shared trait, they’re defined independently on each structure.

So in order to write a function that sets routes on either App or TestApp, you’d have to wrap them in an enum and write code to manually forward all of the method calls to the inner structs. Looking up the associated type information to get those definitions right is tedious busywork, but a simple macro allows you to skip it instead:

macro_rules! init_routes {
    ($app:expr) => {
        // All the initialisation code stays the same,
        // methods are just called on `$app` instead...
    }
}

Now you can invoke the macro from both your test setup and the production code, without it needing to know anything about the type information for $app:

TestServer::new(|app| {
    init_routes!(app);
})

The compiler will concern itself with type-checking later, after the macro has been expanded. All the macro cares about is whether its match arm is satisfied.

Refactoring types

Macros can also be used to eliminate duplication where there is no executable code involved at all, only type information.

For instance, staying with actix, let’s say you have an actor that handles some messages:

pub struct MyActor {
    // Whatever state your actor needs...
}

impl Handler<Foo> for MyActor {
    type Result = Result<HashMap<String, String>, Error>;

    fn handle(&mut self, msg: Foo, context: &mut self::Context) -> Self::Result {
        let mut result = HashMap::new();

        // Populate `result` somehow...

        Ok(result)
    }
}

impl Handler<Bar> for MyActor {
    type Result = Result<bool, Error>;

    fn handle(&mut self, msg: Foo, context: &mut self::Context) -> Self::Result {
        let mut result = false;

        // ...

        Ok(result)
    }
}

The code to define the Foo and Bar messages might look something like this:

pub struct Foo {
    pub id: String,
    pub wibble: String,
}

impl Message for Foo {
    type Result = <MyActor as Handler<Foo>>::Result;
}

pub struct Bar {
    pub id: String,
    pub blee: String,
}

impl Message for Bar {
    type Result = <MyActor as Handler<Bar>>::Result;
}

And that pattern might be repeated many more times for other message types too. Conventional refactoring is immediately off the agenda here because we only have types, properties and the Message trait to work with. But that’s all meat and drink for a macro:

macro_rules! message {
    ($message:ident {$($property:ident: $type:ty),*}) => {
        pub struct $message {
            $(pub $property: $type,)*
        }

        impl Message for $message {
            type Result = <MyActor as Handler<$message>>::Result;
        }
    }
}

With that in place your message boilerplate now looks like this:

message!(Foo {
    id: String,
    foo: String
});

message!(Bar {
    id: String,
    bar: String
});

An important introduction here was the pair of $( and ),* wrapping the declarations of $property and $type in the match arm of the macro. That denotes repetition and says the enclosed portion of the match can be repeated zero or more times, with a , separating each item. Replacing the * with a + would change that to one or more times and the , could be replaced by anything you like (including nothing at all). But , is a good choice here because it makes the macro more intuitive.

Nested macros

That’s all well and good for straightforward cases, but what about when a refactoring has many levels? Perhaps there is a core pattern to be extracted in addition to higher levels that depend on the core? This can also be achieved, although there are a couple of gotchas to be careful of.

Returning to the previous example, you’ll have noticed that Foo and Bar share a common id property. If there are no other message types, we could just move id into the macro body inline. But what if there are Baz and Qux messages, which don’t have an id property? Nested macros can help you with that:

macro_rules! id_message {
    ($message:ident {$($property:ident: $type:ty),*}) => {
        message!($message {
            id: String
            $(, $property: $type)*
        });
    }
}

id_message!(Foo {
    foo: String
});

id_message!(Bar {
    bar: String
});

message!(Baz {
    baz: String
});

message!(Qux {
    qux: String
});

Another way of expressing the same thing might have been to try and write a higher-order macro, nesting one macro_rules! directly inside another. But in this case that wouldn’t work because nested repetition is ambiguous syntax:

macro_rules! message_macro {
    ($macro:ident {$($common_property:ident: $common_type:ty),*}) => {
        macro_rules! $macro {
            ($message:ident {$($property:ident: $type:ty),*}) => {
                pub struct $message {
                    $(pub $common_property: $common_type,)*
                    $(pub $property: $type,)*
                }

                impl Message for $message {
                    type Result = <MyActor as Handler<$message>>::Result;
                }
            }
        }
    }
}

message_macro!(message {});

message_macro!(id_message {
    id: String
});

Here, it would require some magical thinking to infer that we meant for $($property:ident: $type:ty),* to be interpreted as part of the child macro’s match arm rather than the parent macro’s body. The compiler very reasonably points this out to us with the following error message:

error: attempted to repeat an expression containing no syntax variables matched as repeating at this depth
 --> src/main.rs:4:31
  |
4 |             ($message:ident {$($property:ident: $type:ty),*}) => {
  |

The second obstacle to be wary of with nesting is macro hygiene. Rust’s macros are hygienic, which means they each have their own context for expansion. An identifier introduced by an inner context may not be referenced by outer layers, instead you must pass the identifier in from outside. Even if the expansion looks like it would make sense when rolled out manually, the compiler will still complain.

More concretely, consider the following macros for defining route handlers with actix-web:

macro_rules! endpoint {
    ($handler:ident: $dispatcher:ident ($path_type:ty) {$($property:ident: $value:expr),*}) => {
        pub fn $handler(
            (path, state): (Path<$path_type>, State<ServerState>),
        ) -> FutureResponse<HttpResponse> {
            state
                .actor
                .send($dispatcher {
                    $($property: $value),*
                })
                .from_err()
                .and_then(|res| match res {
                    Ok(body) => Ok(HttpResponse::Ok().json(body)),
                    Err(_) => Ok(HttpResponse::InternalServerError().into()),
                })
                .responder()
        }
    }
}

macro_rules! uid_endpoint {
    ($handler:ident: $dispatcher:ident) => {
        endpoint! {
            $handler: $dispatcher (UidParam) {
                uid: path.uid.clone()
            }
        }
    }
}

uid_endpoint! would fail compilation here with:

error[E0425]: cannot find value `path` in this scope

Even though we know path exists because endpoint! always declares it, we still have to pass the identifier in so that uid_endpoint! is allowed to reference it:

macro_rules! endpoint {
    ($handler:ident: $dispatcher:ident ($path:ident: $path_type:ty) {$($property:ident: $value:expr),*}) => {
        pub fn $handler(
            ($path, state): (Path<$path_type>, State<ServerState>),
        ) -> FutureResponse<HttpResponse> {
            state
                .actor
                .send($dispatcher {
                    $($property: $value),*
                })
                .from_err()
                .and_then(|res| match res {
                    Ok(body) => Ok(HttpResponse::Ok().json(body)),
                    Err(_) => Ok(HttpResponse::InternalServerError().into()),
                })
                .responder()
        }
    }
}

macro_rules! uid_endpoint {
    ($handler:ident: $dispatcher:ident) => {
        endpoint! {
            $handler: $dispatcher (path: UidParam) {
                uid: path.uid.clone()
            }
        }
    }
}

Macro debugging

Something that’s obvious from the examples in this post is that macros can get pretty grawlixy and hard to read at times. If the compiler is complaining about some code in one of your macros and you’re struggling to identify the problem, it can be helpful to look at the rolled-out macro expansions. There is a compiler flag that lets you print them from the command line:

rustc -Z unstable-options --pretty expanded <MODULE PATH>

Where <MODULE PATH> is the path to the source module containing the macro you want to print.

A general rule of thumb I’ve found helpful is to try and keep lower-level macros as simple as you can, limiting all repetition to just the outermost macros where possible.

Comoro islands travel bookmarks