Hacker News Personal Blogs 2020 | All Blog Posts

My favourite books of 2020

Gaganpreet Arora — Thu, 31 Dec 2020 23:34:38 GMT

Working and living from home gave me a lot of free time that I could dedicate to reading. I read 40 books this year. I usually read fiction, however this year I aimed to read a mix of fiction and non-fiction.

A smart guide for your career as a software engineer

www.mootoday.com - RSS Feed — Thu, 31 Dec 2020 21:29:05 GMT

Do you want to earn a six figure income, work from anywhere, live a lifestyle of your choosing and be a part of the people who develop the next generation software applications? Are you a software engineer already, but want to change jobs or advance in your current role to get promoted?

New Year's Resolution 2021

Syd Norpol — Thu, 31 Dec 2020 18:48:02 GMT

Well, here is something more personal and also something that has been kind of a tradition for me: publish my New Year’s Resolution on my blog.

I’ve spent the past couple of days occasionally thinking what my NYR’s could be… My incentive behind writing those down is that I’ve honestly had success in the past following them (in contrary to what many other people say about theirs) and also it helps me focusing on something I want in my life.

start psychotherapy 😒
read 12 books (spoiler I’ve only read one in 2020) 🤓
start studying at a university (remote or in person, I live close to TU Berlin)
get my driver’s license 🚗
get my digital life organized 💽
learn reading/playing piano notes 🎶
become self-organized (calendar, ToDo, notes)
more intimacy 💜
stand up for myself 💪🏼
get pierced 💎
try contact lenses 👀
finish three projects 🙈
add more art in my life 🎨

Besides these 13 things I’m of course hoping to keep this blog active and to making new internet friends.

Soul

🦉 olshansky 🦁 — Thu, 31 Dec 2020 14:08:05 GMT

Olshansky's review of Soul

Object Detection at 2530 FPS with TensorRT and 8-Bit Quantization

Machine Learning Articles on paulbridger.com — Thu, 31 Dec 2020 08:43:23 GMT

This article is a deep dive into the techniques needed to get SSD300 object detection throughput to 2530 FPS. We will rewrite Pytorch model code, perform ONNX graph surgery, optimize a TensorRT plugin and finally we’ll quantize the model to an 8-bit representation. We will also examine divergence from the accuracy of the full-precision model.

Sex workers must go POSSE

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 31 Dec 2020 07:00:00 GMT

And no, I don’t mean “a large group often with a common… interest”.

Implementer's Guide to SOCKS

Cookie Engineer's Web Log — Thu, 31 Dec 2020 02:00:00 GMT

Tutorial on how to implement a SOCKS version 4/5 client and server from scratch, explaining related RFCs and potential quirks along the way.

2020 Year in Review

Sam Solomon — Thu, 31 Dec 2020 02:00:00 GMT

What a wild year 2020 has been. Even those of us that were closely monitoring the pandemic had no idea how it would change things.

My Favorite Privacy Apps and Services

Nelson Figueroa — Thu, 31 Dec 2020 02:00:00 GMT

My favorite privacy-respecting tools, apps, and services.

2020 in 13 Seconds

huphtur — Thu, 31 Dec 2020 02:00:00 GMT

Right after we moved into our new house, I took a photo with my iPhone every morning at 8 am. So, 300+ photos later, here’s a timelapse in ~13 seconds:

https://youtu.be/kHypqsFAysU

The sound is the final track of The KLF’s Chill Out album but reversed (thanks AudioMass) for extra dramatic effect.

Highlights

December 3. The first photo I shot. Freezing cold morning with frost on the roofs.

March 11. The neighbor’s maple tree is now in full bloom, temperatures have risen a bit. Spring is here!

March 14. What spring? It’s snowing! Crazy to see the foliage bend under the weight of the snow. A week later we will bend under the pressure of COVID-19 and enter lock-down.

July 1. SQUIRREL! Gotta zoom in a bit to spot it, but it’s sitting on top of the roof.

September 12. The entire west coast of America is on fire. Crazy thick smoke, and even though our house is well insulated, we still have some smoke seep in.

November 25. Pretty much the end of the year, a seagull flying away…

Lowlights

iPhone was not always aligned straight, so images jump around. Messed around in Affinity Photo for a couple of hours to get them aligned somewhat better. Wish I had used a tripod instead.
Forgot about daylight savings nonsense. Kept shooting at 8 am, so shadows jump around.
There are gaps in the sequence. Traveling (pre-pandemic), and when I forgot to take photos because life got busy. Oh, and that one time when I almost died from eating too much licorice (more on that in a future blog post).
2020

Underusing the NHS

Krishna's blog — Thu, 31 Dec 2020 02:00:00 GMT

A personal reflection on the author's experiences with the UK's National Health Service (NHS) and the challenges of overusing and underusing healthcare.

Dynamic Programming (i): Multiplying Matrices

Alex Molas Blog — Thu, 31 Dec 2020 02:00:00 GMT

Learning how to multiply matrices in the best possible order, ie solving the Matrix Chaing Multiplication problem.

My Path to Developer Relations and Thoughts on the Future of DevRel

swyx's site RSS Feed — Thu, 31 Dec 2020 00:06:02 GMT

Alex Lakatos interviewed me for the Devrel Advent calendar he put together

My Journey from No CS Degree to AWS from Age 30 - 34

swyx's site RSS Feed — Wed, 30 Dec 2020 23:21:47 GMT

I was interviewed by Pete from No CS Degree on my journey, here are some extracts!

Swyx Interview on Daily Dev Tips

swyx's site RSS Feed — Wed, 30 Dec 2020 23:04:35 GMT

I was interviewed by Chris Bongers on Daily Dev Tips, here are some extracts.

Financialization of Big Tech is BIG

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 30 Dec 2020 20:00:00 GMT

It really seems Too Big To Remain.

The Road Chose Me Volume 2: Three years and 54,000 miles around Africa

The Road Chose Me — Wed, 30 Dec 2020 05:04:54 GMT

I’m extremely proud to announce I’ve published my second major book in print! ‘The Road Chose Me Volume 2’ documents my entire three year African expedition from start to finish. It covers the adventures,...

Python write_file utility

Home on Pinecoder blog — Wed, 30 Dec 2020 02:00:00 GMT

Recently I wrote a python script for syncing data to a 3rd party service. I needed to output various text files during its run, both for debugging and audit history. Ideally, I wanted a simple function which would allow the following: # Set some base directory via config working_dir = 'data' # Pass filenames and contents, with any directories # automatically created as necessary: # should output to: data/job_2/stats.csv write_file('/job_2/stats.csv.', raw_policy_data) # should output to: data/job_2/applications/results.

The ideal blog platform

nutcroft — Wed, 30 Dec 2020 02:00:00 GMT

WordPress, Tumblr, Blogger, Medium, Ghost. Posthaven, Svbtle, Write.as.

Blogging. The most elegant and natural process of idea diffusion on the web—since its infancy. Started barebones; developed massively; evolving continuously.

I’m excited about the next step. Lately, in my social bubble of technology and startups, I notice a revisited focus to simplicity and minimalism. It’s probably part of a constant cycle, regardless, now it exists. In addition to that, there is a much wider, across the world, appreciation for controlling one’s data.

Towards expressing this second part, it’s great if a blogging platform is open source. It’s also a good idea for a blogging web citizen to control their web estate: the blog domain. One can do that with a lot of blog platforms. To account for simplicity and minimalism, though, the most unwasteful and efficient way to run one’s blog is to serve static HTML. That is, to utilise a static site generator, rather than running a backend, a database, and all that entails. It’s a much simpler, more efficient, more elegant, and significantly faster way of web existence.

Self hosted blogs

How to empower people to create their own self-hosted blog? There are some ideas. Drew instated the Make a blog initiative. Stavros made the Quick Site guide. My idea was mataroa.blog, a blogging platform that makes it easy to get off it.

There is a certain difficulty in starting to blog. It is inspiration that lights the fire, and inspiration is not on tap. I remember when I wanted to start a blog or write an article, I would dive into all the technicalities (re-styling, updating, refactoring, etc) and eventually not write anything.

Mataroa.blog is designed for when inspiration comes. This is the time to just write, focus on the essence and get the words out. When done writing, there is lots of uninspired time to think about the best way to design, rework, speed up your blog. Mataroa enables one to do that by having export as a first-class feature. One can export into simple markdown text files, or Zola sources, or Hugo sources with a single click. One can also set a redirect domain, so that their mataroa blog automatically redirects to a new blog home.

The greatest benefit of blogging platforms is that you can start writing immediately. No domain decisions, no servers, no coding. This comes at a cost, though: potential lock-in.

To account for this and the aforementioned need for simplicity, the features I’d like to see in an ideal blog platform are:

absolutely no vendor lock-in
it even helps you get off it
open source
simple
minimal
fast
focus on the essence
follow open standards such as RSS

Mataroa.blog is designed with all these in mind.

But, actually, I don’t want to sell mataroa to you; as much as I want to convince you on the merits of self-hosted personal blogs.

The foundational platform of the web—the web itself—is more democratic than any medium (wink) has ever been—at least until now.

Let’s build directly on it without intermediate layers.

Addendum on hosting

During the last 10 years of massive web centralisation, there have been instances when the internet giants fall. AWS, Google, Cloudflare—bad days exist for them too.

It’s funny to see your blog up when Google is down. Especially when the only reason is the choice of an indie cloud/server/hosting provider¹. I used to depend on EC2 for servers, Netlify for CDN, Google for fonts, other CDNs for JavaScript libraries. Now, this blog loads fully from a single server with just two small requests, one HTML and one CSS file. So much fun!

I have found Hetzner to be pretty reliable during the last couple of years. I think it’s a good recommendation, though, I’m sure they have bad days too. ↩

BOINC dans un conteneur Docker

Accueil on dbeley — Wed, 30 Dec 2020 01:00:00 GMT

J’ai déjà écrit un article sur ce qu’était BOINC sans en détailler l’installation. L’installation classique de BOINC passe par le téléchargement du client BOINC, mais il est aussi possible d’utiliser BOINC dans un conteneur docker.

Cette méthode d’installation est plus facile à scripter et à automatiser, ce qui la rend très intéressante sur un serveur headless par exemple.

Installation de docker

Tout d’abord, installons docker et docker-compose. Sur une distribution linux utilisant le gestionnaire de paquets apt (comme Debian ou Ubuntu), il suffit d’installer les paquets docker.io et docker-compose.

The globalization of evil autocompletion

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 29 Dec 2020 20:00:00 GMT

Beware of cliche' search engines!

A Modernist Christmas

mass:werk – Now Go Bang! — Tue, 29 Dec 2020 17:30:00 GMT

Festive architecture & optimism

Single Sign-On and Resource Separation on AWS

Home on The Log Book of Manuel Kießling — Tue, 29 Dec 2020 13:50:00 GMT

Introduction Most AWS projects start small and simple, with just a single AWS account that contains all the resources and services to build, deploy, and run one or more software projects. Up to a certain point, this is just fine. And even with just one account, resources can be isolated from each other – for example, you can run the EC2 instances of your preproduction stage in a different VPC than the instances of your production stage.

Kendall Peak Lakes Snowshoe

Sam Hooke — Tue, 29 Dec 2020 04:20:00 GMT

Distance	8.59 km / 5.34 miles
Elevation gain	236 m / 773 ft
Duration	4h25m
Moving	2h25m

We were mostly following the Kendall Peak Lakes Trail which starts at Gold Creek Sno-Park, though didn’t plan to actually reach the lakes. The main trail was quite popular, so we followed a side trail instead, which offered good views of the ski resort (smothered in mist) and inquisitive wildlife (camp robbers after our lunch). A tiny waterfall by the first switchback showcased some elaborate icy sculptures.

Swift package code coverage (plus Emacs overlay)

xenodium.com @alvaro — Tue, 29 Dec 2020 02:00:00 GMT

While playing around with Swift package manager, I had a quick look into code coverage options. Luckily, coverage reporting and exporting are supported out of the box (via llvm-cov).

Ensure tests are invoked as follows:

swift test --enable-code-coverage

A high level report can be generated with:

xcrun llvm-cov report .build/x86_64-apple-macosx/debug/FooPackageTests.xctest/Contents/MacOS/FooPackageTests \
      -instr-profile=.build/x86_64-apple-macosx/debug/codecov/default.profdata -ignore-filename-regex=".build|Tests"

llvm-cov can export as lcov format:

xcrun llvm-cov export -format="lcov" .build/x86_64-apple-macosx/debug/FooPackageTests.xctest/Contents/MacOS/FooPackageTests -instr-profile=.build/x86_64-apple-macosx/debug/codecov/default.profdata -ignore-filename-regex=".build|Tests" > coverage.lcov

With the report in lcov format, we can look for an Emacs package to visualize coverage in source files. Found coverlay.el to require minimal setup. I was interested in highlighting only untested areas, so I set tested-line-background-color to nil:

(use-package coverlay
  :ensure t
  :config
  (setq coverlay:tested-line-background-color nil))

After installing coverlay, I enabled the minor mode via M-x coverlay-minor-mode, invoked M-x coverlay-watch-file to watch coverage.lcov for changes, and voilà!

Hiking bookmarks

xenodium.com @alvaro — Tue, 29 Dec 2020 02:00:00 GMT

A growing list of long distance hikes around the world (Hacker News).

LitElement Web Components in React App

Thoughts by Thomas Derflinger — Tue, 29 Dec 2020 02:00:00 GMT

LitElement is a Web Component library by Google. It succeeds the Polymer library and strives for speed and compatibility with the Web Components standard. How to integrate LitElements into your React application and use event handling, I want to show in this article.

Things I worked on in 2020

λm.me — Tue, 29 Dec 2020 02:00:00 GMT

In this post I'd like to look back and mention a few (programming related) things I worked on in 2020.

Accurate Estimations

Cam Hashemi — Mon, 28 Dec 2020 18:15:04 GMT

We’re constantly asked to give estimations:

How long will it take?
How much will it cost?
What time should we leave?
How much do you want?

Estimations are information about the unknown. We constantly use this information to make decisions: allocating resources, changing strategies, and choosing partners. But despite all this practice, we’re horrible at accurate estimations.

Why Estimations are Hard

Estimations are hard for both technical and social reasons.

We don’t know what we don’t know. Naive estimators fail to account for surprises. They estimate based on known factors and best-case scenarios. These estimates may be perfectly accurate beforehand, but they’re instantly broken by the first surprise.

Experienced estimators account for this problem by ‘adding some buffer’. But even then, how much should they add? Even knowing that surprises can happen, it’s impossible to know how many will happen or the impact of those surprises. Choosing the right amount of buffer is a lot like making the right estimation in the first place. We still don’t know what we don’t know. And adding too much buffer can be as expensive as failing to account for surprise at all.

In addition to this technical problem, there’s a strong social problem. Let’s imagine two common scenarios.

In the first scenario, you’re giving an estimate to your team. You perfectly estimate that a project will take three weeks; but your manager gives you a puzzled look. Your teammate snickers, claiming they could do it in a week, tops. The feeling is that you must be either lazy or incompetent to give such a padded estimation. Your newly shortened estimate is wrong, so you go on to extend the project’s deadline twice in three weeks. Rather than holding you accountable to that one-week estimate, your manager commends you for being able to handle the unforeseen surprises on such a complicated project.

In the other scenario, you’re giving an estimate to a potential client. You perfectly estimate that a project will take three weeks; but your competitor only estimates it’ll take a week. The client signs with your competitor. Since their estimate was wrong, your competitor goes on to extend the deadline twice in three weeks. Even though your estimate was accurate, your client’s estimate got them paid.

I’ve been in countless scenarios like this. Sometimes people outright pressure us into shortening our estimations, and sometimes the voice in our head pushes us to. Either way, giving accurate estimates is both technically hard and socially challenging ¹.

Two Types of Estimators

In response to this hard problem, we become systematic underestimators or systematic overestimators.

Underestimators fail to give enough buffer. This strategy has two key benefits. First, it signals (unrealistically) high performance. Like our virtue-signalling teammate, we can underestimate ahead of time, then point at concrete surprises for our eventual underperformance. And like our overpromising competitor, we can underestimate during a bid and do whatever we want after the contract is signed. Second, tight estimates demand efficiency. Underestimators set deadlines that they and their teams must work hard to meet. Underestimation works well when the costs of going over-budget are small. But when those costs are large, underestimations lead to disasters. On the whole, underestimators systematically run the risk of being burnt out, past-deadline, and over-budget.

Overestimators are instead biased towards large buffers. Extreme overestimators might send you articles titled “Estimations are a Scam”, or claim that estimations are simply tools for worker exploitation. Overestimation works when the costs of buffer are low and the costs of going over budget are high. But overestimators are constantly taxed by Parkinson’s Law. Parkinson’s Law is the pattern where projects fill the time and resources they’re allocated, instead of the time and resources they need ². Rather than pushing towards peak performance, overestimators systematically move at a bored, leisurely pace. Overestimators are also demotivating. Rather than inspiring the team to reach competitive goals, they disparage those who do. So while underestimators run the risk of their teams burning out, overestimators run the risk of their teams shutting down.

This line between underestimators and overestimators forms a classic “spectrum problem”. A spectrum problem occurs when we oversimplify the solution to a given tradeoff. In this case, we’re splitting the spectrum of estimation strategies in half. Underestimators fall on one side of the line, and overestimators fall on the other. With many spectrum problems, the better solution is to cut the spectrum into three pieces, choosing the middle strategy between extremes. In doing so, we acknowledge the costs and rewards of both sides, maximizing the upsides and minimizing the downsides systematically.

Playing Single-Pointed Darts

Imagine a game of darts with very simple rules. I throw my dart first, and you only score by hitting that same exact dart-sized point. This game is very simple, but so difficult that nobody would play it. To make darts playable, we specify scoring ranges. These same ranges are missing from our everyday estimations.

The most common estimate sounds something like: “I’ll have it ready by 5pm.” But this is just like playing single-point darts! Imagine that this estimate is perfectly precise: the project can’t be ready one second before or one second after 5pm. While impressive, there’s zero room for error. If we end up sick, or if the project ends up more complex than expected, our estimate becomes instantly wrong.

To account for surprises, we can add buffer. But adding buffer only shifts the dart board over a few inches. “I’ll have it ready by 5pm tomorrow” faces all the same problems as the first estimation. Two days of surprises still makes me wrong. We’re still playing single-pointed darts.

It’s a losing game. And yet we see it played again and again, day after day, project after project.

Playing Darts with Ranges

To enjoy this game of darts, we need to change the rules. Rather than giving a single-pointed estimate, we give two points: one for the best case, and one for the worst case. These two points create a range of targets to hit, just like the game of darts we know and love.

To demonstrate, I can give an example from my personal life. My girlfriend is very punctual, and I’m not. She’s a classic overestimator, giving as much buffer as we can afford. And I’m a classic underestimator, giving the most optimistic estimates. So whenever we have somewhere to be, and she asks me “what time should we be ready?"… it’s a classic estimation problem!

Recently, I’ve started giving her two answers. The first answer is “the green time”. The green time tells us when we should leave so that we’re pleasantly early. The second answer is “the red time”. The red time is the last minute we can leave, without definitely being late. If we can be ready by the green time without stress or shortcuts, that’s perfect. Once we pass the green time, we go into the “yellow zone.” The yellow zone is the buffer between early and late. There’s no need to take shortcuts or change plans yet, but we’re getting close. Once we approach the red line, we start discussing the need to take shortcuts, or to start telling others that we may be a little late.

Having a green time, a yellow zone, and a red time transforms our game of single-pointed darts into a proper dartboard, with zones of success. The green time makes my girlfriend happy: she can be ready early without worry. The red time makes me happy: I can fill my buffer time up with other activities. And the yellow zone is a signal for both of us to get focused or to start taking shortcuts ³.

Ranges make uncertainty explicit. Adding buffer doesn’t just shift the dartboard a few inches, it expands our range of accuracy. Larger buffers signal more uncertainty and require less precision, while smaller buffers signal more confidence and require more precision. The expression of certainty and confidence isn’t possible to communicate with a single-pointed estimate.

By estimating with two numbers instead of one, a richness of new information and strategies become available to us.

Simple Changes

Using two numbers instead of one, these so-called “confidence intervals” aren’t complicated. Then why are they so absent from everyday life?

Although nothing prevented us from discovering them earlier, the first mention of confidence intervals in scientific literature wasn’t until 1937. And it wasn’t until the 1980s that they were required in scientific journals. So if it took forty years for the most knowledgeable people to apply a simple solution towards the most urgent problems, it’s not surprising that it’s taken the rest of us at least as long. That said, the goal of this essay is to speed up that process.

We can move towards accurate estimations by practicing two simple rules. The first: When giving an estimate, state the best- and worst-case answers. The second: Whenever getting an estimate, ask for the best- and worst-case answers.

“It’ll take two weeks” should raise a red flag. A better answer sounds like: “in the best case, one week; in the worst case, three”. Boom! Now we have a range of days to deliver the project, instead of one. Not only do we have a picture of the optimal scenario, we have an idea for the hidden risks lurking in the future.

As a bonus, we’re capable of using the “green line, yellow zone, red line” imagery to guide our actions ⁴. The green line is how we strive for the best and signal our competence. The red line is our insurance policy in case of surprises. The yellow zone is where time is critical, and we need to coordinate in case our plans need to change.

In a world of uncertainty, this extra byte of information is enough to create more accurate thought and action.

Footnotes

Though questionable, accurate estimations are worth the effort. First of all, accurate estimations build trust. After a few false promises, people catch on. Like compound interest, commitments founded on years of accrued trust are worth much more than short-term deals based on deceptive estimates. Secondly, inaccurate estimations kick off a vicious cycle of degrading quality. Underestimations lead to insufficient resources, insufficient resources lead to shortcuts, shortcuts lead to poor quality, and poor quality leads to greater consumption of resources. These teams work harder and harder to do less and less. Inversely, overestimations lead to too many resources, too many resources lead to inefficiency, inefficiency leads to overestimations. These teams take longer and longer to do less and less. ↩︎
Parkinson’s Law is the insight that projects tend to consume the resources given to them, be it time, money, or people. Fear of Parkinson’s Law motivates people to justify tight deadlines. They claim that without tight deadlines, people tend to get lazy or unfocused.But if Parkinson’s Law was the whole story, we’d never see projects go over budget! But since we see projects go over deadline and budget all the time, adjusting deadlines can’t be the only factor. Instead, keeping deadlines as tight as possible leads to burnout, in addition to all the other costs of underestimating. Though useful for timeboxing projects in general, Parkinson’s Law should not be used as the sole guiding principle for estimations. ↩︎
In professional projects, the same concepts apply a bit differently. The green time is the earliest a project can be ready, satisfying the underestimators; and the red time would be the latest a project can be ready, satisfying the overestimators. Underestimators can still push for the green time, while overestimators still find peace of mind in knowing the red time exists. ↩︎
“One week, give or take two days” in a common way to convey a similar amount of information. I focused on the “best case / worst case” framing, because it opens the door to the green / yellow / red style of project management. That said, two answers are always better than one, so if adding a “give or take” is an easier way to make this transition in how we estimate, I’m all for it. ↩︎

Plan Your Year

trv.rs — Mon, 28 Dec 2020 16:33:00 GMT

This will be the fourth year my wife and I purchased and used Plan Your Year from The Focus Course. This year I am going paperless… I even spent some time this morning scanning in all the previous year's editions.

I really like this tool. Some quick thoughts:

I love that this is a system designed to be used as a family to build a shared vision.
It's encouraging to read back over these as an overview. Lots of reasons to praise the Lord for his faithful hand.
That which we planned for… largely happened. Even in 2020!
Any month we didn't have a design for… fell into default behaviors. Good or bad. (Mostly bad.)
Plan Your Year does a great job of balancing personal growth vs. accomplishing projects. Both are critical.

Shawn and the team at The Focus Course built a great tool and wisely modified this version for this specific and stressful year… flipping through its pages I'm excited to learn and grow with my little family, and I recommend it to y'all, especially if you don't have a shared vision for investing your time and talents in 2021.

node.example.com Is An IP Address

Reilly Tucker Siemens — Mon, 28 Dec 2020 16:13:05 GMT

Hello! Welcome to the once-yearly blog post! This year I'd like to examine the most peculiar bug I encountered at work. To set the stage, let's start with a little background.

Working at Amazon

The Technical Blog of James on purpleidea.com — Mon, 28 Dec 2020 15:27:35 GMT

TL;DR: I’ve got a new job working for Amazon, in the “Open Source Program Office”. I’m now an Amazoner! More information below…

The Amazon. Source: https://en.wikipedia.org/wiki/Amazon_rainforest

Background:

A little over three months ago, I decided to get back into full-time work. For the past 2.5 years since I left my job at Red Hat, I’ve been mostly working full-time on Mgmt Config and my DevOps/SWE Mentoring Program. To do such things, I had to live off of my savings, which isn’t sustainable long-term. I am thankful that I was able to do so for so long. It was time to dive back in, and here I am.

jlink's Missing Link: API Signature Validation

Gunnar Morling — Mon, 28 Dec 2020 13:20:00 GMT

Table of Contents

The Example
The API Signature Check jlink Plug-in
Summary

Discussions around Java’s jlink tool typically center around savings in terms of (disk) space. Instead of shipping an entire JDK, a custom runtime image created with jlink contains only those JDK modules which an application actually requires, resulting in smaller distributables and container images.

But the contribution of jlink — as a part of the Java module system at large — to the development of Java application’s is bigger than that: with the notion of link time it defines an optional complement to the well known phases compile time and application run-time:

Link time is an opportunity to do whole-world optimizations that are otherwise difficult at compile time or costly at run-time. An example would be to optimize a computation when all its inputs become constant (i.e., not unknown). A follow-up optimization would be to remove code that is no longer reachable.

Speedrunning Computer Games History with a 6yr Old - Part 1

duncanlock.net — Mon, 28 Dec 2020 08:13:51 GMT

Figure 1. Galaxians.

As a family gift for Christmas 2020, I set up a Raspberry Pi 400 with Retropie. The plan is to load a selection of the “best of the best” games from computer game history, starting with games up to, roughly, 1985.

The kid doesn’t really know any better, so we can play Space Invaders unsullied by time and expectations and enjoy a speedrun through gaming history, playing just the highlights.

I’ll add more games as we go along, progressing through gaming history, one classic game at a time.

Why?

Figure 2. The Mighty BBC BASIC prompt. This is what you saw when you booted it up - straight into an interactive programming environment: a 1980s REPL. Just type in BASIC code/commands and press Enter.

When I was around 8 yrs old, my parents bought a BBC Micro Model B - an early 8-bit home computer. Home computers were very new back then and I’d never used one before. We played games on it. A lot. Asteriods, Frogger, Missile Strike, Twin Kingdom Valley, Racer, Repton, Chuckie Egg, Elite, etc, etc… all loaded (very slowly) from cassette tape. There were also lots of games and other programs that came as code, printed out in books or magazines, which you then typed in.

Later, I started writing my own little BASIC programs. I think partly because that was what Mum & Dad actually wanted us to do with this rather expensive computer, and partly because it was interesting to get the computer to do things - and gratifying when they finally worked.

Like many others, this little machine changed the trajectory of my life. I’ve been fortunate to have made a reasonable living out of programming computers. It’s still interesting to get computers to do things that you thought up - and it’s still gratifying when it finally works, even after all this time.

I have no illusions that we can re-create this experience - you can’t step into the same river twice. However, I hope that we might be able to capture a few of the best bits, the most fun & most useful, while making new memories of our own.

I think that the 6yr old is currently in a narrow window where this might be possible. They haven’t really had much exposure to computer games, and the COVID-19 pandemic will keep them out of their peers houses for a while longer. But, because kids these days are surrounded by advanced computers, with modern 3D graphics and sound, I can’t leave it until they’re 8 or 9 - the window will have closed and modernity will have forever changed expectations.

I think that I’ve got about a year or so, at the most, before Minecraft and Roblox sweep this all away and close the window. I’m going to see if we can get to 1992 before then - it’s around this time that I think the games will start to stand up on their own, graphics & fun wise, against Minecraft. Maybe? Alright, probably not. We’ll see how it goes - either way, it’ll be fun while it lasts.

It’s also around that point that the Raspberry Pi 400 will start to struggle to emulate the more demanding games & systems, so we’ll have to do something else then anyway.

What do you have against Minecraft/Roblox?

Absolutely nothing, I think they’re great! The problem is, once you play modern games with modern(-ish) 3d graphics, your expectations are forever changed. It makes it harder to appreciate games which don’t have similarly modern graphics. This isn’t a fault of older games, it’s just a change in expectations - it’s hard to put that toothpaste back into the tube. When we get there, I hope we’ll play Minecraft & Roblox together too - and maybe use them to introduce coding.

A Modern “Home Computer”

I’m intending the Raspberry Pi 400 to be “the family/kids/home computer” for a while:

Featuring a quad-core 64-bit processor, 4GB of RAM, wireless networking, dual-display output, and 4K video playback, as well as a 40-pin GPIO header, Raspberry Pi 400 is a powerful, easy-to-use computer built into a neat and portable keyboard.
— https://www.raspberrypi.org/products/raspberry-pi-400/

We can use it to play games on - but it’s also a regular Linux box, running Raspbian (a Debian distro) and can run a huge array of Linux software. It’s easy to change its personality but just booting it off a different USB stick/SD card, with a different OS & apps on. I’m intending to use it to introduce the kids to programming, as well as basic electronics and robotics - all of which are well-supported by the Pi’s hardware and huge maker community.

Hardware

We live in a pretty small apartment, so everything needed to be able to pack away as small as possible, without breaking the bank. The combination of the Raspberry Pi 400, which released at just the right time for this project, and a portable monitor - means that the system packs away fairly neatly.

Figure 3. The assembled system, with Raspberry Pi, monitor and controllers, sitting on an Ikea Lack table. Underneath the table are two boxes and a small pouch; the system packs away into these when not in use.

The system itself is a Raspberry Pi 400
The monitor is a Portable 15.6” TN Screen Full HD 1920×1080
The controllers are some retro NES style USB ones. I actually bought two pairs, these ones and some Buffalo ones from this awesome seller on eBay.

The monitor arrived without the bent metal wire stand, so I ended up making some decorative stands out of cardboard and covering them with retro game artwork:

Figure 4. These stands came out pretty well, if I do say so myself - and they’ve unexpectedly been a hit with the kid, too. I often get “Which game is this?” or “I want to play this one” - pointing at one of the pictures on these stands.

Software

It’s running Retropie, which is a nice retro gaming system, which makes installing and configuring everything pretty simple:

[Retropie] builds upon Raspbian, EmulationStation, RetroArch and many other projects to enable you to play your favourite Arcade, home-console, and classic PC games with the minimum set-up. For power users it also provides a large variety of configuration tools to customise the system as you want.
— Retropie

I used the Retropie image to install this, which means you just need to write the image to an SD card and boot it up. You can use Balena Etcher, the Raspberry Pi Imager, or dd to do this. Installation instructions are here.

The games were selected from my memory, and various lists:

We started with the following emulated systems & games:

Arcade

These all work flawlessly in lr-mame, with perfect controller mappings out of the box.

Galaga ‘88 (1988)
Galaxian (1979)
Pac-Man (1980)
Frogger (1981)
Qix (1981)
Defender (1981)
Qbert (1982)
Donkey Kong (1981)
Centipede (1981)
BurgerTime (1982)
Arkanoid (1986)

ZX Spectrum

The Zesarux emulator is a pretty great, but the speccy graphics are a bit basic, and they’re mostly intended for keyboard play, so we have to remap them for the controllers.

Bubble Bobble (1987)
Tetris (1988)
R-Type (1988)
Manic Miner (1983)
Knight Lore (1984)
Jet Set Willy (1984)
Head Over Heels (1987)
Dizzy (1987)
Chuckie Egg (1983)
Chase H.Q. (1989)
Chaos - The Battle of the Wizards (1985)
Atic Atac (1983)
Deathchase (1983)

Atari 2600

Pitfall! (1982)

Apple II

This is a keyboard oriented emulator, so controller mapping, etc… Also, the sound is pretty bleepy.

Lode Runner (1983)
Choplifter (1982)
Cavern Creatures (1983)
Where in the World is Carmen Sandeigo (1985) - this has two floppy disks and it’s annoying to swap disks in the emulator. Also text heavy.

Commodore 64

This emulator doesn’t seem to work very well and the games are difficult to get working & figure out controls. Consequently, we really haven’t played any of these so far. I’m sure it’s just a matter of me sitting down and figuring out how to configure it properly.

Zak McKracken & the Alien Mindbenders (1988)
Wizball (1987)
Winter Games (1985)
Turrican 1 (1990)
Turrican 2: Final Fight (1991)
Turbo OutRun (1989)
The Sentinel (1986)
M.U.L.E. (1983)
Manic Miner (1983)
Maniac Mansion (1987)
Lemmings (1991)
Laser Squad with Expansion Missions (1988)
International Karate + (1987)
Boulder Dash (1984)

We should probably play the LucasArts games on ScummVM anyway.

ZMachine

These are all text adventures, so require reading & writing.

Zork I (1980)
Zork II (1981)
Zork III (1982)
Starcross (1982)

I’ll report back later…

I realize that’s too many games, but… I got a bit carried away? Probably we won’t get around to playing all of these before we pass them by for newer pastures, but that’s life.

So, we’ll try it out and see how we get on – and I’ll report back later on our progress.

Horrors and errors of SOME independent book sellers

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 28 Dec 2020 07:00:00 GMT

And what they tell us, of course.

Wenatchee Crest Snowshoe

Sam Hooke — Mon, 28 Dec 2020 04:30:00 GMT

Distance	12.26 km / 7.62 miles
Elevation gain	222 m / 728 ft
Duration	5h25m
Moving	2h50

We left Seattle early, but due to the long drive across the Cascades it was already 10:30am by the time we set off.

I had read that the views from this hike were spectacular, but there was a thick mist as far as we could see. This was quite atmospheric and felt like being lost in a cloud.

Patience

xenodium.com @alvaro — Mon, 28 Dec 2020 02:00:00 GMT

Via Orange Book, a reminder to myself:

In investing, patience is rewarded.
In growing a talent, patience is rewarded.
In building a business, patience is rewarded.
In love and friendships, patience is rewarded.
Patience = success

I feel like there's an Emacs lesson somewhere in there…

Futureproof

Matt Keeter — Mon, 28 Dec 2020 02:00:00 GMT

A shader editor in Zig, WebGPU, and Neovim

The Pragmatic Programmer - Review

Luke Singham — Mon, 28 Dec 2020 02:00:00 GMT

📚 Book: The Pragmatic Programmer

⭐️ Rating: 4/5 Sagely advice that is written in an engaging and highly consumable form.

The Pragmatic Programmer is a very good book full of practical advice. It's written with little Tip boxes which are the tl:dr; of a particular section. Plus, there's a healthy sprinkling of Dad jokes throughout the book 😁.

Unlike, a Mythical Man Month, I would recommend this to seasoned professionals as well as newbies. Though this also has a lot of content for the seasoned professional, it is much easier to parse. Instead of having long anecdotes about long since dead technology to glean the lesson, this book is to the point, with helpful tip boxes, exercises and the examples are mostly relevant to today.

❗️ Some criticisms of the book: #

Tip 17: Program close to the problem domain #

Whilst I agree with the principle, the accompanying pages advocate for the creation of mini languages in order to get closer to the domain. The project manager senses in me (I am not a project manager) can see this quickly becoming a rabbit hole distraction where developers end up on an academic pursuit of these mini languages, the necessary language parsers, documentation etc. The suggestion to create languages for all makes me cringe,

There's the end user, who understands the business rules and the required outputs. There are also secondary users: operations staff, configuration and test managers, support and maintenance programmers, and future generations of developers. Each of these users has their own problem domain and you can generate mini-environments and languages for all of them.

I've not observed anyone creating a mini-language during a project. What I think is more practical and occurs in real-world projects is creating packages of functions whose scope is limited to a particular domain. Such that the ontological representation, the digital recreation of real world concepts happens with a set of classes and/or functions that map to the domain.

Chp 3 - The Basic Tools #

This chapter discusses tools that a pragmatic programmer should have in their tool kit. Inevitably the technological tools have changed in the intervening years. Well, Emacs and Vi are still around, but the recommended plugins etc. are out of date. If you don't have a good toolset already, I'd recommend going through MIT's The Missing Semester of Your CS Education.

✅ Major (Useful) Points of the Book #

⚖️ Involve Your Users in the Trade-Off #

It's quite a common occurrence for technical people when asked by a stakeholder to feel anxious or pressured to provide an estimate of when something can be done on the spot. For a particular data request, this might mean involving the stakeholder in the tradeoff between how accurate vs how quick the deliverable's turnaround is.

🍸 DRY #

Don't repeat yourself (DRY) is a commonly held maxim of programming. The Pragmatic Programmer exposits the common causes for duplication. This can arise from multiple places, project requirements, multiple target platforms, multiple programming languages being used in one system, laziness etc. A programmer may have to duplicate the schema of a database table to represent it as a class in their program. Documentation in code as comments and as a separate accompanying document. But these can be duplicative of each other.

good code has lots of comments

☝️ A slight digression from DRY, but this go-to book for the industry is in favour of lots of comments in code. I've come across a few developers who like to argue the code is documentation. But code will never tell you why the code has been written this way, or the trade offs considered in choosing a particular approach. I've never heard anyone complain 'there are just too many comments in this codebase'.

With all rules there are exceptions and trade-offs with DRY.

📐 Orthogonality #

Tip 13: Eliminate effects between unrelated things

I know orthogonality as modularity or decoupling. A great example of modularity is the way unix command line was designed with lots of modular tools that you can chain together using | pipes.

There are two main benefits of orthogonality increased productivity and reduced risk.

🎯 Tracer Bullets #

Better known these days as 'agile development', 'MVP' and 'iteration'. Tracer Bullets are discussed over several pages. Most people could probably skip this as the doctrine of Agile has mostly replaced Waterfall.

🤔 Estimates #

Tip 18: Estimate to avoid surprises

Involve your users and ask if they need high accuracy or a ballpark figure for the estimate. First understand the problem space, this will involve working with your user to understand not just what they are asking, but what their actual problem is. Then whiteboard what the system is you will build to solve it. If this is large enough a system, then estimate how long each component could take to build. Keep track of this estimate so you can track the accuracy of your estimates. As you go through each project iteration (sprint) update the estimate based on any change in delivery schedule that may now be apparent.

A fun fact from a footnote:

What is the value of π? "3"... if you are a legislator. In 1897, Indiana State Legislature House Bill No. 246 attempted to decree that henceforth π should have the value of "3".

Lastly, when asked for an estimate you say, "I'll get back to you".

🐛 Debugging #

This is a good chapter and great if you're new to tech. The first rule of debugging:

Tip 25: Don't Panic

I've done this, and seen others do it. But it really is counterproductive. Take a deep breath 🧘‍♀️, and approach it with the Zen of a master. This will help avoid the myopia of jumping on fixes for the symptoms that you immediately see rather than taking a step back to assess the bigger picture here. This is something I usually call, developing hypotheses of the root causes or, what are the possible solution paths we could go down - it's best to get a range of options before jumping down one that could prove fruitless. Instead, rank the hypotheses in order of most likely, then proceed with the highest ranked hypothesis.

Various debugging tips #

Reproduce the bug on your machine or a dev environment.
Visualise your data. Log it to a console, use code breaks.
🐥 Rubber Duck with someone else - if they have the time, doing this at the outset can help develop more hypotheses as to what is causing the problem and get a better assessment on which is most likely.
If there's not an obvious place to start, apply the principle of binary search.
Tip 27: Don't assume it, prove it, this can help avoid putting fixes in place that aren't really fixes. This can happen because your dev environment is different to prod. Or you accidentally were on the wrong branch of code.

📃 Design by Contract #

The book talks about Java Object Oriented examples. However the principle applies to all code. Does your code return the expected data with the expected types? Does the logic do what it promises to do? Include assertions, tests and document assumptions. An example of the principle applied to a data pipeline where each model is some Spark or SQL code A -> B -> C. The final select of model A uses SELECT * or SELECT *EXCEPT(x,y,z) this is not enforcing model A's side of the contract. When model B consumes from model A, if similar shorthand SELECT * is applied, you can end up with a minor code fix that introduces a new column, changing the schema of tables flowing right through this simple pipeline all the way to model C. This can be easily avoided by explicitly naming the columns to be returned such that downstream models won't accidentally have their schemas changed by an upstream model. Even better, enforce the contract when you select columns from an upstream data model.

Tip 33: If it can't happen, use assertions[or tests] to ensure that it won't

Tip 36: Minimise coupling between modules

🎛 Design for Concurrency #

The book uses the example of making a cocktail and all the tasks that are required from 'open blender' to 'get pink umbrellas'. If you analyse all of these tasks, some are dependent on others to complete. But many tasks are able to happen in parallel to another task. Diagrams can help design for concurrency.

🧹 Refactoring #

Code needs to evolve. It is not static. And unfortunately many people perceive software as done once it is delivered. Software is more like gardening - you might start a gardening project with a plan, plant seedlings according to the plan, but later some plants didn't like the soil others have overgrown. You monitor and regularly tend to the garden to keep it at its best.

When to refactor? #

These are common qualifiers:

Duplication
Highly coupled components
Outdated knowledge i.e. does the code still reflect the real world problem
Performance

Tip 47: Refactor Early, Refactor Often

If time pressure is used as an excuse from management, explain technical debt is like a cancer that grows and is riskier and more time consuming to remove the longer you leave it.

How to refactor? #

Don't try to refactor and add functionality. Leave TODOs if you must.
Make sure you have good tests before starting to refactor, otherwise you may not know when you have broken something.
Take short deliberate changes and test them. Rather than large sweeping changes all at once.

📚 Requirements #

To understand what the user needs:

Tip 52: Work with a user to think like a user

If you have the problem of too many requirements, then point out how much each suggested requirement is going to increase project delivery by.

🧪 Testing #

Test Early. Test Often. Test Automatically.

Unit Tests - is code that exercise a particular piece of code in isolation.
Integration Tests - shows that the major subsystems that make up the project work and play well with each other.
User Testing - does it meet the users expectations?
Performance Testing / Stress Testing - does it scale? Is it going to crash as soon as 100 users are using it at the same time?
Usability Testing - This is about UX. It might pass user testing but does it feel like the extension of the users hand? Does it feel more like an Apple product or another product that technically does the same but just feels awkward.

Notes #

I'd recommend the entirety of chapter 6 While you are coding - particularly the 'How to program deliberately section'
One of the simplest explanations of O notation is in this book.
There's a great section in Chp 8 Pragmatic Projects on team organisation advocating for teams to be organised around functionality which also reflects the decoupled principle.

Art Exercise: Lupine Forest Cabin

Code & Visuals — Mon, 28 Dec 2020 02:00:00 GMT

1. Introduction
2. Initial Blocking
3. Forest Floor
4. Making Moss

5. Filling out the Clearing
6. Mist and Atmospherics
7. Final Lighting
8. Conclusion

9. References
10. Footnotes

Introduction

I recently did a small personal art exercise to experiment with building out a detailed forest type of environment. I have worked with a detailed forest scene before, when I used one while I was developing Takua Renderer’s mipmapping and texture caching system, but in that case I didn’t actually make the scene, I just took an off-the-shelf scene and ported it to Takua. I also made a forest for the background of my Woodville RenderMan Art Challenge piece, but to be honest, I wasn’t very happy with how that turned out. I think the background forest in my Woodville piece looks decent when viewed from a distance, as it is seen in that scene, but it doesn’t come even remotely close to holding up when viewed close up. Also, that forest was incredibly difficult to author. For the Woodville piece, I chose Maya’s MASH toolset over the XGen toolset to create the background forest; I chose MASH because MASH generally feels much more native to Maya and doesn’t have all of the extra file management that XGen requires. However, I found that MASH has performance problems when the things being instanced have really heavy geometry and there are a lot of instances, as is the case when instancing trees and bushes and the type of stuff generally found in forests. For this exercise, I wanted to focus on building a forest environment that would be detailed enough to hold up at closer distances, which means including things like leaf litter and twigs and small rocks on the ground, displacement-based bark on trees, moss on various surfaces, and so on. I also wanted to find a more efficient way to author these kinds of environments.

This post is essentially a collated version of my notes from this project, published here mostly because having them all in one place is useful for me to refer back to later but also in case anyone else finds this stuff interesting. Before I go into the details of how this project went, here’s what the final result looks like:

Initial Blocking

I didn’t really come into this project with much of a plan on what the final image would look like. To be honest, the goal of this project wasn’t even necessarily to make the most visually or artistically interesting image; the goal was primarily a technical exercise to practice building a complex scene. I actually started not with forest related things at all but instead I started with just trying to build an interesting groundplane. My plan was to use as many off-the-shelf premade assets as possible and focus on assembling everything together into a nice scene and focus on final lighting. Quixel Megascans has a lot of amazing large-scale 3D scans of rock formations and cliff faces; I had used a few of these back on the Woodville RenderMan Art Challenge and had made a note to myself back then to play with the Megascans large-scale rock formation assets more later. I started with placing a bunch of Megascans rock formations to get a sense of initial shaping for the scene. The overall shaping I went with for the scene draws from the principle of “compression and release”- I wanted the foreground to be through a narrower passage opening up into a larger space in the background, which I think helps give a sense that we’re looking into a larger world:

The next step was to put in a groundplane of dirt and soil. Since at the time RenderMan didn’t have hex-tiling [Heitz and Neyret 2018, Burley 2019] yet, I used the same technique that I used on the Woodville RenderMan Art Challenge to hide tiling in the groundplane textures: the groundplane texture is made up of several different tiled soil and dirt textures, projected using PxrRoundCube with different rotations per texture, blended together using a couple of different noise projections. One useful trick I added in this time was to also use the displacement maps to drive which texture “wins” at each blend point; this allows for nice things like larger rocks poking through layers of dirt and stuff.

Overall though I didn’t put too much effort into the groundplane since I expected it to be almost entirely covered with vegetation in the final scene, meaning very little of it would be directly visible. After getting the groundplane to a good-enough state, I fired up Maya’s XGen Geometry Instancer and started roughing in what all of the vegetation in the scene would look like.

The XGen toolset that ships with Maya today is really two essentially completely different toolsets packaged into a single plugin- XGen Interactive Grooming, and XGen Geometry Instancer. XGen Geometry Instancer was forked from Disney Animation’s internal XGen instancing system over a decade ago, while XGen Interactive Grooming was then added some years later [Todd 2013] as a totally separate authoring interface for hair workflows. For this project, I only used the Geometry Instancer half of XGen. I’ve had a fair amount of experience with Disney Animation’s version of XGen, so using the Maya version of XGen Geometry Instancer has been an interesting experience- the version in Maya was fairly heavily reworked to be more integrated into Maya compared with Disney Animation’s standalone version, and the two versions have since diverged even more due to many years of independent development and evolution, but there are still clearly a lot of Disney-isms in Maya’s XGen, such as the use of SeExpr [Disney Animation 2011] and Ptex [Burley & Lacewell 2008] to control instancing distributions and properties.

The basic workflow for Xgen Geometry Instancer is to first create a bunch of archives of geometry that the user wants to instance, and then create a description, which is a set of instances of the archives scattered across some area of geometry, called a patch. When the underlying geometry that XGen instances are being scattered on is a polygon mesh, patches bind to the faces of the mesh. Multiple decriptions can be organized together as a collection. Something that is a bit frustrating about XGen Geometry Instancer is that its integration into Maya is… somewhat strange. Maya’s XGen Geometry Instancer doesn’t really feel or behave like a native part of Maya- XGen descriptions and collections show up in the Maya outliner, but are edited through a dedicated separate interface instead of the standard Attribute or Channel editors, and instead of storing all data as part of the Maya file, XGen Geometry Instancer creates and depends on a bunch of external sidecar files, and even worse, paths baked into these external files don’t respect the project workspace system that Maya uses, making reusing XGen description between different Maya projects kind of annoying to do. All of this is the result of XGen Geometry Instancer being derived from an internal standalone tool at Disney Animation- the internal version of XGen makes a lot of sense integrated within the overall Disney Animation pipeline, but none of this was originally designed to be a deeply integrated part of Maya outside of Disney Animation, and even with all of the work that Autodesk put into making the commercially available Maya version of the system better integrated, the outside origin still shows through. However, I found on this project that XGen Geometry Instancer is nonetheless still super powerful and for huge numbers of instances is vastly more performant than the native Maya MASH instancing toolset.

The first vegetation test I did was to put down some basic grass, scattered using a super basic XGen Description setup where the base geometry used for scattering was just the groundplane itself. The grass is made up of just a couple of off-the-shelf grass models from Evermotion and Megascans, reshaded for the project’s needs, exported as XGen archives and scattered with random rotations applied per instance:

Next, I started blocking in trees since this scene was meant to be a forest scene. Originally I had planned on scattering trees using XGen as well, but I wound up placing the trees by hand. Part of the reason for placing by hand was simply that there aren’t that many trees in this scene, but the primary reason was just for more direct control since the trees play a really large role in establishing the overall framing of the scene.

The base spruce tree models I used are from Evermotion, but I opted to reshade them using the Evermotion textures as a starting point but then improved using some stuff from Megascans. The main reason for reshading was to improve how well the trees would hold up to being close to the camera, which requires more detailed displacement on the tree bark. I also put some work into making sure the spruce needles transmit light in a convincing way, which required some initial reverse-engineering of how the tree’s original Vray materials work in order to understand how to get a similar effect in RenderMan’s PxrSurface material, and then required further tweaking and adjustment to both the spruce needle textures and the material.

Here are a couple of lookdev test renders of one of the spruce trees; the first two show the overall look of the trees with my reworked shading, and the last one shows the close-up detail and also how the spruce needles look with backlighting:

One interesting thing I ran into was RenderMan’s opacity caching system. The opacity caching system stores an opacity value per micropolygon, under the assumption that micropolygon sizes in world space should be close to one micropolygon per screen-space pixel. As long as the one-micropolygon-per-screen-pixel assumption holds more or less true, this opacity caching approach works well; the average opacity value across a single screen-pixel sized micropolygon should match what the filtered mip-mapped texture lookup for opacity should be across the micropolygon, therefore, caching this value on the micropolygon allows for skipping a lot of texture lookups over the course of the render. However, as soon as micropolygons are larger than a single screen-pixel, the assumptions that allow for opacity caching to work without visual artifacts break, because the frequency that opacity caching can store values at drops below the frequency that the filtered mip-mapped signal from texturing represents; the net visual result is blurring in opacity edges, which can look super strange.

PxrSurface in RenderMan for Maya enables opacity caching by default, but for the spruce needles on the trees, I had to disable opacity caching. The spruce needles on the trees are modeled as a single quad strip per bundle of needles with opacity mapping; I left the quad strips unsubdivided to keep memory usage lower, but this meant that spruce needle polygon sizes were often way larger than one pixel in screen space for trees closer to the camera. Before doing this project I didn’t realize that RenderMan has an opacity caching system and enables it by default, so figuring out why the spruce needles looked all messed up took me a moment. This wasn’t a problem in the Woodville project because in that scene, all of the trees are in the background and therefore are so far from camera that all unsubdivided quad strips are subpixel in screen space anyway.

Here are the spruce trees blocked out along with the initial grass and rock formations. While nothing here is as close to the camera as in my lookdev tests, nonetheless the nearest foreground trees are close enough to the camera where the reworked high resolution displacement mapping and exact opacity evaluation on the spruce needles instead of cached opacity start to matter a great deal:

Forest Floor

At this stage, with conifer trees and large rock formations, the scene was starting to remind me of trips to Yosemite (even though technically Yosemite’s conifers include pine, fir, and sequoia but notably not spruce). I’ve seen lupine blooms in Yosemite before, which struck me as really visually interesting since large fields of blue-purple flowers feel somewhat unusual and unexpected normally. So, to make the foreground in this scene more interesting than just a grassy plane, I added in a couple of layers of lupine flowers to make a purple lupine meadow. The lupine flowers are placed as a couple of different overlapping XGen descriptions, with varying clustering rules per description. I also carved a footpath through the grass and flowers by painting a simple density mask.

I also realized here that the overall massing of the scene was all vertical but in a horizontal framing, which was starting to make the scene feel claustrophobic to me. Having the rock formations be completely vertical also didn’t make sense to me for how they probably would have been formed in real life; a completely vertical cut through rock cliffs like that feels like it would have had to been man-made. So, I tried sloping the rock formations back to give more of an impression of a small valley or something that could have been carved glacially or by water erosion or something. Sloping the rock formations back also helped bring some more light into the foreground of the scene. Here is a comparison of what the scene looked like at this point with vertical rock formations versus sloped rock formations:

Figure 9: Blocking in lupine flowers and carving out a footpath through the grass and flowers, with vertical rock formations (left) versus sloping rock formations (right). For a full screen comparison, click here.

One nice thing about XGen Geometry Instancing is that it treats driving properties through painted masks as a first-class citizen, as opposed to the MASH toolset where using textures as masks is a lot more annoying than I feel like it should be. In my experience, MASH encourages procedural placement or hand placement using the MASH Placer tool over using painted masks to drive everything; using a painted mask only really works well with MASH’s World node, but the World node in turn doesn’t always play well with everything else in MASH’s toolkit; in this sense I really prefer how easy using painted masks is in XGen. However, in XGen, painted masks also come with a minor complication- one Disney-ism that remains in XGen to this day is that XGen expects painted masks to all be Ptex files, which means the masks either have to be made using a paint tool that natively supports Ptex, or means that masks created as regular UV textures need to be converted to Ptex. Within Disney Animation, which has a native Ptex workflow, this is not a problem at all, but in a vanilla Maya workflow in the outside world, this Ptex dependency adds some friction and an additional layer of file juggling to the XGen workflow if one is more used to a UV-based painting workflow.

Here is what the painted density mask for the footpath looks like in Maya’s viewport. Since leaving this mask pretty coarse was fine for the use case, I just painted it directly in Maya using the built-in simple surface paint tool:

Having just the lupine flowers and base grass looked kind of monotonous, so I also threw in some small shrubs here and there to help visually break up the fields of purple. In the areas where the groundplane show through because of the footpath, I also added several more layered XGen descriptions to scatter various small bits of ground debris that I pulled from Megascans; the debris includes things you’d expect from a conifer forest, such as sticks and twigs and clusters of dead conifer needles, in addition to some pebbles and small rocks and stuff.

With all of the debris on the groundplane, even as instanced geometry, the scene was starting to get really heavy to render, so as a small optimization I took the density mask for the footpath through the lupine flowers and grass and simply inverted it and applied it to the debris XGen descriptions. This way, there’s only debris where it’ll actually be visible. Putting all of this together got the scene to a point where I think the groundplane was looking pretty good:

Here’s a top-down view of what all of the footpath debris looks like in the final version of the scene (I’ll get to the moss and volumes next). The nice thing about using instanced geometry instead of just adding decals to the groundplane texturing, or even adding displacement, is that the instanced geometry provides much more of a feeling of depth, especially in all of the areas where cavities and holes are visible under piles of dried needles:

Making Moss

In order to add a bit more detail to the scene, I wanted to add moss to some of the tree trunks and rock formations and stuff. On the Woodville piece, the vast majority of the mossy surfaces in the scene are made using a purely textured approach, with only a small amount of geometric moss in specific places. For this scene, I wanted to move to a 100% geometry based moss approach. Geometry based moss simply holds up better close to camera, and even far away from camera, geometric moss with proper transluscency allows for much more realistic lighting responses completely automatically.

I made the pieces of moss by just grabbing a bunch of Megascans atlases of moss bits and cutting out each piece from each atlas as its own little card. The cards aren’t simple rectangles; each card is shaped to cut as closely to the actual moss shape as possible while still maintaining a very low triangle count. Using simple rectangular cards means that the vast majority of each card is actually going to be opacity mapped out, and in most raytraced renderers (including RenderMan), somewhat counter-intuitively we’d rather have a few more triangles to deal with in exchange for cutting down opacity map lookups. The reason is because triangle intersection using a BVH acceleration structure means that the runtime cost of intersecting N triangles is typically less than N times the cost of a single triangle, but having to either interrupt BVH traversal to evaluate an opacity map in-line or having to evaluate opacity as part of shading and then generate continuation rays for hits shading points where opacity is less than 1 is really expensive in the aggregate. Here’s what some of the moss cards looked like:

At rendertime each moss card is actually subdivided and displacement mapped as well, so the total number of triangles per moss card actually can end up being relatively high, allowing for more detail. This card-based technique produced several dozen unique pieces of moss, which when combined with some random hue shifts in the material and some flipping and scaling was more than enough to create the impression of tons of unique moss bits when instanced and scattered.

To get an initial sense of how well this approach would work, I did a small-scale test on a single rock before expanding to the whole scene, using a simplified version of the rock for the underlying geometry to grow the XGen descriptions from and using a simple density mask painted directly in Maya:

One small problem with using XGen Geometry Instancer with Megascans assets is that XGen can be somewhat difficult to use with super high-resolution meshes. The reason high-resolution meshes get tricky to use with XGen comes back to XGen needing to bind a patch to every face in the mesh; with a really high-resolution mesh, this can result in a ton of patches, and since Maya and XGen are not particularly well multithreaded in some operations, have a ton of patches in XGen can result in really slow update times as Maya loops over and updates all patches. XGen in Maya does have a multithreading option, but I found that even with this option enabled, really high-resolution meshes still bogged down instance generation a lot. So, my workaround was to take all of the meshes that I wanted to grow moss on and create highly decimated versions to serve as base geometry for XGen.

Below is a comparison of what the full-resolution geometry that I wanted moss on looks like in the viewport, versus the decimated version of that geometry for XGen. Note that the decimated XGen base geometry obviously isn’t visible in the final render; XGen allows for seprarately hiding the underlying base geometry versus hiding the generated instances.

Figure 15: Full-resolution geometry that I wanted moss on (left) versus the decimated version used as base geometry for XGen Geometry Instancer (right). For a full screen comparison, click here.

For the moss density masks, I reached for Substance Painter instead of just painting in Maya using the basic built-in paint toolset. Painting the masks in Substance Painter made getting in nice bits of detail and complexity much easier. In order to make the Substance Painter workflow slightly easier, I also re-UV’d the decimated meshes to try to reduce the number of different UV islands; this step was much easier to do on low-resolution decimated meshes than it would have been on the full-resolution geometry!

Here’s what the painted masks for the “hero” foreground tree trunks look like in Substance Painter:

In order to use UV textures as masks in XGen, the textures have to be converted to Ptex. Maya’s paint tool actually has functionality for this; when using the paint tool, there’s an option to save and load from UV textures, which XGen then converts to Ptex upon hitting the map save button in the XGen UI.

I also used Substance Painter to paint the moss density masks for all of the rock formations and cliff walls. The viewport screenshot shows that the final resolution of the density masks actually isn’t that high, but in this case that wound up being completely okay:

One downside of using low-resolution decimated geometry as the base to grow XGen instances from while keeping high-resolution geometry for rendering is that there can be mismatches between where XGen instances are and where the high-resolution surface is; this mismatch means instances can either float in space or be embedded too far down in the high-resolution surface. XGen Geometry Instancer allows for specifying a displacement map on the base geometry to help correct for this, but in this case I found that the mismatch didn’t really matter. To address floating geometry I just applied a universal offset back along the surface normal, and for something like moss, pieces being embedded too far down in the high-resolution surface just look like shorter pieces of moss.

Here is a test render of what the moss looks like on just the smaller foreground rocks, which I did to confirm that at least for this use case, mismatches between the low-resolution surface used for XGen and the high-resolution render surface weren’t important:

Rendering everything together at this point produced the following image; things were starting to look pretty good at this point! I think the moss really helps with adding in a ton of additional detail, which in turn makes the whole scene look and feel more believable. In retrospect I suppose I should have adjust the moss so that it only grows in areas that stay in the shade longer, but even without taking that additional real-world factor into account, I think the moss looks pretty convincing:

This scene really shows the importance of having a robust instancing system in a production renderer- with all of the XGen instances added in, the scene at this stage took up about 50GB of memory in RenderMan for geometry alone, but 50GB is actually not too bad when considering that the total number of visible triangles is around a billion. According to RenderMan’s stats breakdown, of that 50GB of geometry memory, around 33GB is used just for storing instancing information! Most of the triangles just belong to instanced geometry, which is crucial to keeping geometry memory under control.

I ran some analytics to count up the total number of XGen instances (debris, grass, lupine flowers, and moss) in this scene; the total tally is 11,433,11. Here’s a comparison of what the final viewport looks like with only non-instanced geometry, and with every single XGen instance drawn as a blue bounding box on top:

Figure 20: Final viewport with only non-instanced geometry (left) and with every single XGen instance drawn as a blue bounding box on top (right). For a full screen comparison, click here.

The original stated goal of this project was to learn to create forest detail that could hold up well when viewed close up, but admittedly the closest tree in the final composition for this project is only in the mid-ground at best. I wanted to see how ell the moss would hold up when viewed closer, so after I finished the rest of the project, I set up an alternate camera much closer up to one of the trees. Here’s what the final version of the scene looks like with the alternate close-up camera. It’s far from perfect; in order to really look convincing this close up, I think the moss actually needs to be even smaller and even denser, but at least at a glance I think it does an okay job of holding up:

Filling out the Clearing

To finish building the scene, I wanted to put something in the clearing that the footpath leads to. Because I started this project with no real plan whatsoever, I was a bit stumped at this point what to put in the clearing! One initial idea I played with was throwing in a hot air balloon- I figured that a hot air balloon could provide a visually interesting splash of color into the background. I actually fully carried out this idea, including adding in some emissive volume flames for the hot air balloon’s burners. I also filled out the background with some more trees:

While I think the hot air balloon concept wasn’t the worse idea, I decided against it after seeing the hot air balloon fully in-context with the rest of the scene. Visually the balloon stood out, but I felt that it stood out too much and distracted from the overall composition. Also, in terms of the internal logic of the scene’s world-building, I thought that the balloon didn’t make a whole lot of sense- the clearing is pretty small, which when combined with all of the surrounding tall trees, makes for a hazardous landing site for a hot air balloon! I moved on to trying to find something more appropriate to the scene both visually and logically. I did keep the additional trees though.

Since the scene’s original layout principle of “compression and release” comes from the architecture world, I figured it would be fun to turn this scene into a bit of an architectural scene (which tend to be my favorite type of scene to make anyway). Evermotion released a wonderful 15th Anniversary scene earlier this year that includes two small cabins; I thought that something along the lines of those cabins would fit the bill well for the clearing in my scene. I was looking for something that would be small and relatively unassuming, but also still be visually interesting; I didn’t really have any preference for what the style of the building would be, but something super modern felt like a nice change of pace after working on the Woodville project. I didn’t want to just grab one of the Evermotion 15th Anniversary cabins and drop it into the scene as-is though. To keep things more visually interesting, I wanted to be able to see into the cabin more than is possible in the vanilla Evermotion 15th Anniversary cabins, and I also wanted the cabin to look cozy and inviting on the inside. To build the cabin for this scene, I kitbashed together parts from the two Evermotion 15th Anniversary cabins and made the front of my cabin one gigantic glass sheet:

To give the giant front window a bit more realism, I added a subtle amount of unevenness and wobbliness to the glass sheet, much like how giant glass sheets on skyscrapers often have small amounts of visible unevenness. In the following test render, note the back window behind the staircase; the wobbliness of the front glass is most noticeable by the effect it has on how the back window’s straight frame looks:

To integrate the cabin in the scene, I expanded the footpath further to reach the cabin and expand out into a small sort of pad surrounding the cabin. One major mistake I made in putting together this scene though was that I didn’t keep track of the scale of anything, so finding the appropriate scale for the cabin to sit well in the scene took a few tries; for future projects I’ll have to keep track of everything’s scale more closely.

Overall I think compared with the hot air balloon idea, the cabin does a much better job of integrating into the scene without overly drawing attention to itself while still being visually interesting:

Much like in the Shipshape Art Challenge project, I used PxrSurface exclusively as the shading model for everything in this project. However, for the cabin, using PxrSurface became kind of cumbersome because the shader parameterization that Evermotion assets come with doesn’t match up directly with PxrSurface’s parameterization. Specifically, Evermotion assets typically come with Vray, Corona, or generic “PBR” materials. Corona and generic “PBR” materials are the easiest to adapt to other renderers because they both follow the roughness/metallic/basecolor workflow that was originally introduced by the Disney BRDF [Burley 2012] and has since become the de-facto norm across many popular and widely-used physically based shading systems [Karis 2013, Kulla and Estevez 2017, Georgiev et al. 2019, Häussler et al. 2020, Noguer et al. 2021, Stockner 2022, Andersson et al. 2024]¹. PxrSurface, however, notably does not follow the roughness/metallic/basecolor parameterization [Hery et al. 2017]; roughness is the same in PxrSurface as in pretty much every other modern physical shading model, but instead of metallic, PxrSurface uses a combination of face color, edge color, and fresnel exponent to control how metallic a material looks. In order to convert metallic/basecolor maps into something PxrSurface could use, the approach I took was:

\[diffuseColor_{PxrSurface} = baseColor_{PBR} * (1-metallic_{PBR}) \]

\[specFaceColor_{PxrSurface} = baseColor_{PBR} * metallic_{PBR} \]

\[specEdgeColor_{PxrSurface} = 1 \]

When creating textures from scratch using Substance Painter, as I did for the Shipshape project, PxrSurface’s parameterization isn’t an issue because Substance Painter has built in export templates to save out textures in PxrSurface’s expected parameterization even though Substance Painter uses a standard metallic/basecolor workflow. When bringing in existing metallic/basecolor textures though, the above conversion has to be done within the material’s node network. I had to wire up a small node subnetwork to do this for every single Evermotion material that had metallic, which got really old really quickly. At least as of RenderMan 23 (the current version while I was working on this project), RenderMan doesn’t have any utility node to do this conversion from metallic/basecolor to PxrSurface’s diffuse/specular face/specular edge color paramterization²; such a node would be really helpful! Alternatively, a full implementation of the Disney BSDF [Burley 2015] (as opposed to the reduced-paramter Disney BRDF that is already available) would be useful too since a full implementation of the Disney BSDF would both be able to accept roughness/metallic/basecolor parameterized inputs and provide the full range of abilities that a production-quality shading model needs.

Mist and Atmospherics

With the surface geometry and shading for the scene complete, the last things to do were atmospherics and final lighting. Over the last two RenderMan Art Challenges, I’ve evolved my atmospherics and volume rendering workflow a lot. For the Woodville Art Challenge, I relied on a combination of homogeneous mist and heterogeneous fog made procedurally using Maya’s volume noise nodes plugged into the density field of a PxrVolume. This workflow was simple, but ultimately challenging to control and slow to iterate on since seeing what anything looked like required doing a full render. For the Shipshape Art Challenge, I moved to an entirely VDB-based volumes workflow, even for mist and stuff. Using an entirely VDB-based workflow allowed for more direct control, and because RenderMan for Maya can preview VDBs in Maya’s viewport, I was able to get a good sense of how volumes were placed and how they massed together in the scene without having to wait for full renders.

For this project, the volumes workflow was similar to the one on the Shipshape Art Challenge; everything is VDB-based, even the background haze. In making the mist and atmospherics in this scene, I knew I wanted to break things down into three distinct types of volumes: background haze, foreground godrays and atmospherics, and some mist low down clinging to the ground. I broke out and rendered these three types of volumes in their own passes/layers to give more control during compositing.

Here is the background haze layer, which is mostly uniform but does have a small amount of variation that creates a bit of a cascading effect:

For the main foreground godrays and atmospherics layer, I use similar types of volumes as in the background haze layer, but with a different lighting scheme. The main lighting setup for the entire scene is basically just a single super high-resolution skydome IBL with a sun baked in. However, I wanted the godrays to be more prominent than could be achieved by just using the skydome IBL as-is. For for this layer, I created a separate distant light with an orientation matched to the IBL’s sun position, cranked up the exposure of the distant light, and twiddled with the size (angle) of the distant light until the godrays became more visually prominent:

The low ground-hugging mist is made up of VDBs with higher-frequency detail than the previous two layers. There are only a handful of unique VDBs here, but I instanced them around a bunch using native Maya instancing to cover the entire valley floor. I really like how the low mist settles down in between the lupine flowers and floods the footpath area:

Finally, here’s what the three volumetrics layers look like stacked together, with the per-layer adjustments used in the final composite. One thing that I think is kind of neat about looking at the composited volumes together is that we can actually clearly see the shadow cast by the cabin; this detail isn’t actually necessarily clear in any of the volumes layers in isolation, but becomes apparently obvious once all three are stacked since each layer brings something different into the whole:

The clouds in the background in the final render are just part of the skydome IBL; in the future I’m hoping to build background cloudscapes from scratch, but that’ll have to be for a future art exercise.

Final Lighting

The final lighting setup for this scene was super simple; effectively the whole scene is lit by just a simple sun + skydome IBL setup. My goal was to keep the lighting as naturalistic as possible, so a simple sun + sky setup fit the bill, and even with the simple setup, the complexity of the forest geometry still provided a lot of interesting areas of highlight and shadow. For final lighting I did split out the sun as a distant light from the skydome IBL, much like what I did for the volumetrics passes, and I painted out the baked-in sun in the skydome IBL. The reason for splitting out the sun in this case wasn’t to adjust its brightness independently from the sky though, but instead was to allow for slightly adjusting the color temperature of the sun without affecting the fill provided by the sky. The sun is just a touch warmer in the final lighting than it was in all of the in-progress renders up to this point.

There are a bunch of practical lights in the cabin as well; the interior of the cabin is actually lit only using the practical lights and contains no other stage lighting. I initially calibrated all of the practical lights against the sun’s brightness, but that wound up making the interior of the cabin a bit too dark to be visible in a bright sunny day, so I cheated and applied a uniform exposure boost to all of the cabin’s practical lights to brighten up the inside of the cabin.

Putting everything together with the volumetrics produced the following main daylight version of the scene. I also did a small amount of color correction in Lightroom, but in this case the color correction was limited to some basic contrast and saturation adjustments. The color correction for this piece wasn’t nearly as extensive as what I did for the Woodville and Shipshape projects. Here is the final piece, followed by a 50% grey clay shaded version with the same final lighting, comp, and color correction:

I had some fun with making some alternate time-of-day versions of the final lighting setup. These were quick and just for fun, so the sun angle is the same as in the daylight version even though in reality that would make zero sense. I kept the sun angle the same in the variants since they use the same volumetrics passes, just color corrected differently. These variants were made quickly on a whim, which is why I didn’t put in the additional effort to redo everything to work with more plausible sun angles.

Here’s a late afternoon golden hour variant, followed by a 50% grey clay shaded version. A lot more of the look of this variant was achieved by extensive color correction in Lightroom. I think this variant shows off the godray volumetrics better than the main daylight variant, even if the angle of the godrays and by extension the angle of the sun don’t make much sense for golden hour:

The other variant I played with is a cooler overcast morning. A lot of the look of this variant also comes from Lightroom, possibly even more so that for the golden hour variant. I also boosted the contribution of the low ground-hugging mist in this variant to make it more visible, since low mist is typically characterstic of overcast mornings. I left the godrays in, even though having godrays at all on an overcast morning is fairly implausible; again, these two variants were quick and just for fun.

Here’s the overcast morning variant, followed by a 50% grey clay shaded version:

Conclusion

I learned a lot from this project; the goal of the project was to figure out how to put together a detailed forest environment that holds up well at relatively close camera distances, and I think the end result accomplishes that well enough. The final image in this project is decent- it’s not the most narratively interesting image, but it makes for a passable archviz type piece, but to be honest a nice looking image was somewhat secondary to the goal of figuring out how to build a scene like this. Maya’s version of the XGen toolset makes building this type of scene relatively easy and the XGen toolset is super powerful, even if it doesn’t feel particularly native to the rest of Maya. XGen certainly feels a lot more scalable for this type of project than MASH, which I’ve run into performance problems with on scenes much smaller than this one. At the same time though, in my opinion, Maya’s version of XGen isn’t nearly as easy to use and isn’t nearly as featureful as tools like ForestPack for 3ds Max, which remains the gold standard for instancing/scatter tools, or the Geo-Scatter plugin for Blender, or Houdini’s native toolset.

At the end of the (metaphorical) day, I had a lot of fun making this piece! The process was much more experimental and aimless compared with working on RenderMan Art Challenge pieces, which was a nice change of pace for an art project. I learned a bunch of techniques that I’m sure I’ll make use of again in the future, and I got in more practice, which is ultimately the most important part of getting better at anything. I’m really happy that I’ve more or less figured out a reliable way of making good trees now, given how much time I spent a long time ago trying to figure out how to make decent trees.

Finally, here is a progression video I put together from all of the test and in-progress renders that I made throughout this entire project:

Lupine Forest Cabin Progression Reel

Figure 36: Progression reel made from test and in-progress renders leading up to my final image.

References

Zap Andersson, Paul Edmondson, Julien Guertault, Adrien Herubel, Alan King, Peter Kutz, Andréa Machizaud, Jamie Portsmouth, Frédéric Servant, and Jonathan Stone. 2024. OpenPBR Surface Specification. Academy Software Foundation white paper.

Autodesk. 2020. XGen Geometry Instancer. Autodesk Maya 2020 User Documentation.

Autodesk. 2020. XGen Interactive Grooming. Autodesk Maya 2020 User Documentation.

Brent Burley and Dylan Lacewell. 2008. Ptex: Per-face Texture Mapping for Production Rendering. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering) 27, 4 (Jun. 2008), 1155-1164.

Brent Burley. 2012. Physically Based Shading at Disney. In ACM SIGGRAPH 2012 Course Notes: Practical Physically-Based Shading in Film and Game Production.

Brent Burley. 2015. Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering. In ACM SIGGRAPH 2015 Course Notes: Physically Based Shading in Theory and Practice.

Brent Burley. 2019. On Histogram-Preserving Blending for Randomized Texture Tiling. Journal of Computer Graphics Techniques 8, 4 (Nov. 2019), 31-53.

Iliyan Georgiev, Jamie Portsmouth, Zap Andersson, Adrien Herubel, Alan King, Shinji Ogaki, Frederic Servant. 2019. Autodesk Standard Surface. Autodesk white paper.

Tobias Häussler, Holger Dammertz, and Bastian Sdorra. 2020. Enterprise PBR Shading Model. Dassault Systèmes white paper.

Eric Heitz and Fabrice Neyret. 2018. High-Performance By-Example Noise using a Histogram-Preserving Blending Operator. Proceedings of the ACM on Computer Graphics and Interactive Techniques (Proc. of High Performance Graphics) 1, 2 (Aug. 2018), Article 31.

Christophe Hery, Ryusuke Villemin, and Junyi Ling. 2017. Pixar’s Foundation for Materials: PxrSurface and PxrMarschnerHair. In ACM SIGGRAPH 2017 Course Notes: Physically Based Shading in Theory and Practice.

Brian Karis. 2013. Real Shading in Unreal Engine 4. In ACM SIGGRAPH 2013 Course Notes: Physically-Based Shading in Theory and Practice.

Christopher Kulla and Alejandro Conty Estevez. 2017. Revisiting Physically Based Shading at Imageworks. In ACM SIGGRAPH 2017 Course Notes: Physically Based Shading in Theory and Practice.

Jeremie Noguer, Paul Edmondson, Michael Bond, and Peter Kutz. 2021. Adobe Standard Material Specification. Adobe white paper.

Lukas Stockner. 2022. The new Principled BSDF Model in Cycles. In BCON 2022: Blender Conference 2022.

Thomas V. Thompson, Ernest J. Petti, and Chuck Tappan. 2003. XGen: Arbitrary Primitive Generator. In ACM SIGGRAPH 2003 Sketches & Applications.

Michael Todd. 2013. Interactive Grooming System: User Friendly Hair & Fur. Michael Todd Portfolio.

Walt Disney Animation Studios. 2011. SeExpr.

Footnotes

¹ Additional post-2020 references added in update to post in May 2025. keyboard_return

² As of RenderMan 24, released in 2021, RenderMan does now have a PxrMetallicWorkflow node that converts metallic/basecolor inputs into the diffuse/specular face/specular edge color inputs that PxrSurface expects. RenderMan 24 also includes a new implementation of the full version of the Disney BSDF. keyboard_return

2020 Year in Review

Ben Overmyer's Site — Mon, 28 Dec 2020 02:00:00 GMT

Last year I didn't write a retrospective. This year I felt it was necessary, given all that's happened.

2020 has been a rough year. Covid-19, George Floyd's murder and the ensuing riots, and Donald Trump's increasingly bizarre behavior are all pretty well known and covered elsewhere. My sister lost three cats this year; two to old age and one small kitten to disease. My wife's grandfather passed away due to Covid.

Going fully remote was relatively easy for my wife and me, as we both have professions that mesh well with it. In March just after the coronavirus lockdown began in the USA I changed clients. I have never seen anyone at my current client in person; we have only ever interacted in Teams and Zoom. Video chat is definitely preferable to voice-only, but we still need spatially-aware video chat before it can get close to replicating in-person interaction. Side conversations and awareness of eye contact are sorely lacking.

My wife and I went on our first cruise together in late February and early March. This was right after the first cruise ship off East Asia got quarantined, but right before lockdown began. The ship was filled with hand sanitizer stations and we were required to wash our hands before entering the buffet restaurants, but everything was otherwise no different from normal operation. The daily email updates about the increasing number of ports that were off-limits due to Covid-19 were disconcerting, though. Thankfully, the cruise was in the Caribbean, so it was one of the last regions to be affected by Covid-19 directly, and our ship was never infected.

Work has been good to me this year. At the end of last year, I won my employer's biggest award, the SDG Award. At the end of this year, I won it again. Throughout the year I got a few smaller awards too. That felt really good, and I've felt very appreciated at Solution Design Group. I joined the Employee Experience Committee and gave a presentation on roguelike development at last year's internal SDG Presents event. The Tabletop Games company community that I lead went quiet for the most part over 2020. I regret that a bit; the events we held for it last year were a lot of fun for everyone. In hindsight, running them virtually would have been a nice change of pace from all the negative events of the year.

From a learning perspective, I dove deep into Chef this year. It went so well that not only do I no longer hate Chef, I've become an advocate for it in some scenarios. I also began studying Estonian history and the Estonian language. Estonia is a major focus in my personal life right now as we gear up to move there. Their history is fascinating and so much of it is unknown outside the Baltic region itself. I was blown away when I found out that Estonia is effectively the Silicon Valley of Europe, with services like Skype being born there. More on all of this will come in a future blog post.

With dramatically reduced in-person social events this year, I needed to find new ways to interact with people. The Dungeons & Dragons campaign that I played in went virtual, and I stayed with it for several months, but ultimately it was too big for me to play in virtually. For technical reasons we ended up dropping video chat in favor of voice chat, and that meant it was difficult to get attention without interrupting someone else. I ended up leaving that campaign and joining a smaller one, and that is working out much better. I also have attended a handful of virtual happy hours over the year, and those have been fun, but sporadic. An unexpected social outlet came about in the form of the tilde community that I belong to - Tilde Town. I interacted with them much more this year than in any of the previous five years I've been a part of it, and started to make some friends. Discord communities have also provided a social environment to a lesser degree, and I've become more involved in the Royal Manticoran Navy, a fan organization dedicated to David Weber's Honorverse novels.

My personal project Iron Arachne has seen a wealth of development in 2020. Near the beginning of the year I started running into limitations of the Golang ecosystem and rewrote the entire project into a Laravel monolith. This worked much better, even if certain operations became much slower. The site expanded to the point where users had accounts and could save their generated results, and the heraldry generator grew and became more useful and complex. Unfortunately, the infrastructure costs to support the site also grew, and in the last couple months I could no longer justify their expense. So I rewrote the site again, this time as a single-page app in Vue.js that I could host for free on Netlify. The site lost a lot of functionality in the process, but it now costs nothing to maintain. In a happy turn of events, the star system generator actually became much faster in the conversion to JavaScript, going from a 3-5 second generation time to under 60ms. I will talk more about my plans for Iron Arachne in my first blog post of 2021.

The last big change of 2020 that I'll mention here is my return to the Apple ecosystem. I replaced my Pixel 2 with an iPhone 11 Pro last year, and this year I replaced my custom Windows/Linux PC with a 2019 16" MacBook Pro, and my Fitbit Charge 3 with an Apple Watch. I don't regret this decision at all. While there have been some periods of adjustment, it has been a positive experience overall. Gaming on the MacBook Pro is not as good as it was on my custom rig, but Bootcamp lets me play most of my PC games without issue. I wish the GPU was better but otherwise it works fine.

All in all, 2020 has been a lot better than it could have been for me personally. I am deeply grateful for being in a position to weather the storm this way. Hopefully 2021 will be better for the world in general. The vaccines and new American leadership give me hope.

How to Concatenate Strings in Lua

Danny Guo — Mon, 28 Dec 2020 02:00:00 GMT

FreeBSD on a Raspberry Pi 4 with 4GB of RAM

Lambda Land — Mon, 28 Dec 2020 02:00:00 GMT

+++

This is the story of how I managed to get FreeBSD running on a Raspberry Pi 4 with 4GB of RAM, though I think the setup story is pretty similar for those with 2GB and 8GB.

I also managed to get Rust built from source, (kind of) which is nice because the default Rust installer doesn’t seem to work for FreeBSD running on a Raspberry Pi.

Se connecter à un serveur VPN Wireguard avec NetworkManager

Accueil on dbeley — Mon, 28 Dec 2020 01:00:00 GMT

Wireguard est un protocole de communication permettant de créer des réseaux VPN. Il se veut plus rapide et plus moderne que son concurrent OpenVPN. OpenVPN garde une longueur d’avance non négligeable puisque la quasi-totalité des VPNs commerciaux l’utilise.

Même si Wireguard a encore du chemin à parcourir, il commence à prendre une certaine envergure : en effet, Wireguard est intégré dans le noyau Linux depuis la version 5.6, ce qui facilite sa démocratisation.

Season's Greetings

mass:werk – Now Go Bang! — Sun, 27 Dec 2020 21:15:00 GMT

Better late than never, an old-school e-card…

Advent of Code

RSS — Sun, 27 Dec 2020 17:37:00 GMT

A quick introduction to Advent of Code for those of you who've never tried it out; as well as vignettes of what I learned from participating in AoC 2020.

How to make digital copies of your old video tapes

./techtipsy — Sun, 27 Dec 2020 17:00:00 GMT

Or 'Wow, I did not know that VLC could do _that_!'

A review of iOS, from an Android user's perspective

bastibe.de — Sun, 27 Dec 2020 16:32:00 GMT

My Pixel 2 phone received its final software update the other day, and its battery took the occasion to really show its age. So it was time for a new phone. Which is not a decision to make lightly, considering that I spend multiple hours per day using it. And this time, I was inclined to give the iPhone another go, mostly because of Google's gross lack of human decency in the last few years. So, after years of using various Android devices, I bought a used, red, iPhone SE (2020). I made this choice with some trepidations, as my last experience with iOS was an iPad 3 from 2012. These are my experiences.

As a seasoned Android user, my first impressions of iOS were a bit of a mess. There are things that swipe up from the bottom, swipe down from the top, to the left of the home screens, to the right of the home screens, and at various permutations of pressing/holding the home button. Everything animates, takes a long time, and moves the viewport around. Particularly annoying is the placement of the back button in the top left corner, i.e. the most-inconvenient place on the entire screen for the arguably most-important gesture of the entire UI¹. A close second are the context menus that slide out from a long-pressed item, with the menu thus in a different position on the screen every time you long press anything. These things may be less flashy on Android, but are seriously simpler.

I also immediately missed Android's customizeable home screen, with freely-positionable app icons and a plethora of useful widgets. iOS is very restrictive in this regard, and seemingly for no good reason. Why is the list of all apps (right-of-homescreen) sorted arbitrarily into nonsensical folders instead of a plain list? Why are app widgets allowed, but only on that weird left-of-the-home-screen screen? Why can't I have a currently-playing widget for my podcast player, a weather radar, or my groceries list? Apparently, iOS 14 has a brand new API that does now allow Android-like widgets in iOS, but at the moment they were only available for Apple's own (useless) apps.

My second big stumbling block was the iOS on-screen keyboard. At first glance, I thought it a terribly clunky thing. Actions as simple as inserting a comma require multiple taps, and positioning the cursor seemed almost comically difficult. But then I discovered that the "123" button in the bottom left can be swiped instead of tapped, which makes commas and periods and hyphens available to a quick swipe. That is seriously cool, if slightly hampered by my accidentally activating Control Center instead of swiping from "123" a bit too often. And precise cursor positioning is hidden behind a long-press of the spacebar. Very cool indeed. With these gestures, the iOS on-screen keyboard is actually not bad at all.

Autocorrect seems capable as well, and multi-language aware (hear that, Android?). And, mind-blowingly, a triple-swipe on the content area engages undo and redo in the current text area. Albeit a nigh-undiscoverable gesture, this is miles better than Androids undo/redo system (there isn't one). Actually, Android text fields do support undo and redo if you press Ctrl-Z on the keyboard, it's just that no on-screen keyboard has a Ctrl key. This is my number one grievance with Android at the moment (although there are workarounds).

As a summary to the keyboard situation, I came to respect the iOS keyboard. I still miss a German "�" key, more control about the autocorrect system, and a more modern design. But it works reasonably well. Better than Android's stock keyboard for sure.

My second big hurdle with iOS was the camera. Just like my Pixel's camera, the iPhone takes perfectly acceptible photos. But everything else is just plain worse than on Android. It starts with the way you get access to the camera, either by swiping right on the lock screen, or by tapping the camera app on the home screen. On Android, you double-click the lock button to start the camera, then hit the volume button to take a photo. Not only is this significantly faster, it also requires no ungloved finger, and no look onto the screen. And it doesn't make that abominable shutter sound, either, that the iPhone requires (unless the entire phone is silenced or you take video).

When I take a picture with my phone, it is because I don't have a dedicated camera at hand. Considering the number of cameras I own, this is most likely to happen when I need a camera fast. Thus the speed from pants to picture is extremely important to me, and this is a clear victory for Android. And additionally, Pixel phones have supported stacked RAWs since the original Pixel², and offer quite a comprehensive image editing suite right in the stock camera app. And I prefer the Pixel's relatively neutral rendering over the iPhone's oversharpened, over-denoised, waxy images. But that might be just me.

At this point, I had more or less given up on iOS, and bought a Pixel 4a, which is probably Android's closest competitor to the iPhone SE.

Beyond the camera, there were a number of annoyances with iOS that I found hard to get used to. Like the fingerprint reader on the SE being very unreliable for me (50% miss rate), and awkwardly requiring a distinct push to activate the home button, where the Pixel's is quicker and in a more convenient location. The home button is in fact not a button, but an immovable touch pad that fakes a button-like behavior with a little rumble effect when pressed uncomfortably hard. Pressing a hard surface really hard did not get comfortable to me, especially not when invoking the multi-tasking switcher with a double-press. And it's awkwardly placed at the very bottom of the device, where my thumb does not rest with any kind of force in normal usage. Another real problem is the lack of third-party browsers³, which just leaves you in the cold should a website not work in Safari, or should you dislike Safari's rendering or its very limited ad-blockers. I was particularly annoyed by Safari's context menu for links, which slowly slides out from the link's location, often extending far outside of the screen. Thus opening links in new tabs is just awkwardly slow and annoying in Safari, where Android's more utilitarian menu gets the job done much quicker. Although a phone with a bigger screen might mitigate this problem somewhat.

On the topic of animations in general, one of my favorite features in Android is the animation speed setting in the Developer settings. If you want to feel like you got a new, faster phone, just set the animation speed to 0.5x, and marvel at the newfound snappiness of everything.

Apps in general feel annoyingly restrictive on iOS. Where is a true Firefox, with addons? Where is a Youtube player that can play a YouTube video in the background while the screen is locked, or block ads (NewPipe is the bee's knees!). Or a file synchronization tool that can actually synchronize useful data outside of its own sandox? In fact, my original hopes for iOS were driven by my memory of the fabled higher quality apps available only on the App Store. Looking at some of my old favorites on the SE however, I was forced to take off those rose-tinted glasses. These apps might have been radical back in the day, but the world has moved on. I don't see a pervasive difference in app quality between iOS and Android any longer. Of course iOS apps do still cost a more money on average, and often can't be test-driven without paying. That two-hour free return policy on the Play Store is seriously genius.

Additionally, there were a number of odd problems with the iPhone SE that I found hard to make sense of. For example, apps were very frequently getting booted out of memory. So much so in fact, that often my RSS feeds would not refresh in the background, and podcasts would not download. Even though the iPhone SE sure does have a lot less memory than the Pixel 4a, I hadn't expected this to be an actual problem⁴. And the iPhone would frequently misunderstand a vertical swipe for a horizontal one, or initiate an edge swipe when my finger was still clearly on the screen; My bluetooth headphones sounded noticeably worse, with a strange clipping artifact that should not be there; Significantly worse cell reception and voice quality; Low contrast and tiny text on lock screen notifications; And only barely adequate battery life. And let's not even talk about the stupid Lightning cable, the (comparatively) laggy and tiny TFT screen, the lack of a podcast client I like, and my horrible experiences during the setup process.

So, in summary, the iPhone SE was not for me. Don't get me wrong, it's a nice enough phone, and probably enough of a smartphone for most people. But there were a number of issues with both its hardware and its software, where I found Android and the Pixel 4a plainly more productive. Which is surprising, as my mind still had iOS pegged as the premium alternative, that I would totally buy if I wasn't too cheap for it. But in actual use I was annoyed at its preference of form over function, with many a distracting animation and a number of glaring ergonomic mistakes. Well, another lesson learned. The only truly significant advantage of iOS remains its vastly superior software support story: The SE will probably receive software updates until 2025 or 2027, where the best-in-class Pixel 4a will definitely expire in 2023.

in some places, a right swipe can be used instead of the back button. But the behavior is too inconsistent to be useful↩
what Apple calls "ProRAW", and only makes available in the iPhone 12. iOS does support ordinary RAW, but only in third-party camera apps, and anyway non-stacked RAW files are rather useless from a tiny phone sensor↩
there are different browsers, but they are all required to use Safari's web renderer.↩
I hear this might have been a bug in iOs 14.3? What this says about the software quality of iOS in general is even more troubling, however.↩

Using Dhall to manage my docker-compose.yml

Imran's Blog — Sun, 27 Dec 2020 02:00:00 GMT

I saw Dhall mentioned on a job posting for NoRedInk. I had never heard of it before but NoRedInk did create Elm and are fond of functional tooling so it piqued my interest.

For reference, Dhall is a simple language. It's not Turing complete, but it's strongly typed. It's primarily used for managing configuration files. It has some helpers to output JSON/YML/bash, but you can also output to plain text.

The website does a good job of showing what it can do and its use cases so I wont dive into those. Converting my docker-compose.yaml that I use to manage my media server was the first usecase I could think of.

I wont be posting the full source but I hope I convey how much fun I had working with Dhall.

# Experiment

I'll just be focusing on a small part of the docker-compose file, namely the ports.

Let's say this is my compose file

services:
  plex:
    image: linuxserver/plex:latest
    ports:
      - 32400:32400
      - 32400:32400/udp
    environment:
      VERSION: "latest"

A quick translate over would get us

let plex =
      { image = "linuxserver/plex:latest"
      , ports = [ "32400:32400", "32400:32400/udp" ]
      , environment.VERSION = "latest"
      }

in  { services.plex = plex }

When passed into dhall gets us this:

>>> dhall --file test.dhall

{ services.plex
  =
  { environment.VERSION = "latest"
  , image = "linuxserver/plex:latest"
  , ports = [ "32400:32400", "32400:32400/udp" ]
  }
}

dhall-to-yaml produces the following:

>>> dhall-to-yaml --file test.dhall

services:
  plex:
    environment:
      VERSION: latest
    image: linuxserver/plex:latest
    ports:
      - "32400:32400"
      - "32400:32400/udp"

Which is the exact same file (keys are sorted so the output is deterministic).

I always forget what order the ports should go (host:container) so let's take advantage of Dhall to make it easier on me. Before I make any changes to the file I'm going to be grabbing the Dhall SHA so I can compare it to my refactored code's SHA, if they are equal then the output has not changed.

>>> dhall hash --file test.dhall

sha256:db23b00a8e79e084306b72ecc788f0b95579080c883cb72045a3ae6ea43999fd

First lets move ports into it's own variable and verify the hash

let ports
    : List Text
    = [ "32400:32400", "32400:32400/udp" ]

let plex =
      { image = "linuxserver/plex:latest"
      , ports
      , environment.VERSION = "latest"
      }

in  { services.plex = plex }

>>> dhall hash --file test.dhall

sha256:db23b00a8e79e084306b72ecc788f0b95579080c883cb72045a3ae6ea43999fd

OK we can be sure we didn't change our output at all. Don't worry about : List Text that is just a type annotation.

Lets start by making a type that we can store the port mapping in and a function to convert it to text.

let PortMapping
    : Type
    = { host : Natural, container : Natural }

let PortMapping/show
    : PortMapping -> Text
    = \(pm : PortMapping) ->
        "${Natural/show pm.host}:${Natural/show pm.container}"

Now we can supply the ports we want in any order and be confident the output order is correct. Let's update ports to use this new functionality.

let ports
    : List Text
    = [ PortMapping/show { host = 32400, container = 32400 }
      , "${PortMapping/show { host = 32400, container = 32400 }}/udp"
      ]

Our hash is still sha256:db23b00a8e79e084306b72ecc788f0b95579080c883cb72045a3ae6ea43999fd so we haven't broken anything yet.

This is better but we still need to do some string interpolation if we want to create a UDP port mapping. The docker documentation says the notation without /udp or /tcp will default to tcp.

Let's create an enum that will represent these two and update our code to use it.

let Port
    : Type
    = < tcp : PortMapping | udp : PortMapping >

let Port/show
    : Port -> Text
    = \(p : Port) ->
        merge
          { tcp = PortMapping/show
          , udp = \(pm : PortMapping) -> "${PortMapping/show pm}/udp"
          }
          p

let ports
    : List Text
    = [ Port/show (Port.tcp { host = 32400, container = 32400 })
      , Port/show (Port.udp { container = 32400, host = 32400 })
      ]

Much nicer! And our hash is still producing sha256:db23b00a8e79e084306b72ecc788f0b95579080c883cb72045a3ae6ea43999fd.

Now what would be even nicer is we did not have to specify the port twice if they are the same value. Let's do that by expanding our Port enum to supply a tcp_mirror and a udp_mirror.

let Port
    : Type
    = < tcp : PortMapping
      | udp : PortMapping
      | tcp_mirror : Natural
      | udp_mirror : Natural
      >

let Port/show
    : Port -> Text
    = \(p : Port) ->
        let udpPortMapping = \(pm : PortMapping) -> "${PortMapping/show pm}/udp"

        in  merge
              { tcp = PortMapping/show
              , udp = udpPortMapping
              , tcp_mirror =
                  \(port : Natural) ->
                    PortMapping/show { host = port, container = port }
              , udp_mirror =
                  \(port : Natural) ->
                    udpPortMapping { host = port, container = port }
              }
              p

let ports
    : List Text
    = [ Port/show (Port.tcp_mirror 32400), Port/show (Port.udp_mirror 32400) ]

-- sha256:db23b00a8e79e084306b72ecc788f0b95579080c883cb72045a3ae6ea43999fd

We can include the List/map function from Prelude so we do not have manually transform our port list.

# Final `test.dhall`

let List/map = https://prelude.dhall-lang.org/List/map

let PortMapping
    : Type
    = { host : Natural, container : Natural }

let PortMapping/show
    : PortMapping -> Text
    = \(pm : PortMapping) ->
        "${Natural/show pm.host}:${Natural/show pm.container}"

let Port
    : Type
    = < tcp : PortMapping
      | udp : PortMapping
      | tcp_mirror : Natural
      | udp_mirror : Natural
      >

let Port/show
    : Port -> Text
    = \(p : Port) ->
        let udpPortMapping = \(pm : PortMapping) -> "${PortMapping/show pm}/udp"

        in  merge
              { tcp = PortMapping/show
              , udp = udpPortMapping
              , tcp_mirror =
                  \(port : Natural) ->
                    PortMapping/show { host = port, container = port }
              , udp_mirror =
                  \(port : Natural) ->
                    udpPortMapping { host = port, container = port }
              }
              p

let portsList
    : List Port
    = [ Port.tcp_mirror 32400, Port.udp_mirror 32400 ]

let plex =
      { image = "linuxserver/plex:latest"
      , ports = List/map Port Text Port/show portsList
      , environment.VERSION = "latest"
      }

in  { services.plex = plex }

-- sha256:db23b00a8e79e084306b72ecc788f0b95579080c883cb72045a3ae6ea43999fd

# My Conclusion

Using Dhall is super fun. I can totally see the usefulness in a shared team environment for managing configurations. I think I went a bit overboard as my ~120 line compose file is now ~400 lines of Dhall code 😅. Except now it's statically typed and it will be super easy to add new services to my server (or in context of the experiment, new ports to the Plex service).

Stringer Self Hosted RSS Reader

Brent's Blog — Sun, 27 Dec 2020 02:00:00 GMT

My setup is using the docker compose file and a local build of the docker image (since docker hub image is years old). Here is a dark theme (solarized) for the CSS. Just paste this over your stringer/app/assets/stylesheets/application.css file. I did not spend much time on it but it looks good enough for my taste.

The year in books: 20 to recommend in 2020

Notes on software development — Sun, 27 Dec 2020 02:00:00 GMT

This year I finished 47 books, up from last year but not a personal best. The breakdown was 17 non-fiction and 30 fiction. Another 20-30 remain started but unfinished this year.

Non-fiction

The 8 non-fiction books I most recommend are:

Fashionapolis: The Price of Fast Fashion and the Future of Clothes (Must read)
Effective Python: 90 Specific Ways to Write Better Python (Must read; truly excellent for Python programmers, I recommend this to anyone I work with)
The Machine that Changed the World (Must read)
Europe: The Struggle for Supremacy from 1453 to the Present
Wind, Sand and Stars
American Colussus: The Triumph of Capitalism, 1865-1900
Making Common Sense of Japan
The German Genius

The 3 books I recommend you not to waste time on are: "The Two Koreas", "The Price of Inequality", and "Ninety Percent of Everything: Inside Shipping".

The whole list

The Two Koreas by Don Oberdorfer
- Interesting but not a huge fan, seemed pretty biased against South Korea somehow
Forbidden Nation: A History of Taiwan by Jonathan Manthorpe
Effective Python: 90 Specific Ways to Write Better Python by Brett Slatkin
A Philosophy of Software Design by John Ousterhout
- Came as a recommendation from someone on Twitter, ultimately not a huge fan. Still looking for high quality books on software design
The Price of Inequality by Joseph E. Stiglitz
- Agreed with the premise but the book was incoherent and too self-assuring
Paris Reborn: Napoléon III, Baron Haussmann, and the Quest to Build a Modern City by Stephane Kirkland
Fashionapolis: The Price of Fast Fashion and the Future of Clothes by Dana Thomas
A Moveable Feast by Ernest Hemingway
- I normally love Hemingway's writing but this particular book was not very coherent
Europe: The Struggle for Supremacy from 1453 to the Present by Brendan Simms
- Such an excellent introduction to the continent for Americans who otherwise don't have great background
Wind, Sand and Stars by Antoine de Saint-Exupéry
- A beautiful memoir of flights by the author of The Little Prince, very similar in style to Hemingway
American Colussus: The Triumph of Capitalism, 1865-1900 by H.W. Brands
- Baby's first primer on unions, (I need more recommendations on the history of unions)
Making Common Sense of Japan by Steven R. Reed
- It can be difficult to find English translations of Korean, Japanese history by Korean and Japanese authors; this is a good one by an American professor
The Machine that Changed the World by James P. Womack
- An excellent, well-researched history of automobile manufacturing in the US, Europe and Japan from the 1900s to 1990; how Japan ate everyone's lunch
The United States of Europe by T.R. Reid
- Very light introduction to the European Union
The Soul of a New Machine by Tracy Kidder
- Overhyped by the internets, but not bad
The German Genius by Peter Watson
- Dense but excellent introduction to many famous Germans in many fields throughout time
Ninety Percent of Everything: Inside Shipping by Rose George

Fiction

I'm trying to read more from non-English authors. If you see non-English authors in the vein of these here that you can recommend, I'd love to hear from you.

The 12 fiction books I most recommend are:

Planet of the Apes (Must read, yes even if you've seen the film)
All Quiet on the Western Front (Must read)
The Mouse That Roared (Must read)
The Dead Mountaineer's Inn
The Golem and the Jinni
Neverwhere
Dubliners
Old Man's War
The Inspector Barlach Mysteries: The Judge and His Hangman and Suspicion
Fantômas
Foundation
Out of the Silent Planet

The only book I really didn't like was "Invisible Cities".

The whole list

March Violets by Philip Kerr (Scottish)
Liberty Bar by Georges Simenon (Belgian)
The Late Monsieur Gallet by Georges Simenon (Belgian)
Dubliners by James Joyce (Irish)
Tales of the City by Amistead Maupin (American)
The Third Policeman by Flann O'Brien (Irish)
44 Scotland Street by Alexander McCall Smith (British-African)
Knots and Crosses by Ian Rankin (Scottish)
I Hear Your Voice by Kim Young Ha (South Korean)
The Golem and the Jinni by Helene Wecker (American)
The Tokyo Zodiac Murders by Shimada Sōji (Japanese)
The Screwtape Letters by C.S. Lewis (English)
Neverwhere by Neil Gaiman (English)
Old Man's War by John Scalzi (American)
Tales from Earthsea by Ursula K. Le Guin (American)
Solaris by Stanisław Lem (Polish)
A Wizard of Earthsea by Ursula K. Le Guin (American)
Planet of the Apes by Pierre Boulle (French)
The Dead Mountaineer's Inn by Arkady Strugatsky (Russian)
Invisible Cities by Italo Calvino (Cuban-born Italian)
The Inspector Barlach Mysteries: The Judge and His Hangman and Suspicion by Friedrich Dürrenmatt (Swiss)
Fantômas by Marcel Allain (French)
All Quiet on the Western Front by Erich Maria Remarque (Germany)
A Crime in Holland by Georges Simenon (Belgian)
The Wonderful Adventure of Nils Holversson by Selma Lagerlöf (Swedish)
Foundation by Isaac Asimov (Russian-born American)
Out of the Silent Planet by C.S. Lewis (English)
The Spy Who Came in from the Cold by John le Carré (English)
The Bat by Jo Nesbø (Norwegian)
The Mouse That Roared by Leonard Wibberley (Irish-born American)

Out of 47 books read this year, here's the 20 I recommend to you (gave them 4/5 stars or better). I'm trying to read more non-English authors so I'd love to hear if there are authors with similar style on this list you'd recommend!https://t.co/FjHcvHpRSr
— Phil Eaton (@phil_eaton) December 27, 2020

Migration from FreeNAS to TrueNAS

Kevin Deldycke — Sun, 27 Dec 2020 01:00:00 GMT

The upgrade itself is quite straightforward by the way of the GUI. But things gets complicated if you’ve got an encrypted pool.

Encrypted pool migration ¶

Upgrading from FreeNAS to TrueNAS will render your encrypted pool unavailable. FreeNAS relies on geli ’s block-layer disk encryption and is considered legacy. TrueNAS …

You're Allowed To Make Your Own Tools

swyx's site RSS Feed — Sat, 26 Dec 2020 20:04:11 GMT

Making personal side projects for fun and profit.

Beaver Lake Preserve

Sam Hooke — Sat, 26 Dec 2020 09:00:00 GMT

No trip report yet.

Hazel Wolf Wetlands.

A map of the woods.

What went wrong with software for prisons

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 26 Dec 2020 09:00:00 GMT

It became just another business, in one more market.

Chess bookmarks

xenodium.com @alvaro — Sat, 26 Dec 2020 02:00:00 GMT

On the UX of Microwave Ovens

one mikro2nd — Fri, 25 Dec 2020 12:23:39 GMT

Who the hell decided on four? Why four beeps? That tinny, high-pitched electronic squeak that grates on the nerves like the gravel tearing at your knee that time you came off your bike. Two beeps would have been plenty. Hell, one would be enough. What does it take for the microwave to say, “I’m done. Your food is hot”. Hottish anyway. Hot around the edges but never in the middle. Actually you don’t really need any beeping noises at all — the machinery inside the thing, the fan, the motor that drives the turntable, the synchrotron’s whirl, all those come to a stop.

Our need for rituals, and one thing that really upsets it

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 25 Dec 2020 10:00:00 GMT

Guess what that thing is?

My 2020 reads

Alexander's Blog Blog — Fri, 25 Dec 2020 02:00:00 GMT

Flow

2020 - Year in Review

www.mootoday.com - RSS Feed — Thu, 24 Dec 2020 19:57:09 GMT

A look back at what was 2020. Highlights, what worked and what didn't.

Nanoscale is where chipmaking must go

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 24 Dec 2020 15:54:00 GMT

This is important. Really.

Advent of Haskell 2020: Santa Wrap

Lucas DiCioccio's Blog — Thu, 24 Dec 2020 14:00:00 GMT

We help Santa Elves optimize cutting wrapping paper to accomodate various shapes. The difficult optimization requires a dedicated solver in MiniZinc. Haskell allows to integrate with MiniZinc, reads JSON and outputs a PNG image with the footprint of the positionned shapes to cut. This article shows that a one-day-of-work project can reach pretty far.

Big Tech = Feudalism + Animal Farm, digitized. Via tax havens

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 23 Dec 2020 20:00:00 GMT

Because, INNOVATION!!!!, of course.

2020

trv.rs — Wed, 23 Dec 2020 19:10:00 GMT

Hindsight is 2020… but I'm glad to see 2020 in hindsight. 🥁 Truthfully, my family has been very blessed and protected in so many ways through this wild year.

We welcomed another little one safely, God continues to provide for us through my job (which I enjoy!), and our little family has had lots of time together... sometimes socially distanced but otherwise wonderful.

This has also been a challenging year... mostly the external factors of The Unknown¹ preying on my mind and creating the usual symptoms of stress on my body and mind. Running to the Lord first, and wisely limiting some of the sources of stress has really helped.

Creating

This year I wrote 60 blog posts including this one.

While writing has been creatively satisfying in and of itself (and useful!²) even more wonderful has been likes, comments, reblogs and the occasional email from a new internet friend. I got to help a blogging hero of mine set up webmentions, got a lot of folks talking about zettelkasten, and a few crazy souls actually tried my hammerspoon setup. This year I've received about 12 emails from y'all… and each made my day and inspired me to keep going. I have removed analytics from the site… getting emails from y'all is much nicer than tracking top keywords.

It's really hard to tell how much I added to my zettelkasten… the modified time has been clobbered by running the transition script too many times. Looking forward to investing in that more this year.

While I don't presently code for a living, GitHub counts 872 contributions… probably mostly me dinking around on my blog, hammerspoon, or QMK. :-P

Reading

I only read 13 books this year. :( I'll blame a variety of factors, including discovering long-form podcasts and getting overly absorbed that I can borrow graphic novels on Hoopla that I can borrow comic books from the library (which don't count for me… although I definitely maxed out my 10 per month every single month this year. I suppose if I did count comic books, it'd be closer to 120?)

I've responded by deleting Hoopla off my phone, which seems to be helping. 😜

While I don't know how many articles I read this year, I apparently saved 1,360 articles and videos in my instapaper… so something less than that. I saved 67 annotated links into my zettelkasten… although many more went into blog posts and newsletters.

Health

According to strava I worked out 214 days in 2020. I doubled up on running, and set another sub-25m 5k just last week.

While (God be praised!) we've been well and haven't really gained any weight, I'm still wrestling with my diet, although a combination of macro tracking and looking at the graph every morning seems to be helping me in the last two months. Turns out sugar is the single most correlated stat for me and weight. I'll write more about that automation later I'm sure.

In all, I think this year I finally started to add sleep, water intake, and diet moderation to the mix. Now to make it an easy part of my life.

Work

In February, my wife and I got away to talk about the next four years and what we wanted to accomplish personally and in our family life. One of those things was that we wanted me to work from home by the end of those four years... and we got our wish the next week. 🤷

COVID remote has been a challenge, but a welcome one. I work well from home, and I've been glad that I could be home for the first six months of our newest. While I certainly pray for the pandemic to end, I kinda want to stay home.

Our team has grown both in size and capability. I don't think I've ever been this excited to get back into work for the next year, I know great things are ahead.

I've been tracking my time this year, although it really didn't become a habit till June (Headspace was born in May.)

These are mostly since June:
Dark Blue is High Value. (298h, some of it hidden in the gray.)
Light Blue is shallow work (238h) and meetings (266h).
Green (47h) is "play" which includes blogging and working on dotfiles.

Looking at my stats... I'm kind of appalled of where the time was invested. I'm happy with the results, and so I guess this is an encouragement that I can do even better.

Spiritual Life

I'm still not where I want to be here. Half way through the year I realized my commitment to working out was choking my quiet times, so I've started turning that around: trading my workout time for prayer and the word. I still haven't gotten back to my study habit I had five years ago, I may need to find another time in the day because mornings are difficult with two little ones.

I think the next step is to reconnect with my wife and really prioritize shared time in the word and prayer. With the two smidgens it's easy to have a split schedule.

Relationships

Mid-COVID, I started realizing that while my introverted self was happy, there would be others that would be struggling, so I started reaching out to my friends for virtual lunches. I wound up doing quite a few... too many actually, I think that kind of undercut my family time. Going to find a sustainable pattern going forward, but definitely going to continue something intentional and personal over likes and comments on social media.

Unsurprisingly, I cut off personal social media. I'm still pretty active on twitter and linkedin for semi-professional reasons, but instagram... well I only posted twice this year. I think I'll probably close it down eventually. I'm already collecting an email list for those who want more personal family updates, but don't feel like feeding FB's giant algorithm monster.

While the wife and I got a lot of time together, it wasn't very restful for her... when I work from home I can leave my office, but if you work on the home it is hard to find that life-giving distance in COVID-land, especially with a newborn. We went on our first date in six months this past month, and we are prioritizing more dating time in 2021.

2021

In 2020 chose the theme of "Clarity." I often asked myself when stressed if I had clarity on what was most important, and that really helped. If I knew what was most important (often my family) then I prioritized that. If I didn't know what was most important, then discovering my priorities was the next thing.

I look forward to revealing my 2021 theme... at another time. 😉

I still haven't finished my full yearly personal review and planning... that usually comes in the days before New Years, but I am excited to see what the Lord will do in my heart in 2021. He has been faithful through a very strange and stressful year, and I fully expect to see His hand clearly in 2021.

He is good. I trust him. Let's do this!

Politics. Pandemics. Other P words. ↩
I have been secretly pleased every time I send someone a previously written link to answer a question! ↩

Books and Links: December

trv.rs — Wed, 23 Dec 2020 17:33:00 GMT

This year is almost done! I'm working on a year in review thingy... how many years have I done that now?

Books

I haven't finished any books this month... I've been burning down the remaining 15+ hours of a podcast entitled Fear and Loathing in the New Jerusalem, a podcast series by Darryl Cooper. It examines the birth and growth of Zionism, starting in the late 19th century all the way up to the birth of the modern Israeli state.

I went into this idly interested, but wound up being enthralled… and horrified. I appreciated how he confronts the listener with the experience of people in that time and context, honestly asking the hard question: "what would you have done?"

On deck: The 12 Week Year, House of Leaves.

Links

What’s a Linkable App and Why Does Linkability Matter? – Hook

I'm obsessed with this weird app at the moment.

People have become so accustomed to the weird fact that, without Hook, information resources are not easily addressable on the Mac that they tend not to realize that its just as important to be able to address local (or cloud synced) resources as it is to address remote resources.

What’s a Linkable App and Why Does Linkability Matter? – Hook

Walmart and autonomous trucks - Marginal REVOLUTION

Since last year, those trucks have been operating on a two-mile route between a “dark store” (a store that stocks items for fulfillment but isn’t open to the public) and a nearby Neighborhood Market in Bentonville, Arkansas. Since then, the vehicles have racked up 70,000 miles in autonomous mode with a safety driver.

I guess that future is here.

Walmart and autonomous trucks - Marginal REVOLUTION

By The Numbers: A Year and Half with Atomic CSS

The results speak for themselves. Refactoring our CSS to the functional approach led to a 30% decrease in size. It could have been even smaller, but we left some legacy styles in place to refactor at a later time.

By The Numbers: A Year and Half with Atomic CSS

Revised Bloom’s Taxonomy • Center for Excellence in Learning and Teaching • Iowa State University

The authors of the revised taxonomy underscore this dynamism, using verbs and gerunds to label their categories and subcategories (rather than the nouns of the original taxonomy). These “action words” describe the cognitive processes by which thinkers encounter and work with knowledge.

A statement of a learning objective contains a verb (an action) and an object (usually a noun).

While this is generally fascinating, this has direct application for us productivity nerds: how we write project and to-do titles. I heard CGPGrey say on Cortex that it's good to phrase projects or to-dos in the past tense… some taxonomy like this could be helpful.

Revised Bloom’s Taxonomy • Center for Excellence in Learning and Teaching • Iowa State University

Lanham Lake Snowshoe

Sam Hooke — Wed, 23 Dec 2020 05:00:00 GMT

Distance	8.22 km / 5.11 miles
Elevation gain	365 m / 1198 ft
Duration	4h45m
Moving	2h20m

No trip report yet.

Looking along the line of pylons.

A close look at the pylon base.

Lanhm Lake with the sun just about to vanish.

Snowy lumps amidst the trees.

Snow logs amidst the trees.

An Annotated Guide to React Server Components

swyx's site RSS Feed — Wed, 23 Dec 2020 03:18:46 GMT

React Server Components are a nuanced, sweeping addition to React's existing capabilities. This is my guide.

2020 Recap

James Brooks — Wed, 23 Dec 2020 02:00:00 GMT

I recap on what happened over the year of 2020.

2020 in review

Jason Eckert's Website and Blog — Wed, 23 Dec 2020 02:00:00 GMT

I know what you’re thinking - 2020 sucked! Worst year ever! The COVID-19 pandemic, no social gatherings, no dine-in restaurants, no gyms, the US Trumpster fire, and most importantly to those of us in the tech world: Red Hat terminating CentOS development.

Of course, most people will remember 2020 for the pandemic, and that’s completely understandable. None of us have experienced a pandemic in our lifetime, and both our society and economy were not prepared to deal with it gracefully. Many businesses folded or downsized, and the lack of social contact caused mental health issues for many people of all ages.

Hotwiring the Web: What does Basecamp's New Magic bring to the table?

Davis Haupt's Blog — Wed, 23 Dec 2020 01:38:55 GMT

Hotwire, which seems to be short for (H)tml (O)ver (T)he (Wire), is a collection of frameworks just announced by Basecamp that work together to help build “traditional” server-rendered web applications that look and feel to users like modern, Single-Page Applications (SPAs) built in React, Angular, Vue or other frontend frameworks. Basecamp’s CTO put out a blog post on why he believes in Hotwire, but most of the justification seems to be handwavy claims that JavaScript is inherently “complex,” never mind that Ruby’s syntax and dynamic type system can be just as head-scratching to a newcomer. I think that Basecamp’s built a really interesting tool, and a better argument for Hotwire can be made by fully engaging with the benefits that SPA “thick clients” bring to the table, their specific shortcomings, and all the different ways framework developers are trying to address those shortcomings today.

Problems with Web Application Development Today

I want to separate “web app” development from general “web” development, simply because most websites aren’t single-page applications, or web apps in the sense we think of gmail, or hey.com. It’s put most concisely in “How the Web is Really Built”. Here, though, I’ll be talking about that subset of the web which can be categorized more as an application than simply a web site.

Building a web app is always going to be a complex endeavour. SPA view libraries like React have emerged in the past five or so years as an awesome way to handle that complexity on the client side, allowing developers to build interactive user-interfaces in a composable way, re-using components easily in different parts of the site.

Duplication of Concerns

Generally, when building a React app, you get any dynamic data through a loosely coupled JSON API (in my personal preference, probably written with Django REST Framework). If you’re using a web framework like Django or Ruby on Rails, then you’re going to have a lot of complexity and business logic on both the frontend and backend. Whenever you add a feature, you need to ask yourself where it should be added. The loose coupling between parts that may very well be written in different languages means that it’s really difficult to share code, and even if you’re doing something as simple as validating a form, you’ll probably end up with a decent amount of duplicated business logic, a big no-no if you follow the DRY principle!

Data Fetching Boilerplate

Loosely-coupled APIs present another challenge when building web apps: the boilerplate that comes along with getting the data out of your database and displayed in a browser. The general process for an SPA follows this outline:

[User] Navigates to a page, submits a form, or clicks on a resource.
[Browser] Send a request, along with any route parameters and arguments, to a backend API endpoint.
[Server] API endpoint is called, fetches data from the database using the parameters in the request.
[Server] After retrieving data from the database, marshal that data into JSON to be sent back to the client.
[Browser] Receive JSON from request, un-marshal the JSON into data structures that the web app can understand. For a React app, this would include passing the data as props to each component that uses the data to for render itself.

And that’s only the happy path! In the case of a validation error, for example, the server would return a 400 response with some JSON details, and the client again would need to catch that error, and render it properly. It really is a lot of boilerplate for any given request.

There are frontend libraries like SWR which factor out as much boilerplate as possible and encourage declarative patterns for your own data fetching, but “thick client” web apps will always need to marshal data to JSON on the server, and un-marshal to some view-able format on the client.

Managing SPA Complexity in 2020

There’s a bunch of different solutions to these problems for those who still want to provide an SPA experience, and they fall broadly into two separate categories.

Frontend-First: The JAMStack

In recent years, there’s been a big push towards the JAMStack: pushing all your complexity and business logic onto your frontend, and relying mostly on commodified API services for your backend work. JSON is still the lingua franca here, but reliance on pre-built backends as a service for different functionalities, along with query languages like GraphQL, means that there isn’t any backend boilerplate handled by developers. All the complexity and business logic is pushed to the client-side, which alleviates worries about duplication of concerns.

JAMStack and associated “micro-backends” like Auth0, or “backends-as-a-service” like Supabase and Google Firebase allow people who haven’t done too much backend work in the past to build truly full-stack apps on their own. m3o is even building a constellation of JAMStack-oriented “micro-services” to provide the batteries to power most web apps. Hmm… “batteries included”? Where have we heard that before…

Backend-First: Re-discovering the batteries

MVC Frameworks like Django and Rails came about during the heyday of Web 2.0 to abstract away a lot of the boilerplate associated with building CRUD web applications, the exact issues that we spoke about above. Back in 2006, all those interactions were through normal browser HTTP requests. SPAs, built with anything from Backbone or Ember to React and Vue, were more responsive. These web frameworks became frameworks for JSON API servers, and for many web app developers, functionality like Django’s templates and forms and the battle-tested abstractions for linking them together became vestiges of an earlier age. Django’s Form{.verbatim} classes can render validation errors in templates with virtually no boilerplate written by developers. As soon as you want to put that form action over a JSON API, any responses from your server, which were previously just the HTML that the browser displayed, now have to be un-marshalled from JSON on the client and handled specifically. How much was Django really a “batteries included” framework if you needed to pull in REST Framework and OAuth Toolkit whenever you wanted to work with a “modern” frontend?

Many people, myself included, enjoy modeling business logic in the ways Django and its ilk allow for. Backend-first fullstack frameworks have begun to proliferate built on top of these existing frameworks. Phoenix LiveView and Laravel Livewire are two that come to mind immediately, and have been around for a year or more.

On Monday, even the React Core team at Facebook threw their hat in the ring, with their Server Components that have the opportunity to allow for React components to be rendered much like PHP templates, interspersing database calls and server-side JavaScript with the layout description inside a server component’s render{.verbatim} function.

These fullstack frameworks go a long way towards solving both of the concerns with traditional SPAs listed above. Separation of concerns is no longer an issue, since there is no separate, loosely coupled frontend codebase. Data fetching is drastically simpler in this paradigm. No longer does all your data need to be serialized to JSON before being converted into HTML; your data-fetching flow looks a lot more like this:

[User] Navigates to a page, submits a form, etc.
[Server] Backend route is called, fetches/stores appropriate data from the database based on the request.
[Server] Data is used to populate an HTML template, which is sent to the client and rendered with the help of the framework.

Your backend business logic renders HTML directly, completely replacing steps 3, 4 and 5 above with a single step: map your data into its visual representation in HTML. The two server steps remaining are the actual business logic in your application: the full-stack framework handles the smooth transitions, without the developer having to worry about serializing and de-serializing their own data. The logical flow of your application becomes a lot simpler for a single developer to follow and to handle.

Even React’s Server Components fit this new paradigm: Data fetching no longer happens in AJAX requests, but by declaring a child server component which fetches the data and displays it in its own DOM tree without the developer having to serialize to/from JSON themselves. After the component renders on the backend, its virtual DOM gets sent to the frontend by React itself for display. The developer’s interface into this whole process remains high-level and declarative.

As the Pendulum Swings

We started the decade with frameworks like Meteor.js with extremely tight couplings between the client and server, and after a long time wandering in the wilderness of duplicated compelxity across loosely-coupled frontend and backend, it seems like we’re entering the twenties with a renewed push towards a more monolithic approach to web development. When even a frontend framework like React is beginning to bridge the gap with the backend, you know it’s an interesting idea to explore right now.

So what is Hotwire?

The folks at Basecamp, the company behind Hotwire, have always been skeptical of thick clients with loads of JavaScript. Hotwire is Basecamp’s latest answer to the challenge of building modern, responsive, “snappy” single-page applications where the domain logic lives entirely on the server. They used it to build out their new email service, Hey.com.

At Hotwire’s core is Turbo, a new library that takes HTML from AJAX requests and dynamically modifies the currrent page. It comes out of an existing library called Turbolinks, now called “Turbo Drive” as of today, which is a utility that intercepts all click events on anchor tags, loads the resources over AJAX, and swaps out the <body>{.verbatim} tags, all while handling browser history.

Turbo Frames, one of the new components, is pretty intriguing. Turbo Drive will still AJAX-ify form submissions and link clicks behind the scenes, but instead of swapping out the entire webpage each time, Turbo will look for matching <turbo-frame>{.verbatim} tags on the current page and in the new page’s content. If there’s a match, it’ll dynamically replace that section of the page. Basically, you can compose webpages together, using <turbo-frame>{.verbatim} to delineate template partials as scoped components, similarly to how you’d think of components in a React app. The benefit here being that all the logic is handled on the server-side rather than split between two code-bases.

Trying out Hotwire with Django

What’s special about Turbo when compared to Phoenix LiveView and Laravel Livewire is that Turbo is completely backend-agnostic: Drop the JS bundle into a <script>{.verbatim} tag in your page’s <head>{.verbatim}, and Turbo Drive works its magic without any co-operation from the server. Turbo Frames can be adapted by wrapping <turbo-frame>{.verbatim} tags around template partials in any backend framework. Turbo Streams, the solution for incremental data updates, can also be used in the context of HTTP requests without any co-operation from the server beyond modifying your template partials. It’s only if you want to use Turbo Streams over WebSockets where you’ll need some custom code for your specific backend framework.

Since the push behind Hotwire came from DHH and Basecamp, it makes sense that their examples are with Ruby on Rails, and that’s where they’ve made their supporting libraries. I decided to take a shot at building a demo app similar to what’s shown in Hotwire’s demo video using Django rather than Rails. It really didn’t take long! I got my start in Web Development with a JQuery app with a REST API, and even after moving on to Django, I always used Django REST Framework. I never really took advantage of the templating functionality, or the super-useful built-in CRUD operations with Django forms. It was an interesting experience working with CreateView{.verbatim} and DetailView{.verbatim} rather than ModelViewSet{.verbatim}, and I’ll be excited to keep exploring this going forward.

After an hour or so more of experimentation and digging into the turbo-rails codebase, I got a working prototype of a Turbo Streams Broadcastable{.verbatim} mixin for Django! I’m working on something similar for Django REST Framework right now, which definitely helped in hitting the ground running. I’ll probably look to clean up the code and make sure it works for the other actions, and split it out into its own pypi package.

I’ll have to look at how the Rails integration handles authorization — right now, anyone would be able to subscribe to any stream for a given model, which is obviously not ideal for actual production applications.

I was really surprised at how easy it was to set up Turbo to work effectively with a Django backend. The chat app that I built was really simple, but it also was just not many lines of code: not having to worry about moving data around from the frontend to the backend really decreased the amount of time spent on the implementation. Turbo though is in a pretty early beta, and the one main thing I’d like to see be addressed would be a good fallback mechanism for Turbo Streams over websockets. Right now, if you want to broadcast updates over websockets, then you can’t also send Turbo Streams in HTTP responses to form actions without getting duplicate data appended. The solution in the Hotwire demo video is simply to not send updates over HTTP, and only stream over websockets. This doesn’t seem particularly robust, however, in the case that a websocket connection fails or a client simply doesn’t support it. In addition to the five actions, there should probably be an append-or-replace{.verbatim} action that looks for an element with a matching id{.verbatim}, performs a replace{.verbatim} action if one is found, and otherwise performs an append{.verbatim} action. The duplicate updates from the HTTP response and the websocket stream wouldn’t conflict in that case, since one will append, and the other will replace with identical data.

Closing Thoughts

This is definitely an exciting time for frontend development! I’m hoping to do some more experimentation in the coming weeks, and I’m glad that framework authors accross the board are putting effort into thinking about how to move the web app developer experience to the next level.

Software will eat Hollywood, bringing two risks with it

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 22 Dec 2020 20:00:00 GMT

Not everybody’s software, by the way. Not if things remain as today.

Indiehacking: a review of my 3rd year

Max Rozen — Tue, 22 Dec 2020 11:48:48 GMT

A review of my third year of trying to start an internet business.

Gold Creek Pond Snowshoe

Sam Hooke — Tue, 22 Dec 2020 04:30:00 GMT

Distance	5.87 km / 3.65 miles
Elevation gain	34 m / 112 ft
Duration	4h
Moving	1h40m

No trip report yet.

Following the trail from the trailhead to the lake.

The snow covered bridge at the north side of the lake.

View of Gold Creek Pond from the south shore.

ADHD 🧠

Blog posts of Andreas Klinger — Tue, 22 Dec 2020 02:00:00 GMT

I want to preface this by saying that I have only a mild case of ADHD, eg I don’t take any medication. I get regularly asked about...

Why broot switched to Hjson for its configuration files

dystroy|Canop / blog — Tue, 22 Dec 2020 02:00:00 GMT

TOML was originally the only format for configuring broot.

Here I explain why Hjson is now the preferred one.

Tech salaries probably aren't dropping from remote work

ntietz.com blog - technically a blog — Tue, 22 Dec 2020 02:00:00 GMT

Not even a year ago, most software companies and software engineers were some form of remote work skeptical. Remote work existed (I've been working remote for most of my admittedly short career!) but it was not widespread. When I talked to recruiters at big tech companies they would all insist that remote work was not feasible for them, and even at the companies I worked for, there was pushback that this definitely wouldn't work for us because reasons. But now, I think we're seeing one of the real reasons people were skeptical:

Money.

There are lots of very real reasons to like colocation, and there is a lot that's hard about remote work. But I haven't seen a lot of discussion around the role of salaries in affecting whether or not people prefer remote work. Imagine this (and for some of you, this might be really easy): you're working in New York or San Francisco, pulling down an absurdly high salary. You know that in the Midwest, salaries are much lower, not even comparing to other countries where salaries might be much lower. In this situation, it seems pretty rational to oppose remote work. If you only allow colocation, then you're competing against other workers who all have the same cost of living as you and all have the same sky-high salary demands as you. But if you allow remote work, that great dev from Akron, Ohio might be willing to do the same job for less money. Right?

Well... it isn't quite that simple. Right now, tech salaries have been going up consistently for a while (anecdotally, I've seen this as long as I've been here, so at least since 2013). This is consistent with demand for software engineers outpacing supply of them. Companies like Basecamp and Stripe are employing software engineers anywhere at California rates. These companies aren't charities: they are doing it because they understand that, in the market conditions we have right now, to get the employees they need for their business to be successful, they have to pay those rates.

Now the pool of software engineers who are available for remote work has expanded dramatically. So has the competition, as many companies are leaning into this advantage in hiring (including my employer) by looking for the best talent, regardless of where it is. When you do that, you have to pay what the competition is willing to pay, and right now that's going to be... *checks watch*... a pretty high salary and good equity compensation.

The gut check on this is to consider outsourcing to other countries. This has been a trope and fear as long as I've been aware of computers. For most of my life, people have been talking about how software development is all going to go overseas to whatever the current country of interest is: India, Russia, Ukraine, China... you hear people saying these countries with cheaper labor are going to eat our industry and we'll be out of jobs. Well, it hasn't happened. So why not? There are a few advantages to hiring an engineer in the US: you get someone who is on a compatible timezone (you could also get this in South America), who shares the same language, who has shared cultural touchpoints. And you're working within the local legal framework, which is easy: hiring across US state lines is harder than hiring in the same state, and hiring out of the country is harder than hiring within the same country. So it's just far more convenient to work with people who are in the same market, usually. But we are hiring people globally, and many teams are distributed around the globe. The effect of that? It's buoying salaries everywhere. The countries which once had the cheapest labor have been getting wealthier and salaries have been rising, and before you know it, the labor arbitrare will be hardly worth it (if it even is now).

So you can rest easy, probably: your tech salary is safe with remote work.

But of course I'd say that, because I work remote and like my salary 😉. So make your own judgments, and take your and others' biases into account. From where I sit, it looks like remote work doesn't dramatically change the market forces that are keeping salaries up. I'd be more afraid of bootcamps (and maybe when those threaten our salaries, you'll see a sudden push for licensing to keep salaries up and add barriers to entry).

ByteBuffer and the Dreaded NoSuchMethodError

Gunnar Morling — Mon, 21 Dec 2020 18:45:00 GMT

Table of Contents

How to Prevent This Situation?

The other day, a user in the Debezium community reported an interesting issue; They were using Debezium with Java 1.8 and got an odd NoSuchMethodError:

NO proprietary formats in Italy, please!

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 21 Dec 2020 18:00:00 GMT

It’s so sad that this is still necessary…

Focus

one mikro2nd — Mon, 21 Dec 2020 15:54:58 GMT

Our days on this Earth are limited. Death is coming for us all. (Except maybe Bezos who is planning to upload his mind to AWS.) It behooves us, therefore, to spend what time we have on the things that: matter interest us we are able to affect we have (or can attain) competence in Importance. Significance. Influence. Competence. Needless to say there is a large dollop of subjectivity in judging these things, not to mention a high dose of personal preference, aptitude, inclination and Dunning-Kruger effect.

My Engineering Axioms

Martin Rue — Mon, 21 Dec 2020 14:00:00 GMT

A few months back I gave a talk in which I shared a list of my personal engineering axioms – things that, over the years, I've come to think of as generally true and useful to have in mind when writing code, building things, and working with others.

Stop showing me ads on my TV player

Imran's Blog — Mon, 21 Dec 2020 02:00:00 GMT

I am a huge fan of dumb TV's made smart via a TV player, something along the likes of an apple TV or a Roku.

The first device that I owned was the first generation of a google chromecast. It worked decently enough, but it would struggle when trying to play back some of the more higher definition media that I had.

I upgraded from the chromecast to a Roku 3. It worked splendidly for a while, up until they decided to take up half the home screen with a giant ad. This pissed me off. I already paid around $100-ish dollars for the damn thing. And now they want to shove ads down my throat.

I then upgraded to an amazon fire TV. These things were not available in Canada so I had to order it from the states. It came out to about $150 and also worked fine for a short period of time. That is until, you guessed it, they started showing ads on the home screen.

I am not against advertising and I understand why it exists and its use in free products. The issue is that these things are not free (or cheap). It sucks to pay for something and to have ads shown on top of that. The shittier part is that ads are sometimes for amazon's own products like prime video or prime music. I already have amazon prime. Why are you advertising these things to me? I paid for the hardware and have a subscription on top of that. Why am I getting ads? How much money is enough money?

This pissed me off enough that I spent my evening trying to get rid of them. FTVLaunchX is an project that will let you set your launcher to anything else. There's also aptoide that's worth mentioning (install untrusted apps at your own risk).

When I feel like spending some more money (yet again) for an ad free experience I will go for a vera from https://osmc.tv/

Please for goodness sake, stop showing me ads on my TV streamer! All I need you to do is stream content from my local network to my TV! If you want to show me ads, at least let me get the hardware without paying, otherwise bugger off.

How to React to the Ledger Data Breach: 5 Recommendations

SerHack - Security Researcher — Mon, 21 Dec 2020 02:00:00 GMT

If you think you are amongst the users affected by the latest Ledger data breach, it is important not to panic! Follow a few simple steps and you can check the extent of the damage and keep an eye on the security of your account. -Continue to read the article on SerHack.me

Blog Post Index

rob.sh — Sun, 20 Dec 2020 21:47:06 GMT

40 Coolest neighbourhoods in the world

xenodium.com @alvaro — Sun, 20 Dec 2020 02:00:00 GMT

Via TimeOut's 40 Coolest Neighbourhoods in the World Right Now:

Esquerra de l’Eixample, Barcelona
Downtown, Los Angeles
Sham Shui Po, Hong Kong
Bedford-Stuyvesant, New York
Yarraville, Melbourne
Wedding, Berlin
Shaanxi Bei Lu/Kangding Lu, Shanghai
Dennistoun, Glasgow
Haut-Marais, Paris
Marrickville, Sydney
Verdun, Montreal
Kalamaja, Tallinn
Hannam-dong, Seoul
Bonfim, Porto
Ghosttown, Oakland
Chula-Samyan, Bangkok
Alvalade, Lisbon
Noord, Amsterdam
Centro, São Paulo
Holešovice, Prague
Lavapiés, Madrid
Opebi, Lagos
Narvarte, Mexico City
Uptown, Chicago
Little Five Points, Atlanta
Wynwood, Miami
Phibsboro, Dublin
Nørrebro, Copenhagen
Bugis, Singapore
Gongguan, Taipei
Soho, London
Binh Thanh, Ho Chi Minh City
Melville, Johannesburg
Kabutocho, Tokyo
Porta Venezia, Milan
Taman Paramount, Kuala Lumpur
Allston, Boston
Bandra West, Mumbai
Arnavutköy, Istanbul
Banjar Nagi, Ubud

Static analysis with semgrep: practical examples using Docker

Notes on software development — Sun, 20 Dec 2020 02:00:00 GMT

In this post we'll get a basic semgrep environment set up in Docker running some custom rules against our code.

Existing linters

Linters like pylint for Python or eslint for JavaScript are great for general, broad language standards. But what about common nits in code review like using print statements instead of a logger, or using a defer statement inside a for loop (Go specific), or the existence of multiple nested loops.

Most developers don't have experience working with language parsing. So it's fairly uncommon in small- and medium-sized teams to see custom linting rules. And while no single linter or language is that much more complex than the other (it's all just AST operations), there is a small penalty to learning the AST and framework for each language linter.

Semgrep

Semgrep is a generic tool for finding patterns in source code. Unlike traditional regex (and traditional grep) it can find recursive patterns. This makes it especially useful as a tool to learn for finding patterns in any language.

An advantage of semgrep rules is that you can learn the semgrep pattern matching syntax (which is surprisingly easy) and then you can write rules for any language you'd like to write rules for.

And while the online rule tester is awesome, I had a hard time going from that to a working sample on my own laptop with Docker. We'll do just that.

Catching print statements in Python

Let's say we want a script to fail on any use of print statements in Python:

$ cat test/python/simple-print.py
def main():
  print("DEBUG: here")
  print("DEBUG: ", "now here")

The current default example shown in the online editor happens to be for just this. Click the Advanced tab and you'll see the following:

rules:
- id: fail-on-print
  pattern: |
    print("...")
  message: |
    Semgrep found a match
  severity: WARNING

Copy this into config.yml. Let's modify the pattern to warn on all print calls, not just print calls with a single string argument:

rules:
- id: fail-on-print
  pattern: |
    print(...)
  message: |
    Semgrep found a match
  severity: WARNING

The editor doesn't mention it (nor do any docs I can find) but we also need to include two keys in the individual rule object: mode and languages.

rules:
- id: fail-on-print
  pattern: |
    print(...)
  message: |
    Semgrep found a match
  severity: WARNING
  mode: search
  languages: ["generic"]

Semgrep fails really weirdly if you set mode to anything other than search, but it won't warn you that what you set is garbage. The languages setting is similarly fickle and doesn't give you much feedback if you set it incorrectly.

Also, I'm using the "generic" language here because I don't understand the difference between languages and as far as I'm concerned the syntax I'm using here is already pretty generic.

We run the semgrep Docker image:

$ docker run -v "${PWD}:/src" returntocorp/semgrep --config=config.yml test/python
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
test/python/simple-print.py
severity:warning rule:fail-on-print: Semgrep found a match

2:print("DEBUG: here")
ran 1 rules on 1 files: 1 findings"")

And there we've got our warning!

Not completely clear to me why we're getting warned about a new version when we've pulled latest as the linked docs suggest. Maybe there's a newer version that hasn't made it into a Docker image yet.

Catching fmt.Print* statements in Go

Let's say we also want to fail on print statements in Go (because we should use a logger instead):

$ cat test/golang/simple-print.go
package main

import "fmt"

func main() {
  a := fmt.Sprintf("here")
  fmt.Println(a)
  fmt.Printf("%s\n", a)
  e := fmt.Errorf("My crazy error")
}

We could try to look for any import "fmt" code in a file but that would fail on uses of fmt.Sprintf or fmt.Errorf which are fine. Instead we'll just focus on uses of fmt.Printf or fmt.Println:

$ cat go-config.yml
rules:
- id: fail-on-print
  pattern-either:
    - pattern: fmt.Printf(...)
    - pattern: fmt.Println(...)
  message: |
    Semgrep found a match
  severity: WARNING
  mode: search
  languages: ["generic"]

Run the Go config against the Go files:

$ docker run -v "${PWD}:/src" returntocorp/semgrep --config=go-config.yml test/golang
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
test/golang/simple-print.go
severity:warning rule:fail-on-print: Semgrep found a match

8:fmt.Printf("%s\n", a)
--------------------------------------------------------------------------------
7:fmt.Println(a)
ran 1 rules on 1 files: 2 findings

Cool! Making some sense. Now let's try a harder pattern.

Catching triple-nested for loops

Let's try to warn on the triple-nested loop in this code:

$ cat test/golang/loopy.go
package main

import "log"

func main() {
  doneFirst := false
  for i := 0; i < 10; i++ {
    log.Print(i)

    for j := 0; j < 100; j++ {
      c := i * j

      going := true
      k := 0
      for going {
        if k == c {
          break
        }

        k++
        log.Print(k)
      }
    }

    doneFirst = true
  }
}

If we want to catch the use of nested for loops here then we'll need to search for the loops surrounded by arbitrary syntax. Semgrep's ... syntax makes this easy.

$ cat go-config2.yml
rules:
- id: fail-on-3-loop
  pattern: |
    for ... {
      ...
      for ... {
        ...

        for ... {
          ...
        }
        ...
      }
      ...
    }
  message: |
    Semgrep found a match
  severity: WARNING
  mode: search
  languages: ["generic"]

And run semgrep:

$ docker run -v "${PWD}:/src" returntocorp/semgrep --config=go-config2.yml test/golang
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
test/golang/loopy.go
severity:warning rule:fail-on-3-loop: Semgrep found a match

7:for i := 0; i < 10; i++ {
8:              log.Print(i)
9:
10:             for j := 0; j < 100; j++ {
11:                     c := i * j
12:
13:                     going := true
14:                     k := 0
15:                     for going {
16:                             if k == c {
-------- [hid 10 additional lines, adjust with --max-lines-per-finding] --------
ran 1 rules on 2 files: 1 findings

That's just swell.

Limits of static analysis

Now let's say we refactor one of the inner loops into its own function.

$ cat test/golang/loopy.go
package main

import "log"

func inner(i, j int) {
  c := i * j

  going := true
  k := 0
  for going {
    if k == c {
      break
    }

    k++
    log.Print(k)
  }
}

func main() {
  doneFirst := false
  for i := 0; i < 10; i++ {
    log.Print(i)

    for j := 0; j < 100; j++ {
      inner(i, j)
    }

    doneFirst = true
  }
}

And run semgrep again:

$ docker run -v "${PWD}:/src" returntocorp/semgrep --config=go-config2.yml test/golang
 A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
 running 1 rules...
 ran 1 rules on 2 files: 0 findings

Well great. The 3-nested loop still exists but we can't find it anymore because it's not syntactically obvious anymore.

At this point we'd need to start getting into linting based on runtime analysis. If you know of a tool that does this and lets you write rules like semgrep for it, please tell me!

In summary

In the end though, it's still very useful to be able to learn a single language for writing syntax rules at a high level to enforce behavior in code. Furthermore, a generic syntax matcher helps you write easily write rules for things that don't already have linters like YAML or JSON configuration or Vagrantfiles.

It can be annoying to work around some missing docs in semgrep but overall it's a great tool for the kit.

#semgrep is a really neat tool for syntactic analysis. Here are a few simple examples (catch print statements, triple nested loops, etc.) using Docker. Includes some necessary info the docs don't get intohttps://t.co/UDHEH5JmOa
— Phil Eaton (@phil_eaton) December 20, 2020

Sad follow-up on that small problem with e-voting in Italy

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 20 Dec 2020 01:00:00 GMT

Sad news, indeed.

2010 Prius Microphone

Andrew Ferguson dot NET — Sat, 19 Dec 2020 22:12:14 GMT

This is an update to Notes on Installing Sony XAV-AX100 in a 2010 Prius, specifically trying to integrate the factory microphone. Unfortunately, the microphone is pre-amplified with what I believe to be a New Japan Radio Co. 2140 Op-Amp (the black IC in the middle on front side, pin 1 is upper left corner). When… Read More »

How to find a missing Sonos

I Learnt — Sat, 19 Dec 2020 14:00:00 GMT

We recently purchased a Sonos Beam and hooked it up to 2 Sonos Play Ones we already had. It works really well and we are really pleased with it. We expected to mostly use it with the TV but have found ourselves using it more and more to play music. However every few days the Sonos app seems to lose the setup. The TV still works with it but the application does not show the system - this is very frustrating.

Changes - Ending 2020

Greg Herlein — Sat, 19 Dec 2020 10:00:01 GMT

As 2020 thankfully winds down, I leave Cisco and join AWS.

Programming Serial in C

Greg Herlein — Sat, 19 Dec 2020 10:00:01 GMT

It’s back to the future again. Fidding with node.js a lot, but you know, C is still my favorite language. Dug up some old code to read/write from a serial port.

JavaScript Objects Cachification

Alexander's Blog Blog — Sat, 19 Dec 2020 02:00:00 GMT

Wrapping your JavaScript Objects with caching capabilities

13 random reasons why i still use atom

ThatXliner's Blog — Sat, 19 Dec 2020 02:00:00 GMT

Just a little rant/vent.

2020–12–19: Away from keyboard for the rest of the month

megi's PinePhone Development Log — Sat, 19 Dec 2020 01:00:00 GMT

How to Design Almost Any Backend and Deploy It to AWS with No Code

swyx's site RSS Feed — Fri, 18 Dec 2020 21:42:48 GMT

With the Amplify Sandbox, it is really easy to model and think through any app backend scenario

The Big Tech Model

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 18 Dec 2020 20:00:00 GMT

Knowing exactly how they harm you is the first step of liberation.

Sheffield Is The New Derby; Also Arsenal

infrequent oscillations — Fri, 18 Dec 2020 12:20:42 GMT

In which we make dire predictions.

Visiting the United Kingdom

Ben Bolte — Fri, 18 Dec 2020 02:00:00 GMT

Reflections on our time in the United Kingdom during the Covid-19 pandemic.

Convos: A Persistent Self-Hosted Web Client for IRC

SerHack - Security Researcher — Fri, 18 Dec 2020 02:00:00 GMT

Throughout 2020, in large part due to the lockdown, many people have started to become familiar with internal communication tools. Microsoft Teams, Slack, Telegram, Mattermost, and a few others are the most widely used and recognizable of these tools, but there still seems to be some people using Internet Relay Chat (IRC) ― one of the first online messaging services. -Continue to read the article on SerHack.me

The Mom Test: How to talk to customers & learn if your business is a good idea when everyone is lying to you

🦉 olshansky 🦁 — Fri, 18 Dec 2020 02:00:00 GMT

Olshansky's review of The Mom Test: How to talk to customers & learn if your business is a good idea when everyone is lying to you by Rob Fitzpatrick

Perceived age velocity. Or: Why every year is getting shorter

nutcroft — Fri, 18 Dec 2020 02:00:00 GMT

Time is scary.

Everybody is afraid they will grow old. Even if death was not on the menu, being non-young is not fun. Fatigue, fragility, lack of agility.

In this quest of fear facing one could turn to stoicism. One of the arcane techniques of this resurgent philosophy is negative visualization. One could imagine they are old and wary and experience it in their imagination, which will then—presumably—cause them to make peace with it.

Every year is getting shorter, never seem to find the time.
– Pink Floyd

But spooky time has another trick. Getting older feels faster as one grows older. As if the older one gets, the faster time passes.

This is completely expected, a time scientist would say a posteriori. When one is 10 years old, a new year would include 1/10th of their memories. When they would be 11 years old, the last year would comprise 10%¹ of their lives. However, when one is 50 years old, a new year would comprise 1/50th, which is 2% of new memories when measured in time¹. Of course, then, it feels that time speeds up. Every new year will always be smaller as the already lived memories will only keep growing.

Which brings a second negativity to visualise for the stoic response: one should also assume that the next year will go by even faster than the previous one.

Wicked.

Assuming they remember everything since birth. ↩↩

Quote 94

A Holding Company — Fri, 18 Dec 2020 02:00:00 GMT

Yesterday I was clever, so I wanted to change the world. Today I am wise, so I am changing myself.

-Rumi

Setting up Matomo in such a way to bypass adblockers

JonLuca's Blog — Thu, 17 Dec 2020 21:10:04 GMT

Configuring self-hosted Matomo analytics behind custom routes and POST tracking.

Product-Market Fit is Lindy

Who is Nnamdi? — Thu, 17 Dec 2020 20:20:27 GMT

The longer you search for product-market fit, the less likely you will find it.

What THIS "Google Down" put in full view

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 17 Dec 2020 20:00:00 GMT

Google is like the One Ring of Sauron.

Siamese Neural Networks (Video)

Luke Salamone's Blog — Thu, 17 Dec 2020 19:22:43 GMT

The following is a transcript of the above video

In this paper, the authors present a novel neural network architecture to enable audio search via sounds humans are able to make, for example humming and whistling. This is an important capability when searching through audio for a specific sound.

Motivation

Imagine you have hundreds of unlabeled sound effects on your computer, and you are looking for a specific one. It could be very tedious to listen to every single one until you can find the right sound. Even if the sounds do have some kind of word labels, it could be hard to pinpoint exactly which words to search for. A lot of sounds don’t exactly lend themselves to text descriptors, so finding the right sound can be difficult with a text search.

Joshua Tree, 2020

Avy Faingezicht — Thu, 17 Dec 2020 02:00:00 GMT

In November, we rented a house in Indio with our friends, and drove down to Southern California. Oddly, I didn't take any photos at the house, but I took a bunch on the couple of trips we did into Joshua Tree National Park that week.

How Changing One Sentence led to 2.5x more Reviews

A Holding Company — Thu, 17 Dec 2020 02:00:00 GMT

June 2019
Fifteen years in business and we had a measly three reviews on Google.

Sigh.

I reminded myself that this is exactly why I moved back from the Bay area to work on the family business – to use technology to expand. Let’s fix this!

I’d read somewhere that people are most willing to review right after their experience – good or bad. I’d been itching to use Twilio, and this was the perfect use case! Our inside sales person saves a visitor’s contact information right after a visitor leaves. Their phone number is included in this.

This election and the insanity of betting markets

TODO — Thu, 17 Dec 2020 02:00:00 GMT

If you were following the 2020 U.S. presidential election, you might remember the barrage of ”experts” offering predictions on the outcome…

Making statistics lie for the 2020 Presidential election

Bytepawn - Marton Trencseni — Thu, 17 Dec 2020 01:00:00 GMT

After the 2020 US presidential election, the Trump campaign filed over 50 lawsuits and attacked the integrity of the elections by claiming there was voter fraud. One of the last lawsuits was filed in the Supreme Court of the United States by the state of Texas. Here I look at the statistical claims made in this lawsuit that were supposed to show irregularities in the Georgia vote.

A key finding from OSOR's report on Open Source

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 16 Dec 2020 20:00:00 GMT

Here is a curve that must not flatten!

Towards Continuous Performance Regression Testing

Gunnar Morling — Wed, 16 Dec 2020 18:45:00 GMT

Table of Contents

Getting Started With JfrUnit
Case Study 1: Spotting Increased Memory Allocation
Case Study 2: Identifying Increased I/O With the Database
Discussion
Summary and Outlook

Functional unit and integration tests are a standard tool of any software development organization, helping not only to ensure correctness of newly implemented code, but also to identify regressions — bugs in existing functionality introduced by a code change. The situation looks different though when it comes to regressions related to non-functional requirements, in particular performance-related ones: How to detect increased response times in a web application? How to identify decreased throughput?

These aspects are typically hard to test in an automated and reliable way in the development workflow, as they are dependent on the underlying hardware and the workload of an application. For instance assertions on the duration of specific requests of a web application typically cannot be run in a meaningful way on a developer laptop, which differs from the actual production hardware (ironically, nowadays both is an option, the developer laptop being less or more powerful than the actual production environment). When run in a virtualized or containerized CI environment, such tests are prone to severe measurement distortions due to concurrent load of other applications and jobs.

This post introduces the JfrUnit open-source project, which offers a fresh angle to this topic by supporting assertions not on metrics like latency/throughput themselves, but on indirect metrics which may impact those. JfrUnit allows you define expected values for metrics such as memory allocation, database I/O, or number of executed SQL statements, for a given workload and asserts the actual metrics values — which are obtained from JDK Flight Recorder events — against these expected values. Starting off from a defined base line, future failures of such assertions are an indicator for potential performance regressions in an application, as a code change may have introduced higher GC pressure, the retrieval of unneccessary data from the database, or SQL problems commonly induced by ORM tools, like N+1 SELECT statements.

Personal news, 2020 edition

Separate Concerns — Wed, 16 Dec 2020 14:50:00 GMT

I haven’t posted anything here for 6 months so I thought it would be a good idea to post a personal news update before the end of 2020.

Shutting down Chilli

I haven’t posted about it here yet, but my co-founder Julien and me decided to shut down Chilli at the end of last year, in agreement with eFounders. Basically, some of our initial hypotheses about the state of the market we were in (french SMBs) were wrong and the business couldn’t be profitable enough.

In retrospect, it was an interesting experience. I really appreciated being part of eFounders, I do recommend it for people who want to start a B2B SaaS company.

However, I wanted to get out of my comfort bubble too much with this one, by tackling a problem in a market I didn’t know. Because of that, I had to rely on others regarding core business choices, and while that was fine with me initially it ended up being frustrating in the end when things didn’t work well and I couldn’t help much. So if I start another company someday, it will be one where I know the problem domain better.

Joining Inch

After we shut down Chilli, I decided to join a small startup editing a SaaS CRM and ticketing solution for property managers called Inch, only available in the French market for now.

I didn’t pick Inch randomly, I have known one of the founders for years since he was a Moodstocks user! Fun fact: at the time, I had met him and he was looking for a technical co-founder for his project. I told him he should learn to code instead and gave him Ruby sample code… and this is why I am back to Rails today. Karma? :)

Anyway, I had been following Inch with interest since its creation, because it is the company I like: solving a real need in a market where tools were either terrible or non-existent. Now they have a unique place in the market and some interesting technical challenges to solve, so I decided to join and help.

Starting a family

And here comes the last piece of news: despite all the madness that happened this year, the biggest change for me is that I am now a father! My son was born last week, and I took a holiday until the end of the year to spend time with him and his mother. I don’t intend to post too much personal news here, but this one deserved it. =)

Fix bin/magento taking all the RAM

Baptiste Fontaine’s Blog — Wed, 16 Dec 2020 13:08:00 GMT

While working with Magento 2.3.6 on Bixoto I hit a weird issue: the bin/magento command-line tool was always eating all the RAM, even with a simple command:

$ ./bin/magento --help
PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 262144 bytes) in /home/baptiste/.../vendor/magento/module-store/Model/Config/Placeholder.php on line 146
Check https://getcomposer.org/doc/articles/troubleshooting.md#memory-limit-errors for more info on how to handle out of memory errors.

The issue, as weird as it sounds, is an empty configuration value that causes Magento to end up in an infinite loop.

When I installed Magento on my local machine, I deactivated HTTPS by setting web/secure/base_url to NULL in the table core_config_data. This alone is the cause of the issue.

Check in MySQL:

SELECT * FROM core_config_data WHERE path = 'web/secure/base_url' LIMIT 1;

If this shows a line with a NULL value, either delete it or replace it with some non-null value:

UPDATE core_config_data SET value='http://...' WHERE path = 'web/secure/base_url' limit 1;

This has been reported to Magento but was closed because “it’s not a bug”. I don’t think falling in an infinite loop on --help because some config value is NULL should really be a normal behavior, but at least now you know how to solve it.

ImageMagick on Azure Functions for Linux with Node.js

Kaetemi — Tue, 15 Dec 2020 23:46:37 GMT

Since you cannot install any additional dependencies on function instances, you have to include any missing tools, that you need, as static binaries in your deployment. To prepare a static...

Why You Should Pre-Sell

swyx's site RSS Feed — Tue, 15 Dec 2020 20:00:25 GMT

My thoughts on preselling, from having done it successfully exactly one time.

VM Port Forwarding

Geography & Coding — Tue, 15 Dec 2020 17:55:03 GMT

Port Forwarding in Virtual Box - Postgres example

The Stop! 2020-2021 fundraiser

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 15 Dec 2020 12:00:00 GMT

Important request/Public Service Announcement:

Guidelines for developing Custom Hooks in React

Max Rozen — Tue, 15 Dec 2020 11:48:48 GMT

You've just gotten over the fact we call them 'Hooks', but how do you stop repeating the same code everywhere? Let's learn about Custom Hooks.

Automatic Newsletter Using Drafts

trv.rs — Tue, 15 Dec 2020 09:39:00 GMT

In my day of automation post I mentioned a simple newsletter automation. It's grown since then. Some background:

A year ago, we were using Monday.com to post status updates on hard-to-see work streams for executives. Monday.com didn't catch on, but I thought there was value in having a historical log that stakeholders could check to stay current.

In addition, I spammed links to "good to read" blog posts in Slack… at all hours of the day. This was an excusable habit when I was just a colleague, once I became a manager I realized that it was very disruptive. 🤦

I started using the blog system in Confluence for both the stakeholder statuses and collect all my links and commentary for the week. At first I manually typed up each "issue," occasionally copy/pasting the format of the previous week's entry, and finding links out of Instapaper's archive and making commentary by hand. Like an animal.

Until now.

As I read throughout my week, I save little snippets using my snip tool in Hammerspoon or the Share API on iOS, and tag them link and dayjob. This forms the a queue of the links to share.

On Monday I'm greeted by a prompt in my Task Manager with a URLscheme link to Drafts that runs the following action:

let template = Draft.find("uuid-of-my-template-draft").content;

let today = new Date();
let week_ago = today.getDate() - 7;

let workspace = Workspace.find("Day Job");
let drafts =
  workspace
   .query("inbox")
   .filter(d => d.tags.includes("link") && d.createdAt > week_ago);

let links =
  drafts
    .map(function(d) &lbrace
      let content =
        d.content
          .trim()
          .split("\n");
      content[0] = "### " + content[0];
      return content = content.join("\n");
    &rbrace)
    .join("\n\n")

drafts.forEach(function(d) &lbrace d.isArchived = true; d.update() &rbrace);

let newsletter = Draft.create();
newsletter.content = `$&lbracetemplate&rbrace\n$&lbracelinks&rbrace`;
newsletter.update();

app.setClipboard(newsletter.content);

Short and sweet! After I've clicked the link from my Task Manager, I open Confluence, type c to create a page, choose the Blog template in the sidebar¹, type ,,weekly in the title field that Alfred.app text expands to "UXWEEKLY (2020.51)", and then I paste the newsletter since the last line of my Drafts action copied the draft to the system clipboard. Voila!

Just what is in my clipboard? The contents of my template Draft! Let's take a (redacted) peek.

# ProctorU Weekly Template

## 👷‍♂️ In Progress

[UX Kanban Board](link to our kanban board)

### Project #1: Fictional Title

📁: http://linktoprojectpageonconfluence.com/project-1
💬: [#proj-1-ux](https://dayjob.slack.com/archives/L337)
📈: [Link to dashboard or metrics associated with UX outcomes](http://fakelink.com/)

### Project #2: Fictional Boogaloo

📁: http://linktoprojectpageonconfluence.com/project-2
💬: [#proj-2-ux](https://dayjob.slack.com/archives/nicetry)
📈: [Link to dashboard or metrics associated with UX outcomes](http://fakelink.com/)

## 🔗 Links

That's it! I'll write my tweet-length update in-between the header and the links of the projects, @mention anyone who needs to see it, and the links magically appear below the ## 🔗 Links section of the newsletter.

Projects don't change that much, so as they come and go I'll add them and their pertinent information to the Draft Template.

This simple automation has cut down the time it takes to provide this "hub" for our work from 45 minutes per week to around 3 minutes… and that makes me happy. I'm sure there's a bunch of ways to automate this, but keeping it all in Drafts makes it cross platform and easy to change the template on the fly if I get new information.

Honestly, this is the step I mess up every week and I should automate. ↩

Adding ambient light support to Linux and GNOME

Max Leiter — Tue, 15 Dec 2020 02:00:00 GMT

A couple of a weeks ago I picked up a Surface Pro 7 (i3 1.2 mhz) at a Black Friday sale. I love making devices run software they shouldn't, so I got to work dual-booting Linux on it. To my surprise, almost everything worked with the default Fedora 33 kernel, not including the touchscreen, the ambient light sensor, and the camera. After compiling and loading the great linux-surface kernel the touchscreen started to work but I found a GitHub issue someone had made detailing that the ambient light sensor (a MSHW0184) isn't detected. This normally wouldn't matter a lot, but I'd discovered that GNOME supported adjusting the screen brightness based on the ambient light through iio-sensor-proxy.

I knew nothing (and still know very little) about the Linux kernel or drivers, but this seemed like a great place to start. Luckily, GitHub user archseer discovered that the MSHW0184 registers align with the APDS9960 device, which already has an upstream kernel driver, and he mentioned all that should be needed is a small change to allow the driver to detect the new device.

Here I realized I had to do some research into how the kernel loads drivers. I knew that drivers could either be statically built into the kernel, or built as kernel modules so they can be dynamically loaded when they're needed. But how does the kernel know when to load specific drivers for certain hardware? You register the ID with a matching table.

Initially, I tried adding the MSHW0184 device ID to the existing match table in the APDS9960 driver for the i2c protocol. This consisted of the following one-line change:

static const struct i2c_device_id apds9960_id[] = {
                    { "apds9960", 0 },
+                   { "MSHW0184, 0 }
                    {}
}

However, this didn't work and I realized the conventions seemed a little odd: no other i2c device ids in other drivers contained capitals or were ambient light sensing devices. Thankfully, someone on IRC helped me out:

00:53 <djrscally> That's an acpi ID

00:53 <djrscally> You probably need to add an ACPI match table

A few minutes later (well, probably closer to 30 minutes after compiling and testing the kernel on-device) I had the following, largely copied from other ACPI drivers:


+ static const struct acpi_device_id apds9960_acpi_match[] = {
    +	{ "MSHW0184" },
    +	{ }
    +};
+ MODULE_DEVICE_TABLE(acpi, apds9960_acpi_match);

  static struct i2c_driver apds9960_driver = {
    .driver = {
        .name	= APDS9960_DRV_NAME,
        .of_match_table = apds9960_of_match,
        .pm	= &apds9960_pm_ops,
+       .acpi_match_table = apds9960_acpi_match,
    },
    .probe		= apds9960_probe,
    .remove		= apds9960_remove

I recompiled the kernel module, rebooted, and verified the driver was matched with lsmod and ensured the driver paired with the device by navigating to the IIO device (finding the path to the device took a lot more work than I'd like to admit) and reading in the `in_intensity_clear_raw` file:

 max@surface ~> cat sys/bus/iio/devices/iio:device0/in_intensity_clear_raw
    33

The joy I felt seeing that file exist and the output was monumental. I had something to work with!

I wasn't done yet though — GNOME still didn't show me an option to automatically adjust the screen brightness. After someone else verified they also had the device loaded with the modified driver but not in GNOME I determined the problem was somewhere in iio-sensor-proxy. I cloned the iio-sensor-proxy repository and started digging.

The first thing I always do when I clone a new repo is expand and quickly look over every folder (assuming the repo is a reasonable size) and that served me well here. I found the following file `80-iio-sensor-proxy.rules`:

# iio-sensor-proxy
# IIO sensor to D-Bus proxy

ACTION=="remove", GOTO="iio_sensor_proxy_end"

# Set the sensor type for all the types we recognise
SUBSYSTEM=="hwmon", TEST=="light", ENV{IIO_SENSOR_PROXY_TYPE}+="hwmon-als"
SUBSYSTEM=="iio", TEST=="in_accel_x_raw", TEST=="in_accel_y_raw", TEST=="in_accel_z_raw", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-poll-accel"
SUBSYSTEM=="iio", TEST=="scan_elements/in_accel_x_en", TEST=="scan_elements/in_accel_y_en", TEST=="scan_elements/in_accel_z_en", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-buffer-accel"
SUBSYSTEM=="iio", TEST=="scan_elements/in_rot_from_north_magnetic_tilt_comp_en", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-buffer-compass"
SUBSYSTEM=="iio", TEST=="in_illuminance_input", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-poll-als"
SUBSYSTEM=="iio", TEST=="in_illuminance0_input", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-poll-als"
SUBSYSTEM=="iio", TEST=="in_illuminance_raw", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-poll-als"
SUBSYSTEM=="iio", TEST=="scan_elements/in_intensity_both_en", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-buffer-als"
SUBSYSTEM=="iio", TEST=="in_proximity_raw", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-poll-proximity"
SUBSYSTEM=="input", ENV{ID_INPUT_ACCELEROMETER}=="1", ENV{IIO_SENSOR_PROXY_TYPE}+="input-accel"

ENV{IIO_SENSOR_PROXY_TYPE}=="", GOTO="iio_sensor_proxy_end"

# We got here because we have a sensor type, which means we need the service
TAG+="systemd", ENV{SYSTEMD_WANTS}+="iio-sensor-proxy.service"

As you might be able to figure out, each TEST file is checked for existence and is used to determine whether or not the device should be used by iio-sensor-proxy.

I hadn't seen any of the seemingly relevant files, inilluminance*, so I added my own line:

SUBSYSTEM=="iio", TEST=="in_intensity_clear_raw", ENV{IIO_SENSOR_PROXY_TYPE}+="iio-poll-als"

After another small iio-sensor-proxy change and running again the device was now discovered by iio-sensor-proxy! The full merge request can be seen here. The option appeared in GNOME and I now have automatic adjusting brightness!

It kind of sucks though and isn't very consistent. As it turns out, intensity_clear != illuminance, which is what most programs expect from ambient light sensors, so I need to figure out and perform some math in iio-sensor-proxy to translate the RGBC values to lux.

If you've made it this far (or are here to get the code yourself), the modified driver (with basic proximity sensing support too) can be found here: https://github.com/maxleiter/MSHW0184.

Major thanks to everyone that helped me in the ##linux-surface IRC channel. It was great to finally get a little low-level with Linux.

*This post was originally written on the old maxleiter.com, and formatting has slightly adjusted

2020–12–15: Battery thermistor measurements

megi's PinePhone Development Log — Tue, 15 Dec 2020 01:00:00 GMT

Five critical things for our oceans

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 14 Dec 2020 20:00:00 GMT

and their “digital” implications.

Disable Google Autoupdater on macOS

Caius Theory — Mon, 14 Dec 2020 18:14:53 GMT

From reading Chrome is Bad, it seems in some situations the updater (also known as keystone) can chew up CPU cycles. Whilst I’m not 100% convinced keystone continuously chews CPU, its launchctl configuration suggests it runs at least once an hour. Given I don’t use Chrome as my main browser, this is undesirable behaviour for me.

With that in mind, I’ve decided to disable the background services rather than delete Chrome entirely. (I need it occasionally.) Stopping/unloading the services and fettling the config files to do nothing achieves this aim (and stops Chrome re-enabling them next launch), whilst leaving Chrome fully functional when needed.

Unload the currently loaded services

launchctl unload -w ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
launchctl unload -w ~/Library/LaunchAgents/com.google.keystone.agent.plist

Empty the config files, so if launchd ever tries to launch them they’ll just error out

echo > ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
echo > ~/Library/LaunchAgents/com.google.keystone.agent.plist

Change ownership and permissions of these files so only root can write to the files

chmod 644 ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
chmod 644 ~/Library/LaunchAgents/com.google.keystone.agent.plist
sudo chown root ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
sudo chown root ~/Library/LaunchAgents/com.google.keystone.agent.plist

Now when I want to update Chrome once in a blue moon when I need it, I can navigate to chrome://settings/help to update (or from the UI, Chrome -> About Chrome.)

Benchmarking Rustpython With Criterion

Tom Forbes — Mon, 14 Dec 2020 15:27:30 GMT

I’ve been very excited about RustPython since I saw the project being presented at FOSDEM 2019. Last week I finally got to contributing something to the project ! For context, RustPython benchmarks specific Python files against CPython, the “reference implementation” for Python (read: the only real ...

Ethan Magnass' sermons from Grace Anglican Church in Grove City, PA

Josh Thompson — Mon, 14 Dec 2020 15:00:00 GMT

update from a few years later: I tried a bunch of stuff to keep myself within evangelicalism. failed, so iterated through another option, sorta in the style of exit, voice, and loyalty: responses in decline of quality of firms. I’ve firmly left the club, and warmly invite anyone else finding themselves within it to do the same: On Leaving Evangelicalism and Opposing It

I’ve been recommending a collection of sermons to many people recently.

I’ve listened to each of these sermons quite a few times. They’re worth your time.

Ethan Magness is the rector at Grace Anglican Church in Grove City, PA.

Sermon Series on Joseph

Grace Anglican Church podcast feed (PocketCasts, Apple Podcasts)

Sermon on Matthew 5:43-48

From November 4, 2018: Gollum & the Outstretched Hand - Matthew 5

Relevant scripture:

“You have heard that it was said, ‘Love your neighbor and hate your enemy.’

But I tell you, love your enemies and pray for those who persecute you, that you may be children of your Father in heaven. He causes his sun to rise on the evil and the good, and sends rain on the righteous and the unrighteous.

If you love those who love you, what reward will you get? Are not even the tax collectors doing that?

And if you greet only your own people, what are you doing more than others? Do not even pagans do that?

Be perfect, therefore, as your heavenly Father is perfect.

I have far more thoughts on the topic, but for now, this should help you, the reader, find these excellent sermons.

Todo

I’d like to get high-quality transcripts of these sermons. There are services that do this (Like Rev), I’ll pay for it sometime soon. I want good transcripts, not just AI-generated transcripts. It took me like two hours to clean up a few minutes of AI-generated text for My Thoughts on Eric Weinstein’s Thoughts on Pia Kalani’s Thoughts

At some point I’ll weave together other threads I’ve got sketched out, but are not ready for public consumption.

Additional Reading

Sketchnotes: UX Vision Intensive

trv.rs — Mon, 14 Dec 2020 08:37:00 GMT

This past week our team attended the Leaders of Awesomeness UX Vision Intensive. Jared Spool led the sessions, and I was very impressed and grateful to the time he gave us for free.

If you want a higher-resolution copy, you can download the pdf.

I struggled with the difference between a vision an an envisionment. If I understand correctly, a UX vision is a depiction of a user's timeless experience, with major friction alleviated by a nearly magic product you are imagining. An envisionment is a near future implementation of a UX vision with possible technology. I have been keeping it straight by saying that Star Trek TOS is a UX Vision, where Star Trek TNG is an envisionment. 🤷‍♂️¹

We are super excited to take the lessons to heart and bring a UX Vision to life… talking about that this week. I'm especially looking forward to some of the workshop exercises… "despicable design"² sounds like fun!

I've been playing around with taking notes in the sketchnotes style for a year or so…³ this digital sketchnote I made in GoodNotes, and am very happy with the tool. It has a good mix of power and simplicity that works really well for me… and even my handwriting is searchable! Highly recommended.

If I'm misrepresenting these, please let me know… I'm a TNG guy myself. ↩
In Despicable Design, you ask your team the question: "how could we make the users experience worse?" Sort of an ice breaker, and it quickly highlights which elements upon which the user's experience hinges. ↩
I may go back and find my first couple of sketchnotes. The first 10 were not nearly this good… and the ability of a digital note to zoom and undo makes it a lot easier to make something neat and clean. Just keep at it! ↩

Announcing the AutoEPM Patch Tracker

Subclassed — Mon, 14 Dec 2020 03:05:16 GMT

UPDATE: This service has been discontinued; this post is left as historical reference. If you need this sort of thing, contact me on LinkedIn.

With the recent release of EPM 11.2, I thought it would be a good time to make public a tool I've built and used "behind the scenes" for quite some time, the AutoEPM Patch Tracker.

It's a patch tracker focused on EPM products, automatically aggregating updates and pointing to readmes, hopefully a bit faster to peruse than My Oracle Support. It also recommends Critical Product Updates that can be applied to the EPM stack, and the most common "standard" updates (only for 11.1.2.4 at the moment). With a free account via LinkedIn sign-on, all lists can be downloaded in CSV, JSON, and XML (Atom).

It's only a small showcase of what we can do at AutoEPM by Targlet. If you'd like to find out how to integrate this level of automation and intelligence to your EPM enviroments, feel free to reach out! I'll be at OpenWorld London next week as well, always happy to have a chat.

Mimicry vs Reflexivity

swyx's site RSS Feed — Mon, 14 Dec 2020 00:35:02 GMT

Two opposing forces that describe how perception and reality do battle.

Smaller, Faster-starting Container Images With jlink and AppCDS

Gunnar Morling — Sun, 13 Dec 2020 21:30:00 GMT

Table of Contents

Creating a Modular Runtime Image for a Quarkus Application
Adding an AppCDS Archive to a Custom Runtime Image
Creating a Linux Container Image
Let’s See Some Numbers!

A few months ago I wrote about how you could speed up your Java application’s start-up times using application class data sharing (AppCDS), based on the example of a simple Quarkus application. Since then, quite some progress has been made in this area: Quarkus 1.6 brought built-in support for AppCDS, so that now you just need to provide the -Dquarkus.package.create-appcds=true option when building your project, and you’ll find an AppCDS file in the target folder.

Things get more challenging though when combining AppCDS with custom Java runtime images, as produced using the jlink tool added in Java 9. Combining custom runtime images with AppCDS is very attractive, in particular when looking at the deployment of Java applications via Linux containers. Instead of putting the full Java runtime into the container image, you only add those JDK modules which your application actually requires. (Parts of) what you save in image size by doing so, can be used for adding an AppCDS archive to your container image. The result will be a container image which still is smaller than before — and thus is faster to push to a container registry, distribute to worker nodes in a Kubernetes cluster, etc. — and which starts up significantly faster.

SSH in Virtual Machine Ubuntu Server 20.04

Geography & Coding — Sun, 13 Dec 2020 18:58:46 GMT

A quick guide to setup a SSH-conncection to Virtual Machine Ubuntu Server 20.04

Secrets of Sand Hill Road: Venture Capital and How to Get It

🦉 olshansky 🦁 — Sun, 13 Dec 2020 02:00:00 GMT

Olshansky's review of Secrets of Sand Hill Road: Venture Capital and How to Get It by Scott Kupor

Beep, beep, I'm a sheep

zserge's blog — Sun, 13 Dec 2020 02:00:00 GMT

Audio I/O is a complex topic, which scares away many inspired musicians who are into programming (or programmers who are into music). Let’s try making it look less complex and see how audio works on each modern desktop platform. Our show case today will be a simple beeper. Remeber that annoying thing inside your PC boxes producing the irritating buzzing sound? It’s long time gone, but I suggest to sumon its soul and make a library that plays beeps across all the platforms.

The Perfect Neapolitan Pizza

Brent's Blog — Sun, 13 Dec 2020 02:00:00 GMT

In Italy a society; Associazione Verace Pizza Napoletana makes the rules when it comes to every aspect of the Naples pizza and they describe the rules of a Naples pizza in extreme technical detail here this is a great read if you are a pizza nerd and it details the exact procedure to cook a perfect Napoletana pizza.

Peri Peri Chicken

Brent's Blog — Sun, 13 Dec 2020 02:00:00 GMT

Chicken Thigh (With or without skin / bone - to taste)

The immensely stupid fragility of our "high-tech" way of living

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 12 Dec 2020 20:00:00 GMT

It’s embarrassing. It really is.

Talapus Lake

Sam Hooke — Sat, 12 Dec 2020 07:30:00 GMT

Talapus Lake. The best photo I have, because for some unknown reason I barely took any photos on this hike.

Distance	7.6 km / 4.72 miles
Elevation gain	233 m / 765 ft
Duration	3h10m
Moving	2h10m

Normally we like to start hiking bright and early, but today we went on an afternoon hike with friends.

After turning off the I-90, the drive up FR 9030 was pretty hairy. It’s a gravel road, with sharp blind bends, and at some points it’s rather narrow with a steep drop to the side through dense forest. By itself this wouldn’t have been an issue, but we weren’t the only people driving here. We were caught in a perfect storm between the morning hikers, who had got those precious spaces at the trailhead but were now trying to drive home; the midday hikers, who had found the trailhead car park full and opted to park in various places on the side of the narrow road; and the hopeful afternoon hikers (us included) who were playing slow motion dodgems on the side of a mountain.

Something to monitor about e-voting in Europe

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 11 Dec 2020 20:00:00 GMT

A compendium of inherently wrong things…

You Can Run Containers on AWS Amplify Now

swyx's site RSS Feed — Fri, 11 Dec 2020 16:24:37 GMT

AWS just added the easiest way to deploy long running serverful code alongside a serverless architecture.

Preventing auto-locking and sleeping by simulating user activity

Carey Metcalfe — Fri, 11 Dec 2020 06:12:00 GMT

This post will detail how to simulate activity on your computer in order to prevent it from auto-locking or going into sleep mode. Note that this will usually also prevent the "auto-away" functionality of various chat programs from ever marking you as "away".

Generally you should change your operating system's settings to disable sleeping and auto-locking if possible instead. These methods are for when you don't have the access or permission to change those settings (ie. locked-down devices).

Windows

This method runs a Powershell script on login that toggles scroll lock on/off every minute. Since scroll lock mostly doesn't do anything on modern systems and the script will press it twice to immediately unlock/relock it, this is basically unnoticeable to the user. However, if you have issues simply swap the two {SCROLLLOCK}s in the below command to something else. A full list of special keys can be found here.

Open Explorer (shortcut: Win+E)
Paste %APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup into the location bar and hit enter to navigate there.
Right click in the folder and click New > Shortcut to open the shortcut creation wizard.

Paste the following as the shortcut's location:

powershell.exe -windowstyle hidden -command "$myshell=New-Object -com \"Wscript.Shell\";while(1){$myshell.SendKeys(\"{SCROLLLOCK}{SCROLLLOCK}\");Start-Sleep -Seconds 60}"

Give it any name you like and hit Finish

macOS

This method runs a shell script on login that uses a tool called cliclick to move the mouse one pixel left and right every minute. This is such a small and fast movement that it's usually not noticeable unless you're really looking for it.

Install cliclick from the project website or via brew (brew install cliclick)
Create a script called jiggle with the following contents:

#!/bin/sh
while true; do
    cliclick 'm:-1,+0' 'm:+1,+0'
    sleep 60
done

Make the script executable (chmod +x jiggle)
Open System Preferences, search for "login items" and hit enter.
Click the + button, select the jiggle script and hit "Add". You should see it appear in the list of programs as a "Unix executable". Check the "Hide" checkbox beside it and exit.

Xorg-based Linux

Much like the macOS version above, this moves the mouse one pixel left and right every minute. If you would prefer to instead use a keyboard-based method like the above Windows version, use xdotool key Scroll_Lock twice instead of the xdotool mousemove_relative * commands in the following script. More special key names for xdotool can be found here.

Install xdotool (usually available via your package manager)
Create a script called jiggle with the following contents:

#!/bin/sh
while true; do
    xdotool mousemove_relative --sync -- -1 0
    xdotool mousemove_relative 1 0
    sleep 60
done

Make the script executable (chmod +x jiggle)
Configure your OS to run the script at login. Usually this would be done through the desktop environment's settings or via something like systemd.

Wayland-based Linux

Some preliminary research suggests that this is possible on Wayland using ydotool as a replacement for xdotool. I don't currently run a Wayland-based setup so I can't test it. If you manage to find a solution for Wayland feel free to send it to me and I'll update the post.

Educational Logic Gate Board

Kaetemi — Fri, 11 Dec 2020 03:41:41 GMT

This is a board I based roughly on a setup we had in high school, that I was taught logic gates with, once upon a time. I designed the layout...

I received first-ever donation on my open-source side project and it felt great!

Gourav Goyal — Fri, 11 Dec 2020 02:00:00 GMT

I decided to build an extension for VSCode. That would add handy buttons for many common commands like save file, format file, show opened files etc.

Capitalism and the Jews

🦉 olshansky 🦁 — Fri, 11 Dec 2020 02:00:00 GMT

Olshansky's review of Capitalism and the Jews by Jerry Z. Muller

Dr. Loopback, Or: How I Learned to Stop DOSing Myself and Love the Pi-Hole

Lambda Land — Fri, 11 Dec 2020 02:00:00 GMT

I noticed that my Internet was acting strangely: whenever I visited a web page, my browser would hang for a good second or two before it started loading anything. Zoom calls worked without a problem for school, so this tipped me off that something was wrong with the DNS lookup or the handshake.

Sure enough, I popped open my Pi-Hole admin console, and was greeted with this:

The green number in the Total Queries box would jump by 10, 20, or sometimes even 100 every second.

I tried turning off, disconnecting, and rebooting my various computers and devices to see if it was just some rogue process messing everything up. No change. I rebooted my router. No change. I updated and restarted the Pi-Hole. Still no change.

I noticed that a lot of requests to the domain lb._dns-sd._udp.0.0.0.10.in-addr.arpa. I did a web search and found a single post on a forum that suggested turning off conditional forwarding on the Pi-Hole.

I went into admin settings, turned off conditional forwarding, and was rewarded with this:

Now my pi isn’t overloaded with DNS requests, and my Internet is as snappy as it used to be.

What happened, I think, was that a device on my network would try and lookup some name. The router would forward this request to the pi, which would then forward it back to the router, which would then send it back to the pi, etc. Boom. Infinite loop, and all my DNS requests got bogged down.

I’m not very good with networking; that’s probably my biggest weakness. I learned a thing today, though!

When algorithms meet debt...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 10 Dec 2020 20:00:00 GMT

Everyone must be aware of four (no, five) problems. Especially lawyers.

Email whitelisting

infrequent oscillations — Thu, 10 Dec 2020 19:17:53 GMT

Settings -> Mail -> Accounts -> [appropriate account] -> turn on Notes Go to iOS Notes. Go to the Folders view and find the same account as step 1. Create a note with the first line “Whitelist”, then a blank line, then each domain you want to whitelist on its own line. Wait for the automatic processing to pick up the changes (a few times a day.) Note this isn’t a pure whitelist - instead it adds a base score of -12.

Sounds from Silence: Graeme Clark and the Bionic Ear story

Macleod Sawyer | Web Portal Feed — Thu, 10 Dec 2020 14:06:36 GMT

This story has a very deep place in my heart, my mother was born deaf and after countless surgeries in her early youth she was able to regain (or to have the start of) the hearing in her left ear. She is still entirely deaf in her right, but I am very happy that she can hear at all - if she hadn’t, both her and my own life would be far different (and I wouldn’t likely have existed at all!).

Growing up I always had to subtitles on, driven to speak loudly and clearly, and learn just enough sign language (not much, but being italian grants you the starting ability to hand wave at all hours of the day). This story catalogues all the attempts to create and invent a medical device that could restore or grant the power of hearing. The story is excellently written, and written by the very man who has changed millions of lives over the past few decades, Professor Graeme Clark.

This story goes into the thought process of the device, the manufacturing, the attempt to raise funding (many times! many failures!), the stress of integration within the deaf community, and many more. If you have an interest in technology and how it is a net-benefit of humanity, or are worried about the use of technology, this should be your first choice to see how great the world can be.

Link to https://amberwilson.co.uk/blog/grow-the-indieweb-with-webmentions/

Qubyte Codes — Thu, 10 Dec 2020 12:45:11 GMT

I really like webmentions, so it makes me happy when folk blog about them! Grow the IndieWeb with Webmentions - Amber Wilson

Haskell coreutils - tee

Anardil — Thu, 10 Dec 2020 10:00:00 GMT

Implementing tee in Haskell

Downloading your Favorite YouTube Playlist Automatically

Thu, 10 Dec 2020 02:00:00 GMT

In light of Youtube-DL being taken down from GitHub, I decided to give it a go with a use case I happened to have.

Lately, I've been listening to lots of concerts/festival sets that are not available for traditional purchase. Although I have listened to them on youtube, I didn't want to have the web page open and the auto-play/queuing features are not as fledged out as a proper music player.

I decided to write a quick script that I could run in a cron to pull down the latest version of the playlist I maintain. To keep an eye on it, I included a Slack webhook notification that let's me know when the playlist has been downloaded.

My first version worked but I quickly realised that the script did not handle updates, only a complete re-download. To resolve this, youtube-dl has a --download-archive flag that keeps track of downloaded videos.

Check out the code below and hopefully you find it useful.

#!/bin/bash

playlist_url="https://www.youtube.com/playlist?list=ZZZ"

cd ~/Projects/music

youtube-dl --embed-thumbnail --download-archive downloaded.txt --no-post-overwrites --extract-audio --audio-quality 0 --format bestaudio --audio-format mp3 --yes-playlist --output "%(title)s-%(id)s.%(ext)s" $playlist_url

curl -X POST --data-urlencode "payload={\"username\": \"youtubeb0t\", \"text\": \"Completed download of all Music playlist.\", \"icon_emoji\": \":tv:\"}" https://hooks.slack.com/services/ZZZ/YYY

Bloom Filters - Much, much more than a space efficient hashmap!

Ben E. C. Boyter — Thu, 10 Dec 2020 02:00:00 GMT

A bloom filter is one of those data structures you are probably already aware of, or have at least heard about. For those looking for a simple recap, they are a probabilistic data structure which can be used to determine if something is in a set or not, giving a slight chance of returning a false positive result for some checks but while using less space than a full hashmap.

The planet is obese

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 09 Dec 2020 20:00:00 GMT

No, not humans. The WHOLE planet. We just take the merit.

Requests in Python with requests package and Torpy

Geography & Coding — Wed, 09 Dec 2020 19:54:50 GMT

1
2
3


import requests
response = requests.get("https://webscraper.io/test-sites/e-commerce/allinone")
response

The Strange Case of the Danish Aarhus Mafia

swyx's site RSS Feed — Wed, 09 Dec 2020 17:46:36 GMT

Why are so many CS pioneers from Denmark?

Leading with Data - Cascading Metrics

blog.harterrt.com — Wed, 09 Dec 2020 10:00:00 GMT

It's surprisingly hard to lead a company with data. There's a lot written about how to set good goals and how to avoid common pitfalls (like Surrogation) but I haven't seen much written about the practicalities of taking action on these metrics.

I spent most of this year working with …

Emulating a Computer: Timing and Input

River Gillis [riv.dev] — Wed, 09 Dec 2020 04:44:00 GMT

Correctly emulating time in a CHIP-8 emualtor.

#!shebangs

voussoir.net/writing — Wed, 09 Dec 2020 04:35:08 GMT

Back to writing

#!shebangs

Introduction
1. Windows file extensions
2. Unix shebangs
I don't like shebangs
Hedges
1. The registry
2. .mp3.exe
Conclusion

Introduction (§)

Windows file extensions (§)

On Windows, the file extension of a filename is used to automatically determine how the file should be opened when you double-click on it or run it on the command line. .exe are the kings, because they are Windows's native executables. For other extensions, the shell checks the registry to find the exe associated with the extension, and runs that program with your file as the first argument, plus the rest of your arguments. For example:

> ipconfig.exe → runs directly
> whatsmyip.py → python.exe whatsmyip.py
> dns.py www.example.com → python.exe dns.py www.example.com
> clock.pyw → pythonw.exe clock.pyw
> report.doc → winword.exe report.doc

If you remove the .exe extension from an executable file's name, the shell won't know what to do with it (but deeper APIs can still invoke it (one, two)). That is to say, Windows does not inspect the content of files to determine their type for the purpose of double-click launching. Every file is either an exe or has an extension which maps to an exe, or has no extension and can't be associated to a program. [1]

At no time does this prevent you from using any other executable to open a file. You can always pythonw.exe dns.py or winword.exe clock.pyw. Extensions are not restrictions, they are just shortcuts to launching your executable of choice.

[1] There is also .com, a legacy executable format which is natively executable, yet has a passthrough HKCR entry anyway, presumably because the shell is programmed to consult HKCR for anything that's not .exe. HKCR\comfile\shell\open\command\(Default)="%1" %*

Unix shebangs (§)

On Unix operating systems, the extension is not used to help execute the file. Instead,

The file must have the x permission, aka "executable bit", enabled by chmod +x filename.
If the file is a native executable, it will run directly. Otherwise, the system will look for the shebang. The system reads the first few bytes of the file to figure this out.

The shebang is a line which must be at the very top of the file and must start with #!. Then, a path to an executable is given, which will actually interpret and run the remainder of the file. A Python file with a shebang might look like this:

#!/usr/bin/env python
import sys
print(sys.executable, sys.argv)

You can save this file using any extension you want, or no extension at all, and run it by calling chmod +x ./filename and then ./filename. As always, you're welcome to pass the file as an argument to any other executable if you need to use a different interpreter.

One of the principles here is that executables become abstractions, black boxes. The idea is that as a user, when you run some program on the command line, you shouldn't need to know or care whether program is a shell builtin, or an alias, or an executable on the PATH; and you shouldn't need to care what language it's written in. You could swap out a dns program written in bash with one written in Python, and your callers won't know the difference.

But...

I don't like shebangs (§)

And you won't find a shebang on any of the code I write.

For these arguments, suppose that the file in question is checked into version control (e.g. git), or is seeding on IPFS, or is otherwise being hash-validated, or is restored to factory condition on every software/OS update, such that "just change the shebang" is not an appropriate solution. The file is handcuffed to the interpreter specified by the shebang.

My dislike for shebangs is part principle and part practical. Principles like:

As a user, why should I be expected to agree with the developer's choice of interpreter and pathing (/bin/x vs /usr/bin/x vs /usr/bin/env x ...)? You can search stackoverflow and find plenty of discussions about the pros and cons of each technique. But I'm the user and I'll do what I want, stop subjecting me to your chosen set of pros and cons when I'm the one running the program.
As a developer, why should I add this boilerplate to every file I write? Why should I be encoding "this is how you run my program" in the file, when the user can simply configure their system to do it however they like best?

And practicals like:

My python is really not the same as your python, so stop assuming /usr/bin/env python is the right thing.
The shebang handcuffs also apply to hardlinks / symlinks which I might intentionally want to behave differently.

I will continue to use Python as the example language for this argument because that's what I'm most familiar with, but you may extend these issues to any case where there are alternative interpreters — even if the alternatives haven't been invented yet!

Developer over user (or else bye-bye checksums) (§)

Back in the day, Python code was written in Python 2, and it was fair to assume that the executable named python on the PATH was the Python 2 interpreter, so people wrote shebangs pointing to python expecting P2. Now, code is written in Python 3 and systems are shipping with python referring to P3, but some P2 tools are still around and in use. So now their shebang is wrong, and will be wrong for the rest of time, and the user will always need to explicitly call python2 program.py for as long as they hang on to that file. Great, thanks.

On an extension-based system, the solution is easy. Although the program.py file might be checked into version control, there's nothing stopping me from creating a symlink or hardlink on my PATH called program.py2 that points to the file, and an extension mapping from .py2 to python2.exe. This requires no changes to the file content itself, so it won't interfere with the file's hash.

To be clear, the file layout I would use for maintaining a Python2 and Python3 program simultaneously is:

D:\
├───cmd\ on the PATH
│   ├───a_program.py2 => D:\software\a_program\a_program.py
│   └───b_program.py => D:\software\b_program\b_program.py
└───software\
    ├───a_program\
    │   └───a_program.py
    └───b_program\
        └───b_program.py

So even if a_program.py was written when .py implied Python2 instead of Python3, it doesn't matter because the names I actually execute are those in \cmd on my PATH. I can name my links any way I want to take advantage of the file extension mapping, while the real software is not affected whatsoever. See Master of my domain for more about my software management practices.

Sure, you can say that nobody should be using P2 programs any more, but that's moving the goalposts. Let's say someone wants to run some particular set of Python scripts through PyPy or any other implementation. What then? To give one non-Python example, here is an SO thread about the right shebang for bash. To my eyes it's a disaster:

Some people use /bin/sh under the assumption that all people have that symlinked to bash, which is wrong.
/bin/bash or /usr/bin/bash might be ok, but there's no guarantee that's the path for bash.
/usr/bin/env bash is probably most common, but there's still no guarantee that's the path for env!

On a shebang-based system, the choices are:

Edit the shebang in the file, thus changing its hash.
Create a symlink for the interpreter executable that matches the developer's shebang but actually points to the the user's preferred interpreter. E.g. the program expects /usr/bin/python so I create a symlink /usr/bin/python => /.../python3
Create a separate launcher script, a one-liner shell file that calls the preferred interpreter with the script as an argument — essentially a handwritten symlink. This option is most similar to the extension-based solution, except that if the preferred interpreter changes you've got to update all of the launcher files manually.

Anti-Windows dogmatism (§)

I don't like shebangs, but it's not such a big deal — all I have to do is not use them and that's that. They pose no restrictions upon me as long as I call the interpreter myself.

What actually bothers me are the Unix elitists who in a single breath will mock Windows users as being stupid, helpless lusers, pushed around by the Big Bad File Extension Who Dictates How Files Are Executed... then write a shebang that handcuffs their file to the name of the anticipated interpreter executable. A .rose by any other extension would smell as sweet.

To demonstrate this isn't a strawman, here are some actual quotes gathered from around the 'net:

I know that files are identified by an identifier on their first line, which is why Linux files don't have file endings. But I liked file endings a lot. Isn't it nice to be able to tell what a file is/does without having to open it? In Windows I could open up a folder and immediately have a lot of information about the folder and the files within it, just from the file endings.

If you're asking if there's some kind of setting that will "show" file extensions on system files, no, that is not a thing. You seem really hung up on the idea that filename extensions are a superior method of associating type information with files. Most Linux applications just don't use this method, but have other ways of determining that information. Your question is based on the assumption that this is some kind of feature which has been disabled and can be re-enabled, which is fundamentally untrue.

I will not: I will not allow the ignorance that spews from Redmond to infect my software.

I will not: I will not cater to users with a broken OS.

I will not: I will not bother because it represents the minimal technical hurdle for Windows users.

[A developer renames his readme to readme.txt]

While it may make logical sense to some people, the ultimate implication when you see a .txt file is that the developer uses Windows. I'm not sure a lot of developers are comfortable with that.

No. File extensions are stupid and broken.

Why? What's wrong with encoding metadata about the content in the filename?

Because it then prevents me from changing the filename the way I want to. Because legacy garbage.

Interesting... most commenters here (so far) seem to be against the extension. Can someone explain why? "README" doesn't tell me if it's text, or markdown, or HTML, or anything. Having ".txt" lets me know the format without opening it first to see.

I hate them because they're an abstraction leak. The type of a file's content is different from the application that I want to use to manipulate it, which both are in turn different from the name I chose to give the file. That current file systems and the GUI representations on top of them are so badly designed as to confuse three unrelated concepts is a failing of software and as clear an indication as you could ask for of the triumph of Worse Is Better.

These two in particular point out a problem that's already solved on Windows:

You should not use an extension for executables, as they they are not interchangeable. Imagine that you have a shell script a.sh, then re-write in python a.py, you now have to change every program that calls your script, you have leaked an implementation detail.

If you give a shell script a .sh extension, you'll have to type that .sh as part of the command name when starting it. That's the main reason why I don't like putting that extension in.

We need to talk about PATHEXT.

PATHEXT (§)

On Windows, there is an environment variable called PATHEXT. It contains a semicolon-separated list of file extensions that you don't have to type when running programs on the command line. For example, if .py is on PATHEXT, then you can just ask for dns on the commandline and dns.py will be a valid candidate. On the other hand, if PATHEXT is blank, then you must type the file extension for every executable, even .exe. At that point, the shell builtins will be the only extensionless commands you can run.

The order of extensions on PATHEXT indicates the order of preference for candidate files.

C:\anywhere>set PATH=C:\cmd;C:\Windows\System32
C:\anywhere>set PATHEXT=.py;.bat;.exe
C:\anywhere>where dns
C:\cmd\dns.bat
C:\cmd\dns.exe
C:\cmd\dns.py
C:\anywhere>dns
You ran dns.py!

C:\anywhere>set PATHEXT=.bat;.py;.exe
C:\anywhere>dns
You ran dns.bat!

C:\anywhere>set PATHEXT=.exe;.bat;.py
C:\anywhere>dns
You ran dns.exe!

C:\anywhere>set PATHEXT=.py
C:\anywhere>where.exe dns
C:\cmd\dns.py
C:\anywhere>where.exe dns.bat
C:\cmd\dns.bat
C:\anywhere>dns
You ran dns.py!

Notice I am now calling where.exe, since .exe is no longer on PATHEXT, thus where no longer matches C:\Windows\System32\where.exe.

C:\anywhere>set PATHEXT=
C:\anywhere>where.exe dns
INFO: Could not find files for the given pattern(s).
C:\anywhere>where.exe dns.py
C:\cmd\dns.py
C:\anywhere>dns.py
You ran dns.py!

Regardless of the value of PATHEXT, you are always allowed to specify the full name of the file, like dns.py. Even if .py is not on PATHEXT, dns.py itself is still on the PATH. PATHEXT only affects the evaluation of the bare name, dns.

And just like that, we've achieved the black box principle of executables. You can rewrite a .bat program in .py, or you can supercede a System32 exe with a script of your own. The only thing you can't do is supercede the shell builtins — dir always does dir even if you have a candidate in the cwd or on the PATH — you'll have to call those files by their extensioned names.

Between my public cmd repository and my non-public cmd folder, I've got more than 180 .py and .bat files making up my command line toolkit. Of course I'm not specifying the extensions when I run them, guys.

Hedges (§)

It's time for me to bat for the other team and explain why Windows's solution is also bad.

The registry (§)

The Windows registry is not a beautiful place, and the mapping of extensions to executables is not as easy as one would hope. With Windows 10, Microsoft has introduced some new APIs for dealing with extension associations, mostly because with the old APIs, inconsiderate software developers would non-optionally dive into the registry and set their program as the default for a bunch of filetypes during install, leaving the average user without a clear understanding of how to revert. However, I haven't learned these new systems because Windows refers to them as "default apps" and I have a vendetta against the word "app" so I avoid that whole interface at all costs. When I need to edit extensions, I always just open the registry and do it myself.

The most straightforward technique that I'm familiar with is to create a key HKEY_CLASSES_ROOT\.ext. You can give it a (Default) value containing the name of another HKCR key, and this will act as a redirect for the remainder of the process. Then, beneath either the .ext or the redirect destination, you create shell\open\command\(Default)="C:\myprogram.exe" "%L" %* for the most simple invocation.

Beyond that, it's all a little confusing with keys like OpenWithList, OpenWithProgids, and values like Content Type and PerceivedType (do they do anything for files without an explicit shell\open\command?), and if you want to make two extensions that run the same program but have different icons (.txt, .srt, .css → text editor) you wind up duplicating the whole shell command for each of them because you can't do multiple-inheritance style mappings. Unless I'm missing something, which is also likely.

Also, there are even more keys in HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.ext\UserChoice. When dealing with extensions that I didn't make myself, NirSoft's filetypesman helps me locate their hodgepodge of keys.

The point is that while I prefer extension-executable systems over shebang systems, that does not mean I'm praising the registry as it exists today.

.mp3.exe (§)

A common criticism of Windows is that non-savvy users can be tricked into downloading some file called "Hotel California.mp3.exe" and running it because they think it's an mp3 file. This is made 1,000 times worse by the unfathomable, unforgiveable, bone-headed decision by Microsoft to hide file extensions of "known file types" by default, hiding the .exe but leaving the .mp3. It boggles the mind and stupefies the senses that this default is still in place after all these years, but it is.

On Unix, the downloaded file would not be executable until you chmod +x, which is something you can't trick most people into doing. On Windows, the .exe requires no activation and is ready to detonate as soon as you double-click on it.

That's why UAC was added to Windows, but users will Accept any dialog box that stands between them and their mp3 without reading it, so that hasn't changed much.

Conclusion (§)

Let me re-quote two of the comments I shared above:

Because it then prevents me from changing the filename the way I want to.

The type of a file's content is different from the application that I want to use to manipulate it, which both are in turn different from the name I chose to give the file

These comments make a good point, that it would be nice to have the ability to associate files to programs without having to name them a certain way. Computing is still a relatively young field, and I wouldn't mind seeing new ways of solving this problem. I mentioned that I'd like to see some multiple-inheritance, which would simplify icon management, default programs, and human-friendly type names. But, by my insistence on maintaining file hashes, you can infer that I'd like a solution out-of-band from the file content itself. Current implementations of alternate data streams are not ready to steal this stage because they aren't transmitted when sharing the file over the internet, for better and worse.

For now, I am satisfied with file extensions (as long as I wrangle the registry myself), and I want the Unix elitists to see how I use them in my favor; that they impose no friction on my workflow at all. inb4 Stockholm syndrome.

View this document's history

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

Quote 1

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

For my part, I travel not to go anywhere, but to go. I travel for travel’s sake. The great affair is to move.

-Robert Louis Stevenson

Quote 10

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

We do not win our strength and stability by mastering ideas, but by being mastered by them—held in their grip.

-Charles Henry Parkhurst

Quote 11

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Purpose, and to be thoroughly wedded to that purpose, is three quarters of salvation. It is sad to reflect how much motiveless insipidity there is among us that is steadily resolving itself into ethical rot, for no other reason than that it has never been awakened into vigor and electrified into effect by the touch of a supreme purpose.

-Charles Henry Parkhurst

Quote 12

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Money is made on a competitive plane. Wealth is made on a creative plane.

-Wallace Wattles

Quote 13

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Truth = Honesty

Love = Compassion

Courage = Valor

Truth + Love = Justice

Truth + Courage = Honor

Love + Courage = Sacrifice

Truth + Love + Courage = Spirituality

The absence of Truth, Love, and Courage is Pride, the opposite of which is Humility.

-Steve Pavlina

Quote 14

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

For every one of us, living in this world means waiting for our end. Let whoever can win glory before death. When a warrior is gone, that will be his best and only bulwark… Endure your troubles today. Bear up and be the man I expect you to be.

-Beowulf

Quote 15

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Self-respect is the fruit of discipline; the sense of dignity grows with the ability to say no to oneself.

-Abraham J. Heschel

Quote 16

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Seek freedom and become captive of your desires, seek discipline and find your liberty.

-Frank Herbert (Dune)

Quote 17

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man.

-George Bernard Shaw

Quote 18

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

“The sole cause of man’s unhappiness is that he does not know how to stay quietly in his room.

-Pascal

Quote 19

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

I came over from Greece when I was 17 and started out working in a market. I worked so much that I supported my mother, my father, and put my brother through college. Now my kids have everything, and don’t appreciate anything. You Americans cut ’em off when they’re 18. And that’s probably for the best. But we can’t do that, cause we’re Greek. They’re ours until they’re married.

-Greek Immigrant from the site “Humans of New York

Quote 2

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

I am in the habit of looking not so much to the nature of a gift as to the spirit in which it is offered.

-Robert Louis Stevenson

Quote 20

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.

-Robert A. Heinlein

Quote 21

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Do not pray for an easy life. Pray for the strength to endure a difficult one.

-Bruce Lee

Quote 22

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Make mistakes of ambition and not mistakes of sloth. Develop the strength to do bold things, not the strength to suffer.

-Machiavelli

Quote 23

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

A master in the art of living draws no sharp distinction between his work and his play; his labor and his leisure; his mind and his body; his education and his recreation. He hardly knows which is which. He simply pursues his vision of excellence through whatever he is doing, and leaves others to determine whether he is working or playing. To himself, he always appears to be doing both.

Quote 24

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

When I was a boy of 14, my father was so ignorant I could hardly stand to have the old man around. But when I got to be 21, I was astonished at how much the old man had learned in seven years.

-Mark Twain

Quote 25

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

We are in an age that assumes the narrowing trends of specialization to be logical, natural, and desirable. Consequently, society expects all earnestly responsible communication to be crisply brief. Advancing science has now discovered that all the known cases of biological extinction have been caused by overspecialization, whose concentration of only selected genes sacrifices general adaptability. Thus the specialist’s brief for pinpointing brevity is dubious. In the meantime, humanity has been deprived of comprehensive understanding. Specialization has bred feelings of isolation, futility, and confusion in individuals. It has also resulted in the individual’s leaving responsibility for thinking and social action to others. Specialization breeds biases that ultimately aggregate as international and ideological discord, which in turn leads to war.

Quote 26

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

What man actually needs is not some tension-less state but rather the striving and struggling for some goal worthy of him.

-Viktor Frankl

Quote 27

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Man’s commonest weakness, his aversion to being unpleasantly conspicuous, pointed at, shunned, as being on the unpopular side. Its other name is Moral Cowardice, and is the supreme feature of the make-up of 9,999 men in the 10,000.

-Mark Twain

Quote 28

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

The growing good of the world is partly dependent on unhistoric acts; and that things are not so ill with you and me as they might have been, is half owing to the number who lived faithfully a hidden life, and rest in unvisited tombs.

-George Eliot (Middlemarch)

Quote 29

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Nobody should start to undertake a large project. You start with a small trivial project, and you should never expect it to get large. If you do, you’ll just overdesign and generally think it is more important than it likely is at that stage. Or worse, you might be scared away by the sheer size of the work you envision. So start small, and think about the details. Don’t think about some big picture and fancy design. If it doesn’t solve some fairly immediate need, it’s almost certainly over-designed. And don’t expect people to jump in and help you. That’s not how these things work. You need to get something half-way useful first, and then others will say “hey, that almost works for me”, and they’ll get involved in the project.

Quote 3

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

If your morals make you dreary, depend on it, they are wrong.

-Robert Louis Stevenson

Quote 30

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Organizations are not a source of security but they are a source of identity.

-Bill Taylor, on Loyalty at Work

Quote 31

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Our forefathers had civilization inside themselves, the wild outside. We live in the civilization they created, but within us the wilderness still lingers. What they dreamed, we live, and what they lived, we dream.

-T. K. Whipple

Quote 32

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Understand the difference between dreams, goals, and expectations. Dreams live in far off places. They may come to us while we walk, hike, bike or take a shower. I like to think of dreams as the shining gemstones, just out of reach. Knowing that we have the desire to reach out and grab these gemstones gives us the motivation to get out of bed each morning.

Goals are specific, measurable, achievable, results-focused and time bound items. Goals are things we’re actually going to do and commit to doing. I equate expectations to a yardstick.

Quote 33

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

The customers are buying the product just as fast as you can make it – or usage is growing just as fast as you can add more servers. Money from customers is piling up in your company checking account. You’re hiring sales and customer support staff as fast as you can.

-Marc Andreesen on product/market fit

Quote 34

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

If your friend was standing next to you and their hair was on fire, that fire would be the only thing they really cared about in this world. It wouldn’t matter if they were hungry, just suffered a bad breakup, or were running late to a meeting—they’d prioritize putting the fire out. If you handed them a hose—the perfect product/solution—they would put the fire out immediately and go on their way. If you handed them a brick they would still grab it and try to hit themselves on the head to put out the fire. You need to find problems so dire that users are willing try half-baked, v1, imperfect solutions.

Quote 35

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

To burn always with this hard, gem-like flame, to maintain this ecstasy, is success in life.

-Walter Pater

Quote 36

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Our language has wisely sensed the two sides of being alone. It has created the word loneliness to express the pain of being alone. And it has created the word solitude to express the glory of being alone.

-Paul Tillich

Quote 37

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

All the world’s a stage, And all the men and women merely players; They have their exits and their entrances, And one man in his time plays many parts.

-William Shakespeare

Quote 38

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

..if you simply change your goal from “writing well” to “writing badly,” you will be a veritable fucking fountain of material, because guess what, man, we don’t like to admit it, because we’re raised to think lack of confidence is synonymous with paralysis, but, let’s just be honest with ourselves and each other: we can only hope to be good writers. We can only ever hope and wish that will ever happen, that’s a bird in the bush. The one in the hand is: we suck. We are terrified we suck, and that terror is oppressive and pervasive because we can VERY WELL see the possibility that we suck. We are well acquainted with it.

Quote 39

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

I want to be my own reader.

-Kartik Subbu when asked who does he want to write for

Quote 4

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Whatever you can do or dream you can, begin it. Boldness has genius, power and magic in it.

-Goethe

Quote 40

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Remade website to be more professional looking, faster, more focused towards SEO, mobile friendly. I could write many pages on this :) but following online guides will get you in the ballpark
Started writing interesting content. Truly good content is a PITA and you need to be a decent writer to make something compelling, but our first good article got us more traffic than the entire site with over 100 content pages.
New tools. everything linked together to see where our sales were coming from. Phone system, website, live chat, in person. All of our leads funneled into one system. We could see what was getting people and what wasn’t
Online advertising. Locally first, expand AD reach as campaign becomes more nationally competitive. Make sure all the tools you setup continue to track where all these leads are coming from
Better “convincing” of clients that we’re legit. Company T-Shirts and swag. Hotel-type exaggerated pictures of our team and office. Meet the team page. Professional website like I mentioned earlier. FB, google maps, yellowpages, wherever we could get a free listing. Automated scheduling systems for client meetings, phone tree for calls, tons of previous work repurposed as marketing material for clients to see. Listing our bigger clients on our website. 800 phone #. All the hallmarks of a big successful business even though we weren’t exactly gave clients a much bigger confidence in our abilities and enabled us to charge more.
Marketing toward high value targets – we needed to get the ear of C level and VP’s. Concentrating on the fact that we were a US company without outsourcing. Marketing to wealthy areas “doctorville” we called it. Targeted marketing to business corridors. We went deep, going as far as checking the AS of your IP address and whether any services were running on it. A surprising amount of the time would could link an IP back to a company, so we could quietly market to your company just from you visiting our site.
competitive research. Found our biggest local competitors. Quietly outbid them on all their ad keywords. Even more quietly made sure we had all the backlinks they did on search. Got a list of their clients one way or another, marketed directly to their clients already knowing their weaknesses from our clients who previously worked with those competitors. One competitor hosted their own servers, so I figured out who owned the IP and tracked ownership of the company through their email server. Another put their email address at the bottom of all public sites they wrote, a simple targeted google search returned all/most of their clients’ sites. Another had a “client login” page setup on all of their sites that had a redirect to their main page. Searching for backlinks using SEO tools revealed most of these client sites. Lots more. It took a lot of creative thinking and some failures :)

-79d697i6fdif answering “what’s your top growth strategy?”

Quote 41

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Nobody tells this to people who are beginners, I wish someone told me. All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase, they quit. Most people I know who do interesting, creative work went through years of this. We know our work doesn’t have this special thing that we want it to have. We all go through this. And if you are just starting out or you are still in this phase, you gotta know its normal and the most important thing you can do is do a lot of work. Put yourself on a deadline so that every week you will finish one story. It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions. And I took longer to figure out how to do this than anyone I’ve ever met. It’s gonna take awhile. It’s normal to take awhile. You’ve just gotta fight your way through.

Quote 42

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

A man is honorable in proportion to the personal risks he takes for his opinion.

-Nassim Taleb

Quote 43

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Life gets interesting when we fail, because it’s a sign that we’ve surpassed ourselves.

-Clemenceau

Quote 44

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Of course I agree with the preacher of Ecclesiastes that there is a time to love and a time to die—and when my skein runs out I hope to face the end calmly and in my own way. For most situations, however, I prefer the more martial view that death is the ultimate enemy—and I find nothing reproachable in those who rage mightily against the dying of the light.

-Stephen Jay Gould

Quote 45

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

You cannot overestimate the unimportance of practically everything.

-Greg McKeown

Quote 46

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Be everything to everybody and you’ll be nothing for yourself.

-John Rushton

Quote 47

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Nearly all men can stand adversity, but if you want to test a man’s character, give him power.

-Abraham Lincoln

Quote 48

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Most people will never be successful because most people remain children, consciously, and never evolve. They remain purely self-indulgent and self-absorbed. They never develop convictions that drive them to dedicate their lives to a particular service and audience. They aren’t willing to learn and transform themselves for a commitment. Instead, they only commit to something so long as it benefits them. Once things get tough, their ‘commitment’ disappears. In order to be truly successful, and to make an enormous impact on the world, you must give yourself fully to something. It can’t be about you anymore. You must be driven to serve. To help other people, and to solve specific problems.

Quote 49

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Some people want it to happen, some wish it would happen, others make it happen.

-Michael Jordan

Quote 5

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better.
The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.

Quote 50

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Whatever you really want, wants don’t get met consistently, standards do.

-Tony Robbins

Quote 51

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Tell me and I forget. Teach me and I remember. Involve me and I learn.

-Benjamin Franklin

Quote 52

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

How you do anything, is how you do everything.

-Unknown

Quote 53

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

We are human beings, not human doings.

-Doylene Chan

Quote 54

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Be patient. No matter what.
Don’t badmouth: Assign responsibility, not blame. Say nothing of another you wouldn’t say to him.
Never assume the motives of others are, to them, less noble than yours are to you.
Expand your sense of the possible.
Don’t trouble yourself with matters you truly cannot change.
Expect no more of anyone than you can deliver yourself.
Tolerate ambiguity.
Laugh at yourself frequently.
Concern yourself with what is right rather than who is right.
Never forget that, no matter how certain, you might be wrong.
Give up blood sports.
Remember that your life belongs to others as well. Don’t risk it frivolously.
Never lie to anyone for any reason. (Lies of omission are sometimes exempt.)
Learn the needs of those around you and respect them.
Avoid the pursuit of happiness. Seek to define your mission and pursue that.
Reduce your use of the first personal pronoun.
Praise at least as often as you disparage.
Admit your errors freely and soon.
Become less suspicious of joy.
Understand humility.
Remember that love forgives everything.
Foster dignity.
Live memorably.
Love yourself.
Endure

-John Perry Barlow on Life and Work

Quote 55

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Don’t ask what the world needs. Ask what makes you come alive. For what the world needs is more people who have come alive.

-Rev. Dr. Howard Thurman

Quote 56

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

If you want to build a ship, don’t drum up the people to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.

-Antoine de Saint-Exupéry

Quote 57

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

The design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.

- http://jehanne.io

Quote 58

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

You will come to understand that you were taking care of them because it made YOU feel good, it made YOU happy to see them smiling and without a care in the world — and that was extremely selfish of you. While you were feeling satisfied with yourself, you were slowly eating away at their own dreams and ambitions. You were adding material things to their lives, but subtracting the most precious gifts of all: independence and growth.

Quote 59

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Someone once told me the definition of Hell: The last day you have on earth, the person you became will meet the person you could have become.

-Unknown

Quote 6

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

I think positivity, and good, is practical advice to building an empire and I want to be the poster child of the person that built the biggest, baddest empire and did it being a good dude along the way.

-Gary Vaynerchuck

Quote 60

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

One day, you will be dead. Each year that passes after your death, fewer and fewer people will remember who you were, what you did or what your face looked like. Eventually, you’ll be completely forgotten. Who cares if that side project you put out sucked or not. Might as well put it out there and see what people think. There’s a good chance too that the project won’t even be remembered even by the time you die. The best way to get experience is to do things, so given that no one is going to remember, might as well try to do it. Your successes won’t be remembered along with your failures.

Quote 61

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

So one day Jeff Bezos issued a mandate. He’s doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion – back around 2002 I think, plus or minus a year – he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.

Quote 62

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Hard times create strong men. Strong men create good times. Good times create weak men. And, weak men create hard times.

-G. Michael Hopf

Quote 63

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

We thought of life by analogy with a journey, a pilgrimage, which had a serious purpose at the end, and the thing was to get to that end, success or whatever it is, maybe heaven after you’re dead. But we missed the point the whole way along. It was a musical thing and you were supposed to sing or to dance while the music was being played.

-Alan Watts

Quote 64

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Retirement is when you stop sacrificing today for some imaginary tomorrow. When today is complete in and of itself. How do you get there? One is, that you have so much money saved up that your passive income off of that covers your burn rate. Second is, you just drive your burn rate down to zero. Become a monk. Third is, you’re doing something you love. You’re enjoying it so much that it’s not about the money.

Quote 65

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

There are two great addictions - heroin and a monthly salary.

-Nassim Taleb

Quote 66

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Every man has two lives. And the second starts when he realizes he has just one.

-Confucius

Quote 67

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

A healthy person wants 10,000 things. A sick person wants just one.

-Confucius

Quote 68

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Destroy the idea that you have to be constantly working or grinding in order to be successful. Embrace the concept that rest, recovery, and reflection are essential parts of progress towards a successful and ultimately happy life.

-Unknown

Quote 69

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

If you can keep your head when all about you

Are losing theirs and blaming it on you,

If you can trust yourself when all men doubt you,

But make allowance for their doubting too;

If you can wait and not be tired by waiting,

Or being lied about, don’t deal in lies,

Or being hated, don’t give way to hating,

And yet don’t look too good, nor talk too wise:

Quote 7

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

I want my pedestal moment, I want to become that big, and what I want to come from that is that kids that aren’t even born today think that they can build a $5 billion dollar company and be a great guy or great gal.

-Gary Vaynerchuck

Quote 70

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

The only way to succeed is if the pain of doing something is less than the pain of doing nothing. You must, therefore, increase the pain of doing nothing.

-Jon Morrow

Quote 71

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

The mind is a kite and the body is the string. In order to move the kite you must move the string.

-Unknown

Quote 72

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Make your interests gradually wider and more impersonal, until bit by bit the walls of the ego recede, and your life becomes increasingly merged in the universal life. An individual human existence should be like a river — small at first, narrowly contained within its banks, and rushing passionately past rocks and over waterfalls. Gradually the river grows wider, the banks recede, the waters flow more quietly, and in the end, without any visible break, they become merged in the sea, and painlessly lose their individual being.

Quote 73

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

It is useless work that darkens the heart. The delight of the nursing mother, of the scholar, of the successful hunter, of the good cook, of the skillful maker, of anyone doing needed work and doing it well, - this durable joy is perhaps the deepest source of human affection and of society as a whole.

-Ursula K LeGuin

Quote 74

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

I wonder, I wonder, what you would do if you had the power to dream at night any dream you wanted to dream? And you would, of course, be able to alter your time sense, and slip, say, 75 years of subjective time into 8 hours of sleep.

You would, I suppose, start out by fulfilling all your wishes. You could design for yourself what would be the most ecstatic life: love affairs, banquets, dancing girls, wonderful journeys, gardens, music beyond belief.

Quote 75

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

To be a warrior is not a simple matter of wishing to be one. It is rather an endless struggle that will go on to the very last moment of our lives. Nobody is born a warrior, in exactly the same way that nobody is born an average man. We make ourselves into one or the other.

-Carlos Castaneda

Quote 76

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

And in truth, I’ve never known a man worth his salt who, in the long run, deep down in his heart, didn’t appreciate the grind, the discipline. The difference between a successful person and others is not a lack of strength, not a lack of knowledge, but rather… a lack of will.

-Vince Lombardi

Quote 77

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Work is love made visible.

-Kahlil Gibran

Quote 78

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

If you are depressed you are living in the past. If you are anxious you are living in the future. If you are at peace you are living in the present.

-Lao Tzu

Quote 79

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

I divide my officers into four classes as follows: The clever, the industrious, the lazy, and the stupid. Each officer always possesses two of these qualities.

Those who are clever and industrious I appoint to the General Staff. Use can under certain circumstances be made of those who are stupid and lazy. The man who is clever and lazy qualifies for the highest leadership posts. He has the requisite nerves and the mental clarity for difficult decisions. But whoever is stupid and industrious must be got rid of, for he is too dangerous.

Quote 8

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Massive action without clearly defined objectives is a form of laziness.

-Timothy Ferriss

Quote 80

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Above all, don’t lie to yourself. The man who lies to himself and listens to his own lie comes to a point that he cannot distinguish the truth within him, or around him, and so loses all respect for himself and for others. And having no respect he ceases to love… “Above all, avoid falsehood, every kind of falsehood, especially to yourself.

-Zosima

Quote 81

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

When you go out into the woods, and you look at trees, you see all these different trees. And some of them are bent, and some of them are straight, and some of them are evergreens, and some of them are whatever. And you look at the tree and you allow it. You see why it is the way it is. You sort of understand that it didn’t get enough light, and so it turned that way. And you don’t get all emotional about it. You just allow it. You appreciate the tree.

Quote 82

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Step 1: Consider the habit you want to build, e.g. “I want to meditate 10 minutes every day”

Step 2: Make it the absolute smallest possible version of itself; so small that it requires zero motivation/willpower, e.g. “I want to close my eyes and take 3 deep breaths every day”

Step 3: Place this habit immediately following an existing habit, e.g. “After I brush my teeth in the morning, I will close my eyes and take 3 deep breaths”

Quote 83

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Motivation is what gets you started. Habit is what keeps you going.

-Jim Ryun

Quote 84

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

There are only two mistakes one can make along the road to truth; not going all the way, and not starting.

-Buddha

Quote 85

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

If your plan is for 1 year, plant rice. If your plan is 10 years, plant trees. If your plan is for a 100 years, educate children.

-Confucius

Quote 86

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

A ship is always safe at the shore - but that is NOT what it is built for.

-Albert Einstein

Quote 87

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Find ecstasy in life; the mere sense of living is joy enough.

-Emily Dickinson

Quote 88

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

The successful warrior is the average man, with laser-like focus.

-Bruce Lee

Quote 89

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

You will either step forward into growth or you will step back into safety.

-Abraham Maslow

Quote 9

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Income without time has no value.

-Timothy Ferriss

Quote 90

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Discipline is doing something you hate, like you love it.

-Mike Tyson

Quote 91

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Children do not belong to you, and they do not come from you. They only come through you. It is a privilege that we must appreciate and enjoy, rather than thinking it is our right or trying to determine where they should go and what they should do. As long as they are working towards their wellbeing and they are not doing something negative against their own life, you must wait. For the entire time until the child becomes twenty-one, you must feel like you are still pregnant. Just wait. When the child was inside, you did nothing, right? Just nourished yourself well and waited. Just like that – provide the atmosphere and wait.

Quote 92

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

You want to know the certainty of death in life. This is called belief, belonging, identification.

-Sadhguru

Quote 93

A Holding Company — Wed, 09 Dec 2020 02:00:00 GMT

Your children are not your children.

They are the sons and daughters of Life’s longing for itself.

They come through you but not from you,

And though they are with you yet they belong not to you.

You may give them your love but not your thoughts,

For they have their own thoughts.

You may house their bodies but not their souls,

For their souls dwell in the house of tomorrow,

SMZ: Updated Drafts Actions

trv.rs — Tue, 08 Dec 2020 22:55:00 GMT

Today was the first day I used the whole revamped SMZ system, and it feels real good.

I've made a few tweaks to it, but the migration script has stayed stable and I haven't needed to run it in a little while… I believe that's over.

The final step was updating my JS actions for my beloved Drafts.

I'm only a couple of weeks into using it, but so far the new version has been much more stable and usable. In the highly unlikely chance that you were using my library to build your actions, here's what's changed:

The "library" step has been renamed to Zettelkasten API. Some of the booknotes specific steps have been moved to another "library" step called Booknotes API.

In the previous version, the library treated the active draft as the "data model." This caused weird heisenbugs related to what functions called draft.update() and which tried to not thrash the view constantly. In the V2, (Zettelkasten API,) the only methods that touch the active Draft object are Zettel.parse(draft) and Zettel.update(). All the other transformations happen on a simple JS object (built by Zettel.parse/1 that takes the form of &lbracemeta: &lbrace … &rbrace, content: "foo" &rbrace.

So far this approach has meant much easier debugging and predictability. I'm a fan.

The previous versions had dependencies on "template" drafts that couldn't possibly exist on another person's database… so all the "templates" are now in the actions themselves. I think there's still a dependency on having a workspace named "Booknotes," but that's pretty solvable by an end user.

The use-cases are still about the same. As I move through my day, I throw quotes, links, and snippets of ideas in Drafts. During the daily review at the end of the day I go through Drafts, expand and save the thoughts that need to be saved. The new actions don't add all the kluge metadata that blocks the view of the title until it's saved.

If it's something you are interested in... message me on twitter and I'll try to fix it for you. Until then, add it to Drafts here.

Really smart cities help their stores to go Dark

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 08 Dec 2020 20:00:00 GMT

The digital dark side or retailing is more smart than dark.

Emulating a Computer: Graphics and Texture Streaming

River Gillis [riv.dev] — Tue, 08 Dec 2020 05:51:00 GMT

Displaying graphics using SDL for our emulator.

Solving my fun, frustrating docker-machine error

ntietz.com blog - technically a blog — Tue, 08 Dec 2020 02:00:00 GMT

Last Saturday, I ran into a problem doing a routine backup of a web app I maintain. In fact, this was the second time I ran into the exact same issue, so it's time to write it down. (Hopefully, the third time I run into this, I have the presence of mind to look up my own solution!)

My web app is deployed using docker-machine and docker-compose. This is not a great production setup, but it works for me and there are just a handful of users. Every week, I manually run a backup script that copies down the database and all the images from this web app. (I could set up a cron job, but I consciously chose to keep it manual so I would, every week, be able to see that the backups are working: this has paid off, since I saw when it did NOT work!)

When I ran the backups, I ran into a mysterious error message:

$ backup.sh
Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "xx.xx.xx.xx:2376": dial tcp xx.xx.xx.xx:2376: i/o timeout
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which might stop running containers.

Error response from daemon: Container 06916a79c735b152c287c8aaa57ff65958898f819c604ceee83fadf3f502922f is not running

First thought: Okay, well, that's weird that the certs are expired but let's just follow what it says. Let's regenerate those. So, I did, and then... the entire app was down, because it shut down the containers but could not start them! Now a routine backup has turned into an outage.

Aaand I can't see the machine:

$ docker-machine ls
NAME        ACTIVE   DRIVER    STATE     URL   SWARM   DOCKER   ERRORS
picklejar            generic   Timeout

The strange thing? docker-machine ssh <host> worked.

So... I cannot see the machine, I cannot validate the certs, but ssh works.

If you're screaming at the monitor right now because the answer is obvious, I know. I missed it in the moment, but it was right there in front of me (sort of) in that first error message: dial tcp xx.xx.xx.xx:2376: i/o timout: This means that we can't establish a TCP connection on that port, which could be... caused by the firewall. Let's not talk about how long it took me to realize this, and how many other things I tried before I had that head smack moment: doh!

The problem was: I have the instance firewalled in a way that allows my home network to establish the TCP connections needed for docker-machine, but no external traffic. BUT I have ssh allowed from any port, so that I can get into the host while I'm on the go (or that was the idea, when travel was a thing). So when my ISP issued me a new IP address, suddenly I could do some things on the machine (ssh) but could not do others, leading to this confusing situation of docker-machine kinda sorta half working.

So if you get an error message from docker-machine about an error validating certificates, don't just assume (as I did) that its suggested fix is a good idea: verify that you don't have a network/firewall issue first.

IPython embed and autoreload

Gaganpreet Arora — Mon, 07 Dec 2020 22:01:38 GMT

IPython makes for very productive and effortless sessions working interactively with Python. From the numerous extensions for IPython, autoreload is one I use often. It’s very useful for working with existing code and being able to edit code from my code editor and have changes reflected automatically within the IPython terminal.

Pumpkin pie recipe [Video]

David Schlachter — Mon, 07 Dec 2020 21:23:00 GMT

One of my favourite desserts! Recipe from my great-grandmother.

You are losing easy publishing, consuming and remixing

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 07 Dec 2020 20:00:00 GMT

Where? Right here on the web, of course.

AoC day7

blog — Mon, 07 Dec 2020 19:35:56 GMT

Emulating a Computer: Images and Rendering

River Gillis [riv.dev] — Mon, 07 Dec 2020 04:21:00 GMT

Adding images to our CHIP-8 emulator.

Go Tour Summary

Luke Singham — Mon, 07 Dec 2020 02:00:00 GMT

These are my summary notes of A Tour of Go - which is meant for people who are familiar with programming to have a quick tour of the Go language. I'm most familiar with dynamic languages (R, Python, JavaScript) so there are some go-lang features that naturally feel new by the nature of Go being a statically typed language. A subsequent post will expand upon some of the more advanced features like pointers, concurrency and polymorphism.

Named Return Values #

A return statement without arguments returns the named return values. This is known as a "naked" return.

func split(mynumber int) (x, y int) {
	x = mynumber * 4
	y = mynumber - x
	return
}

func main() {
	fmt.Println(split(17))
}
// This will print x and y:
// 68 -51

Variables #

The var statement declares a list of variables; as in function argument lists, the type is last.

Variables with initializers #

A var declaration can include initializers, one per variable.

If an initializer is present, the type can be omitted; the variable will take the type of the initializer.

var a int // will have a value of 0 as a placeholder
var f float64 // 0
var b bool // false
var s string // ""

var i, j int = 1, 2 // explicit

var k, l = 1, 2 // implicit

`:=` #

Inside a function, the := short assignment statement can be used in place of a var declaration with implicit type.

Outside a function, every statement begins with a keyword (var, func, and so on) and so the := construct is not available.

func main() {
  // := only within fn
  k := 3
}

Types in go #

bool

string

int  int8  int16  int32  int64
uint uint8 uint16 uint32 uint64 uintptr

byte // alias for uint8

rune // alias for int32
     // represents a Unicode code point

float32 float64

complex64 complex128

Type conversions #

var i int = 42
var f float64 = float64(i)

//or
i := 42
f := float64(i)

Constants #

Constants are declared like variables, but with the const keyword.

const Pi = 3.14

For #

Go has only one looping construct, the for loop.

func main() {
	sum := 0
	for i := 0; i < 10; i++ {
		sum += i
	}
	fmt.Println(sum)
}

The init and post statements are optional.

sum := 1
for ; sum < 1000; {
  sum += sum
}

If #

Go's if statements are like its for loops; the expression need not be surrounded by parentheses ( ) but the braces { } are required.

If with a short statement #

Like the for loop, the if statement can start with a short statement to execute before the condition.

Variables declared by the statement are only in scope until the end of the if.

func pow(x, n, lim float64) float64 {
  // v doesn't exist outside of the if statement scope
  if v := math.Pow(x, n); v < lim {
		return v
	}
	return lim
}

Switch #

A switch statement is a shorter way to write a sequence of if else statements. It runs the first case whose value is equal to the condition expression.

Go's switch is like the one in C, C++, Java, JavaScript and PHP, except that Go only runs the selected case, not all the cases that follow. In effect, the break statement that is needed at the end of each case in those languages is provided automatically in Go. Another important difference is that Go's switch cases need not be constants, and the values involved need not be integers.

func main() {
	switch 1 {
	case 1:
		fmt.Println("matches")
	case 1 + 1:
		fmt.Println("doesnt match")
	}
}
// matches

Switch with no condition #

Switch without a condition is the same as switch true. This construct can be a clean way to write long if-then-else chains.

func main() {
	t := time.Now()
	switch {
	case t.Hour() < 12:
		fmt.Println("Good morning!")
	case t.Hour() < 17:
		fmt.Println("Good afternoon.")
	default:
		fmt.Println("Good evening.")
	}
}

Defer #

Defer evaluates but doesn't execute

func main() {
	fmt.Println("hello1")
	defer fmt.Println("world")
  fmt.Println("hello2")
}
// hello1
// hello2
// world

Stacking defers #

Deferred function calls are pushed onto a stack. When a function returns, its deferred calls are executed in last-in-first-out order.

func main() {
	fmt.Println("counting")
	for i := 0; i < 4; i++ {
		defer fmt.Println(i)
	}
	fmt.Println("done")
}
// counting
// done
// 3
// 2
// 1
// 0

Pointers #

Pointers are useful if you have read something into memory and you don't want to duplicate it. So you can just point to the memory address. Useful for optimising your code.

So if you want to change a variable stored in memory, you can 'mutate' it by using pointers.

The type *T is a pointer to a T value. Its zero value is nil.

var p *int

The & operator generates a pointer to its operand.
n.b. In 3+6, 3 and 6 are operands. And the + is the operator.

i := 42
p := &i // a pointer p=0xc00002c008
z := *p // dereferences the pointer. z=42
fmt.Println("i",i,"p",p, "z", z)
// i 42 p 0xc00002c008 z 42

The * operator denotes the pointer's underlying value.

fmt.Println(*p) // read i through the pointer p
// 42
*p = 21         // set i through the pointer p
fmt.Println(i)
// 21

This is known as "dereferencing" or "indirecting".

Structs #

A struct is a collection of fields.

type Vertex struct {
	X int
	Y int
}

func main() {
	fmt.Println(Vertex{1, 2})
}
// {1 2}

Pointers to structs #

Struct fields can be accessed through a struct pointer.

type Vertex struct {
	X int
	Y int
}

func main() {
	v := Vertex{1, 2}
	p := &v
	p.X = 10
	fmt.Println(v)
}
// {10 2}

Struct Literals #

A struct literal denotes a newly allocated struct value by listing the values of its fields.

You can list just a subset of fields by using the Name: syntax. The order of named fields is irrelevant.

The special prefix & returns a pointer to the struct value.

type Vertex struct {
	X, Y int
}

var (
	v1 = Vertex{1, 2}  // has type Vertex
	v2 = Vertex{X: 1}  // Y:0 is implicit
	v3 = Vertex{}      // X:0 and Y:0
	p  = &Vertex{1, 2} // has type *Vertex
)

func main() {
	fmt.Println(v1, p, v2, v3)
}
// {1 2} &{1 2} {1 0} {0 0}

Arrays #

The type [n]T is an array of n values of type T. The expression
var a [10]int
declares a variable a as an array of ten integers. An array's length is part of its type, so arrays cannot be resized.

func main() {
	var a [2]string
	a[0] = "Hello"
	a[1] = "World"
	fmt.Println(a[0], a[1])
	fmt.Println(a)

	primes := [6]int{2, 3, 5, 7, 11, 13}
	fmt.Println(primes)
}
// Hello World
// [Hello World]
// [2 3 5 7 11 13]

Slices #

An array has a fixed size. A slice, on the other hand, is a dynamically-sized, flexible view into the elements of an array

func main() {
	primes := [6]int{2, 3, 5, 7, 11, 13}

	var sliced []int = primes[1:4]
	fmt.Println(sliced)
}

A slice does not store any data, it just describes a section of an underlying array. Changing the elements of a slice modifies the corresponding elements of its underlying array. Other slices that share the same underlying array will see those changes.

func main() {
	names := [4]string{
		"John",
		"Paul",
		"George",
		"Ringo",
	}

	a := names[0:2]
	b := names[1:3]
	fmt.Println(a, b)

	b[0] = "XXX"
	fmt.Println(a, b)
	fmt.Println(names)
}
// [John Paul] [Paul George]
// [John XXX] [XXX George]
// [John XXX George Ringo]

Slice literals #

A slice literal is like an array literal without the length.
This is an array literal:
[3]bool{true, true, false}

And this creates the same array as above, then builds a slice that references it:
[]bool{true, true, false}

	q := []int{2, 3, 5, 7, 11, 13}
  fmt.Println(q)
  // [2 3 5 7 11 13]

Slice length and capacity #

The length of a slice is the number of elements it contains.
The capacity of a slice is the number of elements in the underlying array, counting from the first element in the slice.
The length and capacity of a slice s can be obtained using the expressions len(s) and cap(s).

Nil slices #

The zero value of a slice is nil.

func main() {
	var s []int
	fmt.Println(s, len(s), cap(s))
	if s == nil {
		fmt.Println("nil!")
	}
}
// [] 0 0
// nil!

Creating a slice with `make` #

Slices can be created with the built-in make function; this is how you create dynamically-sized arrays. The make function allocates a zeroed array and returns a slice that refers to that array:

a := make([]int, 5)  // len(a)=5

// To specify a capacity, pass a third argument to make:

b := make([]int, 0, 5) // len(b)=0, cap(b)=5

Appending to a slice #

func main() {
	var s []int
	printSlice(s)

	// append works on nil slices.
	s = append(s, 0)
	printSlice(s)

	// The slice grows as needed.
	s = append(s, 1)
	printSlice(s)

	// We can add more than one element at a time.
	s = append(s, 2, 3, 4)
	printSlice(s)
}

func printSlice(s []int) {
	fmt.Printf("len=%d cap=%d %v\n", len(s), cap(s), s)
}
// len=0 cap=0 []
// len=1 cap=1 [0]
// len=2 cap=2 [0 1]
// len=5 cap=6 [0 1 2 3 4]

Range #

The range form of the for loop iterates over a slice or map.
When ranging over a slice, two values are returned for each iteration. The first is the index, and the second is a copy of the element at that index.

n.b. verbs in string templates.

var pow = []int{1, 2, 4, 8, 16}

func main() {
	for i, v := range pow {
		fmt.Printf("2**%d = %d\n", i, v)
	}
}
// 2**0 = 1
// 2**1 = 2
// 2**2 = 4
// 2**3 = 8
// 2**4 = 16

You can skip the index or value by assigning to _.

for i, _ := range pow
for _, value := range pow

Map literals #

Map literals are like struct literals, but the keys are required.

type Vertex struct {
	Lat, Long float64
}

var m = map[string]Vertex{
	"Bell Labs": Vertex{
		40.68433, -74.39967,
	},
	"Google": Vertex{
		37.42202, -122.08408,
	},
}

// m can also be expressed
var m = map[string]Vertex{
	"Bell Labs": {40.68433, -74.39967},
	"Google":    {37.42202, -122.08408},
}

func main() {
	fmt.Println(m)
}
// map[Bell Labs:{40.68433 -74.39967} Google:{37.42202 -122.08408}]

Mutating Maps #

// Insert or update an element in map m:
m[key] = elem

// Retrieve an element:
elem = m[key]

// Delete an element:
delete(m, key)

// Test that a key is present with a two-value assignment:
elem, ok = m[key]

Function values #

Functions are values too. They can be passed around just like other values. Function values may be used as function arguments and return values.

Function closures #

Go functions may be closures. A closure is a function value that references variables from outside its body. The function may access and assign to the referenced variables; in this sense the function is "bound" to the variables.

func adder() func(int) int {
	sum := 0
	return func(x int) int {
		sum += x
		return sum
	}
}

func main() {
	pos, neg := adder(), adder()
	for i := 0; i < 10; i++ {
		fmt.Println(
			pos(i),
			neg(-2*i),
		)
	}
}

Methods #

Go does not have classes. However, you can define methods on types.
A method is a function with a special receiver argument.
The receiver appears in its own argument list between the func keyword and the method name.
In this example, the Abs method has a receiver of type Vertex named v.

type Vertex struct {
	X, Y float64
}

// as a method
func (v Vertex) Abs() float64 {
	return math.Sqrt(v.X*v.X + v.Y*v.Y)
}

func main() {
	v := Vertex{3, 4}
	fmt.Println(v.Abs())
}

// as a normal function
func Abs(v Vertex) float64 {
	return math.Sqrt(v.X*v.X + v.Y*v.Y)
}

func main() {
	v := Vertex{3, 4}
	fmt.Println(Abs(v))
}

Pointer receivers #

Methods with pointer receivers can modify the value to which the receiver points (as Scale does here 👇). Since methods often need to modify their receiver, pointer receivers are more common than value receivers.

With a value receiver, the Scale method operates on a copy of the original Vertex value. (This is the same behavior as for any other function argument.) The Scale method must have a pointer receiver to change the Vertex value declared in the main function.

type Vertex struct {
	X, Y float64
}

func (v Vertex) Abs() float64 {
	return math.Sqrt(v.X*v.X + v.Y*v.Y)
}

// method on a pointer
func (v *Vertex) Scale(f float64) {
	v.X = v.X * f
	v.Y = v.Y * f
}

func main() {
	v := Vertex{3, 4}
	v.Scale(10)
	fmt.Println(v)
}
// {30 40}

// method on value receiver
func (v Vertex) Scale(f float64) {
	v.X = v.X * f
	v.Y = v.Y * f
}

func main() {
	v := Vertex{3, 4}
	v.Scale(10)
	fmt.Println(v)
}
// {3 4}
// the result {3 4} is because the the original values are not changed. A copy is made on execution. As opposed to the pointer receiver, which modifies the original values.

Methods and pointer indirection (2) #

Functions that take a value argument must take a value of that specific type:

var v Vertex
fmt.Println(AbsFunc(v))  // OK
fmt.Println(AbsFunc(&v)) // Compile error!

Methods with value receivers take either a value or a pointer as the receiver when they are called:

var v Vertex
fmt.Println(v.Abs()) // OK
p := &v
fmt.Println(p.Abs()) // OK

Interfaces #

An interface type is defined as a set of method signatures.
A value of interface type can hold any value that implements those methods.

A type implements an interface by implementing its methods. There is no explicit declaration of intent, no "implements" keyword

Interface values #

Under the hood, interface values can be thought of as a tuple of a value and a concrete type:
(value, type)
An interface value holds a value of a specific underlying concrete type.

Interface values with `nil` underlying values #

If the concrete value inside the interface itself is nil, the method will be called with a nil receiver.
In some languages this would trigger a null pointer exception, but in Go it is common to write methods that gracefully handle being called with a nil receiver.

The empty interface #

The interface type that specifies zero methods is known as the empty interface:
interface{}
An empty interface may hold values of any type. Every type implements at least zero methods. Empty interfaces are used by code that handles values of unknown type. For example, fmt.Print takes any number of arguments of type interface{}.

Type assertions #

A type assertion provides access to an interface value's underlying concrete value.
t := i.(T)
This statement asserts that the interface value i holds the concrete type T and assigns the underlying T value to the variable t.
If i does not hold a T, the statement will trigger a panic.
To test whether an interface value holds a specific type, a type assertion can return two values: the underlying value and a boolean value that reports whether the assertion succeeded.
t, ok := i.(T)

func main() {
	var i interface{} = "hello" // implicit string type

	s := i.(string)
	fmt.Println(s) //"hello"

	s, ok := i.(string)
	fmt.Println(s, ok) //"hello", true

	f, ok := i.(float64)
	fmt.Println(f, ok) //0, false

	f = i.(float64) // panic
	fmt.Println(f) //panic: interface conversion: interface {} is string, not float64
}

Type switches #

A type switch is a construct that permits several type assertions in series.
A type switch is like a regular switch statement, but the cases in a type switch specify types (not values), and those values are compared against the type of the value held by the given interface value.

func do(i interface{}) {
	switch v := i.(type) {
	case int:
		fmt.Printf("Twice %v is %v\n", v, v*2)
	case string:
		fmt.Printf("%q is %v bytes long\n", v, len(v))
	default:
		fmt.Printf("I don't know about type %T!\n", v)
	}
}

func main() {
	do(21)
	do("hello")
	do(true)
}
// Twice 21 is 42
// "hello" is 5 bytes long
// I don't know about type bool!

Errors #

Go programs express error state with error values. The error type is a built-in interface similar to fmt.Stringer:

type error interface {
    Error() string
}

As with fmt.Stringer, the fmt package looks for the error interface when printing values.
Functions often return an error value, and calling code should handle errors by testing whether the error equals nil.

i, err := strconv.Atoi("42")
if err != nil {
    fmt.Printf("couldn't convert number: %v\n", err)
    return
}
fmt.Println("Converted integer:", i)

Goroutines #

A goroutine is a lightweight thread managed by the Go runtime.

func say(s string) {
	for i := 0; i < 3; i++ {
		time.Sleep(100 * time.Millisecond)
		fmt.Println(s)
	}
}

func main() {
	go say("world")
	say("hello")
}
// hello
// world
// world
// hello
// hello

Observations #

unused variables are not allowed by the compiler - very cool
hard concepts (read more on)
- pointer indirection
  - performance
  - modify in place
  - when optional, pointer to a string... no empty strings in go. But the pointer to the string will be nil
- interfaces and polymorphism (types)
- concurrency
- stringer, but why? Pretty printing? https://tour.golang.org/methods/17
skipped
- exercises
- reader
- images
- routines, channels, buffered channels, range and close, select,sync.Mutex
typing a particular concept into youtube usual has a human to explain it.
- sentdex is pretty good at explaining in 5mins
is go functional? Nope and it's not OO. It's procedural. Very explicit.

A Call

ho.dges.online — Mon, 07 Dec 2020 02:00:00 GMT

‘Hold on,’ she said, ‘I’ll just run out and get him.
The weather here’s so good, he took the chance.
To do a bit of weeding.”

So I saw him
Down on his hands and knees beside the leek rig,
Touching, inspecting, separating one
Stalk from the other, gently pulling up
Everything not tapered, frail and leafless,
Pleased to feel each little weed-root break,
But rueful also…

Then I found myself listening to
The amplified grave ticking of hall clocks
Where the phone lay unattended in a calm
Of mirror glass and sunstruck pendulums…

Why I turned my homepage into a timeline

www.mootoday.com - RSS Feed — Sun, 06 Dec 2020 22:21:32 GMT

A short explanation of why I streamlined the homepage into a timeline showcasing my blog posts, Twitter stats, YouTube videos, and projects.

Learning to Program by Making a Game

Mike Kasberg's Blog — Sun, 06 Dec 2020 18:00:00 GMT

This year, Computer Science Education Week will be December 7-13, 2020. In honor of CS Education Week, I wanted to write this blog post about the first computer game I wrote, and what I learned along the way. I was about thirteen, and this story picks up near where I left off in my previous blog post about my first computer program. When I wrote that blog post in 2017, I was working mostly from memory. Since then, I actually found an old floppy disk with my game on it. And finding that old floppy disk got me thinking about about how much I learned from writing computer games in my childhood, and how much others might be able to learn from writing their own first computer games today.

At a young age, I became fascinated with computers because by programming, you can make something from nothing. You don’t need to pay money for a new Lego set, and you don’t need a trip to the craft shop or the hardware store. You don’t need any building material. With computers, you can make something that’s interesting and valuable with just your brain. Most thirteen-year-olds probably just want to play computer games, but I wanted to build my own.

Building Games with Game Maker

My desire to build my own computer game led me to a piece of software called Game Maker. This software made it relatively easy to program computer games from scratch. At least, easy enough that a thirteen-year-old could figure it out. Although it wasn’t quite as easy as the drag-and-drop programming Scratch provides today, the principles were largely the same. In Game Maker, you create rooms and objects, and the objects can display as sprites. Game Maker provides a UI for programming the game by making the objects respond to different events. You can make objects move by responding to keyboard events, and you can make objects interact with each other by responding to collision events. The combination of these inputs and interactions makes a complete game.

When I was a young teenager, Game Maker was the perfect program to satisfy my curiosity about programming video games. It was relatively easy to learn, but it was also very powerful. I wanted to program a multiplayer game so several people could play at once on the same machine, like a Nintendo or a PlayStation. My idea was simple – I was going to program a computer version of the playground game “tag”. Fortunately for me, the logic for a tag game isn’t too complex. Each player needs to be able to move, and you need to know when someone runs into someone else. A perfect programming challenge for a young teenager!

Programming Games Is… Educational?

Yes! Programming computer games can be very educational! As a lot of research around project based learning shows, there’s no better way to learn something than to jump right in and do it. By getting into a project a little outside your comfort zone, you quickly realize what you don’t know and you’re motivated to learn so you can finish your project. You’ll remember what you’ve learned better too because you understand how to apply it. Project based learning can be significantly more effective than other forms of education.

Scratch

Today, there are a variety of platforms that people (both young and old) can use to make their own games while building skills in software development. Scratch is probably the easiest to get started with and also the most well-known. Unfortunately, I’ve personally known a few kids who developed a bad impression of Scratch at a young age. Their first experience with it might have been a silly project that they didn’t like, or perhaps they only used it for animations in the past (which they weren’t interested in). Scratch isn’t limited to silly animations or simple programming like making a robot follow a path. Advanced Scratch projects can be very complex and look like “real” video games – see Flappy Bird, The Crusty Quest, or Chess. So while programming games won’t appeal to everyone, most kids get excited when they see what Scratch can do. I’ve taught an Hour of Code for several years using Scratch (usually with some variation of a simple game) and every single time the kids get excited when they see my game and realize they’re going to be given the tools and knowledge to build their own.

Fundamentally, Scratch works with all the same principles I used when I made my first game in Game Maker. You create a scene, you put objects in the scene, and you program those objects to respond to input and events. I even re-wrote my tag game (the one I made with Game Maker) in Scratch, just to prove to myself that it wouldn’t be hard to do. In addition to that, I’ve written an asteroids game, a drawing program, a platform game, and a simple Frogger clone. I designed most of these as part of lessons I was teaching, but the important thing is that any of these games could be written by a motivated 5th grader. When I wrote these projects, my goal was to make a fun game with simple code that’s easy to understand. Scratch lets you look at the code behind any game, and the code for most of mine is usually no more than about 50 lines long (take a look)! This makes it easy to modify other people’s projects (turn Asteroids into Dodgeball) or learn from them to make your own!

Other Tools

Of course, in addition to Scratch, there are several other more advanced tools available. At some point, Scratch becomes too limiting and too boring, and you want to build better things with more advanced tools. MIT App Inventor uses the same kind of programming blocks that Scratch uses, but provides a wider variety of blocks and allows you to create phone apps that will actually run on real phones. It’s a great tool to move on to when you get bored with Scratch! Alternatively, you could give GDevelop a try. GDevelop is a much more advanced tool for building real computer games, but it operates on many of the same of the program I used back in 2003. No matter what tool you’re using, finding a good tutorial (either written or on YouTube) isn’t a bad place to start.

Building computer games can be as fun as playing them, and I hope I’ve inspired you (or your children!) to try building something and to learn something new along the way! If you like my Scratch games, come follow me on Scratch!

Using Jupyter Notebook as GIS

Geography & Coding — Sun, 06 Dec 2020 13:23:22 GMT

Clipping a geodataframe and displaying it on a basemap in Jupyter.

Update on Old Posts

infrequent oscillations — Sun, 06 Dec 2020 10:52:43 GMT

In which we look back and update ourselves.

CSS Variables are cool

secretgeek.net — Sun, 06 Dec 2020 09:29:36 GMT

(Technically they're called properties, but everyone calls them variables because of the var(--mycolor) syntax -- and I'm a descriptivist not a prescriptivist, so I'm gonna call 'em variables too.)

Now... CSS Variables are COOL.

The hooded figures who lurk near the Forbidden Dog Park don't want you to know about CSS Variables. The Sherriff's Secret Police would rather we did not discuss CSS Variables. But I Must Press On With This Blog Post.

I started using CSS Variables when revamping NimbleText with dark mode.

But I didn't even begin to scratch the surface then. I didn't get it (I didn't need to) but now the deeper I dig, the more power I find. No wonder shadowy helicopters hover outside the window of my office tower, even now.

Basic usage

First, I used Css Variables as a way to make sure I could define useful colors "just once", and reuse them consistently.

e.g. you can define a "main" background color like this:

:root {
	--main-bg: #FFF;
}

And apply it to any selectors you want:

body {
	background-color: var(--main-bg);
}

And you can override that... for example for your dark theme:

@media (prefers-color-scheme: dark) {
	:root {
		--main-bg: #222;
	}
}

And that's fun. But the fun doesn't stop there!

Sidebar: wait what's the `:root` ??

You may have noticed, above, that we applied our variable definition to a :root selector... what is that strange thing? and what other options do we have?

The colon prefix tells us it's a "pseudo-class" (like :hover or :focus), in this case it's a special pseudo-class meaning the root of the document. So the variable we created applies to the whole document.

We could also have "scoped" the variable to just a part of the document...

footer {
	--bg: #222;
	background-color:var(--bg);
}

I'm not using that piece of knowledge yet, but I mention it because many authors gloss over it, and also, perhaps because a skeletal figure just dragged his ragged body across the floor of my office, rose to his bony feet, brandished a glistening metallic weapon with a bright logo from "Cranial Extracto Matic" and screamed, in a voice like a thousand metal polar bears shrieking in agony as they are fed into a terrible grinding machine, that it would perhaps be wisest if I refrained from mentioning ":pseudo elements" in this particular blog post. And I am a bit of a contrarian on these things.

Now for something cool

The cool thing I've noticed more recently (and why I'm writing to you about this now) is this:

You can reference variables from variables.

🤯

... and referencing variables from variables must have many uses, an infinite number of uses.

For me, an immediate and powerful use is for doing custom themes properly.

In a real world theme, on a real world site, you don't end up with just 2 or 3 colors. You end up with many, many slightly different variations on a smaller number of hues. (A good recent article on the topic is here). If a button has a gradient, then it has several variations of the initial color. A panel may have a less saturated version of a background color, pushing it into the distance. Many variations, many colors -- but very few hues.

With Css Variables we can reuse an underlying hue with different saturation, lightness or opacity.

Instead of putting an entire color into a variable, start by putting your hue into a variable.

For example:

:root {
	--main-hue: 124; /* a green hue */
}

If we have a "greenish" theme, perhaps a variation on the hue of the green slime that has begun to bleed from the shadowy apparition who hovers even now abaft the cavernous black void that threatens the rusted children's play equipment in the boarded up park across the road, then we can store our "hue" and make sure that every "greenish" color on our site is based upon that hue.

Write your colors in hsla format... e.g:

:root {
	--main-hue: 124; /* a green hue */
	--main-bg: hsla(var(--main-hue),100%, 13%, 1); /* background */
	--main-fg: hsla(var(--main-hue), 17%, 86%,1); /* foreground */
}

... and so on.

Thus your whole theme can build up and having many colors based on a small number of hues.

The dark theme might even use the same hue, but flip the saturation/lightness of the colors that build upon it. It's flexible enough to achieve anything you need. And all the while, a single fact such as the base hues of your theme are not repeated.

It's DRY, it's separation of concerns. All those "principles" people used to go on about back in the Olden times of Before.

With your underlying hue defined in a variable, you can change a single line of CSS and completely alter your theme.

Now make it dynamic...

Oh but wait, it's cooler than that.

In one line you can also change those variables with javascript.

var root = document.documentElement;
root.style.setProperty('--main-hue', 0); // now the theme color is pure red (`hue = 0`), not that slime-like green.

Or if -- like me -- you're a fan of taking things too far, you can have a timer that randomly tries out different hues, and a stop button for when you're happy.

Or you can have several hues all rotated by various amounts from each other, in triads, or complements etc. You can "AB test" different values, or survey a small group to see the best possible themes.

This is much closer to how things should be (and it's also much more satisfying than using a pre-processor). The stylesheet encodes the underlying relationships between the properties, instead of just hard-coding the final values everywhere.

The art of science is the science of turning art into a science and the art of turning science into an art of science. Or something.

All of this is powerful stuff that no one is talking about. Maybe because THE MAN wants to keep us DOWN, or because BIG CSS wants us to be silent and happy with our rems and our ems. But I'm not going to be quiet about this anymore. Variables are there: do something extraordinary with them today.

Oh and bonus points if you also use calc() with them -- i'd love to hear about some interesting uses of calc with var.

My lawyer also wishes me to make it known that certain elements of this blog post would not have been possible if I had not recently begun listening to far too much "Welcome to Night Vale"; a very, very, very good podcast. There are also books.

Type classes with Scala 2

Functional[Justin] — Sun, 06 Dec 2020 02:00:00 GMT

Photo by Crissy Jarvis on Unsplash

Make your own Numeric type class

This is the companion blog for a Functional Justin video which you can find here https://youtu.be/pJFfXhZlR5o. This is a series of coffee-break sized videos that each explore a topic in the world of Scala Functional Programming.

In this video I talk about Type classes and here is the blog version for those that would rather read.

What are type classes

Here are some good references for reading about type classes in Scala (they can of course be implemented in other functional programming languages).

Sam Halliday has a great introduction to data types and type classes in his book Functional Programming for Mortals. https://leanpub.com/fpmortals/read#leanpub-auto-data-and-functionality

This idea of separating data from functions can also be found in the Cats functional programming library, with the library itself being organised into distinct sections for data types and type classes. https://typelevel.org/cats/typeclasses.html https://typelevel.org/cats/datatypes

My former colleague and functional programming advocate Francis Toth also has a nice blog on this topic. https://contramap.dev/2020/04/09/typeclasses.html

In my own words, type classes are not explicitly part of the Scala language, but are rather a pattern of implementation that enable you to define behaviours which can then be implemented for data types. Data types are values like strings and numbers, collections like lists and sets and "higher kinded types" like IO, ZIO and so on.

Type classes do not have to be implemented a certain way, but in Cats, Scalaz and other libraries you will find them implemented just as they are here, but additional layers of complexity to manage things like stack safety and take advantage of tools to generate boilerplate code.

In general type classes consist of:

A trait that has one, or sometimes more, type parameters.
It contains one or more abstract methods. These define behaviours that must be implemented for each instance.
Also you will find generalized methods that are built using the abstract ones.
Typically a type class contains no data

Numeric

In the video I implemented the Numeric type class. What is Numeric? It is a type class you'll find in the Scala standard library and is used to build a generic representation of numbers.

https://www.scala-lang.org/api/current/scala/math/Numeric.html

What I develop here is just enough that we can use it for the expression evaluator in the first video (which only requires add and multiply operations).

You can find the complete code here

https://github.com/justinhj/evalexample/blob/master/src/main/scala/Scala2Numeric.scala

Let's look at the pieces one by one…

trait Numeric[T] {
  def Add(a: T, b: T): T
  def Mul(a: T, b: T): T

  def square(a: T): T = mul(a, a)
}

Here we define the trait which decribes the core of our type class; what it can do, and what you need to implement if you want your data type to be an instance of this typeclass.

Add and Mul are abstract, whilst the square function is derived or generalized and does not need to be implemented when making new supported instances.

Instances

Once you have the interface for your type class you can define instances for various data types. Here we define the instance for Long.

implicit val numericLong: Numeric[Long] = new Numeric[Long] {
  def add(a: Long, b: Long): Long = a + b
  def mul(a: Long, b: Long): Long = a * b
}

Note that this is implicit to make it easier to use our instances with other types, taking advantage of the fact that users can import the definititions and then use them in their code.

Here's an example function that is written using the Numeric type class as a parameter. This function displays what we call ad-hoc polymorphism, in that this function does not know all of the instances that exist for the input type T, nor does it need to. All it knows is that to do its work it needs an instance of Numeric[T] so that it can access the add method.

def sumList[T](ts: List[T])(implicit numeric: Numeric[T]): T = {
  ts.reduce((a, b) => numeric.Add(a,b))
}

Now we can sum a list of any T where T has a Numeric instance. Here we use the Int instance (not shown).

val l1 = List(1, 2, 3, 4)
val sum = sumList(l1)

println(s"sum of int list is $sum")
// sum of int list is 10

Improving the ergonomics

So that's all you need to build type classes but we can take a couple of steps to make things more ergonomic. For one we can ditch the implicit parameter and instead make use of [[https://docs.scala-lang.org/tutorials/FAQ/context-bounds.html]context bounds]]. This makes things clearer for the caller of the function.

def sumList[T : Numeric](ts: List[T]): T = {
  val numeric = implicitly[Numeric[T]]
  ts.reduce((a, b) => numeric.add(a,b))
}

Note that we have to summon the implicit Numeric instance using implicitly. This is a function in the standard library which takes advantage of the way context bounds work: the context bound Numeric specifies that there is an implicit Numeric instance in scope, but there is no named parameter as before. The implicitly function lets us access that implicit in a succinct way.

def implicitly[T](implicit e: T): T = e

So using context bounds helps a little with the use of type classes, the next step is to use implicit conversions so that we take the functions in our type class and make them look like ordinary methods on the data type.

object ops {

  implicit class NumericOps[T](a: T)(implicit numeric: Numeric[T]) {
    def add(b: T): T = numeric.add(a, b)
    def mul(b: T): T = numeric.mul(a, b)

    def +(b: T): T = add(b)
    def *(b: T): T = mul(b)
  }

}

Now if we import ops we can take advantage of the implicit conversion from type T to type NumericOps[T] to give us syntax like below.

val s1 = "abcd"
val s2 = "efgh"
val product = s1 * s2
println(s"product $product")
// product aeafagahbebfbgbhcecfcgchdedfdgdh

So you can see that by implementing a somewhat goofy instance of Numeric for string (given below) we now have the ability to use the multiplication operator on it as if it was a regular number.

implicit val stringNumeric: Numeric[String] = new Numeric[String] {
    def add(a: String, b: String): String = a + b

    def mul(a: String, b: String): String = for (
      as <- a;
      bs <- b;
      s <- as.toString ++ bs.toString) yield s
  }

Note that while this implentation of string arithmetic is not very rigorous and just for fun, there's nothing to stop you from implementing Numeric for data types that do have well defined arithmetic operations such as Roman Numerals.

Coherence

Type class coherence is an important concept I'll leave you with. This is a guideline in place to keep programs easy to reason about. It's a good practice to keep common instances together with your type classes so that users can easily find them, and don't duplicate the work. It's also important that you don't try to make multiple instances and let the users select one depending on their needs. The reason for that is the behaviour of your program can change profoundly when you do this, and that's terrible. It means you can't take advantage of local reasoning, one of the benefits of functional programming. You would need to be very careful with imports to make sure you are using the instance you think you are.

Final words

If you're coming from Java or similar OOP language you may recognise some of this as the adapter pattern (except with Scala implicits). Type class traits also have similarities to Go interfaces, although the type class pattern goes a bit beyond them in scope.

I would be amiss not to mention Haskell here, which has type classes implemented as a first-class language construct, and for some people the over-use of type classes in Scala is somewhat of an anti-pattern. We will see in future videos that the pattern will be greatly simplified in Scala 3 however.

References

Typelevel Cats functional programming library documentation https://typelevel.org/cats/

Functional Programming for Mortals https://leanpub.com/fpmortals

Update for December, year of total chaos 2020

ciesie.com — Sun, 06 Dec 2020 02:00:00 GMT

This will be the last update for 2020. We’re almost on the other side! We shouldn’t expect much out of 2021, since the time doesn’t care about our labels. Still, there is a feeling of hope that we’ll be able to, slowly, transition to post pandemic world. How will that world look like? Time will tell.

Last few months have been very productive for me. For better or worse. It’s nice to create and have a lot of fun ideas but… there is only so much time in a day. Self isolation, working from home, etc. help save time but that means I’m working way more than before pandemic. I’ll certainly need some quality vacation soon.

Beach • Art

Muffin Man — Sun, 06 Dec 2020 02:00:00 GMT

Two separate plots, and I'm super happy how I managed to align them. On the last image, you can see the lines I drew to help me align it. My wife insisted we keep this one.

Pigma Micron pens are indestructible, just saying.

Created:	December 2020
Size:	70x35cm
Paper:	Bristol 220gsm
Pens:	Pigma Micron

AWS Sagegemaker Jupyter Notebook

Brent's Blog — Sun, 06 Dec 2020 02:00:00 GMT

I wanted to upgrade the version of pandas that comes with AWS Sagemaker Notebook to > 1.1 because this simplify handling of group by with null or NaN values.

[Computing] Modifying and running a binary by recompiling a reverse engineered disassembly

devever.net/~hl — Sun, 06 Dec 2020 01:36:00 GMT

You're reverse engineering an executable and you want to modify it. Perhaps you want to add or tweak functionality of the executable, or perhaps you want to be able to modify parts of the program to get a better understanding of how it works. Either way, you want to start with a working binary and gradually replace some parts of it (but not others) with C code as you gradually understand more of the binary.

Language Servers are the New Frameworks

swyx's site RSS Feed — Sat, 05 Dec 2020 22:51:05 GMT

Developer Experience is shifting left, all the way to onKeyUp.

Lessons from Sweden about going cashless

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 05 Dec 2020 20:00:00 GMT

The most interesting ones are about the MEANING of money.

Emulating a Computer: The CHIP-8 Interpreter

River Gillis [riv.dev] — Sat, 05 Dec 2020 02:42:00 GMT

How to write a CHIP-8 emulator.

Emacs: Rotate my macOS display

xenodium.com @alvaro — Sat, 05 Dec 2020 02:00:00 GMT

Every so often, I rotate my monitor (vertical vs horizontal) for either work or to watch a movie. macOS enables changing the display rotation via a dropdown menu (under Preferences > Displays > Rotation) where you can pick between Standard, 90°, 180°, and 270°. That's all fine, but what I'd really like is a quick way to toggle between my preferred two choices: Standard and 270°.

Unsurprisingly, I'd also like to invoke it as an interactive command via Emacs's M-x (see Emacs: connect my Bluetooth speaker). With narrowing frameworks like ivy, helm, and ido, invoking these commands is just a breeze.

Turns out, this was pretty simple to accomplish, thanks to Eric Nitardy's fb-rotate command line utility. All that's left to do is wrap it in a tiny elisp ~~function~~ hack, add the toggling logic, and voilà!

The screen capture goes a little funky when rotating the display, but you get the idea. It works better in person :)

…and here's the snippet:

(defun ar/display-toggle-rotation ()
  (interactive)
  (require 'cl-lib)
  (cl-assert (executable-find "fb-rotate") nil
             "Install fb-rotate from https://github.com/CdLbB/fb-rotate")
  ;; #  Display_ID    Resolution  ____Display_Bounds____  Rotation
  ;; 2  0x2b347692    1440x2560      0     0  1440  2560    270    [main]
  ;; From fb-rotate output, get the `current-rotation' from Column 7, row 1 zero-based.
  (let ((current-rotation (nth 7 (split-string (nth 1 (process-lines "fb-rotate" "-i"))))))
    (call-process-shell-command (format "fb-rotate -d 1 -r %s"
                                        (if (equal current-rotation "270")
                                            "0"
                                          "270")))))

Setup VSCode to run and debug C / C++ code

Gourav Goyal — Sat, 05 Dec 2020 02:00:00 GMT

Follow this guide to run, debug, and get IntelliSense for C/C++ files in VSCode

Build a bulletin board using scotty and friends

λm.me — Sat, 05 Dec 2020 02:00:00 GMT

A tutorial for haskell-scotty, building a very simple bulletin board website.

Seldom discussed tips for working in the tech industry

Jason Eckert's Website and Blog — Sat, 05 Dec 2020 02:00:00 GMT

There are plenty of blog posts on the Web that give sage advice for anyone working in the tech industry. However, most of them focus on the same topics: time management, resourcefulness, continuous learning, verbal and written communication skills, caffeine consumption, and so on.

In this blog post, I’ll discuss 6 tech industry tips that are rarely discussed online, but that are equally (if not more) important from my experience over the past three decades in this industry.

Firefox : extensions, configuration et quelques astuces

Accueil on dbeley — Sat, 05 Dec 2020 01:00:00 GMT

Introduction

Mozilla Firefox est un navigateur libre développé par la fondation Mozilla. Il utilise le moteur de rendu Gecko, le seul concurrent au moteur de Google Chrome sur le marché des moteurs web modernes.

Malheureusement Firefox perd constamment des parts de marché face aux autres navigateurs. Selon radar.cloudflare.com, Firefox ne représenterait que 6,40% du traffic.

Pourtant Firefox dispose de beaucoup d’atouts pour qui souhaite un navigateur moderne, éthique et respectueux de la vie privée.

2020–12–05: Alternate EG25G userspace project

megi's PinePhone Development Log — Sat, 05 Dec 2020 01:00:00 GMT

Sometimes the best data are those that do not exist

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 04 Dec 2020 20:00:00 GMT

Or, what you can learn about privacy from Soviet weapon’s philosophy.

Separate core logic from the framework

Writing on Adam Craven — Fri, 04 Dec 2020 19:49:16 GMT

QGIS - How to use custom SVG-Markers

Geography & Coding — Fri, 04 Dec 2020 12:13:18 GMT

Use your own SVG as a QGIS marker!

HttpResponseMessage.Content is non-nullable in NET5

Nicola Iarocci — Fri, 04 Dec 2020 08:05:25 GMT

Today I learned the hard way that NET5 changed the HttpResponseMessage.Content type from nullable to non-nullable

Web Scraping Is Vital

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 03 Dec 2020 20:00:00 GMT

Seriously. Even if you have NO idea what it is. Keep reading:

The UX of Proving Our Humanity To Machines

swyx's site RSS Feed — Thu, 03 Dec 2020 18:13:27 GMT

What happens when we can't convince machines that we are human?

Using Tailwind UI and Next.js Together

Khanna Law Blog — Thu, 03 Dec 2020 02:00:00 GMT

These instructions have been updated to reflect the installation process for TailwindCSS 2.0 and Next.js 10.

Contact Lost • Art

Muffin Man — Thu, 03 Dec 2020 02:00:00 GMT

It ended up being vertically symmetrical because I forgot to change parameters for the bottom half.

Created:	December 2020
Size:	30x42cm
Paper:	Fabriano Bristol 250gsm
Pens:	Pigma Micron

References in Rust

Thu, 03 Dec 2020 02:00:00 GMT

Introduction

Rust references are very simple at runtime: they are plain memory addresses. At compile time, in contrast, references participate in more complex compiler analysis. For example, references help to prove memory safety of a program. But in this post, I will not cover the safety aspects of references. You have already read about that in the post Memory safety in Rust - part 2. In this post, you will understand the syntactical and usage aspects of references. Let's start with the very basics.

Basics

Let's say you have a variable x, which owns a value:

let x: i32 = 42;

To create a reference to x, you'd use the & operator:

let r = &x;

And to get the value of the referent, you'd use the * operator:

let v: i32 = *r;

All the values and references created above were immutable, which is the default in Rust. If you want to change the value through a reference, create a mutable reference. The &mut operator creates a mutable reference:

let m = &mut x;

But this alone is not enough. You will realize this if you try to compile it:

error[E0596]: cannot borrow `x` as mutable, as it is not declared as mutable
  |
2 |     let x: i32 = 42;
  |         - help: consider changing this to be mutable: `mut x`
...
5 |     let m = &mut x;
  |             ^^^^^^ cannot borrow as mutable

The error makes sense because it avoids surprises for the programmer. If the compiler had allowed this, a mutable reference (m) could change an immutable value (x). Such changes in an immutable value would have been confusing. Let's try the fix suggested by the error message:

//make x mut
let mut x: i32 = 42;
let m = &mut x;

Now the code compiles fine. Finally, you can change the value through the reference:

*m = 100;

Shared and mutable references

Let's now try to create another mutable reference to x:

let mut x: i32 = 42;
let m: &mut i32 = &mut x;
let n: &mut i32 = &mut x;
println!("m is {}", m);
println!("n is {}", n);

But it doesn't work:

error[E0499]: cannot borrow `x` as mutable more than once at a time
  |
3 |     let m: &mut i32 = &mut x;
  |                       ------ first mutable borrow occurs here
4 |     let n: &mut i32 = &mut x;
  |                       ^^^^^^ second mutable borrow occurs here
5 |     println!("m is {}", m);
  |                         - first borrow later used here

Again, the error tells us exactly what went wrong. We cannot have two mutable references to a value at the same time. In fact, it goes even further. As long as a mutable reference is alive, it locks the original value. You can't mutate the value through the owner during the lifetime of a mutable reference:

let mut x: i32 = 42;
let m: &mut i32 = &mut x;
x = 100;
println!("m is {}", m);

This results in the following error:

error[E0506]: cannot assign to `x` because it is borrowed
  |
3 |     let m: &mut i32 = &mut x;
  |                       ------ borrow of `x` occurs here
4 |     x = 100;
  |     ^^^^^^^ assignment to borrowed `x` occurs here
5 |     println!("m is {}", m);
  |                         - borrow later used here

On the flip side, you can create many immutable references to a value at the same time:

let x: i32 = 42;
let m: &i32 = &x;
let n: &i32 = &x;
println!("m is {}", m);
println!("n is {}", n);

This is safe since there is no mutation. But what about both mutable and immutable references at the same time? Should the compiler allow that? The answer is no. Again, for safety reasons this is forbidden:

let mut x: i32 = 42;
let m: &i32 = &x;
let n: &mut i32 = &mut x;
println!("m is {}", m);
println!("n is {}", n);

You'll get below error if you try:

error[E0502]: cannot borrow `x` as mutable because it is also borrowed as immutable
  |
3 |     let m: &i32 = &x;
  |                   -- immutable borrow occurs here
4 |     let n: &mut i32 = &mut x;
  |                       ^^^^^^ mutable borrow occurs here
5 |     println!("m is {}", m);
  |                         - immutable borrow later used here

To summarize, the usage rules for immutable and mutable references are as follows. For a value, there can be:

Either many immutable references.
Or one mutable reference.

But not both at the same time. For this reason, another term for immutable references is shared references. And a similar term for mutable references is exclusive references.

So why do references have these rules? You might have guessed by now — to enforce memory safety. (It is in fact due to safety reasons that Rustaceans term creating a reference as borrowing.) To understand how these rules achieve memory safety, read Memory safety in Rust - part 2.

Even though that post covers memory safety, it still leaves one question unanswered. Why does Rust ban more than one mutable reference on the same thread? That shouldn't cause any harm, right? Turns out that it does. Read Manish's excellent post The Problem With Single-threaded Shared Mutability to know how.

Do these rules remind you of something? To me they look exactly like a reader-writer lock. But that's where the similarities end. Unlike a reader-writer lock, the compiler enforces the reference rules at compile time. At runtime, a reference behaves exactly like a raw pointer; It is just a memory address. If you do want a construct with these same rules but at runtime, take a look at RefCell. If you are looking for an actual reader-writer lock, RwLock is your guy.

Implicit dereferencing

Let's say you have a struct instance s and a reference r to that instance:

struct Int {
    i: i32,
}

let s = Int { i: 10 };
let r = &s;

To get s's member i you'd write:

let j = s.i;

Similarly, if you were accessing the member through the reference, you'd write:

let j = r.i;

But wait a minute, shouldn't you dereference r before accessing the members? Like this:

let j = (*r).i;

You could, but it is not necessary because Rust has spared us needless verbosity. The explicit dereference is not needed because the . operator automatically dereferences the operand on its left. Not only that, the . operator also dereferences its left operand as many times as needed to reach the member:

let r = &s;
let rr = &&s;
let rrr = &&&s;

let j = r.i;
let j = rr.i;
let j = rrr.i;

Note Note

The . operator is not the only operator which automatically dereferences its operands. The comparison operators ==, !=, <= and >= also dereference operands on either side as needed.

The . operator also follows any Deref trait implementations on the left operand:

struct Inner {
    i: i32,
}

struct Outer {
    o: Inner
}

impl Deref for Outer {
    type Target = Inner;

    fn deref(&self) -> &Self::Target {
        &self.o
    }
}

let s = Outer { o: Inner { i: 10 }};
//s.i is transformed into s.deref().i by the compiler
let k = s.i;

Here the s struct doesn't have a member i. But this still compiles because the compiler transforms the expression s.i into s.deref().i.

One last trick up the . operator's sleeve is that it automatically takes its left operand's reference if needed:

struct Int {
    i: i32,
}

impl Int {
    fn print(&self) {
        println!("i is {}", self.i);
    }
}

let s = Int { i: 10 };
(&s).print();//calls the print method
s.print();//so does this

You might think that you should call the print method by creating a reference like this: (&s).print(). This sounds sensible because the print method takes a reference to Int (via the &self argument). But the shorter version s.print() works just as well because the compiler implicitly creates a reference for you. Rust goes to great lengths to avoid even such small paper cuts. This shows the attention to detail that goes into making Rust more ergonomic.

Rust references vs C++ references

In some ways Rust's references resemble pointers in C++. For example, the & and * operators are very similar in the two languages. But these syntactical similarities are superficial. C++ pointers (even smart pointers) can't match Rust references' safety.

References in C++, in contrast, are quite dissimilar to Rust references, even syntactically. Creating and dereferencing a reference is implicit in C++:

int i = 42;
int &r = i;//no & operator before i
int j = r;//no * operator before r

While in Rust these are explicit:

let i = 42;
let r = &i;//note the & operator before i
let j = *r;//note the * operator before r

Another difference is that you can reassign a reference in Rust to another object:

let i = 42;
let mut r = &i;//r is a reference to i
let j = 84;
r = &j;//r is now a reference to j

But you can't reseat a C++ reference:

int i = 42;
int &r = i;
int j = 84;
//might look like r is reseated but
//r still refers to i whose value is
//updated to become 84
r = j;

Apart from those differences, the two languages are similar in one aspect: references can't be null in either language.

And that concludes a whirlwind tour of references in Rust.

Conclusion

In this post, I covered an everyday usage of references in Rust. I showed how you can create shared and mutable references, and what rules govern their use. I also covered implicit dereferencing, which makes it easy to write clearer code. At last I compared Rust's references with C++ references, highlighting their superficial similarity. But pointed out that Rust's references are unmatched in safety, even by smart pointers in C++.

Comparing conversion at control and treatment sites

Bytepawn - Marton Trencseni — Thu, 03 Dec 2020 01:00:00 GMT

In real-life, non-digital situations, it's often not feasible to run true A/B tests. In such cases, we can compare before and after rollout conversions at a treatment site, while using a similar control site to measure and correct for seasonality. The post discusses how to compute increasingly correct p-values and bayesian probabilities in such scenarios.

Journal Article: â€œThe Dordrecht Synod at 400: A Guide for the Church in Times of Crysisâ€

The Grey Shadow — Wed, 02 Dec 2020 23:13:56 GMT

The end of the year is nearing, and if it was hectic before, now we are reaching â€œcore meltdownâ€ status. However, in moments of rare respite within this race, I had time to write a small journal article for Espacio TeolÃ³gico, the theological journal of the Evangelical Faculty of Theology with the title stated above....

Migrating a Decade of Plaintext to Zettelkasten

trv.rs — Wed, 02 Dec 2020 22:26:00 GMT

This is going to be a code-heavy post. Use a repeatable script, take advantage of good programming practices to control the data going into your system and normalize it. Borrow liberally from me if that helps. 🙏_

I think I'm done with an eight month-long struggle to import all my previous plaintext notes into the Simple Markdown Zettelkasten format. It was not easy… but I thought I'd share some of the lessons that I learned and tricks that I used.

I've been playing around with Zettelkasten style note-taking for a while now… and I'm pretty convinced that having everything in one place is the right path for me for now.

Abbreviated version of my plaintext adventures to this point…

Notepad documents from Windows and Ubuntu computers.
Plain text on an Android Dev-1 phone… some terrible text editor.
nvALT and Simplenote for years.
Daily devotional note on iPhone using Byword that got so long it routinely crashed my phone.
Vimwiki diary and wiki for five years at my last job.
Experimental zettelkasten using the MMD metadata.

There's some duplication, some craziness… and in the end it's 2,068 files. Let's get to work.

Use a script.

Just like any large scale migration, using a script has huge advantages over a series of manual "find and replace." If you write it correctly, it's repeatable, and you'll get the same results each time, which means that if you get more data to migrate, or change your mind about a format... you change the script and run it again. I changed my mind half a dozen times while I was working on this script, and I was actively using the zettelkasten as I went.

For instance... while working on the script, Obsidian released the aliases feature that lets backlinks work with Zettelkasten notes... I just adjusted my script and kept on rolling.

(Also, running the script while having Obisdian open on graph mode made for a really cool building animation that was definitely fun!)

I did flex some computer science muscles, and defined a class with a bunch of (inconsistently named) convenient methods and a standardized metadata format.

require 'Zettel'

zettel = Zettel.new

zettel.add_tag('knowledge')
zettel.set_meta('title', 'Zettelkasten is cool!')
zettel.content = "This note is useful, but short."
zettel.set_meta('id', '20201228369')

# this would render as:
"""
---
title: 'Zettelkasten is cool!'
tags: ['#knowledge']
id: 20201228369
---

This note is useful, but short.
"""

(using zettel.rb)

The code is not pretty... you can rip it off, but don't look too closely. 🤣

Having a single code-path to make a zettel makes the next step easy...

Adjust your script per source.

I wrote a customized script per "source" that I listed in my history above. Each one has details specific to that system...

My nvALT notes have titles in their filenames:

title = File.basename(entry, ".*")

(from migrate.rb)

But, vimwiki diary entries have <h2>s for some reason:

title = content.scan(/^#+ .*/).first.to_s.gsub(/^#+ /, "")

(from migrate.rb)

This kind of flexibility is super handy.

Guarantee unique IDs.

This is critical... and hard. I used a Set in Ruby like I have in the past, and it worked pretty well. I added a bunch of instrumentation to help me troubleshoot.

class Migrator
  DST = '../../wiki/'

  def initialize
    @unique_ids = Set.new
  end

  # [ ... ]

  def ensure_uniqueness(zettel)
    if @unique_ids.include? zettel.id then
      # we've got a dup
      puts "🔴 Duplicate ID! #&lbracezettel.title&rbrace: #&lbracezettel.id&rbrace"

      zettel.set(:id, zettel.id.to_i + 1)
      puts "trying #&lbracezettel.id&rbrace..."
      self.ensure_uniqueness(zettel)
    else
      if zettel.id.size > 12 then
        puts "#&lbracezettel.id&rbrace 🟡 ID is too long! #&lbracezettel.title&rbrace"
      elsif zettel.id.size < 12 then
        puts "#&lbracezettel.id&rbrace 🟡 ID is too short! #&lbracezettel.title&rbrace"
      end
      @unique_ids << zettel.id
    end
  end
end

(from migrate.rb)

Filter tags in place.

I also used the Zettel class to change and filter data going into the knowledge system graph:

def add_tag(tag)
  if tag then # not nil
    tag = tag
          .gsub("#", "")
          .gsub(" ", "_")

    if
      !tag.match(/\d+-?/) && # not just a number
      tag.size > 1 # not a single character
    then

      # blacklist
      unless [
        'responsible',
        'todos_archived',
        'wizardsetupstep',
        'tags',
        'id',
        'create_tickets',
        # [ ... ]
        'todo',
      ].include? tag then

        tag =
          case tag
          when 'pmux'
            'ux'
          when 'interview'
            'career'
          when 'project'
            'projects'
          when 'book'
            'books'
          # [ ... ]
          else
            tag
          end

        if @meta[:tags] then
          @meta[:tags].push(tag)
        else
          @meta[:tags] = [tag]
        end
      end
    end
  end
end

(from zettel.rb)

By having a single "point of contact" with the Zettel class, I was able to filter out or modify tags as they went into the final product.

Use the internet to supplement metadata.

Probably the cleverest thing I did was to grab data from Google Books to add metadata to my booknotes.

filename = File.basename(booknote)
query  = filename.gsub(/\..&lbrace2,3&rbrace$/, '')
title  = query.split(" by ").first.strip
author = query.split(" by ").last.strip

books =
  JSON.parse(Net::HTTP.get_response(
    URI("https://www.googleapis.com/books/v1/volumes?q=#&lbraceURI.encode(query)&rbrace")).body)

book = books["items"].min_by do |b|
  levenshtein_distance(b["volumeInfo"]["title"], title)
end["volumeInfo"]

zettel.set(:title, title)
zettel.set(:subtitle, book['subtitle'])
zettel.set(:author, book['authors'].join(', ')) # close enough to MLA
zettel.set(:publisher, book['publisher'])
zettel.set(:identifer, book["industryIdentifiers"][0]["identifier"])

(from migrate.rb)

I'm pretty proud of using using the Levenshtein distance to choose the most likely search result. It works fine for me, and my "database" has a little more information.

Go do it!

I'm really happy that I'm done, and I've got the script if I ever discover something isn't quite out of whack. I can now start using my knowledge system fearlessly... except I have to finish writing my Drafts actions.

What is more important, reading or job prospects?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 02 Dec 2020 20:00:00 GMT

Lies, damn lies, international agencies and priorities.

Log Pattern Recognition: LogMine

Sayr.us — Wed, 02 Dec 2020 16:00:00 GMT

When retrieving logs from an application, you are often looking for outliers (such as a unique error) or what pattern occurs the most. However, defining patterns manually is time-consuming and requires rigour across projects. In this series of blog posts, we are going to explore automated Log Pattern Recognition.

Master of my domain

voussoir.net/writing — Wed, 02 Dec 2020 10:15:01 GMT

Back to writing

Master of my domain

Introduction
Prefer portable over installers
With installers, always choose your directory
Separate subdirectories for each version
Use 'latest' junctions for program shortcuts
Reserve PATH for nobility only
Use junctions to reign in appdata
Back up your software
Dealing with uncooperative software

Introduction (§)

This article is about how I manage the software on my computer to stay organized and in control. I've developed these habits over years of dealing with software installations that each want to behave a little differently and go their own way. But I'm the master of my domain — queen of my castle, if you will — and I run the show around here. So here's how I put software in its place:

Prefer portable over installers (§)

If a software download page offers a choice between "portable"/.zip/.7z downloads and "setup"/.exe/.msi downloads, always pick the portable option. The reason they're called portable is because you can unzip them to a flash drive and just run the .exe on any computer you plug it in to, so that you don't have to actually install the program on every computer you use.

As a result, a well-behaved portable program will:

Store configuration files in the same folder as the .exe, instead of the computer's central registry or other paths.
Not rely on files outside of the program's folder — the whole thing is self-contained.
Not leave much, or anything, behind on the main computer — usually just some log files in appdata.

These properties are what I want from all software! The benefits even for non-portable use are many. Portable software makes it easy to:

find and modify config files by hand — no hunting for registry keys or configuration paths.
uninstall — just delete the folder.
run multiple simultaneous instances of the program, even of different versions or with different configurations.
sync with Syncthing.
back up and restore — just zip the folder. No need to worry about important files hiding anywhere else.

There is even a website, portableapps.com, dedicated to promoting portable software and preparing portable-ized versions of software that isn't normally available as portable. In the second case, however, you rely on portableapps being trustworthy.

Ostensibly, the benefits of installers over portables is they can set file associations, integrate with the right-click menu, and so forth. Well, I can handle file associations myself (HKCR\.ext\shell\open\command\(Default)), and even the most well-meaning context menu items really just contribute to an ever-growing list of junk so I'd rather not have those anyway. I can write those myself, too, if I really need them (HKCR\.ext\shell\mycommandname\command\(Default)). Then I'll know exactly where they're defined should I ever decide to remove them. This is in contrast to properties set by installers which are always in crazy places and mapped through GUIDs and whatnot.

With installers, always choose your directory (§)

Some software isn't available as a portable download, so you've got to suffer through the installer. If it gives you the opportunity to change the install directory, always take that chance. Software installers on Windows have a nasty habit of scattering themselves throughout C:\Program Files, C:\Program Files (x86), C:\ProgramData, and %appdata%. I'm king of this county and I hereby declare that I'm not gonna go on a wild goose chase looking for my software on my computer — they'll go where I tell them to go.

If I have to install a program from an installer, I'll usually try to zip up the resulting folder and keep that as my own portable copy for later. A lot of installers are nothing more than extractors, so as long as the program doesn't rely on some system changes during install, keeping the portable copy works fine (besides, system changes are often anti-user anyway). And I really don't need an installer acting as a glorified file extractor, I can do that myself from a zip. Unfortunately, some software doesn't work after re-extracting it, and I do have to keep the installer.

Some software authors take this Lèse-majesté a step further by not even offering a chance to change the install directory. There is no justifiable reason for this. I operate on a two or three strikes policy depending on how important the software is: I expect simply moving the installed folder, possibly leaving a junction in its place, to work properly. But now that the program is on my bad side it takes a lot less for me to uninstall.

The worst offenders are the likes of Adobe. A search on my computer right now shows:

C:\Program Files\Adobe
C:\Program Files\Common Files\Adobe
C:\Program Files (x86)\Adobe
C:\Program Files (x86)\Common Files\Adobe
C:\ProgramData\Adobe
C:\Users\All Users\Adobe
C:\Users\voussoir\AppData\Local\Adobe
C:\Users\voussoir\AppData\LocalLow\Adobe
C:\Users\voussoir\AppData\Roaming\Adobe
C:\Users\voussoir\AppData\Roaming\Adobe\Adobe
C:\Users\voussoir\Documents\Adobe
C:\Users\Public\Documents\Adobe

You're not the lord of this manor, Adobe!

Separate subdirectories for each version (§)

My Software folder looks like this, with a subdirectory for each version of every program:

D:\Software\Git\2.25.0
D:\Software\Inkscape\0.92.3
D:\Software\QuiteRSS\0.19.4
D:\Software\WinSCP\5.17.8

This way, I can maintain multiple versions of the software at the same time, which makes updating totally stress-free because I can stick with the old version as long as I want if the update has a regression. Contrast that with an in-place updater where you might regret the update after finding a new bug. My Android phone has several apps that I simply don't bother to update because I don't want to deal with reverting the .apks by hand through adb should I regret the update [1].

The only problem occurs when multiple versions expect %appdata%\Program to belong to themselves alone. If there is no commandline switch to change the appdata directory, I am somewhat out of luck on that point. One thing I had never though of until now is modifying the environment variables before running the program to manipulate %appdata% to a separate directory, similar to this SE thread. However, that solution didn't work for me, but I'll keep an eye out for something similar. This seems fragile since the program might (wrongfully) construct appdata from %userprofile%\appdata\roaming, or cache the value of %appdata% during install to a registry key, etc.

[1] F-Droid offers historical apks for download so I've got no concern there, but Google doesn't. Raccoon can help you download present-day apks from Google for your future benefit, though.

Use 'latest' junctions for program shortcuts (§)

Here's a fantastic trick I started using just recently. Since all my software is separated by version number, when I download the newest version of a program, all of my shortcuts or scripts which call that program will still be using the old path, and I'd have to update them manually. I came up with this solution: just mklink /j __latest X.Y.Z inside the program folder, and have all your shortcuts point to the exe inside __latest. Since it's a filesystem junction, it will behave transparently to any calling program. In the past I had tried to achieve this by making a single shorcut and pointing any future shortcuts to that one, but that didn't work because when you assign a shortcut to a shortcut it resolves to the real path and saves that path forever instead of folowing the link every time.

When I download a new version of a program, I just delete that junction and make it again with the new version number, and all linked paths work as normal, as long as the name of the .exe itself hasn't changed.

I don't like filling up my desktop or start menu with shortcuts, so I wrote PGUI which acts as a launcher for any .lnk files located inside my PGUI folder. Each of these shortcuts actually point to the __latest\program.exe path. As a bonus, my PGUI folder is one of the few folders to enjoy the honor of being on my PATH, which means I can just start them from the command line or Win+R — that's all possible without a gui launcher of course, but it's a great symbiosis.

Reserve PATH for nobility only (§)

The entire contents of my PATH are:

C:\Windows
C:\Windows\system32
D:\cmd
D:\git\cmd
D:\git\cmd\PGUI

Two folders for system executables, and three folders owned by me, because this is a monarchy and I'll appoint whoever I want. No application, under any circumstance, gets to put its own install directory on the PATH. If I need global commandline access to that program, I'll put an .lnk or symlink in \cmd.

At one point, I had some issues with programs that I wanted to subprocess from Python. subprocess.run(shutil.which('ffmpeg')) was breaking because subprocess.run wants to receive executable files and of course the only thing on my path was an .lnk. So, I wrote winwhich.py and now that problem is solved. The software bends to my will, not the other way around.

Use junctions to reign in appdata (§)

I've mentioned that even portable software has a tendency to leave a few traces in %appdata%. Sometimes it's inconsequential stuff like log files, but sometimes it's config, and that I want to keep in the program folder.

The solution here is in three steps:

Move %appdata%\Program into D:\Software\Program.
Rename that folder to e.g. "appdata" so you don't have Program\Program like some kind of Adobe.
mklink /j "%appdata%\Program" appdata.

Now you are closer to portable enlightenment and can zip up the program folder without losing your appdata files.

Back up your software (§)

Although I run regular backups, I don't include my whole Software folder in the backup routine because I already keep all the portable zips and installers backed up separately. The only items of value in the install directory are config files, most of which are not highly personalized and I don't mind losing either.

But, there are a few programs which I've heavily configured and I'd hate to start from scratch with them. Sublime, Putty, and WinSCP for example. For those I wrote a wrapper around WinRAR, rarpar.py, which I use to create highly compressed archives with a datestamp in the filename. Because the __latest files will be included in duplicate, you might think it'd waste a lot of space, but with a Solid archive and a very high dictionary size, that duplication causes no extra disk usage.

Dealing with uncooperative software (§)

If you find that one of your programs isn't cooperating with the above techniques, consider these more delicate approaches:

Say in a loud, commanding, and controlled voice, "YOU WORK FOR ME" (requires microphone).
Spawn several child processes. Kill them.
Reduce process priority to Below Normal and insist, "You can have your cycles back after you've cleaned up your act".

View this document's history

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

2020–12–02: Fixing fallout from the new WiFi driver

megi's PinePhone Development Log — Wed, 02 Dec 2020 01:00:00 GMT

RSS Feed written in HTML + JavaScript

Navan's Archive — Tue, 01 Dec 2020 22:52:00 GMT

Short code-snippet for an RSS feed, written in HTML and JavaScript

Please promote ODF with these posters!

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 01 Dec 2020 20:00:00 GMT

You really have no more excuse to not use and PROMOTE ODF.

A complete guide to preprocess Instagram post data

Geography & Coding — Tue, 01 Dec 2020 18:14:42 GMT

Scripts and tricks to preprocess Instagram data mined with Fast-Instagram-Scraper. Get it on a map!

AoC day1

blog — Tue, 01 Dec 2020 14:03:45 GMT

Guidelines for Deploying React

Max Rozen — Tue, 01 Dec 2020 11:48:48 GMT

These days there are a heck of a lot of options for deploying React. It can be a bit overwhelming. This article simplifies your choices.

How to tag data in Redis

Alexander's Blog Blog — Tue, 01 Dec 2020 02:00:00 GMT

What are Sets and how to use them as secondary indexes

Direct democracy features of the Swiss kind

nutcroft — Tue, 01 Dec 2020 02:00:00 GMT

There is some colloquy in political discussions that Switzerland has the best kind of democracy.

Introducing the Confederation

Officially the Swiss Confederation, Switzerland is a federal democracy, theoretically, resembling that of the United States¹. Instead of states, there are cantons. However, Switzerland, as a semi-direct democracy, also has a lot of mechanisms that enable direct democracy. Even though I have often invoked the Swiss democracy as worth copying, I was in the dark—until recently—about specific details of how the political system works. Except, only, for the famous high frequency of their referenda.

Three of the main voting mechanisms in Switzerland that enable directness are:

Mandatory referenda, for important changes, eg. in the constitution.
Legislative referenda, optional, for changing laws passed by the parliament.
Popular initiatives, which are initiated by the people to change the constitution in any way.

It would be great for other western countries to have a higher quality democracy, like the Swiss. But, it seems infeasible. Something that would require an unimaginable amount of change both in policies and in perception of politics by the people.

Assumptions

I’d love to try it, though. Nowadays, I'm under the impression that the vast majority of the world’s democracies are spiralling down rather swiftly. But, technology is booming, so what’s wrong? It would seem it's not a case of humanity being out of luck and experiencing a one-off event. Rather, something is wrong at a foundational level. Is technology not enough to solve all of humanities problems?

In addition, by enabling more directness in a democracy, the power is more divided among the people, which—in case of dangerous people—makes direct democracy more dangerous, not more desirable!

Both the importance of technology and the case of freedom versus safety are philosophical discussions out of the scope of this text. Thus, let’s assume:

Technology is of supreme importance.
We prefer not to distribute power if that means people with an education of lesser quality get it.

Educating voters

Under these assumptions, education and knowledge are the foundational stones of direct democracy. For the common people to decide about policies, they need to know about policies. Are Swiss people more educated than, say, other western countries? Even so, how much more educated?

I would guess not by that much, that it would justify a complete overturn of who holds the political power. To counter for the lack of education and specific knowledge about each referendum, in Switzerland, each ballot is accompanied with a handbook. The handbook explains in great detail what each vote means. For instance, what cost would each option incur, how would it be covered, what is the recommended vote by each level of authority (canton, parliament, municipality), what is the minority’s opinion and, of course, why.

An acceptable way to educate the public on what they vote for, but also an imperfect way. It raises many questions. Namely one, how many people do actually bother to vote, given the handbook's overhead? According to Wikipedia, the number has varied since the 1970s between 42% and 50%, with few extreme cases going over 60%, such as the matter of Switzerland joining the EU.

Those numbers do sound close to an average election turnout in many western countries, if not worse. Which raises another interesting question: how do they manage to maintain their direct democracy quality (and do they?) given these meagre voter turnouts²?

Other features of the system

If we examine the example of a Swiss citizen, who has been living in Zurich since 2000, they have had the opportunity to vote in 548 referenda in the last 21 years. With an average of ~1 referenda every two weeks, that sounds unreal. How could this work?

They vote in bulk. They may hold bulk referenda four times every year, voting for ~6 issues each time. It makes one wonder how many tricks like that they have devised in order to scale direct democracy to 8+ million people³.

Among other interesting properties of the Swiss political system is the fact that Switzerland has no constitutional court. The power of interpreting the constitution lies again only with the people.

But, the most interesting one is probably that there is no single human that is the face and embodies the represented power of the Swiss government. There is a Federal Council that is represented by seven members, each having one vote. There is also a Federal Chancellor, but they are not important at all as they hold no vote of those seven.

I'm inclined to argue that this property of dividing a country’s top executive by seven, doesn't even need that much of a perception shift to happen—it's only a policy to be voted for, with no (?) negative side effects yet with the distribution-of-power positive main effect.

Conclusion

As I'm reading more and more about all the various mechanics the Swiss have, each one solving and improving on the drawback of the previous, my realisation is this: there is no perfect system, of course; the decision is which fundamental ideas we want to adopt and build upon.

At the end of the day, the foremost inquiry is: do we want to try something new or is this indeed the end of history? I understand the need to remain with the devil we know, yet, I'm excited to discover the one we don't. Only because the fundamental idea of power distribution sounds better than the lack of it.

Epilogue

Martin Sústrik's blog posts at 250bpm is where most of the above findings were found. They were extremely interesting to read, and I recommend them especially for the parts that I did not mention. Examples include how the number of popular initiatives soared in the 1970s (before that they hadn't figured out how to do them productively!), how they dealt and are dealing with their serious problem of immigration, how to face dangerous referenda (read Brexit), and one that could be a book of its own: concordance versus competitive democracy (it's in Part 3).

Other references include:

antenna.nl: The history of direct democracy in Switzerland
houseofswitzerland.org: The way to modern direct democracy in Switzerland
admin.ch: Switzerland and its Democracy – a view from inside

SWI swissinfo.ch: Swiss and US democracy: twins separated at birth? ↩
In the last paragraph of Part 3, Martin reveals something interesting: "It turns out that about a quarter votes almost every time, a quarter almost never, and the rest votes occasionally. That being said, 90% of people have voted at least once in the past five years. This suggests that people vote selectively." ↩
Here, one could argue, they have actually scaled to 5.2m people (rather than 8.5m), because that's the amount of Swiss people eligible to vote. Which is yet another interesting problem: how much of a democracy is it, if only ~61% of the population take part? N.B. that in addition to expats, I have included ignorant children in this number, too. ↩

My stack will outlive yours

Steren's essays — Tue, 01 Dec 2020 02:00:00 GMT

Using Topology to Classify Labelled Graphs

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Mon, 30 Nov 2020 22:00:00 GMT

I have written at lengths about certain aspects of topological data analysis, but I have neglected to discuss one of its main applications, i.e. the classification of graphs. In this post, I will therefore take you on a quick tour of our ICML paper A Persistent Weisfeiler–Lehman Procedure for Graph Classification.

Let us assume that we are given a graph with node label information. The graph could, for instance, be a molecule, whose nodes are atoms such as carbon or oxygen, and whose edges indicate chemical bonds. The goal could now be to classify a given molecule into a set of classes, such as ’toxic’, ‘carcinogen’, etc. How can we achieve such a classification? One of the simplest techniques dates back to the 1960s and involves calculating an iterative fingerprint of the graph! This procedure was suggested by Boris Weisfeiler and Andrei Lehman (sometimes also transliterated as ‘Leman’) in their seminal article The reduction of a graph to canonical form and the algebra which appears therein.

At its core, the algorithm is an iteration scheme that works like this:

For a node $v$, collect its label and the labels of adjacent nodes in a multiset.
Assign this multiset a new label by hashing it—with the proviso that the hashing function is perfect, i.e. it maps distinct labels to distinct values with no collisions.
Replace all node labels by their multiset hashes.

Intuitively, each step of the algorithm accumulates more information from nodes that are further removed from the current node. The hashed multiset label is thus an expression of the neighbourhood around a node—and after a sufficiently large number of iterations, the hashed labels will not change any more.

For example, suppose you are dealing with this simple graph (to prevent confusion of node labels and node IDs, I used colours to indicate node labels in this example):

Tabulating the neighbourhood of each node then results in the following table:

Now for the hashing step. In this example, perfect hashing means choosing a set of colours that is distinct for every distinct combination of neighbourhood labels and vertex labels:

Notice how nodes A, B, and G are hashed to the same colour—because in the first iteration of the algorithm, they cannot be distinguished. How can we use the information about the hashed labels in a subsequent comparison task? The answer is lies in counting them in a histogram vector, which is indexed by the unique labels—this is where our requirement of the perfect hashing function is helpful. For the previously-shown graph, it looks like this:

The fingerprint of this graph, according to the first iteration of the Weisfeiler–Lehman scheme is therefore $(3, 1, 2, 1)$. Further iterations just make the feature vector longer (technically, the initial labels already give rise to a feature vector of counts). This procedure can now be repeated for higher-order iterations and the resulting feature vectors can be compared across graphs by evaluating, for example, their dot product. More formalisations of this idea have resulted in the very successful Weisfeiler–Lehman Graph Kernels publication. This method arguably constitutes the basis for graph neural networks—in fact, these networks can be seen as a parametrised version of the Weisfeiler–Lehman iteration scheme. But I digress—if you are interested in these aspects, please read Michael Bronstein’s article on going beyond graph isomorphism for more details.

Now, despite its great practical utility, this feature vector is lacking some structural information about the graph. It does not know whether a certain label contributes much to the topological structure of a graph—such as a ring of carbon atoms would in molecule—or not. To this end, we introduced a notion of topological relevance for each node label! Briefly put, we first developed a distance metric that would permit us turn any labelled graph into a weighted graph. We then calculate a persistence barcode, a topological descriptor of the graph. This descriptor assesses the relevance of a topological feature created by some node label. We use the topological relevance of each feature as an additional weight for the previously-shown feature vector. In essence, labels that contribute a large amount of topological structure in a graph are assigned a higher weight than labels that only contribute a meagre amount!

Here is a graphical depiction of our process:

If you want to brush up your understanding of the barcode calculation, head on over to Christian’s website; he has an excellent article on persistent homology.

The neat thing about our approach is that we can easily integrate information about cycles into the feature vector—this is a functionality that the original Weisfeiler–Lehman Graph Kernels Framework lacks. Moreover, these cycles turn out to be crucial in improving classification performance—we get an increase of more than 3% in classification accuracy by considering them in some data sets!

If this has whet your appetite, I invite you to read our paper or take a look at the code. If you want to learn more about graph classification using graph kernels, take a look at our recent survey on graph kernels, which will hopefully be officially announced in time for NeurIPS 2020. In the best tradition of Fermat, I would very much like to cover the content of the survey here, but this blog is too small to contain all of it—maybe for a subsequent post?

Until next time!

Books and Links: November

trv.rs — Mon, 30 Nov 2020 20:28:00 GMT

Hard to believe that tomorrow starts December of 2020. The years are short and the days are long.

Grateful to be chasing my toddler around the house with matching magnifying glasses, solving "mysteries" that I've scrawled on scrap paper. God is good.

I wrote a fair amount of code this month, mostly around the simple markdown zettelkasten work that I finally unblocked.

Books

I actually read some books this month! Still trying to get my booknotes straightened out, but this month's progress emboldened me.

On deck: Design Systems, Knowing God, The 12 Week Year, and maybe that new Brandon Sanderson action.

Whiskey Tango Foxtrot by Lynne M. Black, Jr.

Content warning: Some language and discussion of the sins of soldiers.

I heard about this on a podcast... and it's a wild ride. It's very hard to believe that all that happened to one person. The aggressive and decisive decision making under fire is really inspiring.

The Pastel City by Michael John Harrison

I'm a fan of "The New Weird," so I had this on the queue to revisit some of the "weird fiction" that inspired some of the books I've enjoyed.

Kind of crazy that this book was in 1971. I can see its echos in Star Wars, Brandon Sanderson, and of course New Weird like the Southern Reach trilogy.

Children of Time by Adrian Tchaikovsky

This book devoured my time until it was done. I don't think I've read it before, but I am fairly certain that I had it explained to me by someone at some point. (Looking at you Robert H.!)

If you like high-concept scifi and haven't read this one, this is a good one to visit. It deserves the accolades it got.

Links

I Spent 11 Years Working on This Line Rider Track | by David Lu | Medium

I remember when I first discovered Line Rider, that Flash game from 2006 with the simple premise of drawing a track for a sledder to ride on. I was immediately sucked into it, doing things like devising elaborate tracks for the rider to overcome, building worlds for the rider to explore, and manipulating the rider to perform stunts.

[…]

Eleven years later, after I reversed engineered and recreated Line Rider, after I developed as an artist and explored all types of creative mediums, I finally completed the project and even went beyond my original vision, reclaiming the project to tell a new story. I present to you, Omniverse II.

😱😱😱

I Spent 11 Years Working on This Line Rider Track | by David Lu | Medium

Copying is the way design works || Matthew Ström: designer & developer

There’s another word in Chinese for a copy: shanzhai (山寨). It’s translated to English as “fake,” but as with most Chinese words, the translation is lacking. Shanzhai literally means “mountain stronghold;” the word is a neologism, a recent invention, inspired by a famous novel in which the protagonists hide in a mountain stronghold to fight against a corrupt regime. Shanzhai products are playful, drawing attention to the fact that they aren’t original, putting their makers’ creativity on display.

Take the popular shanzhai novel Harry Potter and the Porcelain Doll; in it, Harry goes to China to stop Voldemort and Voldemort’s Chinese counterpart. It doesn’t pretend to be an original. It plays on its fake-ness: Harry speaks Chinese fluently, but he has trouble eating with chopsticks.

It’s easy to think of shanzhai as a Chinese quirk, but there are parallels in Western culture. One in particular, is a staple of the design community: the unsolicited redesign.

Wonderful essay.

Copying is the way design works || Matthew Ström: designer & developer

The Truth about Expressive Individualism

expressive individualism is simply this idea that the most authentic individual, the most real person, is the person who expresses or performs outwardly that which they are inwardly.

I found this to be very helpful… it's a philosophy that spans across our entire culture these days… "that person is so real" is a common refrain.

One of the big differences between what we might call Christian expressive individualism and secular expressive individualism is that Christian expressive individualism acknowledges that there may be a problem with that which is inside.

The Truth about Expressive Individualism

How to Think for Yourself

I wish someone had told me about this distinction when I was a kid, because it's one of the most important things to think about when you're deciding what kind of work you want to do. Do you want to do the kind of work where you can only win by thinking differently from everyone else? I suspect most people's unconscious mind will answer that question before their conscious mind has a chance to. I know mine does.

To try and avoid conventional thinking is an interesting idea… it's particularly weird since people on both sides of the aisle consider the other side to be the convention.

More generally your goal should be not to let anything into your head unexamined, and things don't always enter your head in the form of statements. Some of the most powerful influences are implicit. How do you even notice these? By standing back and watching how other people get their ideas.

This sounds like choosing to develop your worldview more than anything else. I can agree with that.

How to Think for Yourself

The Rise and Triumph of the Modern Self | Tim Challies

It turns finally to the world-changing ideas of Friedrich Nietzsche, Karl Marx, and Charles Darwin who each “in their different ways provided conceptual justification for rejecting the notion of human nature and thus paved the way for the plausibility of the idea that human beings are plastic creatures with no fixed identity founded on an intrinsic and ineradicable essence.” It is through these men that society’s notion of the self became psychologized.

The Rise and Triumph of the Modern Self | Tim Challies

YOU'RE ALL JUST JEALOUS OF MY JETPACK

I love this artist.

YOU'RE ALL JUST JEALOUS OF MY JETPACK | My latest cartoon for New Scientist See more of my...

Jeffrey Paul: Your Computer Isn't Yours

We were all super confused when this happened mid November… but now I'm just concerned. It probably wasn't a malicious design originally, but this sounds like something that a company truly committed to privacy should address.

On modern versions of macOS, you simply can’t power on your computer, launch a text editor or eBook reader, and write or read, without a log of your activity being transmitted and stored.

It turns out that in the current version of the macOS, the OS sends to Apple a hash (unique identifier) of each and every program you run, when you run it. Lots of people didn’t realize this, because it’s silent and invisible and it fails instantly and gracefully when you’re offline, but today the server got really slow and it didn’t hit the fail-fast code path, and everyone’s apps failed to open if they were connected to the internet.

Jeffrey Paul: Your Computer Isn't Yours

A small problem with e-voting in Italy

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 30 Nov 2020 20:00:00 GMT

Some Italians may e-vote soon. Says WHO?

Will we ever get solid systems?

utf9k — Mon, 30 Nov 2020 13:00:00 GMT

Is software reliable? Probably not

Setting up a cross-platform SDL project

River Gillis [riv.dev] — Mon, 30 Nov 2020 05:15:00 GMT

A look at how to set up SDL everywhere.

Focus Is the Most Important Thing

Byte Size Theories — Mon, 30 Nov 2020 03:58:56 GMT

In the present world we live in, most products compete on the same market and for the same resource: our attention. The usual suspects are facebook, twitter, reddit on your phone, netflix, hulu & friends on your other devices. In the “more tolerated” category we have personal messaging: imessage, signal, whatsapp & messenger (hey again facebook). The competition has recently extended to “workplace productivity”. This is a sneaky way of saying “social media, but at work”: slack, teams, email.

A simple neural network

RSS — Mon, 30 Nov 2020 02:20:00 GMT

Belatedly realizing that it's surprisingly simple to implement a neural network from scratch, particularly if you don't care about using it in production -- some developer notes and a snapshot of a basic net in roughly ~150 lines of Rust with minimal dependencies.

(Unrelated aside: A Notebook Style Guide (Slides) -- a short talk I gave at JupyterCon 2020 was released on YouTube. It started as a blog post here earlier this year.)

Goodbye Google Play Music

Imran's Blog — Mon, 30 Nov 2020 02:00:00 GMT

This weekend I spent some time migrating my music off Google Play Music to Spotify.

I am going to miss its auto generated radio stations which helped introduce me to a lot of new music.

The last Google product that I actually enjoyed was Google Inbox, which was subsequently also killed. With this I am probably not going to be buying into any new Google products.

The lesson learnt is that do not pay for a service that is not the primary income for that company. Otherwise you do not have any garauntees that they wont kill the service off once they're bored with it.

Here's to hoping Spotify won't shut down in the immediate future.

2020–11–30: Improved ANX7688 driver and increased I2C speed

megi's PinePhone Development Log — Mon, 30 Nov 2020 01:00:00 GMT

Test emails with Symfony

Swag Industries — Sun, 29 Nov 2020 16:09:47 GMT

Let's see together how to test Symfony emails with PHPUnit, but also with Behat.

My objective function will blow it up

dcz's posts — Sun, 29 Nov 2020 16:00:00 GMT

My objective function will blow it up

I have a preference for playing god. Build a system, observe, disrupt, see what new equilibrium it reaches. This manifests in weirdest places: gardening, cleaning, feeding the birds, computer games. Here's the story of how it turned me into a loss function and how you could become one too.

Computer games define the main thread today. It all played out in the span of mere weeks before publishing the post.

Fuel

It started from an unexpected angle this time: not Black and White, where you literally play god to your people, nor Civilization where your actions determine the prosperity of a city. Instead, what set me on the path to explosion was Crimsonland, namely the freeware 2002 version. As I mourned that it's not playable under Linux, I considered making a Libre clone. However, after succumbing to a few more waves of monsters, I also concluded that it's perfect at what it does, therefore any potential attempt to replicate it would end up a great disappointment. No point trying. I let my attention wander.

Oxidizer

Creatures is be a name you might have heard in the '90s and early '00s. A series of games that could be as sophisticated tamagotchi, they put you in charge of the well-being of humanoid creatures called Norns. I stumbled across a gameplay video, which showed some gameplay, including sharing tips about punishing and rewarding Norns for their behaviour, breeding them, showing the contents of the neural network that makes their brains, and analyzing the chemicals flowing through their bodies. Turns out they have genes too!

To me, that moved the game from "being a parent" to "playing god" territory. I immediately tried it out. Turns out that playing a nurturing god is only exciting when there's something going on, and the Norns learn and explore rather slowly. As I let them be, as they walked back and forth on a second screen, my attention went astray again…

Detonator

My next chance encounter was the link to Bevy, the game engine written in Rust. Nice, I've been wanting to learn to use an Entity Component System for a while, and I'm unwilling to deal with C. I never made games because of how tedious basic graphics are, but if I had anything to use it for, I would give it a try.

Fire in the hole!

And then it all clicked.

What if I took Crimsonland and gave the monsters a set of genes – simple enough to let them sense the enemy and walk – and culled them using the protagonist's weapons, it would be artificial selection!

Selection by gun

What's so cool about that? It's that the programmer is no longer directly in control. A well executed evolutionary algorithm may take you to unexpected places. It may be undesired when it subverts your goals and do something you didn't want it to – there was a group who tried to teach a simulated human to throw a ball, but ended up with one running with it instead (sadly, I lost the link). It may be neutral when the shape of its solution is just weird, or, like I hope to see here, it may bring surprise and variety into the game. That's on top of the fact that I don't really know how to program a smart AI for a monster myself.

Now that we know what evolution is good for, what are the pieces it requires to take off?

First, we need a bunch of creatures: the gene pool. That's a given both in the virtual world and on farms. Evolution does not work on individuals, but on groups, and it's the group of individuals that keeps improving.

Next, in order for the group to improve, there needs to be a way to know the "better" ones from "worse" ones. For a farmer, this may be the number of laid eggs. For a shooter, let's start with the rule that every creature (genome) that manages to score a bite gets a "goodness" point. That's also called the "objective function".

It's no use knowing what is better if we don't do something about. We want improvement in our gene pool, so genomes scoring higher will more often be the source of newly spawned creatures, and those less bitey will be underrepresented. That keeps the balance on the high side.

But those spawned creatures will not just be genetic copies of their ancestors. If they were, our gene pool would only be able to lose genetic variety: some heritage lines (the unsuccessful ones) would die off forever, leaving emptiness in its place. So we need to restore some of that variety. Instead of spawning exact clones, we give them a chance to mutate! Less bitey mutations rarely score high, and so they rarely reproduce. Little harm done. However, better monsters score high often, and they will spawn copies more often – gene pool improvement!

We end up with a dominating loop: creature -> bite -> mutate -> better creature -> bite -> mutate. There are other paths which arrive at a dead end, like: creature -> bite -> mutate -> worse creature -> fail, or bad creature -> fail -> not breed. As long as the loop dominates, however, we're improving.

There, now we are on the path to breed ultimate killing machines, and I can play god with an entire species of virtual beings.

Breedmatic

The result of that all of thinking is a prototype game Breedmatic. Adhering to my principle of starting in the middle, from the most interesting part, I went on to create a bunch of genetically modifiable monsters. They are very simple: sense, turn, walk. The genes describe how they turn relative to their target: their "brain". There's one hidden part of the story though: breeding and maintaining a healthy population.

It turns out that implementing a good genetic algorithm takes a lot of time. There are considerations regarding the size of gene pools, the rates of mutation of different parts of the genome, the selection of the best objective function, and even the strategy for preserving diversity: should reproduction be asexual? Diploid? How many sexes? That is a lot of sessions of shooting monsters. Hands get tired, progress is slow. In a moment of inspiration, I decided to sidestep the problem with… an evolutionary algorithm. As of version 0.2, the shooter itself is driven by evolution, in much similar way as the monsters.

But they start blank, so you can watch the shooters over hundreds of attempts going from barely moving through phases of spinning randomly to somewhat competent, and even sometimes coming up with weird moves which end up scattering the otherwise accurate laser, which end up being nonsensically effective against the incoming hordes.

A video of an evolved shooter that actually kinda gets it. Seizures help when shooting lasers, apparently.

Playing god has never been so comfy: no need to even move a finger.

Is this fun tho?

Sure it is. Watching the struggle of each and every shooter, I can't help but pray to the random number generator gods for luck in the next draw. "May it be 55's offspring next! It was turning the right way, just opposite!". The chance of hitting gold with the brain connections keeps me on the edge of the seat, and I lose track of time. Every session, there's a chance that a previously unseen combination appears – and the brain is only made of 7 neurons!

But you promised I could be the objective function!

And so you can! The most recent version from git lets you take over control. The monsters are not bred for variety, and the gameplay is basic though, so it's rather boring. At least compared to watching evolution go, so I have a better idea going in that direction.

Did you play Massive Chalice? It combines a tactical game with the management of a royal lineage. There are not many games of this sort, and the premise is effectively the same as mine here: to end up with a genetically superior population. Following the inspiration, future versions of Breedmatic will aim to put the player – not the algorithm – in the role of the gene pool manager. Hopefully that's more fun than shooting questionably intelligent monsters.

Evolve

Breedmatic is open, so if you are dismayed with the intelligence or shape of the creatures, feel free to fork it or contribute. Perhaps we can share ideas regarding the actual gameplay. The prototype could still evolve in many directions!

Otherwise, watch this space. While the next post may not be about evolution, I always have some crazy project in progress.

Unlimited free WiFi on flights with T-Mobile

JonLuca's Blog — Sun, 29 Nov 2020 14:12:29 GMT

Investigating an airline Wi-Fi authentication flow that granted T-Mobile connectivity.

Create Ebay Scraper in Python using Scraper API Learn how to create an eBay data scraper in Python to fetch item details and price.

Adnan's Random bytes — Sun, 29 Nov 2020 11:56:52 GMT

In this post of ScrapingTheFamous, I am going o write a scraper that will scrape data from eBay. eBay is an online auction site where people put their listing up for selling stuff based on an auction. Like before, we will be writing the two scripts, one to fetch listing URs and store in a text file and the other to parse those links. The data will be stored in JSON format for further processing. I will be using Scraper API service for parsing purposes which makes me free from all worries blocking and rendering dynamic sites since it takes care of everything. The first script is to fetching listings of a category. So let’s do it! import requests from bs4 import BeautifulSoup if __name__ == '__main__': API_KEY = None links_file = 'links.txt' links = [] with open('API_KEY.txt', encoding='utf8') as f: API_KEY = f.read() URL_TO_SCRAPE = 'https://www.ebay.com/b/Computer-Components-Parts/175673/bn_1643095' payload = {'api_key': API_KEY, 'url': URL_TO_SCRAPE, 'render': 'false'} r = requests.get('http://api.scraperapi.com', params=payload, timeout=60) if r.status_code == 200: html = r.text.strip() soup = BeautifulSoup(html, 'lxml') entries = soup.select('.s-item a') for entry in entries: if 'p/' in entry['href']: listing_url = entry['href'].replace('&rt=nc#UserReviews', '') links.append(listing_url) if len(links) > 0: with open(links_file, 'a+', encoding='utf8') as f: f.write('\n'.join(links)) print('Links stored successfully.') Now let’s write the parse script for parsing individual listing information. Do remember that I am not here to write the entire script and each and everything. I already made many tutorials about it and you can check it all here. SO below is the parse script: import requests from bs4 import BeautifulSoup if __name__ == '__main__': record = {} price = title = seller = image = None with open('API_KEY.txt', encoding='utf8') as f: API_KEY = f.read() URL_TO_SCRAPE = 'https://www.ebay.com/p/5034585650?iid=202781903791' payload = {'api_key': API_KEY, 'url': URL_TO_SCRAPE, 'render': 'false'} r = requests.get('http://api.scraperapi.com', params=payload, timeout=60) if r.status_code == 200: html = r.text soup = BeautifulSoup(html, 'lxml') title_section = soup.select('.product-title') if title_section: title = title_section[0].text.strip() selleer_section = soup.select('.seller-persona') if selleer_section: seller = selleer_section[0].text.replace('Sold by', '').replace('Positive feedbackContact seller', '') selller = seller[:-6] price_section = soup.select('.display-price') if price_section: price = price_section[0].text image_section = soup.select('.vi-image-gallery__enlarge-link img') if image_section: image = image_section[0]['src'] record = { 'title': title, 'price': price, 'seller': seller, 'image': image, } print(record) When I run the script it prints the following: { 'title': 'AMD Ryzen 3 3200G - 3.6GHz Quad Core (YD3200C5FHBOX) Processor', 'price': '$99.99', 'seller': 'best_buy\xa0(698388)97.2% ', 'image': 'https://i.ebayimg.com/images/g/ss8AAOSwsbhdmy2e/s-l640.jpg' } Pretty straight forward. Conclusion In this post, you learned how you can scrape eBay data easily by using Scraper API in Python. You can enhance this script as per your need like writing a price monitoring script. Writing scrapers is an interesting journey but you can hit the wall if the site blocks your IP. As an individual, you can’t afford expensive proxies either. Scraper API provides you an affordable and easy to use API that will let you scrape websites without any hassle. You do not need to worry about getting blocked because Scraper API by default uses proxies to access websites. On top of it, you do not need to worry about Selenium either since Scraper API provides the facility of a headless browser too. I also have written a post about how to use it. Click here to signup with my referral link or enter promo code adnan10, you will get a 10% discount on it. In case you do not get the discount then just let me know via email on my site and I’d sure help you out. If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Create Ebay Scraper in Python using Scraper API
Learn how to create an eBay data scraper in Python to fetch item details and price. first appeared on Adnan's Random bytes.

Emacs: Clone git repo from clipboard

xenodium.com @alvaro — Sun, 29 Nov 2020 02:00:00 GMT

Cloning git repositories is a pretty common task. For me, it typically goes something like:

Copy git repo URL from browser.
Drop to Emacs eshell.
Change current directory.
Type "git clone ".
Paste git repo URL.
Run git command.
Change directory to cloned repo.
Open dired.

No biggie, but why go through the same steps every time? We can do better. We have a hyper malleable editor, so let's get it to grab the URL from clipboard and do its thing.

shell-command or async-shell-command can help in this space, but require additional work: change location, re-type command, what if directory already exists… This is Emacs, so we can craft the exact experience we want. I did take inspiration from shell-command to display the process buffer correctly (git progress, control codes, etc.) and landed on the following experience:

;; -*- lexical-binding: t -*-

(defun ar/git-clone-clipboard-url ()
  "Clone git URL in clipboard asynchronously and open in dired when finished."
  (interactive)
  (cl-assert (string-match-p "^\\(http\\|https\\|ssh\\)://" (current-kill 0)) nil "No URL in clipboard")
  (let* ((url (current-kill 0))
         (download-dir (expand-file-name "~/Downloads/"))
         (project-dir (concat (file-name-as-directory download-dir)
                              (file-name-base url)))
         (default-directory download-dir)
         (command (format "git clone %s" url))
         (buffer (generate-new-buffer (format "*%s*" command)))
         (proc))
    (when (file-exists-p project-dir)
      (if (y-or-n-p (format "%s exists. delete?" (file-name-base url)))
          (delete-directory project-dir t)
        (user-error "Bailed")))
    (switch-to-buffer buffer)
    (setq proc (start-process-shell-command (nth 0 (split-string command)) buffer command))
    (with-current-buffer buffer
      (setq default-directory download-dir)
      (shell-command-save-pos-or-erase)
      (require 'shell)
      (shell-mode)
      (view-mode +1))
    (set-process-sentinel proc (lambda (process state)
                                 (let ((output (with-current-buffer (process-buffer process)
                                                 (buffer-string))))
                                   (kill-buffer (process-buffer process))
                                   (if (= (process-exit-status process) 0)
                                       (progn
                                         (message "finished: %s" command)
                                         (dired project-dir))
                                     (user-error (format "%s\n%s" command output))))))
    (set-process-filter proc #'comint-output-filter)))

Comment on reddit or twitter.

Updates

Added lexical binding.
Checks clipboard for ssh urls also.

Moving my blog to zola

Imran's Blog — Sun, 29 Nov 2020 02:00:00 GMT

I decided to move my blog to Zola instead of Docusaurus. They are both static site generators, but Zola is much faster.

Heres a Docusaurus build:

And heres a Zola build:

Writing on books - Fantastic Beasts and Where to Find Them

Nemo's Home — Sun, 29 Nov 2020 02:00:00 GMT

“Fantastic Beasts and Where to Find Them” is a curious book:

The in-universe book is written by Newt Scamander and was published in 1927.
The first edition of the companion book was published in 2001. This is apparently the 52nd in-universe edition with a foreword from Dumbledore and was released to the muggle world for charity.
The 2001 edition pretends to be Harry’s copy of the book as of the end of Harry’s 4th year at Hogwarts. As such it includes hand-written comments from the trio. (Yes, Hermione writes on books!)

However, with the release of the film of the same name in 2016 - a new edition was released with lots of changes:

6 new beasts that made an appearance in the film were added to the book¹:
- Hidebehind
- Hodag
- Horned Serpent
- Snallygaster
- Thunderbird
- Wampus cat
The hand-lettering was removed.
Dumbledore’s foreword is removed from the book, in favor of a in-universe foreword from Newt Scamander.
“About the Author” section changes Newt’s background. He no longer graduates from Hogwarts, just “leaves” it, as portrayed in the film.

All of these changes are meant to fix the inconsistencies in the book with the canon, however that also makes the book much less charming. I got myself a copy of the Hogwarts Library boxset a few years ago, which includes the newer edition of the book (Bloomsbury) - that means no witty comments from Ron.

Since it didn’t have the hand-lettering, I took it upon myself to fix that mistake. Thankfully, lists of all the comments in 2001 edition are available on the internet. The trickiest part was the “this book belongs to” page, which is missing from the newer edition. I ended up creating a faux-library card for that instead.

Here is what it looks like:

Ron plays hangman and loses.
Hermione writes on books!

Thanks to Bhavya for helping with the troll illustration.

I didn’t like the new additions, they sound less like a textbook and more like a transcript of what happened in the film. ↩

🍳 Vegetarian Bolognese

Prashanth Sadasivan — Sun, 29 Nov 2020 02:00:00 GMT

A simple mushroom based bolognese, with lots of depth.

Measuring Java JVM thread activity in Linux with task state sampling

Tanel Poder Blog — Sun, 29 Nov 2020 00:52:09 GMT

FYI: I will be speaking about similar topics at P99Conf in Oct 2023 (online, free event)

There are plenty of JVM profilers out there, but before attaching with a profiler, you can get a high-level overview by just sampling the Linux /proc/PID/comm and /proc/PID/stat fields to see which JVMs threads are most active and whether they’re burning CPU or are blocked by some I/O.

When creating threads in Java, you can name them using the following syntax: Thread t = new Thread("MyThread") or thread.setName("Worker-123"). This name will show up in the comm fields in Linux /proc. Some developers make good use of this observability feature.

The Problem Mindset

Pete's Points — Sat, 28 Nov 2020 20:32:03 GMT

Everywhere I look in the tech world, I see everything everyone does being framed as "solving problems". "What problem are you trying to solve?" is a question asked at the start of many project inception documents. It occupies an early slide in most presentations. Whole company missions are framed as solving a particular problem. Many engineers list problem solving as one of their core skills.

My motivation for this post is to simply state that I don't think this mindset should be taken as a given and I personally don't think this way about everything. It's not that clearly defining a problem isn't a useful approach to planning/strategy. I just think about the word "problem" in a more specific way meaning something is broken, troublesome, or has gone wrong somehow. I think the world needs people and organizations doing things in addition to solving problems. It's perfectly valid to be refining something that is not problematic. It's fine to be creating a new capability from scratch without having to think of the "I don't have X capability" as the definition of the problem.

OK let's go through some concrete examples.

Are ride sharing apps like Lyft and Uber really solving a problem? I think of them as providing a particular kind of transportation. It's an update to a centuries-old core business in the world of smart phones, GPS, gig economy, etc. There's a raft of interesting changes in their financial models, approach to vehicle inventory, driver relations, driver compensation, driver competition, etc, but fundamentally I don't think of what they are doing of solving the problem of "I would rather use an app than a telephone to summon a taxicab". For that matter I don't think "transportation" when thought of as a problem has a science-fact solution. We can make it better, but not really solve it as a problem. Thus I would think about it more like a need.

WTF problem is instagram solving? It's a thing that exists and has utility for some and is also problematic but I can't bend my brain to think of instagram as solving any particular problem. I guess if arch-capitalists think of "Advertising dollars are being spent with companies I do not own" as a problem, then Instagram solves for that I guess?

Does AirBnB really solve the "It's hard for me to book short-term tenants into my guest room" problem? It makes a lot more sense to me to think of this as creating a marketplace facilitated by the web and smartphone apps.

Here's a couple examples of companies/products that it seems logical to think about as solving a problem.

Insurance comes to mind. Something rare and costly just went wrong and I otherwise would be in financial disaster, but insurance prevents that disaster for me. Roadside assistance. A flat tire is a problem that sometimes happens and while roadside service doesn't "solve" it as in prevent it from ever being a thing that happens, it reduces the impact significantly.

An extension cord solves a problem. In some ways WiFi solves a similar problem for your network cable, but it seems weird to me to think about WiFi as a solution to the problem "my gigantic ethernet cable is annoying to manage while I carry my laptop around the house". It's just a new technology and I don't feel any need to frame it in relationship to a specific problem.

If any of this resonates with you, please join me in rolling our eyes in our next strategic planning meeting.

When you make digital archives in the wrong way...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 28 Nov 2020 20:00:00 GMT

Sooner or later they rot. But only in that case.

Quarkus and Testcontainers

Gunnar Morling — Sat, 28 Nov 2020 18:45:00 GMT

Table of Contents

Bonus: Schema Creation

The Testcontainers project is invaluable for spinning up containerized resources during your (JUnit) tests, e.g. databases or Kafka clusters.

For users of JUnit 5, the project provides the @Testcontainers extension, which controls the lifecycle of containers used by a test. When testing a Quarkus application though, this is at odds with Quarkus' own @QuarkusTest extension; it’s a recommended best practice to avoid fixed ports for any containers started by Testcontainers. Instead, you should rely on Docker to automatically allocate random free ports. This avoids conflicts between concurrently running tests, e.g. amongst multiple Postgres containers, started up by several parallel job runs in a CI environment, all trying to allocate Postgres' default port 5432. Obtaining the randomly assigned port and passing it into the Quarkus bootstrap process isn’t possible though when combining the two JUnit extensions.

Obsidian Fixed Unlinked Mentions for ZK IDs!

trv.rs — Sat, 28 Nov 2020 18:16:00 GMT

I have been working on the migration script to move my plaintext notes to a format that Obsidian and other applications will like.

I determined that it seemed more applications had adopted YAML syntax for their frontmatter¹ Adopting this format would be as app-compatible as possible. With my previous recommendation of using multimarkdown-style frontmatter, you were locked to MMD, but by using YAML I could use multimarkdown, github-markdown, common mark, or just plain markdown.

With the #hashtag style tags and YAML markdown, pretty much everything in Obsidian worked 100% with SMZ style zettels except the incredible useful "Unlinked Mentions."

Until yesterday. 🤷

Obsidian is moving very fast, and each new build update the team posts to Discord brings wonderful toys. I bought the Insider Access to try out the feature I was interested in… they rolled it out to the general public around four hours afterward. I am not sorry at all… they are making an amazing application.

Obsidian v0.9.16 brought the ability to define an aliases attribute in the frontmatter of a note. That means if you have a note named "Artificial Intelligence" with the file name Artificial Intelligence.md, you can set up aliases like:

---
title: "Artificial Intelligence"
aliases: ['AI', 'Expert Systems']
tags: ['#futurism', '#technology']
---

Artificial Intelligence broadly covers a number of technologies ranging from […]

Now… any time in your notes there is the string "AI", "Artificial Intelligence", or "Expert Systems," Obsidian will show it as an unlinked mention, ready to be linked up.

This works for filenames containing a Zettelkasten ID! If your filename is 202011280902-obsidian.md you can define your alias simply as:

---
title: "Obsidian"
aliases: ['Obsidian']
tags: ['#knowledge_management']
---

I'm really starting to like Obsidian[…]

Suddenly, Unlinked Mentions work perfectly! 🥳 This closes a feature-request I that has been floating around for a long while, and really excites me to get the last 10 years of notes into a place where Obsidian can find it. Suddenly I can be working on a note on "Leadership", and find journal entries and project logs that cross reference "leadership" and draw new conclusions.

I included this idea of aliases in my migration script, and I am having a blast re-discovering all my notes (especially book notes!) and seeing all the new ways that they relate.

The only hitch here is that when I create a new note, it would be nice for Obsidian to make it really easy to have a note with the ZK ID and automatically add the title and alias. I have seen a templating system in Obsidian and I'll dig into it, but I don't think it has the simple syncing smarts that I'm looking for… so I may add Obsidian scripting to my todo list right after I finish fixing my Drafts workflows.

Honestly thinking out loud… I'd probably be just as happy using Drafts to write all my zettel and use actions to send them to Obsidian, treating Obsidian as a research tool and Drafts as a capture tool… I think I will definitely prioritize my Drafts actions soon. 👍

I'll eventually fix the table in the original article and link it here. ↩

Scotland: The Story of a Nation

Macleod Sawyer | Web Portal Feed — Sat, 28 Nov 2020 14:06:36 GMT

Hailed as a vast, superb history that relates Scotland’s past over a dozen millennia”, Magnusson draws on a great deal of modern scholarship to redefine a nation’s history. He charts the long struggle toward nationhood, explores the roots of the original Scots, and examines the extent to which Scotland was shaped by the Romans, the Picts, the Vikings, and the English. Encompassing everything from the first Mesolithic settlers in 7000 B.C. to the present movements for independence, Scotland: The Story of a Nation is history on an epic level, essential reading for anyone interested in the rich past of this captivating land.

Why Benford's Law is Obvious

Micro-optimist — Sat, 28 Nov 2020 02:00:00 GMT

Benford’s Law states that in many naturally occurring collections of numbers, the leading digit is likely to be small. I love math factoids like this, and I remember how deeply unintuitive this seemed when I first learned it. Now, I find myself looking for a way to explain how intuitive it really is.

TL;DR

Ok, Benford’s law isn’t obvious. But what would the world look like if it were?

Benford’s law follows unavoidably from a certain type of probability distribution (uniform in the logarithmic domain), which happens to be common in real-world data. I can draw some charts that – to me – make this perfectly clear. Somehow, that doesn’t make it any less surprising to learn. Go figure.

I contend that the surprise comes from a lack of familiarity with the log-uniform distribution. What if we, as a society, internalized that distribution instead of the “usual” linear-uniform one? In this post, I explain what the counterpart of Benford’s law would be in that case. It turns out, it’s almost too obvious to mention. Which is the point.

Benford’s Law

When you first hear of it, the thinking might go something like this: if you choose a number randomly, shouldn’t the first digit be uniformly distributed? Even if the original distribution isn’t uniform, as long as it covers a wide enough range, why would the first digit have any bias?

Explanations abound, but I find that they often miss the crucial point (like this otherwise fantastic video for example). Simply put, it is a direct consequence of the log-uniform distribution, which governs these “naturally occurring collections of numbers”, e.g. measurements, population counts, etc. If you can accept that these natural collections tend to be log-normal distributed, Benford’s Law is “obvious”, that is, an unavoidable consequence. Some simply explain this empirically, giving examples. But I think there is more to it.

Why do some collections of numbers have this tendency? I like to think of it like this:

The logarithmic scale is at least as natural a way to look at the world as the linear scale is.
In a given scale (linear, log, or otherwise), a uniform distribution is the “simplest” option. It’s sort of a default distribution, perhaps in part due to being the maximum entropy distribution for a bounded domain.

I’m aware that this is an academic viewpoint - I’m using “obvious” facetiously here. While the log-uniform distribution may be more “natural” in some sense, the linear-uniform distribution is certainly not unusual. Roll a die, draw a card, pick a ball from an urn - these are all basic processes, easily understood, and obviously uniform. While they may be more common in the built environment rather than in nature, they are still regular occurrences to a human child living in society, helping to reify the linear scale in her mind.

The linked video also presents a counterexample to Benford’s law, plus a counterpart law with its corresponding counterexample. Paraphrased:

Benford’s Law: most-significant digits are expected to be biased toward lower values with a logarithmic trend, if the range of values spans multiple orders of magnitude.

Benford Counterexample (Biden votes): If the data spans just a single order of magnitude (specifically [10^n, 10^(n+1)) so the number of digits is constant, then the first-digit distribution just reflects the data distribution itself. Benford’s law is not violated, it is rather not applicable.

Last-digit test: least-significant digits are expected to be distributed uniformly, if the range of values is wide enough, relative to the precision of the values.

Last-digit counterexample (incumbent votes): If the data spans just a single, low, order of magnitude, e.g. [10, 100), then the last two digits are just the digits, so the last-digits distribution just reflects the data distribution itself. The last-digit test is not violated, it is rather not applicable.

There’s a pleasing sort of duality here, and illustrating each of these with the two political parties’ vote counts is pretty cute.

Another counterpart

Although these are interesting lines of thought, they prompted another question, the real motivation for writing this: is there another counterpart to Benford’s Law, which might be equally surprising to a culture that thinks in the logarithmic scale by default? Of course, human sensory perception is logarithmic, as is our natural perception of integers. But early education beats that out of us, and our culture generally encourages us to consider the linear scale as the default way of thinking about numbers - the Way Numbers Really Are.

If, instead, we were trained from a young age to think about the logarithmic scale as the default, then what sort of distribution would lead to a similarly surprising result, and what would that result be?

After ruminating on this topic over the years, and a concerted hour or two of trying to answer this question, I found a candidate. I’ll restate Benford’s again for comparison:

Benford’s Law: When accustomed to the linear scale, a log-uniform distribution results in a suprising bias in first-digits, logarithmically decreasing from 1 to 9.

And the linear counterpart:

Drofben’s Law: When accustomed to the logarithmic scale, a linear-uniform distribution results in a surprising bias in number-of-digits, exponentially increasing from 1 (to N).

These charts illustrate both laws, as described above – but I’ll elaborate.

Graphical explanation, Benford’s Law

Figure 1: Log-uniform random data

This is a set of three views of the same random data set, following a log-uniform distribution.

1a shows that log-uniform data follows a logarithmically-decreasing trend, when viewed in the linear domain.
1b shows that log-uniform data follows a constant trend, when viewed in the (arguably more appropriate) logarithmic domain.
1c shows Benford’s law directly, with bar heights corresponding to first-digit bias. Note that this shape closely resembles the shape of the distribution in 1a – it isn’t identical, because 1c kind of “layers” the histograms from each level of magnitude in 1a, into a new form that examines only first digits instead of all values.
1d (aside from being kind of busy because I’m a bad graphic designer) shows where these “layers” come from. The black trace represents the determinstic values of the function first_digit(x) for all values of x in [1, 10000]. The green regions cover all numbers that start with 1, and they’re widest – that cumulative width corresponds to the height of the bar for digit 1 in figure 1c. Red regions correspond to numbers that start with 2, with a more moderate cumulative width. Purple corresponds to 5, with a much smaller width, and similarly for all other digits.

This figure explains why Benford’s Law is “obvious”. It may be more clear to explain the logical progression of thought:

This data follows a natural distribution – log-uniform – illustrated by the constant trend in the logarithmic domain (1b).
This necessitates a logarithmic-decreasing trend in the linear domain (1a).
This necessitates a logarithmic-decreasing trend in the first digits (1c, 1d).

Graphical explanation, Drofben’s Law

And here are the corresponding charts for Drofben’s law:

Figure 2: Linear-uniform random data

This is a set of three views of another random data set, following a linear-uniform distribution.

2a shows that linear-uniform data follows a constant trend, when viewed in the linear domain.
2b shows that linear-uniform data follows an exponentially-increasing trend, when viewed in the logarithmic domain.
2c shows Drofben’s law directly, with bar heights corresponding to number-of-digits bias. Note that this shape closely resembles the shape of the distribution in 2b.
2d shows the analogous “layers” of Drofben’s law – these layers just represent magnitude. The gray region, representing numbers with five digits, takes up 90% of the total width. Pink, for four digits, takes 9%, and so on.

In the second case, these layers just reflect the counts of integers in ranges: |[1, 10)| = 9, |[10, 100)| = 90, |[100, 1000)| = 900, etc – this seems almost too obvious to mention. But, that’s what I was looking for – a result that’s surprising to an alien, logarithmic culture should be obvious to our linear minds.

(code for charts)

The digits are not the value

The treachery of digits, with apologies to Magritte… and IEEE

Warning

Philosophical rambling ahead

All of this might seem silly. Consider a linear-uniform distribution over [1, 100). Of course the range [10, 99] is 10x as likely as the range [1, 9] – that’s just the relative size of the intervals. But it’s the arithmetic size, the linear size. If you were conditioned to think first of the geometric size, the logarithmic size, you would instinctively see these two intervals as having (nearly) the same size. You wouldn’t expect the linear-uniform distribution, and this behavior would indeed be surprising. Perhaps it’s hard to imagine this, because the digits themselves make it obvious: for Benford’s Law, it is not obvious that 1234 is any more likely to occur than 5678. In contrast, you can’t help but notice that 5 has only one digit, while 5678 has four digits. However, considering non-integer values allows us to make a more illustrative comparison between 5.678 and 5678. Try to imagine perceiving numbers through their magnitude instead of their digits. Perhaps growing up using a slide rule is the best way to internalize the logarithmic scale. I missed that boat by a few years.

My point is, the digits are only a representation of the number’s value. Despite being a perfectly complete representation, the digit string is not an innate property of the value, but rather an artifact of the representation system. It’s like looking for information in the pixel boundaries of a photograph: anything you find might be explained by the sampling grid, and disappear under a different sampling. Any properties of the digit string of a number might be explained by the choice of base, and disappear under a different base.

Of course, the number of digits is a crude approximation of the magnitude (regardless of the choice of base), whereas the first digit is more reasonably described as a “random” artifact of the representation. I’ll concede that the number of digits is not just an artifact, and therefore Drofben’s Law is more obvious – if you think about numbers primarily in terms of their non-fractional digits.

Tangent

I’m a former mathlete, I love recreational math problems/puzzles. These sometimes concern properties of the base-10 representation of numbers (Project Euler often uses the digit sum, etc., similarly, the Putnam exam usually includes a problem where the current year is used as an upper limit or the like), and I’ve always thought of these as sort of “impure” problems which aren’t as deeply interesting. On the other hand, there is some deep math involved in Goodstein sequences, based on fiddling around with hereditary base-N notation. Although, it comes specifically from not constraining N.

Base effects

When working with numbers, we have to pick some representation system; base 10 is fine, but it might be interesting to think about how these artifacts change when using different systems. While Benford’s Law occurs regardless of the base, there is some interesting behavior at the extremes:

base 2: Benford’s Law doesn’t mean as much in this case – all first bits are 1! The generalization still applies to first-N-bits, but this is still an interesting comparison. In contrast, number-of-bits is now a closer approximation to the magnitude of the number.
base infinity: each numeral is represented by a different symbol. Now, number-of-digits is always 1! Most of the questions become meaningless, because we’ve abandoned the concept of place-value digits, without abandoning the values. The concept of “first digit bias” isn’t even tautological as in the base-2 case. The distribution of digit occurrences still contains some information, however.

Aside from these extremes, there is the zero-padded representation of base-N. Now, regardless of the the value of N, first-digit is uniform, and number-of-digits is constant. As lazy humans, we don’t bother writing out all these leading zeros, but that’s how computers work! I suppose a child raised by computers would think all of the zero-pad-counterparts of these laws are perfectly obvious and devoid of value.

Un an avec SailfishOS

Accueil on dbeley — Sat, 28 Nov 2020 01:00:00 GMT

Introduction

SailfishOS est un système d’exploitation mobile basé sur Linux. C’est aujourd’hui l’une des rares alternatives à Android et iOS sur le marché des smartphones.

Cet article est un retour d’expérience d’un Sony Xperia X sous SailfishOS que j’ai utilisé comme smartphone principal pendant l’année 2019.

Le Sony Xperia X n’a pas été choisi par hasard, puisqu’il s’agit de l’un des smartphones officiellement supportés par SailfishOS. Il est disponible aux alentours de 70€ sur internet en occasion (j’ai acheté le mien sur aliexpress pour 90€ début 2019).

Don't boycott Amazon. Free it, instead.

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 27 Nov 2020 20:00:00 GMT

Something to think about, on this and all future Black Fridays.

A weirdness with tinydns

infrequent oscillations — Fri, 27 Nov 2020 18:36:33 GMT

In which tinydns needs better logging.

My Favorite New Podcasts in 2020

swyx's site RSS Feed — Fri, 27 Nov 2020 18:18:14 GMT

An update to my favorite podcasts list with the ones I have found this year

John Oliver Thirst Bot

infrequent oscillations — Fri, 27 Nov 2020 11:31:08 GMT

In which John Oliver is Thirsty.

PostgreSQL autogenerated interval

Random Notes by agilob — Fri, 27 Nov 2020 02:00:00 GMT

Using postgres specific SQL syntax we can create autogenerated column which subtracts two dates and stores them as interval. The age function is also null-safe, so if time_ended or time_started it will not crash. ALTER TABLE session ADD COLUMN duration interval GENERATED ALWAYS AS (age(time_ended, time_started)) STORED;

RMS Does Not See the Future of Emacs

Lambda Land — Fri, 27 Nov 2020 02:00:00 GMT

I am an avid Emacs user. I’m using it right now to compose this post. I use it every single day for everything from work to school to personal notes. Most of my activity on GitHub comes from me tweaking little things in my configuration files. I now have an editor that perfectly fits my hands. Emacs is a big part of my life.

I’m afraid it’s dying.

Racially biased pulsimeters

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 26 Nov 2020 20:00:00 GMT

More of the depressing same.

Emulating linux/AMD64 userland: interpreting an ELF binary

Notes on software development — Thu, 26 Nov 2020 02:00:00 GMT

In this post we'll stumble toward a working emulator for a barebones C program compiled for linux/AMD64. The approach will be slightly more so based on observation than by following a spec; a great way to quickly become familiar with a topic, and a bad way to guarantee correctness.

The goal:

$ cat tests/simple.c
int main() {
  return 4;
}
$ gcc tests/simple.c
$ go build -o main
$ ./main a.out && echo $?
4

This may look ridiculously simple but when you don't know how to deal with a binary or how instructions are encoded, it will take a few hours to write an emulator that can generally handle this program!

Code for this project is available on Github.

Background

AMD64, x86_64 or x64 are different names for AMD's widely adopted 64-bit extension to Intel's x86 instruction set (i.e. the encoding and semantics of x86 binaries). AMD64 is a superset of x86 (introducing 64-bit registers and operations) and thus backwards compatible with x86 programs.

A year and a half ago I first got into emulation with an AMD64 emulator in JavaScript. The JavaScript emulator interpreted the textual representation of AMD64 programs (e.g. MOV RBP, RSP, Intel's assembly syntax). A C program had to be compiled with -S to produce an assembly file that the JavaScript emulator could read (i.e. gcc -S tests/simple.c) This was a great way to get started with emulation by ignoring the complexity of encoded instructions and executable formats.

If we dig into the binary file produced by gcc on Linux we learn that it is an ELF file.

$ gcc test/simple.c
$ file a.out
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=d0b5c742b9fbcbcca4dfa9438a8437a8478a51bb, for GNU/Linux 3.2.0, not stripped

ELF is responsible for surrounding the actual binary-encoded program instructions with metadata on exported and imported C identifiers and program entrypoint. But for simple programs like this initial emulator, we can ignore export/imports. We'll only use the ELF metadata to find out where the instructions for our main function start.

Where is main?

If we use an ELF reader+disassembler on the binary generated by gcc and search for main we can find its address.

$ objdump -d a.out | grep -A10 '<main>'
0000000000401106 <main>:
  401106:       55                      push   %rbp
  401107:       48 89 e5                mov    %rsp,%rbp
  40110a:       b8 fe 00 00 00          mov    $0xfe,%eax
  40110f:       5d                      pop    %rbp
  401110:       c3                      retq
  401111:       66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
  401118:       00 00 00
  40111b:       0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)

0000000000401120 <__libc_csu_init>:

This means that the function, main, starts at address 0x401106 in memory. Furthermore, this implies that the binary must be loaded into CPU memory such that the CPU can jump here to execute our program.

In truth, main is not this program's entrypoint. If we run objdump -x a.out we can see that the ELF entrypoint is 0x401020.

$ objdump -x a.out
a.out:     file format elf64-x86-64
a.out
architecture: i386:x86-64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000401020

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000400040 paddr 0x0000000000400040 align 2**3
             filesz 0x00000000000002d8 memsz 0x00000000000002d8 flags r--

This is because the actual entrypoint gcc sets up is a function called _start. The libc prelude beginning with _start is responsible for initializing the libc runtime, calling our main function and executing the exit syscall with the return value of main.

objdump -d a.out | grep -A10 '<_start>'
0000000000401020 <_start>:
  401020:       f3 0f 1e fa             endbr64
  401024:       31 ed                   xor    %ebp,%ebp
  401026:       49 89 d1                mov    %rdx,%r9
  401029:       5e                      pop    %rsi
  40102a:       48 89 e2                mov    %rsp,%rdx
  40102d:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp
  401031:       50                      push   %rax
  401032:       54                      push   %rsp
  401033:       49 c7 c0 90 11 40 00    mov    $0x401190,%r8
  40103a:       48 c7 c1 20 11 40 00    mov    $0x401120,%rcx

But because all this libc initialization is relatively complicated we're just going to skip the actual ELF entrypoint for now. Our emulator will locate main, load the binary into memory, jump to the start of main, and set the exit code of the emulator to the result of main.

As you can see, this ELF binary has its own hard-coded view of where it will be in memory. What if our CPU were to run multiple process at once? We might give each process its own virtual memory space and map back to a real memory space so each process (and by extension, compilers) doesn't have to think about how they fit into memory relative to other processes.

The last question to figure out is where to load the ELF binary into emulator memory so that addresses in memory are where the program expects.

As it turns out, there is a piece of metadata called section headers that contain an address and a offset from the start of the ELF file. By subtracting this we can get the location the file expects to be in memory.

$ objdump -x a.out
a.out:     file format elf64-x86-64
a.out
architecture: i386:x86-64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000401020

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000400040 paddr 0x0000000000400040 align 2**3
             filesz 0x00000000000002d8 memsz 0x00000000000002d8 flags r--

That is: 0x400040 (vaddr) - 0x40 (off) = 0x400000. Judging from a Google search this seems to be a pretty common address where ELF binaries are loaded into memory.

ELF and Go

Binary file formats tend to be a pain to work with because, to enable greater compression, everything ends up being a pointer to something else. So you end up jumping all around the file just to stitch information back together.

So the one third-party-ish library we'll use is Go's builtin debug/elf package. With this library we can load an ELF binary and iterate over symbols and sections to discover the location of main and the start address for the binary in memory.

Editing in main.go:

package main

import (
    "bytes"
    "debug/elf"
    "fmt"
    "io/ioutil"
    "log"
    "os"
)

type process struct {
    startAddress uint64
    entryPoint   uint64
    bin          []byte
}

func readELF(filename, entrySymbol string) (*process, error) {
    bin, err := ioutil.ReadFile(filename)
    if err != nil {
        return nil, err
    }

    elffile, err := elf.NewFile(bytes.NewReader(bin))
    if err != nil {
        return nil, err
    }

    symbols, err := elffile.Symbols()
    if err != nil {
        return nil, err
    }

    var entryPoint uint64
    for _, sym := range symbols {
        if sym.Name == entrySymbol && elf.STT_FUNC == elf.ST_TYPE(sym.Info) && elf.STB_GLOBAL == elf.ST_BIND(sym.Info) {
            entryPoint = sym.Value
        }
    }

    if entryPoint == 0 {
        return nil, fmt.Errorf("Could not find entrypoint symbol: %s", entrySymbol)
    }

    var startAddress uint64
    for _, sec := range elffile.Sections {
        if sec.Type != elf.SHT_NULL {
            startAddress = sec.Addr - sec.Offset
            break
        }
    }

    if startAddress == 0 {
        return nil, fmt.Errorf("Could not determine start address")
    }

    return &process{
        startAddress: startAddress,
        entryPoint:   entryPoint,
        bin:          bin,
    }, nil
}

func main() {
    if len(os.Args) < 2 {
        log.Fatal("Binary not provided")
    }

    proc, err := readELF(os.Args[1], "main")
    if err != nil {
        panic(err)
    }

    fmt.Printf("Start: 0x%x\nEntry: 0x%x\n", proc.startAddress, proc.entryPoint)
}

We can test on a basic compiled C program:

$ cat tests/simple.c
int main() {
  return 4;
}
$ gcc tests/simple.c
$ go build -o main
$ ./main a.out
Start: 0x400000
Entry: 0x401106

And verify against objdump:

$ objdump -d a.out | grep -A10 '<main>'
0000000000401106 <main>:
  401106:       55                      push   %rbp'>'

And that's it for dealing with ELF. Now we can sketch out a virtual CPU and how we deal with interpreting instructions starting at this address.

The CPU

AMD64 counts on being able to store values in registers and memory, sometimes through direct addressing and sometimes indirectly using stack operations (push and pop). And userland processes count on being loaded into CPU memory so the CPU can jump to the process entrypoint and process.

type cpu struct {
    proc    *process
    mem     []byte
    regfile *registerFile
    tick    chan bool
}

func newCPU(memory uint64) cpu {
    return cpu{
        mem:     make([]byte, memory),
        regfile: &registerFile{},
        tick:    make(chan bool, 1),
    }
}

The tick channel is so that later on we can wrap the emulator in a terminal debugger. But by default we'll just set up a goroutine to tick forever.

func repl(c *cpu) {
  // TODO
}

func main() {
    if len(os.Args) < 2 {
        log.Fatal("Binary not provided")
    }

    proc, err := readELF(os.Args[1], "main")
    if err != nil {
        panic(err)
    }

    debug := false
    for _, arg := range os.Args[1:] {
        switch arg {
        case "--debug":
            fallthrough
        case "-d":
            debug = true
        }
    }

    // 10 MB
    cpu := newCPU(0x400000 * 10)

    go cpu.run(proc)
    if debug {
        repl(&cpu)
    } else {
        for {
            cpu.tick <- true
        }
    }
}

Registers

To emulate a simple program like our tests/simple.c, we'll only need to support a few common registers. The order is important so that we can use the Go identifiers when we want to refer to the encoded integer value of the register.

type register int

const (
    // These are in order of encoding value (i.e. rbp is 5)
    rax register = iota
    rcx
    rdx
    rbx
    rsp
    rbp
    rsi
    rdi
    r8
    r9
    r10
    r11
    r12
    r13
    r14
    r15
    rip
    rflags
)

var registerMap = map[register]string{
    rax:    "rax",
    rcx:    "rcx",
    rdx:    "rdx",
    rbx:    "rbx",
    rsp:    "rsp",
    rbp:    "rbp",
    rsi:    "rsi",
    rdi:    "rdi",
    r8:     "r8",
    r9:     "r9",
    r10:    "r10",
    r11:    "r11",
    r12:    "r12",
    r13:    "r13",
    r14:    "r14",
    r15:    "r15",
    rip:    "rip",
    rflags: "rflags",
}

type registerFile [18]uint64

func (regfile *registerFile) get(r register) uint64 {
    return regfile[r]
}

func (regfile *registerFile) set(r register, v uint64) {
    regfile[r] = v
}

Of immediate importance will be rip, rsp, and rax registers. rip is used to track the current instruction to process. It will generally be incremented except for when dealing with function calls and returns. rsp is used as a pointer to the top of a stack in memory. It is incremented and decremented as values are pushed and popped on this stack. Finally, rax is used to pass function return values.

Loading a program

Running a program is a matter of loading the program into memory, setting the stack pointer to the last address of memory (in x86 the stack grows down), pointing rip at the entrypoint, and looping until the entrypoint function returns.

func writeBytes(to []byte, start uint64, bytes int, val uint64) {
    for i := 0; i < bytes; i++ {
        to[start+uint64(i)] = byte(val >> (8 * i) & 0xFF)
    }
}

func (c *cpu) loop(entryReturnAddress uint64) {
    for {
        <-c.tick

        ip := c.regfile.get(rip)
        if ip == entryReturnAddress {
            break
        }

        inb1 := c.mem[ip]

        // TODO: deal with instructions

        // move to next instruction
        c.regfile.set(rip, ip+1)
    }
}

func (c *cpu) run(proc *process) {
    copy(c.mem[proc.startAddress:proc.startAddress+uint64(len(proc.bin))], proc.bin)
    c.regfile.set(rip, proc.entryPoint)
    initialStackPointer := uint64(len(c.mem)-8)
    writeBytes(c.mem, initialStackPointer, 8, initialStackPointer)
    c.regfile.set(rsp, initialStackPointer)
    c.loop(initialStackPointer)
    os.Exit(int(c.regfile.get(rax)))
}

We write the initial stack pointer address into the stack so that when the program final returns, it will return to this address at which pointer we can exit the program.

And now we're ready to start interpreting instructions.

Instruction decoding

Using objdump we get a sense for what the program decodes to.

$ objdump -d a.out | grep -A10 '<main>'
0000000000401106 <main>:
  401106:       55                      push   %rbp
  401107:       48 89 e5                mov    %rsp,%rbp
  40110a:       b8 fe 00 00 00          mov    $0xfe,%eax
  40110f:       5d                      pop    %rbp
  401110:       c3                      retq
  401111:       66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
  401118:       00 00 00
  40111b:       0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)

0000000000401120 <__libc_csu_init>:

We see that 0x55 means push %rbp. And we also see that instructions aren't a fixed number of bytes. Some are one byte, some are seven. Some (not shown) are even longer.

Thankfully instructions follow some fairly simple patterns. There are a set of prefix instructions and a set of real instructions. So far we should be able to tell on the first byte whether the instruction is a prefix instruction and, if not, how many bytes the instruction will take up on the whole.

push

To support a new instruction, we'll look up 0x55 in an opcode table like this. Clicking on 55 in the opcode index we see that this is indeed a push instruction. 50+r means that we have to subtract 0x50 from the opcode to determine the register we should push.

The register will be 0x55 - 0x50 = 5 which if we look up in a register table is rbp. Since we set up our register enum in code in this order, we'll be able to just use the constant rbp in Go code.

Finally, since the next instruction numerically is 0x58 we know that this instruction is identified by being between 0x50 and 0x57 inclusive. This is all the info we need to handle this instruction.

// helper for dumping byte arrays as hex
func hbdebug(msg string, bs []byte) {
    str := "%s:"
    args := []interface{}{msg}
    for _, b := range bs {
        str = str + " %x"
        args = append(args, b)
    }
    fmt.Printf(str+"\n", args...)
}

func (c *cpu) loop(entryReturnAddress uint64) {
    for {
        <-c.tick

        ip := c.regfile.get(rip)
        if ip == entryReturnAddress {
            break
        }

        inb1 := c.mem[ip]

        if inb1 >= 0x50 && inb1 < 0x58 { // push
            regvalue := c.regfile.get(register(inb1 - 0x50))
            sp := c.regfile.get(rsp)
            writeBytes(c.mem, sp-8, 8, regvalue)
            c.regfile.set(rsp, uint64(sp-8))
        } else {
            hbdebug("prog", c.mem[ip:ip+10])
            panic("Unknown instruction")
        }

        c.regfile.set(rip, ip+1)
    }
}

If we try this out now we should expect it to panic on the second byte, 0x48.

$ go build -o main
$ ./main a.out
prog: 48 89 e5 b8 4 0 0 0 5d c3
panic: Unknown instruction

goroutine 19 [running]:
main.(*cpu).loop(0xc000086c30, 0x2800000)
        /home/phil/tmp/goamd/main.go:168 +0x16d
main.(*cpu).run(0xc000086c30, 0xc000086c00)
        /home/phil/tmp/goamd/main.go:180 +0xac
created by main.main
        /home/phil/tmp/goamd/main.go:211 +0x286

Looking good.

mov

Taking a look at the next two instructions with objdump we see mov encoded two different ways.

$ objdump -d a.out | grep -A4 '<main>'
0000000000401106 <main>:
  401106:       55                      push   %rbp
  401107:       48 89 e5                mov    %rsp,%rbp
  40110a:       b8 fe 00 00 00          mov    $0xfe,%eax

Looking up 0x48 we see that this is a prefix instruction that turns on 64-bit mode for the instruction. Some instructions like pop and push don't need this prefix to be in 64-bit mode. In any case, this just means we'll have to have a size flag that switches from 32-bit to 64-bit mode on seeing this instruction. This flag will be reset each time we start reading an instruction.

To deal with prefixes in general we'll loop through bytes when processing an instruction until we no longer see a prefix bytes. As we see prefix bytes we'll handle them accordingly.

var prefixBytes = []byte{0x48}

func (c *cpu) loop(entryReturnAddress uint64) {
    for {
        <-c.tick

        ip := c.regfile.get(rip)
        if ip == entryReturnAddress {
            break
        }

        inb1 := c.mem[ip]

        widthPrefix := 32
        for {
            isPrefixByte := false
            for _, prefixByte := range prefixBytes {
                if prefixByte == inb1 {
                    isPrefixByte = true
                    break
                }
            }

            if !isPrefixByte {
                break
            }

            // 64 bit prefix signifier
            if inb1 == 0x48 {
                widthPrefix = 64
            } else {
                hbdebug("prog", c.mem[ip:ip+10])
                panic("Unknown prefix instruction")
            }

            ip++
            inb1 = c.mem[ip]
        }

        if inb1 >= 0x50 && inb1 < 0x58 { // push

...

Moving past this prefix we get to 0x89. This instruction is for copying one register into another. The register operands are encoded in the second byte, 0xe5, called the ModR/M byte. Pulling out the two registers is just a matter of shifting and bitmasking the right 3 bits for each.

With this knowledge we can expand the instruction handling code.

        if inb1 >= 0x50 && inb1 < 0x58 { // push
            regvalue := c.regfile.get(register(inb1 - 0x50))
            sp := c.regfile.get(rsp)
            writeBytes(c.mem, sp-8, 8, regvalue)
            c.regfile.set(rsp, uint64(sp-8))
        }  else if inb1 == 0x89 { // mov r/m16/32/64, r/m16/32/64
            ip++
            inb2 := c.mem[ip]
            rhs := register((inb2 & 0b00111000) >> 3)
            lhs := register(inb2 & 0b111)
            c.regfile.set(lhs, c.regfile.get(rhs))
        } else {
            hbdebug("prog", c.mem[ip:ip+10])
            panic("Unknown instruction")
        }

Try emulating a.out again now. It will panic on the next unknown instruction, 0xb8. From objdump disassembly we see this is another mov instruction.

Hurray! There are apparently multiple ways the same instruction can be encoded. Looking it up in the opcode table, we see 0xB8 is for when the value to be copied is a literal number. The operand will be 32-bits, or four bytes, presumably because it doesn't have the 0x48 prefix.

// helper for converting up to 8 bytes into a single integer
func readBytes(from []byte, start uint64, bytes int) uint64 {
    val := uint64(0)
    for i := 0; i < bytes; i++ {
        val |= uint64(from[start+uint64(i)]) << (8 * i)
    }

    return val
}

func (c *cpu) loop(entryReturnAddress uint64) {

    ...

    for {

        ...

        } else if inb1 >= 0xB8 && inb1 < 0xC0 { // mov r16/32/64, imm16/32/64
            lreg := register(inb1 - 0xB8)
            val := readBytes(c.mem, ip+uint64(1), widthPrefix/8)
            ip += uint64(widthPrefix / 8)
            c.regfile.set(lreg, val)
        }

        ...

Two more instructions to go: pop and ret.

A terminal debugger

Taking a break for a moment, our system is already too complex to understand. It would be helpful to have a REPL so we can step through instructions and print register and memory values.

func (c *cpu) resolveDebuggerValue(dval string) (uint64, error) {
    for reg, val := range registerMap {
        if val == dval {
            return c.regfile.get(reg), nil
        }
    }

    if len(dval) > 2 && (dval[:2] == "0x" || dval[:2] == "0X") {
        return strconv.ParseUint(dval[2:], 16, 64)
    }

    return strconv.ParseUint(dval, 10, 64)
}

func repl(c *cpu) {
    fmt.Println("go-amd64-emulator REPL")
    help := `commands:
    s/step:             continue to next instruction
    r/registers [$reg]:     print all register values or just $reg
    d/decimal:          toggle hex/decimal printing
    m/memory $from $count:      print memory values starting at $from until $from+$count
    h/help:             print this`
    fmt.Println(help)
    scanner := bufio.NewScanner(os.Stdin)

    intFormat := "%d"
    for {
        fmt.Printf("> ")
        if !scanner.Scan() {
            break
        }
        input := scanner.Text()
        parts := strings.Split(input, " ")

        switch parts[0] {
        case "h":
            fallthrough
        case "help":
            fmt.Println(help)

        case "m":
            fallthrough
        case "memory":
            msg := "Invalid arguments: m/memory $from $to; use hex (0x10), decimal (10), or register name (rsp)"
            if len(parts) != 3 {
                fmt.Println(msg)
                continue
            }

            from, err := c.resolveDebuggerValue(parts[1])
            if err != nil {
                fmt.Println(msg)
                continue
            }

            to, err := c.resolveDebuggerValue(parts[2])
            if err != nil {
                fmt.Println(msg)
                continue
            }

            hbdebug(fmt.Sprintf("memory["+intFormat+":"+intFormat+"]", from, from+to), c.mem[from:from+to])

        case "d":
            fallthrough
        case "decimal":
            if intFormat == "%d" {
                intFormat = "0x%x"
                fmt.Println("Numbers displayed as hex")
            } else {
                intFormat = "%d"
                fmt.Println("Numbers displayed as decimal")
            }

        case "r":
            fallthrough
        case "registers":
            filter := ""
            if len(parts) > 1 {
                filter = parts[1]
            }

            for i := 0; i < len(registerMap); i++ {
                reg := register(i)
                name := registerMap[reg]
                if filter != "" && filter != name {
                    continue
                }

                fmt.Printf("%s:\t"+intFormat+"\n", name, c.regfile.get(reg))
            }

        case "s":
            fallthrough
        case "step":
            c.tick <- true
        }
    }
}

Let's try it out:

$ go build -o main
$ ./main a.out --debug
go-amd64-emulator REPL
commands:
        s/step:                         continue to next instruction
        r/registers [$reg]:             print all register values or just $reg
        d/decimal:                      toggle hex/decimal printing
        m/memory $from $count:          print memory values starting at $from until $from+$count
        h/help:                         print this
> r
rax:    0
rcx:    0
rdx:    0
rbx:    0
rsp:    41943040
rbp:    0
rsi:    0
rdi:    0
r8:     0
r9:     0
r10:    0
r11:    0
r12:    0
r13:    0
r14:    0
r15:    0
rip:    4198662
rflags: 0
> m rip 10
memory[4198662:4198672]: 55 48 89 e5 b8 4 0 0 0 5d
> s
> m rip 10
memory[4198663:4198673]: 48 89 e5 b8 4 0 0 0 5d c3
> r
rax:    0
rcx:    0
rdx:    0
rbx:    0
rsp:    41943032
rbp:    0
rsi:    0
rdi:    0
r8:     0
r9:     0
r10:    0
r11:    0
r12:    0
r13:    0
r14:    0
r15:    0
rip:    4198663
rflags: 0
> ^D

Now we can inspect the system interactively.

pop

Reemersing in the state of things, we now panic on 0x5D.

./main a.out
prog: 5d c3 66 2e f 1f 84 0 0 0
panic: Unknown instruction

goroutine 5 [running]:
main.(*cpu).loop(0xc000098ae0, 0x2800000)
        /home/phil/tmp/goamd/main.go:219 +0x2c5
main.(*cpu).run(0xc000098ae0, 0xc000098ab0)
        /home/phil/tmp/goamd/main.go:231 +0xac
created by main.main
        /home/phil/tmp/goamd/main.go:358 +0x286

Looking this up we see this is part of 58+r, pop. Similar to push we subtract 0x58 from the byte to get the register to pop onto. The stack operation is the reverse of push.

func (c *cpu) loop(entryReturnAddress uint64) {

    ...

    for {

        ...

        } else if inb1 >= 0x58 && inb1 < 0x60 { // pop
            lhs := register(inb1 - 0x58)
            sp := c.regfile.get(rsp)
            c.regfile.set(lhs, readBytes(c.mem, sp, 8))
            c.regfile.set(rsp, uint64(sp+8))
        }
        ...

Build and run for the final panic:

$ go build -o main
$ ./main a.out
prog: c3 66 2e f 1f 84 0 0 0 0
panic: Unknown instruction

goroutine 8 [running]:
main.(*cpu).loop(0xc000060c30, 0x2800000)
        /home/phil/tmp/goamd/main.go:224 +0x345
main.(*cpu).run(0xc000060c30, 0xc000060c00)
        /home/phil/tmp/goamd/main.go:236 +0xac
created by main.main
        /home/phil/tmp/goamd/main.go:363 +0x286

ret

Looking up 0xC3 we see that it is indeed ret. This function's responsibilty is to pop the stack onto rip, jumping back to caller.

        } else if inb1 == 0xC3 { // ret
            sp := c.regfile.get(rsp)
            retAddress := readBytes(c.mem, sp, 8)
            c.regfile.set(rsp, uint64(sp+8))
            c.regfile.set(rip, retAddress)
            continue
        }

Build and run:

$ go build -o main
$ ./main a.out
$ echo $?
4

If we modify tests/simple.c?

$ cat tests/simple.c
int main() {
  return 254;
}
$ gcc tests/simple.c
$ ./main a.out && echo $?
254

Not bad!

Process and next steps

Getting this far took a lot of trial and error, much of it hidden in this post. Setting up the REPL was critical to debugging mistakes. But aggressively unit testing would probably have been similarly fruitful. In the end, the most bug-prone aspects are basic arithmetic (off by one errors and converting bytes to/from integers). The part that's not terribly hard is actually interpreting instructions! But it's made easier by greatly simplifying the problem and ignoring legion cases.

Along the way it would have been helpful to also disassemble so that instead of just dumping memory at the instruction pointer we print the instructions we thought we were going to process. That may be a next goal.

Otherwise the typical goals are around getting syscall support, function call support, and porting these simple examples to Windows and macOS for the experience.

Here's take two on writing an emulator for linux/amd64 in Go. This time we're starting with ELF binaries, but still ignoring libc and jumping straight to main.https://t.co/A87r2RY21c
— Phil Eaton (@phil_eaton) November 26, 2020

What is more short-sighted? UBI, or certain critics of it?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 25 Nov 2020 20:00:00 GMT

What is the PURPOSE of life, again?

Reasons why SELECT * is bad for SQL performance

Tanel Poder Blog — Wed, 25 Nov 2020 05:14:58 GMT

Here’s a list of reasons why SELECT * is bad for SQL performance, assuming that your application doesn’t actually need all the columns. When I write production code, I explicitly specify the columns of interest in the select-list (projection), not only for performance reasons, but also for application reliability reasons. For example, will your application’s data processing code suddenly break when a new column has been added or the column order has changed in a table?

Mexico uses asymmetric encryption for university diplomas and taxes.

Daniel Sada Caraveo | Developer Productivity & Culture — Wed, 25 Nov 2020 02:00:00 GMT

I recently graduated, and in order to get my official graduation title I needed to get my private and public key from the government. That’s right, you go to the government’s office, certify that you are who you are, and you’ll get a private and public key in an USB key from the government which you can use to do all your bureaucracy. You also get a TaxID which is comprised of

2020–11–25: New multi-distro and p-boot released, WiFi power savings

megi's PinePhone Development Log — Wed, 25 Nov 2020 01:00:00 GMT

Starlink? All will be well

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 24 Nov 2020 20:00:00 GMT

If they only answer SOME people.

AoC

blog — Tue, 24 Nov 2020 14:09:39 GMT

Fixing Race Conditions in React with useEffect

Max Rozen — Tue, 24 Nov 2020 11:48:48 GMT

If you're using useEffect to fetch data, chances are you've either run into a race condition, or have one without realising it. Let's learn how to fix them in this article.

The Best Garlic Chilli Prawns 🌶🍤

Brent's Blog — Tue, 24 Nov 2020 02:00:00 GMT

These are mediterranean style garlic chilli prawns, normally I double the garlic and chilli for this recipe and the oil is so good to dip with warm fresh crusty bread.

Zettelkasten Updates...

trv.rs — Mon, 23 Nov 2020 22:31:00 GMT

Since I wrote my last post… there has been so much motion in the knowledge-management / note-taking space, largely due to the exploding popularity of Roam Research. Many apps have adopted Roam-like features like linking to blocks, wiki links, or graph views.

This has greatly excited me to learn and code more tooling for this kind of thinking … but it has also had a paralyzing effect. There's so many more options out there for creating this kind of networked knowledge management, whether you call it a wiki, a zettelkasten, or whatever. Like many I don't know exactly where to turn.

For the (three of you) that read this and care, here's what I'm looking for as I watch this next wave of note-taking apps. I'm attempting to balance convenient accessibility and complete ownership… on a budget. :P

I still want to own my content.

This is the main reason that Roam (however wonderful!) is not an option for me. I want to be able to own, export, transfer, and transform my content however I wish. Part of this is because I'm persnickety, part of it is because I'm a programmer. It would be foolish to give up my ability to program/script against the notes that I plan to spend a life on.

I also want the file format to "work" as a Zettelkasten independent of a particular app. A helpful thought: my system should still be usable when printed.

Markdown

Markdown continues to be the way I write and store content. While it is frustrating how many standards there are, markdown still is the best option for writing that I know of (unless you are a LaTeX fan, I suppose. Latechie? Latexan?)

Being a bare text file, Markdown does present the problem of "where do the images/files go?" But that doesn't worry me too much… as long as I have a service or server that supports URLs. Anything is automatable.

Metadata within the content of the file.

I still believe in having a stable ZK style ID. If there is unique ID (or UID) we can rename/retitle the file with low risk of irrevocably broken links.

For example: I started a zettel thought around a particular Bible verse, but over time the thought changed from simply this one verse to a discussion of the "fear of the Lord." If the title of the thought changes, does the file name? If the file name changes, what happens to all the links?

Most knowledge management software solutions handle this using their internal database to update all the links in all the other files if the linked-to title/file name changes.

I also am in favor of all metadata existing within the body of the file. This way if I were to print my knowledge base it would still work.

This is the sticky point for most of the solutions and applications that we are going to look at today.

Trying to consider maximum future compatibility, there are the two changes from my original recommendation:

Use YAML style frontmatter instead of the multimarkdown spec.
Use keywords instead of tags to respect Pandoc.

(Optionally, you can change the filename structure to use spaces instead of kebab-case.)

Filename:

202011182014 Title of the note.md

Content:

---
id: 202011182014
title: Title of the note
keywords:
  - #still
  - #use
  - #hashtags
---

Body of the note should be a single thought, complete and fully formed… but not too big.

References and links can come below.

- [baseline Zettelkasten](https://zettelkasten.de/posts/baseline-zettelkasten-software-reviews/)

Apps and projects I'm watching…

For a bit there, I was trying to write my own system using shell scripts and vim. I'm still messing with that, but I'm paying attention to a couple readily available projects that seem to align with my goals:

Obsidian

This is been a favorite of many, and I really really like it. It checks nearly all of the boxes: It's got a nice interface, graph view, ongoing development, and markdown… and a growing plugin community that is very strong. The discord is active and really helpful.

Obsidian does have support for both markdown and YAML front-matter. It works for 100% of the things I need, but not 100% of its feature-set works for file names that include the ZK ID.

This really shouldn't stop me from using it… it supports everything I could hope to implement myself. I mostly have been holding out to see if a convention supporting stable Zettelkasten-style IDs will be settled on… but if you don't really care about this, you should probably just start using Obsidian.

Neuron

I really like what I see in Neuron. It's philosophy matches up with what I would like and its shell-script approach allows there to be multiple editors like emacs, vim, and VS Code.

I would really like to try it out (it seems nearly perfect for me!), but it has a really strange install method that is prohibitively hard on OS X. It looks like there is some recent motion on it though, so maybe I'll get to test-drive it soon.

Foam

Foam is the most popular (that I can see) of the multitude of plugins for VS Code.

If you use VS Code daily as your editor, something like this makes a lot of sense. I still am hoping for something cross-form, but whatever. It's an impressive plugin.

Logseq

Logseq is an interesting option… it's entirely web-based, but connects to your GitHub. This means you could interact with it using any desktop editor of your choice… but you could also work on it on a mobile device or iPad natively in the browser, as long as you have an internet connection.

Basically, logseq is a roam research clone that stores the data as markdown files on your github. Pretty cool.

Even if you don't use it as your primary editor, you could use this to work on a Simple Markdown Zettelkasten from mobile.

Vanilla Vim / Emacs

There's nothing stopping you from doing this all by scratch. I've started down this path a little bit with the help of much smarter people. At the moment I'm not actively pursuing it… I want to make sure my file format / data structure is more finalized. There are folks with similar but much more mature solutions. I fear I may not have the time to learn lua/vimscript fast enough. :P

Where does that leave me?

I'm in the middle of writing a transition script to transform my existing plaintext notes, book notes, and baby zettelkasten (V1!) into the "new" format with YAML metadata. I haven't given it the attention it deserves, so now I have the old zettelkasten, new notes, old untransformed plaintext notes… I put myself in a place where I have notes everywhere in at least six formats. It's a problem.

Over the next few days, I'm probably going to resume the Sisyphean effort of working on my notes… (I last time I counted there's about 2k files in my untransformed folder!) but I thought I'd post what I'm thinking heading into it all.

If anyone is further down this path, I'd love to hear about it. It's a great time to be refactoring and rethinking your note taking options... just don't get stuck mid-transition. 🤮

Artificial Intelligence is like Jessica Rabbit

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 23 Nov 2020 20:00:00 GMT

It’s not bad. WE just draw it that way.

EXWM on Chrome OS

Reticulated — Mon, 23 Nov 2020 02:08:33 GMT

How to use EXWM on Chrome OS

Pulled pork recipe

xenodium.com @alvaro — Mon, 23 Nov 2020 02:00:00 GMT

Made pulled pork a couple of times. Freestyled a bit. No expert here, but result was yummie.

Grind/blend spices

2 teaspoons smoked paprika
2 teaspoons cumin seeds
2 teaspoons whole pepper corn mix
2 teaspoons chilly flakes

If spices are whole, grind or blend them. Set aside.

Optionally: Substitute 1 teaspoon of paprika with chipotle pepper.

Mix into a paste

2 tablespoons honey
1 teaspoon of dijon mustard

Mix the honey, mustard, and previous spices into a paste.

Rub the mix in

Rub mix thoroughly into the pork shoulder.

Bake (1 hour)

Place in a pot (lid off) and bake in the oven for 1 hour at 200 °C.

Add liquids

1/2 cup of water.
4 tablespoons of apple cider vinegar.

Add liquids to pot.

Bake (3-5 hours)

Bake between 3 to 5 hours 150 °C. Check every hour or two. Does the meat fall easily when spread with two forks? If so, you're done.

Pull apart

Use two forks to pull the meat apart.

How to: Backup a Mastodon instance

nutcroft — Mon, 23 Nov 2020 02:00:00 GMT

I recently started a new Mastodon instance and went through the joy of setting up a Mastodon server. It was easier than expected, the process is very streamlined and detailed in the docs: https://docs.joinmastodon.org/admin/prerequisites/

But, of course, the backup process depends on your setup. The docs here, too, are very helpful—in the sense of what you need to backup: https://docs.joinmastodon.org/admin/backups/

My setup is based on a simple VPS, with AWS S3 for backup storage, and Healthchecks.io for schedule monitoring. Except for the environment secrets, which is just a static file and easily backup-able, the others are postgres, redis, and uploaded files.

This is the full backup-mastodon.sh script:

#!/bin/bash

# postgres
echo "Postgres..."
runuser -l mastodon -c 'cd ~ && pg_dump -Fc mastodon_production -f postgres.dump'
aws s3 cp /home/mastodon/postgres.dump s3://sirodohtbucket/backup-postgres/$(date --iso-8601=seconds)/

# redis
echo "Redis..."
aws s3 cp /var/lib/redis/dump.rdb s3://sirodohtbucket/backup-redis/$(date --iso-8601=seconds)/

# uploads
echo "Uploads..."
now=$(date --iso-8601=seconds)

runuser -l mastodon -c 'cd ~ && tar -czvf accounts.tar.gz /home/mastodon/live/public/system/accounts/'
aws s3 cp /home/mastodon/accounts.tar.gz s3://sirodohtbucket/backup-uploads/$now/

runuser -l mastodon -c 'cd ~ && tar -czvf media_attachments.tar.gz /home/mastodon/live/public/system/media_attachments/'
aws s3 cp /home/mastodon/media_attachments.tar.gz s3://sirodohtbucket/backup-uploads/$now/

runuser -l mastodon -c 'cd ~ && tar -czvf site_uploads.tar.gz /home/mastodon/live/public/system/site_uploads/'

aws s3 cp /home/mastodon/site_uploads.tar.gz s3://sirodohtbucket/backup-uploads/$now/

Notes:

For postgres, we just use pg_dump with the postgres custom format -Fc.
For redis, we just copy the periodically disk-written database dump.
For the uploaded files, we archive three directories accounts/, media_attachments/, and site_uploads/ and then upload them to S3 with the aws s3 copy command.
For all commands we use runuser so that they run as if the mastodon user is executing them.
Also, we upload them into datetime directories nested in the bucket, with date --iso-8601=seconds.

As for the crontab entry, we follow the Healthchecks docs on the dot:

0 3 * * * /home/sirodoht/backup-mastodon.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/f880a11a-4cd8-4b64-98a2-20a510cd429f

Does iOS 14 protect your privacy?

Gingerlime — Sun, 22 Nov 2020 20:12:24 GMT

A few months ago I wrote a post: Does Apple care about your privacy? In the post, I looked at Apple’s IDFA – ID For Advertising, and how it’s abused by companies like Facebook and many more to track you. I believed then, and still believe now, that what Apple is doing is not ethical…

The post Does iOS 14 protect your privacy? appeared first on Gingerlime.

A quick look at Touch Handling APIs in Game Engines

Erico Notes — Sun, 22 Nov 2020 17:53:12 GMT

Since the iPhone happened in 2007, touch input has been at the forefront of human interaction with machines. But Game Engines have existed for a reasonable long life, preceding the mobile boom. Touch as a means of interaction with the game world in engines is provided in many ways. Below, we are going to look at different touch APIs to see how they have been provided to enable gaming on our phones. Tilt, camera, and other sensors are going to be ignored so we can focus on touch.

Browser - Gold standard of Touch APIs

The browser is not a "game engine" but there are games that run in browsers and it has solid handling of touch input that is used across devices every day in different ways. Mostly, though, they are handled through DOM elements and not directly. But let's look at how their direct handling can be done.

Browser handles 4 touch events:

touch start: when the finger starts touching the browser content area
touch move: if the finger moves after touching the browser content area
touch end: whether the finger moved or not when it's released from the browser content area
touch cancel: when the browser tab changes, the finger goes to the browser UI or something else

In code, subscribing to these events requires passing the function that will handle them, and the boolean at the end gives information on how that event propagates. In the browser is also possible to specify the element that is susceptible to that event - these are not specifics for touch but give interesting functionalities.

function startup() {
  var el = document.getElementById("canvas");
  el.addEventListener("touchstart", handleStart, false);
  el.addEventListener("touchend", handleEnd, false);
  el.addEventListener("touchcancel", handleCancel, false);
  el.addEventListener("touchmove", handleMove, false);
}

Each event passes an array of Touch elements, each with the following parameters: identifier, clientX, clientY, pageX, pageY, screenX, screenY, target. More elements are available experimentally depending on the browser. The identifier property is a unique integer for each touch and remains consistent for each event during the duration of each finger's contact with the surface.

Desktop browsers, even when a touch screen is available, can disable these events. A different API for Pointer Events exists and is recommended to use instead, and mixes Mouse Input, Touch input, and Pen input in the same event type.

Unity - The most complete API

https://docs.unity3d.com/ScriptReference/Touch.html

In Unity, you can check Input.touchCount to figure out how many touchpoints are available at that frame, and retrieve each one using Input.GetTouch(i).

Each touch will have the following properties: fingerId, phase, position, deltaTime, radius, pressure, rawPosition, deltaPosition, azimuthAngle, altitudeAngle, maximumPossiblePressure, radiusVariance and type (whether a touch was of Direct, Indirect (or remote), or Stylus type).

The following are the possible touch phases: Began (A finger touched the screen), Moved (A finger moved on the screen), Stationary (A finger is touching the screen but hasn't moved), Ended (A finger was lifted from the screen. This is the final phase of a touch) or Canceled (The system cancelled tracking for the touch).

Because both the index i used in Input.GetTouch(i) and fingerId are integers, in some touch APIs that require passing a integer is not obvious what to use. For example EventSystem.current.IsPointerOverGameObject(int) actually requires passing a touch.fingerId.

GameMaker - Multiple mouses approach!

Game Maker has two separate APIs for dealing with touch. One is the same used for mouse input, but they work the same way a PC would work if it had multiple mice attached, which makes it different from what we have seen so far. You pass a device number that goes from 0 to n, where the number of devices available appears to be 5 - but I can't find the function that retrieves this number. Then 0 is the first finger that touched the screen (or the mouse in a computer) and 1 is the second one and so on, and you can use a function like device_mouse_check_button(device, button); for retrieving if either the finger is pressed (by checking left click) or the mouse is clicking, with the device being this number that ranges from 0 to n.

The following functions are available: device_mouse_check_button, device_mouse_check_button_pressed, device_mouse_check_button_released, device_mouse_x, device_mouse_y, device_mouse_raw_x, device_mouse_raw_y, device_mouse_x_to_gui, device_mouse_y_to_gui, and device_mouse_dbclick_enable.

An additional, second API is available that gives access to all common touch gestures and makes them easy to use them. These include tap, drag, flick, pinch, and rotate finger events.

Godot - A minimal approach

Godot has very lean InputEvent for screen touch, with three parameters: index (a number from 0 to n identifying a finger), position (x and y point in the screen), and a boolean named pressed that is false when the finger is released.

At the same time, since it's an InputEvent, it's can propagate through the SceneTree similar to how the browser allows handling events in elements in the DOM. This makes this simple approach still very powerful.

Additionally, Godot also provides TouchScreenButton to design buttons meant to receive the touch of multiple fingers.

Unreal - Traditional with a timestamp

Unreal TouchEvents are similar to the ones we have seen so far, the touch has a unique Handle per finger, it happens at a TouchLocation (x,y position), it has a Type (similar to previous seen phase), it has a float indicating a Force and it has a DeviceTimestamp so you can precisely know at which time the specific touch occurred.

A touch Type can be one of the following enum values Began, Moved, Stationary, ForceChanged, FirstMove, Ended, NumTypes (used when iterating across other types).

Overall, it's a very standard API. Events are also available for On Input Touch Begin, On Input Touch End and others.

Ogre3D - Very similar to SDL2

https://ogrecave.github.io/ogre/api/latest/struct_ogre_bites_1_1_touch_finger_event.html

Ogre3D provides TouchFingerEvent where each finger has a unique integer fingerId, a type, an x and y position, and additionally a dx and dy is provided for the delta of that position.

Love 2D - Polling and events and Lua

Love2D allows to both poll for the touch on an engine update or to use a function to work with touch events.

When polling, love.touch.getTouches() retrieves a table named touches, which has a list of ids for each touch. The position of a touch can then be retrieved by using local x, y = love.touch.getPosition(id) for each id. Because Love2D uses Lua, polling is very fast.

The events available are love.touchpressed, love.touchmoved and love.touchreleased. In each of them, the passed parameters are id, x, y, dx, dy and pressure.

Ren'Py - Gestures

https://www.renpy.org/doc/html/gesture.html

Using gestures instead of handling the position of where a touch happens is a very different approach, Ren'Py docs explain this neatly:

The gesture recognizer first classifies swipes into 8 compass directions, "n", "ne", "e", "se", "s", "sw", "w", "nw". North is considered to be towards the top of the screen. It then concatenates the swipes into a string using the "_" as a delimiter. For example, if the player swipes down and to the right, the string "s_e" will be produced.

This allows building gestures in string, config.gestures is available as a dictionary of sorts where a string representing a gesture maps to a function (an action): define config.gestures = { "n_s_w_e_w_e" : "progress_screen" } .

Conclusion

When researching for this topic I looked into HaxeFlixel Actions, and while there's no obvious way of handling touch there other than the mouse, it has a very interesting system of matching input to a function that will execute its gameplay result and it seems similar to what is available in both Ren'Py and Game Maker premade gestures. Still, gesture handling in the engine seems to be a minority.

Looking at the touch APIs of most engines above, they seem decided to give you the meaningful data available that is as close as the user input and let you handle and interpret it as you wish. In the early days of multi-touch APIs I believe I saw even one x and y position per "pixel" touched which was very demanding to go through. By giving a position per finger per frame it's data that can be handled without as much effort and from the documentation it appears to be what is being made available in the current APIs. In some engines though, the position is a normalized float between 0 and 1 and you are on your own to convert this in your world coordinates.

Godot seem to work with the leanest at just an ID per finger, a position, and information so you can tell when the finger is down and when it leaves the screen. If you are looking to the minimal you can have to work with mobile devices when building your game, I believe it hits it.

Mining Instagram efficiently with Torpy - Fast Instagram Scraper

Geography & Coding — Sun, 22 Nov 2020 14:42:15 GMT

A fast and efficient Instagram Scraper based on Torpy. Scrapes posts for multiple hashtags and location ids.

Tidying up towing data with Pandas

utf9k — Sun, 22 Nov 2020 13:00:00 GMT

A step by step walkthrough on how I cleaned up some data as part of a side project

Why TUI applications ?

dystroy|Canop / blog — Sun, 22 Nov 2020 02:00:00 GMT

Twenty years ago, most of us thought TUI applications were going to disappear. TUI were just a vestige of low capabilities computers. There was no reason for TUI not to die with the rise of the personal computer.

This didn't happen. We still love to use TUI applications.

Why can be kind of mysterious.

OSS logstash with AWS Opendistro for Elasticsearch

Brent's Blog — Sun, 22 Nov 2020 02:00:00 GMT

To get logstash talking to the Open Distro Elasticsearch the first thing that should be understood is that open distro only works with the OSS (Apache 2.0 Licensed) edition of the Elastic tools and not the Elastic licensed edition (Xpack).

Intercept HTTP/HTTPS TLS traffic

Brent's Blog — Sun, 22 Nov 2020 02:00:00 GMT

Use open source project mitmproxy and have visibility into what iOS apps (or anything else) are sending back to their mother ship.

Building a Sim Racing Wind Simulator

Reticulated — Sun, 22 Nov 2020 02:00:00 GMT

I first started semi-seriously sim racing a few years ago after buying a cheap old Fanatec wheel/pedal/h-shifter combo on Craigslist for $60. I was looking for something new to get into, and had a previous obsession with the Gran Turismo games on PlayStation to guide me past the first hurdles. Fast forward to today and I have heavilyinvested in my setup, with the main goal being realism and immersion.

My Perfect Task App

The David Brownman Blog — Sun, 22 Nov 2020 02:00:00 GMT

I have come to the sad realization that there is no perfect task app. Many are good and some are great, but nothing will ever be perfect. Nevertheless, I spec out my ideal app.

Read the whole thing.

Not ALL COVID-related "state interventions" are the same

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 21 Nov 2020 20:00:00 GMT

There are interventions, and then there are interventions…

Professionalism and the extra mile

I Learnt — Sat, 21 Nov 2020 14:00:00 GMT

We had been planning to have a large garden summerhouse built for a number of months and we were finally in a position to do so. After a lot of research we decided to engage a local company as we liked the style of building they produced and the price was also competitive.

Devlog #4: entity activities

Dom Williams — Sat, 21 Nov 2020 14:00:00 GMT

A technical dive into how living entities do the things the brain tells them to do without crippling game performance.

Novembre 2020 : Revue de mon matériel informatique

Accueil on dbeley — Sat, 21 Nov 2020 02:00:00 GMT

Voici une revue du matériel informatique en ma possession à la date du 21 novembre 2020.

Ordinateur de bureau

Processeur : AMD Ryzen 5 2600
Watercooling AIO : Corsair H60 2019
Carte mère : ASRock Fatal1ty B450 Gaming-ITX_ac
Mémoire vive : 2x8 Go G.Skill RipJaws 5 3200MHz CL16
Carte graphique : AMD Sapphire RX 580 Nitro+ 8Go
Alimentation : Corsair HX 620W (achetée en 2010)
Boîtier : Silverstone SG13
Stockage : SSD Samsung 970 Evo NVMe 500 Go + 3 disques dur interne de 2 To, 500 Go et 500 Go

Ce système est ma machine de jeu principale, je joue exclusivement à des jeux Steam sous Linux (la liste des jeux auxquels je joue est disponible sur mon profil Steam).

Moved blog to alabaster with Sphinx

Brent's Blog — Sat, 21 Nov 2020 02:00:00 GMT

Mostly notes to self on how to deploy and architecture / setup. Inspired by https://github.com/vincentbernat/vincent.bernat.ch

The political threat of WhatsApp

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 20 Nov 2020 20:00:00 GMT

As if WhatsApp wasn’t already bad enough…

High System Load with Low CPU Utilization on Linux?

Tanel Poder Blog — Fri, 20 Nov 2020 18:31:58 GMT

In this post I will show you how to break down Linux system load by the load contributor or reason. You can drill down into the “linux system load in thousands” and “high system load, but low CPU utilization” problem patterns too.

Introduction - terminology
Troubleshooting high system load on Linux
Drilling down deeper - WCHAN
Drilling down deeper - kernel stack
How to troubleshoot past problems
Summary
Further reading

Introduction - Terminology

The system load metric aims to represent the system “resource demand” as just a single number. On classic Unixes, it only counts the demand for CPU (threads in Runnable state)
The unit of system load metric is “number of processes/threads” (or tasks as the scheduling unit is called on Linux). The load average is an average number of threads over a time period (last 1,5,15 mins) that “compete for CPU” on classic unixes or “either compete for CPU or wait in an uninterruptible sleep state” on Linux
Runnable state means “not blocked by anything”, ready to run on CPU. The thread is either currently running on CPU or waiting in the CPU runqueue for the OS scheduler to put it onto CPU
On Linux, the system load includes threads both in Runnable (R) and in Uninterruptible sleep (D) states (typically disk I/O, but not always)

So, on Linux, an absurdly high load figure can be caused by having lots of threads in Uninterruptible sleep (D) state, in addition to CPU demand.

Fix Virtualbox installation for Docker on macOS

Baptiste Fontaine’s Blog — Fri, 20 Nov 2020 18:08:00 GMT

While following a tutorial to install Virtualbox in order to have docker working on macOS, I hit an issue where the docker-machine create command fails with an error that looks like this:

VBoxManage: error: Failed to create the host-only adapter
VBoxManage: error: VBoxNetAdpCtl: Error while adding new interface: failed to open /dev/vboxnetctl: No such file or directory
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component HostNetworkInterfaceWrap, interface IHostNetworkInterface
VBoxManage: error: Context: "RTEXITCODE handleCreate(HandlerArg *)" at line 95 of file VBoxManageHostonly.cpp

If you search on the Web, everybody says you have to open the Security & Privacy settings window and allow the Oracle kernel extensions to run. But I didn’t have it. I tried uninstalling Virtualbox, re-installing through the official website, reboot, uninstall, re-install with brew cask but I always had the issue. Some people reported having a failed Virtualbox installation but mine seemed ok.

I tried the spctl solution but it didn’t change anything.

In the end, I tried this StackOverflow answer:

sudo "/Library/Application Support/VirtualBox/LaunchDaemons/VirtualBoxStartup.sh" restart

It failed, but it told me to check the Security & Privacy setting window. I did, and I had the button everyone was talking about. I enabled the kernel extension, rebooted, and it worked.

Hope this can save some time to anyone having the same issue!

My Manic Pixie Dream Programming Language

janissary — Fri, 20 Nov 2020 02:00:00 GMT

Aider la science avec le projet BOINC

Accueil on dbeley — Fri, 20 Nov 2020 01:00:00 GMT

Le Projet BOINC

Le projet BOINC (Berkeley Open Infrastructure for Network Computing) est un projet de calcul distribué créé par l’université de Berkeley en 2002.

Il permet à des volontaires d’aider la recherche scientifique en faisant tourner des calculs scientifiques en arrière-plan sur leur ordinateur. Les volontaires n’ont qu’à télécharger le client BOINC et s’inscrire sur un ou plusieurs des projets utilisant BOINC.

Chaque projet BOINC propose des unités de calcul plus ou moins gourmandes en ressources, qui lorsque terminées par les utilisateurs, les récompenseront d’un certain nombre de points. Tout un système autour de ces points a été mis en place, avec des équipes, des classements ou encore des badges.

Resilient Git, Part 1: Hydra Hosting

All content on Seirdy’s Home — Thu, 19 Nov 2020 22:23:18 GMT

Efficient redundancy via repository mirroring with nothing but git.

Resilient Git, Part 0: Introduction

All content on Seirdy’s Home — Thu, 19 Nov 2020 22:23:18 GMT

A seires on setting up resilient git-based project workflows, free of vendor lock-in.

Write Errors That Don't Make Me Think

swyx's site RSS Feed — Thu, 19 Nov 2020 21:33:35 GMT

How we plan to fail with Thoughtful Error Message Design at Temporal.

Of Google, enforcing a false sense of democracy

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 19 Nov 2020 20:00:00 GMT

Because the web should not be steered by ANY single company.

Configuration magic

utf9k — Thu, 19 Nov 2020 13:00:00 GMT

Configuration is great! Magic... not so much!

SES Email client for S3

Brent's Blog — Thu, 19 Nov 2020 02:00:00 GMT

You can conveniently browse the SES mail in S3 using this tool

OpenStreetMap is now Too Big To Stay Pure

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 18 Nov 2020 20:00:00 GMT

But is that a problem, or just a warning? And of what?

Capturing Alexa Errors with Sentry and GitLab

Nathan Friend — Wed, 18 Nov 2020 16:22:10 GMT

Diagnosing issues with a live Alexa skill can be tricky.

DevOps Team Charter

smcleod.net — Wed, 18 Nov 2020 16:00:00 GMT

A template team charter for software and platform engineering teams.

Research¹ shows there is great value in people embracing a shared vision and ideas that are bigger than themselves.

When it comes to a team charter, it’s important to remember that it’s not a static document. It’s a living document that should be updated as the team evolves and grows.

Keep in mind that

One size doesn’t fit all.
Consider your (teams) mission, cultural background and values.
While not exhaustive - this is on the longer side, you may wish to distill as is practical, but I implore you not to only pick the easier items - that’s missing the point.
These can give you talking points when performing postmortems, retros and during times of conflict.

CCEC 2020 Reg Friesen Competition [Video]

David Schlachter — Wed, 18 Nov 2020 15:15:00 GMT

Review of pre-print manuscripts on remote instruction since January 2020. How are students and instructors adapting the the global pandemic?

Building a J2EE app using IntelliJ Idea and deploying on Heroku

Online Home of Rajkumar — Wed, 18 Nov 2020 08:53:35 GMT

Scala 3 Context Functions

Functional[Justin] — Wed, 18 Nov 2020 02:00:00 GMT

Some Scala 3 Things: Context functions, Enums and significant whitespace

Updated: for Scala 3.0.0-M3 `as` keyword was removed

This is the companion blog for my first Functional Justin video which you can find here https://youtu.be/J01u_Dmrx5U. I spend around 15 minutes adding some Scala 3 (formerly Dotty) features to an Scala 2 program.

The program itself builds a simple Algebraic Data Type (ADT) to represent a simple arithmetic expressions. We can then build expressions in this algebra and evaluate it using an eval function using pattern matching…

sealed trait Exp
case class Val(value: Int) extends Exp
case class Add(left: Exp, right: Exp) extends Exp
case class Mul(left: Exp, right: Exp) extends Exp
case class Var(identifier: String) extends Exp

Now given an expression like Mul(Var("z"), Add(Val(30), Mul(Var("x"), Var("y")))) I'd like to be able to recursively traverse it and calculate a final Int value at the end.

Val represents an Int value, whilst Add and Mul take care of addition and multiplication. You could go ahead and add more functions. Var is interesting because it takes an a string identifier (i.e., a variable name) and will look it up in an environment. The environment is represented a Scala map of String to Int.

type Env = Map[String, Int]

For the eval function we just use a pattern match to dispatch to functions that handle each particular operation. These handler functions and eval are mutally recursive, and note that every function has to have the Env passed to it as an implicit parameter, yet only Var needs it. This will be important later.

Here's the eval function and handlers.

def eval(exp: Exp)(implicit env : Env): Int = {
  exp match {
    case Var(id) => handleVar(id)
    case Val(value) => value
    case Add(l,r) => handleAdd(l,r)
    case Mul(l,r) => handleMul(l,r)
  }
}

def handleAdd(l: Exp, r: Exp)(implicit env : Env) = eval(l) + eval(r)
def handleMul(l: Exp, r: Exp)(implicit env : Env) = eval(l) * eval(r)
def handleVar(s: String)(implicit env: Env) = env.getOrElse(s, 0)

Note that we could have inlined these functions in eval, but it a larger example it's important to break things out to keep things managable.

That is all the implementation we need, and all that remains is to create an expression, create an environment (declared implicit so Scala knows to include it as an implicit when eval is called) and print the result of evaluating the expression.

val exp1 : Exp = Mul(Var("z"), Add(Val(30), Mul(Var("x"), Var("y"))))

implicit val env : Env = Map("x" -> 17, "y" -> 10, "z" -> 2)
val eval1 = eval(exp1)

println(s"Eval exp gives $eval1")

You can compile and run the code to see this working. The code is here. https://github.com/justinhj/evalexample/blob/master/src/main/scala/Scala2Eval.scala

Fun with Enum

Scala enums have been improved greatly. For one they are very simple to create and use just as in other languages.

enum StatusCode:
   case OK, TimedOut, Error

Here we've defined three enums that have ordinal values 0 to 2. You can access the ordinal value with the .ordinal method, convert ordinal values to Enums using .fromOrdinal and convert Strings to enums (assuming they match) with .valueOf.

println(s"Ordinal value of StatusCode.Error is ${StatusCode.Error.ordinal}")
println(s"StatusCode from ordinal 1 is ${StatusCode.fromOrdinal(1)}")
println(s"StatusCode from string OK is ${StatusCode.valueOf("OK")}")

// Ordinal value of StatusCode.Error is 2
// StatusCode from ordinal 1 is TimedOut
// StatusCode from string OK is OK

You can also add your own parameters and definitions to enums. The underlying ordinal values are still there. For example you could encode Http Status codes as follows.

enum HttpStatusCode(code: Int) {
  case OK extends HttpStatusCode(200)
  case NotModified extends HttpStatusCode(304)
  case Forbidden extends HttpStatusCode(404)

  def isSuccess = code >= 200 && code < 300
}

Scala 3 team also took the opportunity to make Enums a more concise notation for ADTs and GADTs. For our purposes that means we can simply the definition of Exp as follows.

enum Exp {
  case Val(value: Int) extends Exp
  case Add(left: Exp, right: Exp) extends Exp
  case Var(identifier: String) extends Exp
}

In fact you can further simplify to the following (you could also remove the braces).

enum Exp {
  case Val(value: Int)
  case Add(left: Exp, right: Exp)
  case Var(identifier: String)
}

Explicit implicits

A focus of the Scala 3 team is to help beginners access the language and in particular simplifying implicits. There are many subtle changes here but two obvious ones are that you now have different keywords for implicit parameters and creating implicit instances. In our code this means that when we supply the implicit symbol table to eval we now use the new given syntax instead of implicit.

implicit val env : Env = Map("x" -> 17, "y" -> 10, "z" -> 2)

becomes…

given envMap: Env = Map("x" -> 7, "y" -> 6, "z" -> 22)

Similarly, the method parameters now no longer use the implicit keyword and instead you prefix the parameter name with using.

def eval(exp: Exp)(implicit env : Env): Int

becomes…

def eval(exp: Exp)(using env : Env): Int

You don't have to change your Scala 2 code at this point, it is still compatible, but for new code and in the long term you should gradually eliminate implicit.

Context Functions

Last and not at all least are context functions. This gives us one more opportunity to remove boiler plate from the eval code. When you create a regular function value it has a type like Function1[A,B]. In other words it is a function that takes a value A and returns vale of type B. Context Functions are a new function value type (this is synthesized by the compiler so you won't see it your code), with an input and an output type. The difference is that the input is understood to be provided implicitly.

Let's make this more concrete. Assume you have a function that needs an ExecutionContext. We can make a Context Function type that will take an implicit execution context and return some paramaterized type T.

type Executable[T] = ExecutionContext ?=> T

How would that be used in a real program? Let's say you have some deeply nested function (f4 in the code below) and it is only down at that level you need the implicit execution context. Without implicit parameters you'd add the ExecutionContext parameter to every single function call all the way down and then have to take care to pass it along. With Scala 2 implicits you still have to declare the parameter but you can make it implicit and avoid the burden of manually passing it along.

With Scala 3 you can define the function to be of type Executable[T] and then we don't need to even name the implicit parameter, we just know that it will be included automatically all the way down. Here is a complete example.

import scala.concurrent.{Future, ExecutionContext, Await}
import scala.concurrent.duration._
import scala.language.postfixOps

object Executable extends App {

  type Executable[T] = ExecutionContext ?=> T

  def f1(n: Int): Executable[Future[Int]] = f2(n + 1)
  def f2(n: Int): Executable[Future[Int]] = f3(n + 1)
  def f3(n: Int): Executable[Future[Int]] = f4(n + 1)
  def f4(n: Int): Executable[Future[Int]] = {
    val ex = summon[ExecutionContext]
    Future {
      println(s"Hi from the future! n is $n")
      n
    }
  }

  {
    given ec: ExecutionContext = scala.concurrent.ExecutionContext.global
    Await.result(f1(10), 1 second)
    // Hi from the future! n is 13
  }

}

Context functions reduce boilerplate when dealing with implicit parameters in deeply nested code. We can apply this technique to our eval function so that the symbol table itself is the implicit piece of context.

type WithEnv = Env ?=> Int

def eval(exp: Exp): WithEnv =
  exp match {
    case Var(id) => handleVar(id)
    case Val(value) => value
    case Add(l,r) => handleAdd(l,r)
  }

def handleAdd(l: Exp, r: Exp): WithEnv = eval(l) + eval(r)

def handleVar(s: String): WithEnv =
  val env = summon[Env]
  env.getOrElse(s, 0)

You can take a look at the final Scala 3 version of the code here.

https://github.com/justinhj/evalexample/blob/master/src/main/scala/Scala3Eval.scala

Final notes

Of all the new features in Scala 3, I found Context Functions of most interest because of Martin Odersky's blog from 2016 https://www.scala-lang.org/blog/2016/12/07/implicit-function-types.html where this intriguing quote appears near the end. (Context functions were initially known as implicit functions).

There are many interesting connections with category theory to explore here. On the one hand, implicit functions are used for tasks that are sometimes covered with monads such as the reader monad. There’s an argument to be made that implicits have better composability than monads and why that is.

On the other hand, it turns out that implicit functions can also be given a co-monadic interpretation, and the interplay between monads and comonads is very interesting in its own right.

But these discussions will have to wait for another time, as this blog post is already too long.

Somewhat of a Fermat's last theorem moment there, and I am also interested in how we can represent concepts, that are currently implemented in libraries which model category theory, using vanilla Scala 3 or alternative representations.

References

https://en.wikiquote.org/wiki/Pierre_de_Fermat

https://dotty.epfl.ch/docs/reference/enums/enums.html https://dotty.epfl.ch/docs/reference/enums/adts.html

http://dotty.epfl.ch/docs/reference/other-new-features/indentation.html

https://dotty.epfl.ch/docs/reference/contextual/givens.html https://dotty.epfl.ch/docs/reference/contextual/using-clauses.html

https://dotty.epfl.ch/docs/reference/contextual/context-functions.html

Foundations and Applications of Implicit Function Types https://infoscience.epfl.ch/record/229878/files/simplicitly_1.pdf

http://recurse.se/2019/09/implicit-functions-in-scala-3/

The Mythical Man Month - Review

Luke Singham — Wed, 18 Nov 2020 02:00:00 GMT

📚 Book: Mythical Man Month

⭐️ Rating: 3/5 (4/5 if you haven't had experience in tech)

The Mythical Man Month (MMM) is a collection of essays that encapsulate wisdom accumulated by the author Fred Brooks. If, like me, you have been in the tech industry for a number of years you are unlikely to gain much value from this book - as you have either read articles or experienced first hand these lessons.

Some criticisms of the book: #

🤦‍♀️ There's not a female pronoun in sight - one of the few mentions of a female, is for an analogy where the project is a pregnancy, adding more women to the project will not decrease the nine months it will take for the birth to happen. This is to be expected of a book originally written in 1975 and with a title like Mythical Man Month. I read the 1995 edition. I would hope any subsequent editions endeavour to have more gender inclusive language.
👼 There's quite a lot of God mentions, more mentions of God than Women.
📠 Many of the examples are very out of date. And the details on these examples of e.g. IBM 360 are superfluous to the modern reader.

Major (Useful) Points of the Book #

💣 Combinatorial Explosion #

Where the title of the book comes from. Many simpler types of work scale linearly with additional resources. For example, if you were a Farmer of 20 fields employing labour to til those fields. You might imagine doubling labour from 2 to 4 would double results. This is true, as there is little requirement for coordination of the manual labour. Labourers each get an allocation of fields to independently til. However, with work that requires much coordination, alignment of corresponding components of a greater system like software, adding more labour is frequently counterproductive. This leads to an assertion that adding engineers to a late project will make it later because of the communication overhead and the need to onboard.

The effect is known as combinatorial explosion. Where each node (worker) that needs to be aligned with the rest of the team adds many more communication links.
$links = \frac{n^{2} - n}{2}$

A few examples demonstrate this:

when n=2, there is 1 link
when n=4, there are 6 links
when n=8, there are 28 links

This rule is also known as the 'two pizza rule' 🍕 - teams shouldn't be larger than can be fed with two pizzas. Not helpful if you eat an entire pizza to yourself 🤤. However, 5-7 people is what is meant.

🧠 Conceptual Integrity #

... conceptual integrity is the most important consideration in system design

Brooks talks a lot about Conceptual Integrity of software systems. In data analytics this is known by various terms e.g. ontology, entity models. If you have worked with any system with multiple contributors you have likely witnessed the core concepts of a system drift over time. Here Brooks suggests a solution,

If a system is to have conceptual integrity, someone must control the concepts. That is an aristocracy that needs no apology.

📈 10x programmers #

To maintain conceptual integrity smaller teams should be preferred.

one wants the system to be built by as few minds as possible

The type of team you should favour is one of a few senior engineers rather than one that is larger with junior engineers. Brooks cites a study measuring programmer productivity which is probably where the 10x engineers terminology originated. I thought this might be an exaggeration, but I've seen this play out in a workplace. A leader of a project thought that software was a sweatshop and if they just got more cheap juniors fresh out of code bootcamps that software would happen. It didn't. However, hiring one senior and one mid-level engineer saw delivery sky rocket. And yes, the senior was an order of magnitude more productive than juniors and for less than 3x the price!

📄 Documentation #

Write a spec/proposal document prior to commencing on the code. This outlines a number of things, the why, the requirements of a solution, the range of solutions considered and the proposed solution to be implemented. This helps in many ways, the writing process helps expose gaps in thinking, communicates to others, helps align the team and provides a record of what is being done or has been done.

🚧 Maintenance #

A program doesn't stop changing when it is delivered for customer usage

Brooks estimates that maintenance is '40 percent or more' than the initial cost of developing a system. The more users you have the greater the cost as more bugs are found by more users.

Resources #

asciimath.org - used to generate mathML mathjax formula

Indirect Effects of Allocate Direct

SerCe's blog — Wed, 18 Nov 2020 02:00:00 GMT

This article explores the trade-offs between the allocate and allocateDirect methods in Java for allocating memory.

Configuring Git for Work

Philipe Fatio — Tue, 17 Nov 2020 22:38:04 GMT

How to use your work email in your work repositories

It seems we need four more Asimov's laws

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 17 Nov 2020 20:00:00 GMT

OK, but… for whom? Robots, or something else?

Generating HTTPS Certificate using DNS a Challenge through Let's Encrypt

Navan's Archive — Tue, 17 Nov 2020 17:04:00 GMT

Short code-snippet to generate HTTPS certificates using the DNS Challenge through Lets Encrypt for a web-server using DuckDNS.

Fetching Data in React with useEffect

Max Rozen — Tue, 17 Nov 2020 11:48:48 GMT

If you're confused about side-effects and pure functions, it can be hard to understand useEffect. Let's learn them both, to fetch data with useEffect.

My terminal, coding workflow - i3, Kakoune, nnn

ciesie.com — Tue, 17 Nov 2020 02:00:00 GMT

If I’m working on code or tech related tasks, I’m doing it on Linux. That’s because I enjoy spending my time in terminal. I like playing around with new command line tools and fitting them into my workflow. Improving my process is important so I often try to update parts of my setup. The goal is to reduce the friction between my intention and execution.

Linux environment is full of various tools and utilities. I’ve always been a fan of Vim but quickly found NeoVim, which is an attempt to modernize Vim’s codebase. Vim itself is never enough so I’ve started integrating plugins, which in turn introduced me to a lot of smaller tools that further improved my process. Some of those tools have been substituted by a faster and more robust alternative (usually written in Rust).

Seastar's weak pointer

Noah Watkins — Tue, 17 Nov 2020 02:00:00 GMT

The non-owning counterpart to Seastar’s shared pointer.

Online direct democracy? No, thanks

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 16 Nov 2020 20:00:00 GMT

Because the world is not what is used to be.

A quick readthedocs Wiki with mkdocs

Geography & Coding — Mon, 16 Nov 2020 14:41:55 GMT

Setting up a readthedocs wiki like 1,2,3!

Bedroom carbon dioxide buildup

IvyMike.dev — Mon, 16 Nov 2020 10:00:00 GMT

You probably already know that carbon monoxide (CO) from combustion is highly poisonous. Hemoglobin binds to CO tighter than it binds to O2 by a factor of 200, so a small amount quickly prevents your blood cells from being able to carry oxygen, and you die. That's why you need …

Create Amazon Scraper in Python using Scraper API Learn how to create an Amazon scraper in python to scrape product details like price, ASIN etc

Adnan's Random bytes — Mon, 16 Nov 2020 09:51:11 GMT

In this post of ScrapingTheFamous, I am going o write a scraper that will scrape data from Amazon. I do not need to tell you what is Amazon. You are here because you already know about it 🙂 So, we are going to write two different scripts: one would be fetch.py that would be fetching URLs of individual listings and save in a text file. Later another script, parse.py that will have a function taking an individual listing URL, scrape data, and save in JSON format. I will be using Scraper API service for parsing purposes which makes me free from all worries blocking and rendering dynamic sites since it takes care of everything. The first script is to fetching listings of a category. So let’s do it! import requests from bs4 import BeautifulSoup if __name__ == '__main__': headers = { 'authority': 'www.amazon.com', 'pragma': 'no-cache', 'cache-control': 'no-cache', 'rtt': '600', 'downlink': '1.5', 'ect': '3g', 'upgrade-insecure-requests': '1', 'dnt': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'sec-fetch-site': 'same-origin', 'sec-fetch-mode': 'navigate', 'sec-fetch-user': '?1', 'sec-fetch-dest': 'document', 'referer': 'https://google.com', 'accept-language': 'en-US,en;q=0.9,ur;q=0.8,zh-CN;q=0.7,zh;q=0.6', } API_KEY = None links_file = 'links.txt' links = [] with open('API_KEY.txt', encoding='utf8') as f: API_KEY = f.read() URL_TO_SCRAPE = 'https://www.amazon.com/s?i=electronics&rh=n%3A172541%2Cp_n_feature_four_browse-bin%3A12097501011&lo=image' payload = {'api_key': API_KEY, 'url': URL_TO_SCRAPE, 'render': 'false'} r = requests.get('http://api.scraperapi.com', params=payload, timeout=60) if r.status_code == 200: text = r.text.strip() soup = BeautifulSoup(text, 'lxml') links_section = soup.select('h2 > .a-link-normal') for link in links_section: url = 'https://amazon.com' + link['href'] links.append(url) if len(links) > 0: with open(links_file, 'a+', encoding='utf8') as f: f.write('\n'.join(links)) print('Links stored successfully.') So here is the script. I picked the electronics category, you may choose anyone you want. I arranged relevant headers. You can either pick manually via Chrome Inspector or use https://curl.trillworks.com/ to do this for you by copying the cURL request. I am using the ScraperAPI API endpoint by setting both the payload and api_key. I am using h2 > .a-link-normal selector because there are many .a-link-normal links which are not required so I am using h2 > to make sure to pick the required links. Once we have links we will be saving in a text file. Now the next part of the post is about parsing the info: import requests from bs4 import BeautifulSoup def parse(url): record = {} headers = { 'authority': 'www.amazon.com', 'pragma': 'no-cache', 'cache-control': 'no-cache', 'rtt': '1000', 'downlink': '1.5', 'ect': '3g', 'dnt': '1', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'sec-fetch-site': 'none', 'sec-fetch-mode': 'navigate', 'sec-fetch-user': '?1', 'sec-fetch-dest': 'document', 'accept-language': 'en-US,en;q=0.9,ur;q=0.8,zh-CN;q=0.7,zh;q=0.6', } payload = {'api_key': API_KEY, 'url': url, 'render': 'false'} r = requests.get('http://api.scraperapi.com', params=payload, timeout=60) if r.status_code == 200: data = r.text.strip() soup = BeautifulSoup(data, 'lxml') title_section = soup.select('#productTitle') price_section = soup.select('#priceblock_ourprice') availability_section = soup.select('#availability') features_section = soup.select('#feature-bullets') asin_section = soup.find('link', {'rel': 'canonical'}) print(asin_section) if title_section: title = title_section[0].text.strip() if price_section: price = price_section[0].text.strip() if availability_section: availability = availability_section[0].text.strip() if features_section: features = features_section[0].text.strip() if asin_section: asin_url = asin_section['href'] asin_url_parts = asin_url.split('/') asin = asin_url_parts[len(asin_url_parts) - 1] record = {'title': title, 'price': price, 'availability': availability, 'asin': asin, 'features': features} return record if __name__ == '__main__': API_KEY = None with open('API_KEY.txt', encoding='utf8') as f: API_KEY = f.read() result = parse('https://www.amazon.com/Bambino-Seconds-Stainless-Japanese-Automatic-Leather/dp/B07B49QG1H/') print(result) Pretty straightforward. I fetched title, scraped Amazon ASIN, and other stuff. You can scrape many other things like Amazon reviews as well. The data is stored in JSON format. Conclusion In this post, you learned how you can scrape Amazon data easily by using Scraper API in Python. You can enhance this script as per your need like writing a price monitoring script or as an ASIN scraper. Writing scrapers is an interesting journey but you can hit the wall if the site blocks your IP. As an individual, you can’t afford expensive proxies either. Scraper API provides you an affordable and easy to use API that will let you scrape websites without any hassle. You do not need to worry about getting blocked because Scraper API by default uses proxies to access websites. On top of it, you do not need to worry about Selenium either since Scraper API provides the facility of a headless browser too. I also have written a post about how to use it. Click here to signup with my referral link or enter promo code adnan10, you will get a 10% discount on it. In case you do not get the discount then just let me know via email on my site and I’d sure help you out. If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Create Amazon Scraper in Python using Scraper API
Learn how to create an Amazon scraper in python to scrape product details like price, ASIN etc first appeared on Adnan's Random bytes.

Review: NieR:Automata

Elliott Slaughter — Mon, 16 Nov 2020 02:00:00 GMT

This review contains NO SPOILERS except where specifically marked.

Openness, discussions and success

I Learnt — Sun, 15 Nov 2020 14:00:00 GMT

I work very closely with two colleagues - and have done so for a number of years. Together we have experienced some serious ups and downs in the company we work for and also personally. We have each reached a point where we were close to leaving; we have each had something happen in our lives where the others have supported us.

Debugging Python

Geography & Coding — Sun, 15 Nov 2020 13:17:55 GMT

1
2
3


import pdb
from pdb import set_trace as bp
bp()

Debugging Python with pdb 🐞

Logstash ingestion of AWS billing customer usage reports

Brent's Blog — Sun, 15 Nov 2020 02:00:00 GMT

To get the CUR reports into elastic search:

How to improve your mathematical writing

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sat, 14 Nov 2020 23:00:00 GMT

Having been a reviewer for numerous machine learning conferences now, there are some repeated issues I want to point out, concerning the way authors describe mathematical concepts in their papers. I hope that my suggestions will serve in particular those that are new to machine learning and for whom the task of reading a paper with a ‘bad’ mathematical style might seem a daunting prospect. Without further ado, here are some suggestions for improving your research article.¹

Build intuition

Your primary objective when writing mathematics should be to build intuition for readers. This does not mean that you should sacrifice any rigour or precision—I am just advocating for considering that a reader of your paper is not an expert in the subject matter yet. If you want readers to take away some wisdom from the paper, you have to address their intuition first. No one except an expert in your field will be swayed by long-winded proofs.²

I am not advocating for sloppy thinking, sloppy writing, or ‘hand-waving’ in your proofs. I am just saying that the way you lay out your overall argument should be based on intuition, such that even reviewers that are not experts in your domain can follow along. If you only write a paper geared towards a very small audience, you are reducing its impact and make your life harder because it will almost be impossible to find good reviewers for it in the conference cycle.

Strive for consistent notation and terminology

Whenever you introduce notation, try to remain consistent. For example, if you want to name three objects, prefer calling them $a$, $b$, and $c$, instead of $\alpha$, $\mathbf{B}$, and $\mathfrak{c}^\prime$.

Whenever possible, follow established terminology—such as calling loss terms $\mathcal{L}$—but do not assume that the reader is aware of these conventions. Always take the time to at least briefly introduce your notation.

This point is in fact closely related to the second issue, namely…

Less can be more

Some people aim to be extremely precise and always employ index-based notations. For example, instead of referring to objects $a$ and $b$, it is always $a_1$ and $a_2$ or something. There is nothing wrong with this in principle, but I would suggest to always strive for as little index notations as possible and, if only two objects of the same type are needed, why not use a $\prime$ symbol (\prime in LaTeX) instead of indices? This can often increase readability and makes you less error-prone because you do not have to keep track of the indices any more. For example, in our survey on graph kernels, we try to always speak of $G$ and $G^\prime$ when we need to refer to two graphs. This leaves the indices free for when you actually have to index something, for example because you are aggregating over objects arising from a certain set.

In addition, try to adhere to ’less can be more’ in other places as well, by removing unnecessary variables, unnecessary concepts, and unnecessary definitions. It does you credit if you want to explain everything about a certain field in your paper, but if you introduce a concept X as a special case of a concept Y, there is no need to go to great lengths to describe any super-concept Z, unless you actually need to refer to its properties. This does not mean that you should leave out salient information—I am suggesting to focus on pertinent information only. For example, there is no need to expand on properties of metric spaces, topologies, and so on, if all you need is Lipschitz continuity. Aim to get to the point quickly and without too many redirections (unless you are writing a book, but I am assuming that the reader of this post is interested in writing research papers).

Try also not to fall prey to the tendency to make your formulas more complex than they have to be. For example, peppering your formulas with logical quantifiers such as $\forall$ or $\exists$ is more appropriate when you are publishing in logic. Many people, however, treat these symbols as shortcuts. Please do resist that temptation. As Knuth remarks in Mathematical Writing, the overuse of such symbols just decreases readability.

As a corollary to the preceding rule, do not include ‘cosmetic mathematics’ in your paper. Every equation, lemma, proposition, or theorem should be there because it needs to be there. I know that there is a tendency for reviewers to look favourably on a paper whose mathematics seem ‘sufficiently complex’, but this is not the right way. If you want to publish good papers, your first duty is to ensure that readers of that paper can glean some insights from it. Showing off will not help in the long run.

Conclusion

Similar to my previous post on writing in general, the main thing to remember is to know your audience. In machine learning, this audience can come from highly different backgrounds. Make them appreciate the ideas in your paper by guiding them gently into the mathematics. Reserve the ‘heavy stuff’ for an expert audience that can give you some actual feedback for it instead of just skimming over it.

Write well, until next time!

I am assuming throughout this article that your research article is written for a machine learning conference. You might have to adjust your writing considerably if you are a mathematician publishing for other mathematicians—but in machine learning, many people are not exclusively trained in mathematics, making writing research papers a little bit more challenging if you want to do it well. ↩︎
Whether proofs should be included in machine learning papers at all is still a matter of debate. I think that a rigorous, principled exposition in a paper is to be welcomed. Yet, the more our community expands, the more we have to think about how to successfully get the point across in our papers! At some point, there will not be a sufficient number of expert reviewers for highly specific papers, I fear. ↩︎

Exploring a new Postgres database

CRAIG KERSTIENS — Sat, 14 Nov 2020 21:30:56 GMT

At past jobs I’d estimate we had 100 different production apps that in some way were powering key production systems. Sure some were backend/internal apps while others key production apps such as the dashboard itself. At other companies we had a smaller handful of Heroku apps that powered our cloud service, about 5-10 in total. Even just working with those internal apps it’s a number of things to keep context on. But when it comes to interacting with something you don’t know getting a lay of the land quickly is key. In helping a customer optimize and tune, or even just understand what is going on in their app an understanding of the data model is key.

As I just started a few months back at Crunchy Data I found myself digging into a lot of new systems and quickly trying to ramp up and get a feel for them.

Over the past 10 years I’ve pretty well codified my steps to getting a feel for a new database. While I’m not a DBA and a small portion of my job is spent inside a database being able to quickly navigate one saves me hours each month and days out of the year. I’m sure my process isn’t perfect, but hopefully it helps other when navigating a new Postgres database for the first time.

First the tooling

For any new database my go to tool is psql. The built-in Postgres CLI is going to be the quickest thing for me to navigate around. If you use a CLI for anything else then this should be your preference here as well. I’m also going to have a psqlrc file setup that has some good defaults. My go to defaults in my psqlrc are:

-- Automatically format output based on result length and screen
\x auto


-- Prettier nulls
\pset null '#'

-- Save history based on database name
\set HISTFILE ~/.psql_history- :DBNAME

-- Turn on automatic query timing
\timing

Getting a feel for the tables

The first thing I’m going to do is just table a look at which objects exist within the database with \d. This will spit out a mix of tables, views, sequences all within your database. A cleaner version of this may be \dt which is only tables:


\d

 List of relations
 Schema | Name | Type | Owner
--------+-------------------------------+----------+----------------
 public | pg_stat_statements | view | postgres
 public | schema_migrations | table | postgres
 public | sessions | table | postgres
 public | sessions_id_seq | sequence | postgres
 public | tasks | table | postgres
 public | teams | table | postgres
 public | users | table | postgres
(7 rows)

\dt

 List of relations
 Schema | Name | Type | Owner
--------+-------------------------------+----------+----------------
 public | schema_migrations | table | postgres
 public | sessions | table | postgres
 public | tasks | table | postgres
 public | teams | table | postgres
 public | users | table | postgres
(5 rows)

We can also use the describe operation (\d) on specific relations as well such as tables to get a feel for how they look:


\d users

 Table "public.users"
 Column | Type | Collation | Nullable | Default
---------------------------+--------------------------+-----------+----------+--------------------
 id | uuid | | not null | uuid_generate_v4()
 email | text | | not null | ''::text
 encrypted_password | text | | not null | ''::text
 reset_password_token | text | | |
 reset_password_sent_at | timestamp with time zone | | |
 remember_created_at | timestamp with time zone | | |
 last_sign_in_at | timestamp with time zone | | |
 created_at | timestamp with time zone | | not null |
 updated_at | timestamp with time zone | | not null |
 name | text | | not null |
 team_id | uuid | | not null |
 deleted_at | timestamp with time zone | | |
Indexes:
 "users_pkey" PRIMARY KEY, btree (id)
 "index_users_on_email" UNIQUE, btree (email)
 "index_users_on_reset_password_token" UNIQUE, btree (reset_password_token)
Foreign-key constraints:
 "users_team_id_fkey" FOREIGN KEY (team_id) REFERENCES teams(id)

Human readable output

Of course you may want to go one step further and actually get a sense for the data. Here a basic SELECT tends to work with a limit 1. As you don’t quite know the shape of the data this is where having \x auto setup within your .psqlrc file is helpful to autoformat the output to your screen. You can also just manually run \x auto in your SQL session to get cleaner output.


SELECT *
FROM users
LIMIT 1;

-[ RECORD 1 ]-------------+--------------------------------------------------------------
id | 0a7a3cde-3613-4073-86a7-6a19b4e62bbe
email | craig.kerstiens@gmail.com
encrypted_password | $#IJ937Gmsdf00297sEmdfu12234
reset_password_token | #
reset_password_sent_at | #
remember_created_at | 2016-07-14 14:31:01.414795+00
last_sign_in_at | 2020-02-12 21:32:53.629246+00
created_at | 2016-02-18 03:03:26.403108+00
updated_at | 2020-02-14 23:16:16.080729+00
name | Craig
team_id | d46e864-1886-45e6-b538-8991562d2e99
deleted_at | #

Time: 91.592 ms

Most databases I work with now leverage JSONB. It’s a great tool for mixing semi-structured data with more structured data.

If you’re using JSON or JSONB then there is also a handy utility function to clean up that output - jsonb_pretty(yourcolumnhere). This will take care of making that huge JSON blob nice and readable.

Feel more at home when you encounter a new database

It doesn’t take hours of reading an ERD or schema files. In about 5-10 minutes of connecting to a new database I’m able to get a sense of how things are structured and able to actually start digging in. Don’t get me wrong, I’m by no means and expert in that time, but knowing some of these basic commands will really help the next time you encounter a database and are asked to help out or glean insights from it.

The second half of MANY chessboards

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 14 Nov 2020 20:00:00 GMT

I’ts just where we ALL are, right now. And it’s dangerous.

The Roaring 2020's

swyx's site RSS Feed — Sat, 14 Nov 2020 15:25:42 GMT

Five themes I am optimistic about for the next decade - and more open questions I have no idea about.

Blog update

Linux Engineer's random thoughts — Sat, 14 Nov 2020 01:00:00 GMT

I recently updated the server where this blog is hosted, so I tought I'd do an update on the original post explaining the tech stack used to run it. Once in seven years shouldn't be too meta.

The design

First of all, kudos to Pascal Navière, a very talented polymath …

The opaqueness of "Be Not Evil"

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 13 Nov 2020 20:00:00 GMT

It is annoyingly hard to understand how to break up Google.

Writing Your Own Scraper - Simple Instagram Scraper

Geography & Coding — Fri, 13 Nov 2020 18:44:28 GMT

Photo by NeONBRAND on Unsplash

How to write your DYI Instagram Scraper

M1 Memory and Performance

metablog — Fri, 13 Nov 2020 14:46:29 GMT

The M1 Macs are out now, and not only does Apple claim they're absolutely smokin', early benchmarks seem to confirm those claims. I don't find this surprising, Apple has been highly focused on performance ever since Tiger, and as far as I can tell hasn't let up since.

One maybe somewhat surprising aspect of the M1s is the limitation to "only" 16 Gigabytes of memory. As someone who bought a 16 Kilobyte language card to run the Merlin 6502 assembler on his Apple ][+ and expanded his NeXT cube, which isn't that different from a modern Mac, to a whopping 16 Megabytes, this doesn't actually seem that much of a limitation, but it did cause a bit of consternation.

I have a bit of a theory as to how this "limitation" might tie in to how Apple's outside-the-box approach to memory and performance has contributed to the remarkable achievement that is the M1.

The M1 is apparently a multi-die package that contains both the actual processor die and the DRAM. As such, it has a very high-speed interface between the DRAM and the processors. This high-speed interface, in addition to the absolutely humongous caches, is key to keeping the various functional units fed. Memory bandwidth and latency are probably the determining factors for many of today's workloads, with a single access to main memory taking easily hundreds of clock cycles and the CPU capable of doing a good number of operations in each of these clock cycles. As Andrew Black wrote: "[..] computation is essentially free, because it happens 'in the cracks' between data fetch and data store; ..".

The tradeoff is that you can only fit so much DRAM in that package for now, but if it fits, it's going to be super fast.

So how do we make sure it all fits? Well, where Apple might have been "focused" on performance for the last 15 years or so, they have been completely anal about memory consumption. When I was there, we were fixing 32 byte memory leaks. Leaks that happened once. So not an ongoing consumption of 32 bytes again and again, but a one-time leak of 32 bytes.

That dedication verging on the obsessive is one of the reasons iPhones have been besting top-of-the-line Android phone that have twice the memory. And not by a little, either.

Another reason is the iOS team's steadfast refusal to adopt tracing garbage collection as most of the rest of the industry did, and macOS's later abandonment of that technology in favor of the reference counting (RC) they've been using since NeXTStep 4.0. With increased automation of those reference counting operations and the addition of weak references, the convenience level for developers is essentially indistinguishable from a tracing GC now.

The benefit of sticking to RC is much-reduced memory consumption. It turns out that for a tracing GC to achieve performance comparable with manual allocation, it needs several times the memory (different studies find different overheads, but at least 4x is a conservative lower bound). While I haven't seen a study comparing RC, my personal experience is that the overhead is much lower, much more predictable, and can usually be driven down with little additional effort if needed.

So Apple can afford to live with more "limited" total memory because they need much less memory for the system to be fast. And so they can do a system design that imposes this limitation, but allows them to make that memory wicked fast. Nice.

Another "well-known" limitation of RC that has made it the second choice compared to tracing GC is the fact that updating those reference counts all the time is expensive, particularly in a multi-threaded environment where those updates need to be atomic. Well...

fun fact: retaining and releasing an NSObject takes ~30 nanoseconds on current gen Intel, and ~6.5 nanoseconds on an M1
— David Smith (@Catfish_Man) November 10, 2020

How?

We got that working on x86-64 too :) this further improvement is because uncontended acquire-release atomics are about the same speed as regular load/store on A14
— David Smith (@Catfish_Man) November 10, 2020

Problem solved. I guess it helps if you can make your own Silicon ;-)

So Apple's focus on keeping memory consumption under control, which includes but is not limited to going all-in on reference counting where pretty much the rest of the industry has adopted tracing garbage collection, is now paying off in a majory way ("bigly"? Too soon?). They can get away with putting less memory in the system, which makes it possible to make that memory really fast. And that locks in an advantage that'll be hard to duplicate.

It also means that native development will have a bigger advantage compared to web technologies, because native apps benefit from the speed and don't have a problem with the memory limitations, whereas web-/electron apps will fill up that memory much more quickly.

Stashy is a Really simple Key Value store

secretgeek.net — Fri, 13 Nov 2020 01:22:14 GMT

In my various websites and web applications, I often need to store some things and later retrieve them.

Mostly for this I use a really simple key value store called "Stashy", that I built, oh, decades ago.

Imagine I have a "plain old class object", and I want to save it. It might be an object called "myNewBlogPost".

I will call:

stashy.Save<BlogPost>(myNewBlogPost, myNewBlogPost.Slug);

Later, if I want to load that blog post, I will need to know the key I used, let's say it was "hello-world":

var myNewBlogPost = stashy.Load<BlogPost>("hello-world");

And there's also a method for loading all objects of a given type:

var myBlogPosts = stashy.LoadAll<BlogPost>();

And for deleting a single object:

stashy.Delete<BlogPost>("hello-world");

And that's it!

It's not distributed. It doesn't have a query language. It doesn't have built in caching semantics. It's not even async.

I tell you what it does have: utility. It's bloody useful! I would definitely not consider it for cases with millions of records, but 1000 or under? It's fine! And one less thing to worry about.

The Stashy class is an implementation of the IStashy interface. This makes it easier to test, and helps me with the dependency injection stuff. Mostly it serves to make sure I'm keeping the contract small.

The IStashy interface is just this:

public interface IStashy<K>
{
    void Save<T>(T t, K id);
    T Load<T>(K id);
    IEnumerable<T> LoadAll<T>();
    void Delete<T>(K id);
    K GetNewId<T>();
}

The type K is the type of key you want to use. I usually use strings as my key. But you can use integers or guids, or anything else, if you like.

As I tweeted recently, The heart and soul of my most used key-value store fits in a single tweet. Easily!

I've used a couple of different implementations of this interface over the years, but currently what it does is serialize the object as JSON and then store it in a file named after the key, in a sub-folder named after the type of the object. (I also have a completely in-memory implementation, for testing etc.)

Here's an implementation I use at the moment in one of my projects. It is not glorious beautiful wondrous code, but it is working code that I just... well I just never have to look at it. But I rely on it all the time:

see gist: FileStashy.cs

It's about 100 lines of code, written years ago, and relies on Newtonsoft.Json to turn objects into Json and vice-versa.

(10 years ago my FileStashy used XML, on Windows... Now it's all Json, and runs anywhere that .net core runs. Maybe one day, one special day, we'll all switch to CSV.)

Sometimes I build other stuff on-top of Stashy, such as indexes, for faster lookups, and "revisions", so I can look at old versions of an object. That was easy enough to write (and very fun of course!) but I wouldn't want to do too much with it, or I'd switch to a real key value store... but then get all the headaches that come from those.

Got any crufty little things you've rolled yourself instead of using something "better"?

How we switched to 4-day weeks

Gingerlime — Thu, 12 Nov 2020 21:28:43 GMT

We recently made a landmark decision at my company. Starting from Friday, 1st January, 2021, all Fridays are now part of the weekend. We’ll be working Mon-Thu all year round. This is still considered an “experiment”, because we don’t know if it’s something we can commit to long-term. But honestly, I can’t imagine going back.…

The post How we switched to 4-day weeks appeared first on Gingerlime.

Building a HomeKit compatible temperature and humidity sensor

Tommaso Amici — Thu, 12 Nov 2020 21:12:35 GMT

In this short post I'll explain how I set up a HomeKit compatible temperature and humidity sensor with a ESP8266 chip, Homebridge, and an AM2302 sensor

Contact tracing failures, again

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 12 Nov 2020 20:00:00 GMT

Nobody cares. But it still HAS value.

Deleting Browser Cache with Python Selenium and Chromedriver

Geography & Coding — Thu, 12 Nov 2020 17:03:58 GMT

Quick and dirty

AI breaks the game of hide and seek

I Learnt — Thu, 12 Nov 2020 14:00:00 GMT

This is a really fun video where an AI is given a version of the game of hide and seek and each team has to learn ways to win - either by finding the other team or by not being found.

Why I use GNU GPL (v3+)

ThatXliner's Blog — Thu, 12 Nov 2020 02:00:00 GMT

Because it's great.

My First PC

Ben E. C. Boyter — Thu, 12 Nov 2020 02:00:00 GMT

At my current workplace we have been doing themed Friday stand-ups. The most recent was about your first PC love, or whatever technology you look back on fondly. What follows is my one.

My first PC was I think a Commodore PC40-III. I’m a little fuzzy on the details and its hard to find it based on pictures. Was not my first taste of technology, but the one I look on back most fondly.

Why you want OPEN System-on-Chips

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 11 Nov 2020 20:00:00 GMT

Because microelectronics is what makes the world go round.

The Light and Dark Side of the API Economy

swyx's site RSS Feed — Wed, 11 Nov 2020 18:07:36 GMT

The 'API Economy' is a popular term for VC's and tech media, however Developers seem ironically out of the loop despite their central importance to the whole story. Here's my explanation, together with a dash of economics and social implications.

Building Product

ferrucc.io — Wed, 11 Nov 2020 11:00:58 GMT

If you’re working on something new, there are so many directions your work and product can take.

The biggest risk is that you build something that is hard to explain and no one wants.

A good process minimizes this risk. Helping you move in the direction of building something people want.

To build June me and my cofounder Enzo used a very systematic approach. We decided to shared it as you might find useful.

Our work on obtaining dissolution profiles from single controlled-release drug particles using vibrating tube sensors published in <i>Scientific Reports</i>

Grover Lab — Wed, 11 Nov 2020 11:00:00 GMT

Heran Bhakta and Jessica Lin’s paper “Measuring dissolution profiles of single controlled-release drug pellets” was published in Scientific Reports.

Heran and Jessica show that a simple and inexpensive vibrating tube sensor can measure the dissolution of single microgram-sized controlled-release pellets in physiologically relevant fluids with nanogram-scale resolution. Their technique addresses many of the shortcomings of existing USP testing methods, requires no additional analytical instrumentation like UV-VIS or HPLC, and is suitable for both fast-dissolving and slow-dissolving formulations. And by obtaining dissolution profiles for single pellets instead of populations of pellets, their technique is capable of measuring pellet-to-pellet variations in dissolution behavior that are much more difficult to measure using existing methods. Using this technique, we observed significant variations in single-pellet dissolution profiles, not only between different types of drugs in different physiological conditions, but also between generic and name-brand formulations of the same drug, and even between different pellets from the exact same capsule.

This technique provides pharmaceutical researchers and producers with a simple, low-cost, and fully automated tool for obtaining single-pellet dissolution profiles from any drug in any desired fluid. This capability should be powerful in a variety of different scenarios. For example, measurements of the dissolution behavior of pellets from each production batch can provide valuable quality assurance data and illuminate possible production defects before the product reaches consumers. Even within a single batch, single pellet dissolution profiles provide information about the consistency of the pellet manufacturing process. And as a gravimetric (mass-based) method, this technique places no constraints on the chemical or physical composition of the fluid surrounding the pellet, meaning that pharmaceutical developers are free to measure pellet dissolution in any physiologically-relevant fluid without fear that the fluid will interfere with the measurement process. For these reasons, vibrating tube sensors should help facilitate the development of better controlled-release pharmaceuticals with better patient outcomes.

Power Costs

Greg Herlein — Wed, 11 Nov 2020 10:00:01 GMT

A quick measure of the costs of running on-premise computers.

i3 Migration Guide

Cookie Engineer's Web Log — Wed, 11 Nov 2020 02:00:00 GMT

Tutorial on how to migrate to the i3 window manager with explanations for typical keybindings, configuration problems and status bar integrations.

The impact of management teams as a decision-making group, in startups and enterprise

Notes on software development — Wed, 11 Nov 2020 02:00:00 GMT

Ambitious companies form management teams at every level above you, sometimes including you. Management teams meet periodically and have private chat rooms. They discuss customers, product and organizational direction. Sometimes discussions are well documented and periodically public. Sometimes decisions are poorly telegraphed out.

Management teams do no inherent harm in a company with customers; employees outside of the management team can unearth customer usage data to discover meaningful places to contribute. For example, graphing historic server logs to discover slowest requests, figure out why and how to fix. Or even just paying attention to the most frequent questions sales asks and finding ways to clarify. (All of this under the assumption that even when there is solid product direction, good employees tend to have extra time at work and want to make good use of it.)

For the first few years even in a well-funded startup with solid founders, there are few customers. Even under a solid product team, the product direction is not yet completely clear. The management team includes founders and non-engineering executives. As a decision making group they are opaque. Employees outside the management team face a barrier in finding ways to meaningful contribute. Ambitious, dedicated folks outside the team leave.

So what?

It is not clear to me how the natural (and not inherently bad) concept of management teams attracts and retains ambitious, dedicated non-founders at small companies. Maybe disenfranchisement is not important, or even necessary.

Or maybe management teams as a decision-making group are too easily a substitute for developing a grassroots culture of collaboration and trust between marketing, sales, product and development.

New post! "management teams as a decision-making group are too easily a substitute for developing a grassroots culture of collaboration and trust between marketing, sales, product and development."https://t.co/7RukBMI59h
— Phil Eaton (@phil_eaton) November 16, 2020

Watch out! Vokenization is coming!

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 10 Nov 2020 20:00:00 GMT

Computers are learning the colour of sheeps. Here is why it matters.

The growing power of influencers

I Learnt — Tue, 10 Nov 2020 14:00:00 GMT

Quite often I will come across long articles and either put off reading them or just skim through their content. This article is well worth taking the time to read as it completely changed my views on where the media industry is heading. My notes here do not do it justice but will hopefully give you enough to encourage you to read it in full.

My thoughts on PiHole

Heitor's log — Tue, 10 Nov 2020 02:00:00 GMT

My experience with PiHole after one month.

Rust as a productive high-level language

Omar Abid - Personal Blog — Tue, 10 Nov 2020 02:00:00 GMT

High-level productivity with Rust

Michelin Stars for Celiacs: Angler, San Francisco: Not recommended

rob.sh — Mon, 09 Nov 2020 23:30:00 GMT

As some folks might be aware, I became allergic to gluten and dairy through the spring of 2019, and have been avoiding them in my diet since. Becca and I have not lost our passion for eating great foods, and visiting some of the fantastic restaurants in the cities we visit (although, in 2020, this has meant a lot of places in the Bay Area). One of the things that we’ve struggled a bit with is finding out how well some of the Michelin star establishments handle the gluten-free, dairy-free diet. This weekend, after a visit to Angler - down on the Embarcadero, here in San Francisco - we discussed that it’d be useful if there was more information available for folks that are celiac and interested in fine dining. So this is the first post – I’ll try and add some reviews of previous places we’ve eaten through the next few weeks.

The coming, dangerous hyper-segmentation of "news"

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 09 Nov 2020 18:00:00 GMT

It’s bad, and it is on ALL sides.

Borat Subsequent Moviefilm

🦉 olshansky 🦁 — Mon, 09 Nov 2020 10:28:10 GMT

Olshansky's review of Borat Subsequent Moviefilm

Individualism Vs Collectivism

Reticulated — Mon, 09 Nov 2020 02:25:01 GMT

It’s odd when you think about how most of your understanding of the world, your political views, your goals, and much more is defined by just a few experiences. I can lay out the path from being born a southern, Christian-by-default, Republican-by-default kid to a Democratic Atheist person whose not sold on the virtues or unendingness of Capitalism. Funnily, the International House of Pancakes has quite a bit to do with the start of all that, but I’ll save some of those stories for another post.

Towards a Lightweight Jamstack

Nicolas Goutay — Mon, 09 Nov 2020 02:00:00 GMT

This article aims to give directions to curb the JavaScript tax in the Jamstack.

The modern editor war

ThatXliner's Blog — Mon, 09 Nov 2020 02:00:00 GMT

Goodbye, Emacs

Dissonance: Your Tolerance to Change

A Matter of Perspective — Mon, 09 Nov 2020 02:00:00 GMT

👈 Back to all conditions

Our third condition that unlocks the door of real change is the ever-so-stubborn beast of Dissonance.

What’s Dissonance, you may ask? Put simply, how much does the boat need to rock before a change occurs? Do you feel enough tension to get off your but and do something?

Dissonance: The awareness of a gap between life’s present conditions and our current means of solving them.

Put bluntly, Dissonance is the fire under you that makes you move your ass when it gets too hot.

Rocking the boat

Have you ever had one fight too many in your relationship? You know that big one? The one that leaves you thinking, “That’s it! No more”.

What about at work? One task too many, or failed project later, and you feel it’s about time to move on to greener pastures?

Perhaps the climate gets too hot, and you move towns. Or you get enough comments from your friends that it’s time to give up partying and settle down.

In our image above, Mr. Blue comes home to one too many of Mr. Red’s parties. He’s thinking it might be time to find a new flatmate.

We are not strangers to Dissonance. It’s often the driving force that pushes us into new life choices, relationships, ideas, beliefs, and even towns!

Before the moment of change sets in, we feel the tension.

But it’s not yet painful enough for us to do anything about it. Depending on our pain tolerance, Potential, and how much we care about the situation, we may stay where we are for a good while. We’re more stubborn than we give ourselves credit for, us humans.

Many of us have things we want in our lives, but we don’t have a strong enough reason to realize them.

The fire beneath us is not hot enough to warrant moving away. At least for now…

The amount of disharmony (Dissonance) needed to create change is determined by the willingness to change.

How stubborn are you?

Getting to know your tolerance for Dissonance

We all have a level. It’s that point before we break and make something happen. When we get pushed too far, or the world is too much, and where the thermometer bursts. We make a change. Sometimes even a snap-decision!

Of course, we can have different levels for different parts of our life also!

I may have a high Dissonance tolerance for my relationship. This could mean it would take a lot of heat (strong Dissonance) to change the dynamics.

I could also have a low Dissonance tolerance with work, implying I’m jumping from companies all the time due to my frustrations with company politics.

Sometimes these levels work for us, and other times they don’t. Sometimes high tolerance is good, and sometimes it’s not.

Your tolerance could be too high if your mirror’s reflection starts saying:

“I’ve just gotta stick with it” when everyone’s telling you to relax.
“He’ll change one of these days, I know it!” when your friends are praying that you’ll leave him.
“I refuse to change” when the world is demanding it

And your tolerance may be too low when it says:

“This place is already on my nerves” when you haven’t even worked a month at the new job.
“She’s not the same as when we first met”, but you’ve only been dating 2 weeks.
“I give up” when you’ve only just started.

Using Dissonance

As always, A Matter of Perspective likes to provide practical tools take us away from theory alone. How might we employ Dissonance as a catalyst for change?

The first step is to assess the “Potential” of a given system (the first Condition of Change). If the system/person/group is Closed, a heavy dose of Dissonance may be required to shake the shackles loose. It may not even be worth the effort.

However, if they remain Open to new ideas and ways of life, then the boat requires less rocking. They are already willing to change and half out the boat.

In the case that Dissonance IS needed, what might that look like? Consider a personal coach that picks up a client early every morning. Should the lazy client decide they want to sleep in, the coach promptly holds the horn until they drag themselves out of bed and into the car. He is creating such conditions in life that will force you to change. The neighbors love him.

Another example may lie in a corporation attempting to shift it’s work culture. At a management level, it may decide to hire a very new type of employee in hopes of stimulating fresh ideas and energy. With enough new people pushing for a new idea, the old ones may begin to loosen, creating a shift towards a different culture. All provided they’re open, of course 😉 (they’re often not, and this creates a big dilemma).

In our own lives, we may use this key-of-change to push ourselves. Getting ourselves to a meditation retreat when we least want it and most need it. Throwing out all the junk food. Giving ourselves a vacation. Depending on your situation (and if it’s required), there may be many ways to inject a little Dissonance. To put some fire under your belly.

Provided you need to change. And can change.

Often, we can just enjoy where we are and go along for the boat–ride.

👈 Back to all conditions

How-to revert HP printer firmware ban on 3rd-party toner cartridges

Kevin Deldycke — Mon, 09 Nov 2020 01:00:00 GMT

Hewlett & Packard, the founders, had great lessons to teach us (managers in high-tech) about culture. To the extend of quoting them in my awesome list on engineering team management . 👨‍💼

HP Inc., the company, sucks. At least their printer division’s business model . They recently pushed a firmware update to ban …

DSA-2020-209: Dell XPS 13 9370 Improper Exception Handling vulnerability

Sayr.us — Sun, 08 Nov 2020 19:20:00 GMT

In August 2019, I’ve discovered a bug in how the XPS Bios handles a bootable device. This bug allows an attacker with write permission to a bootable device or physical access to the computer to prevent it from booting.

Ferrari F12 Berlinetta

smcleod.net — Sun, 08 Nov 2020 18:00:00 GMT

Last weekend I had a chance encounter with a Ferrari F12 Berlinetta and the owner kindly let me jump in and take a few photos.

Specs

6.3L v12
544 KW (740 Horsepower)
690 Nm at 6000RPM
1525 kg
0-100 in 3.1s
0-200 in 8.5s
Maximum speed: ‘over 340 km/h’
Was ‘The fastest road car Ferrari ever built’

Photos

The loosely held product roadmap

I Learnt — Sun, 08 Nov 2020 14:00:00 GMT

In a recent article by Ryan Singer, Options, Not Roadmaps, he explains why he doesn’t use roadmaps - he uses options instead.

Protecting TimeMachine backups from itself

Gingerlime — Sun, 08 Nov 2020 11:54:20 GMT

Going down the time machine rabbit hole… I love the fact that MacOS comes with TimeMachine built-in, and I also really appreciate its simplicity. It makes backups easy and accessible even for non-technical people. It gets messy though if you also want to have real offsite backups however. TimeMachine works great with a USB external…

The post Protecting TimeMachine backups from itself appeared first on Gingerlime.

Halloween Coding Adventure

Acbart ePortfolio — Sun, 08 Nov 2020 02:00:00 GMT

This has been a challenging semester for my students. I am teaching CISC108, the introductory class for Computer Science majors. The class is tough enough, even for many students with prior programming experience. But, due to the Global Pandemic, this Fall 2020 semester is online. In addition to my existing curricular framework, I have a lot of new course policies meant to make the experience easier: no exams, flexible deadlines, and a huge team of excellent TAs giving 12:1 attention. But even with all this, my students are all still struggling to stay engaged.

Kubernetes tips

Brent's Blog — Sun, 08 Nov 2020 02:00:00 GMT

The problem is solved by removing the leading slash after the podname colon separator:

Enterprise tips for AWS Workspaces and AWS Managed AD

Brent's Blog — Sun, 08 Nov 2020 02:00:00 GMT

::: note ::: title Note :::

I made a website to guess tomorrow’s number of COVID-19 cases, and here’s what happened

code · words · emotions: Daniel Janus’s blog — Sun, 08 Nov 2020 02:00:00 GMT

Before

It seems so obvious in hindsight. Here in Poland, people have been guessing it ever since the pandemic breakout: in private conversations, in random threads on social media, in comments under governmental information outlets. It seemed a matter of time before someone came up with something like this. In fact, on one Sunday evening in October, I found myself flabbergasted that apparently no one yet has.

I doled out $4 for a domain, koronalotek.pl (can be translated as “coronalotto” or “coronalottery” – occurrences of the name on Twitter date back at least as far as April), and fired up a REPL. A few hours and 250 Clojure LOCs later, the site was up.

I wanted it to be as simple as possible. A form with two fields: “your name” and “how many cases tomorrow?” A top-ten list of today’s winners, sorted by the absolute difference between the guess and the actual number of cases, as reported daily on Twitter by the Polish Ministry of Health. The official number, prominently displayed. And that’s all.

On 17 October, I posted the link on my Facebook and Twitter feeds, and waited. The stream of guesses started to trickle in.

After

It never grew to be more than a stream, but it hasn’t gone completely unnoticed either.

The above plot shows daily number of accepted guesses (i.e., those that were used to generate the next day’s winners) over time – a metric of popularity. Each day’s number means guesses cast in the 24 hours up until 10:30 (Warsaw time) on that day, which is when the official numbers are published by the Ministry of Health.

I’ve been filtering out automated submissions, as well as excess manual submissions by the same IP that seemed to skew the results too much – I’ve arbitrarily set the “excess” threshold at 10. The missing datapoint for 19 October is not a zero, but a N/A: I’ve lost that datapoint due to a glitch. More on this below.

The interest peaked on October 23, with more than a thousand guesses for that day (I think it was reposted by someone with a significant outreach back then), and has been slowly declining since.

I have privately received some feedback. One person has pointed out that they found the site distasteful and that making fun of pandemic tragedies made them uncomfortable. (I empathise; for me it’s not so much making fun as it is a coping mechanism—a way to put distance between my thoughts and the difficult times we’re in and to keep fears at bay.) Some people, however, have thanked me for making them smile when they guessed more or less correctly.

Back to data. Being a data junkie, I looked at what I had been collecting. First things first: how accurate is the collective predictive power of the guessers?

Quite accurate, in fact! Data for this plot has only been slightly preprocessed, by filtering out “unreasonable” guesses that don’t fall within the range [100; 50000].

People have over- and underguesstimated the number of new cases, but not by much. There were only a few occasions where the actual case count didn’t fall within one standard deviation of the mean of guesses (represented by the whiskers around blue bars on the plot). Granted, the daily standard deviation tends to be large (on the order of a few thousand), but still, I’m impressed. A paper on estimating the growth of pandemic based on coronalottery results coming soon to a journal near you! ;-)

Just for the heck of it, I’ve also been looking at individual votes. Specifically, names. Here’s a snapshot of unique guessers’ names sorted by decreasing length, on 23 October. (NSFW warning: expletives ahead!)

Let me translate a few of these for those of you who don’t speak Polish:

1 is “Sasin has fucked over 70 million zlotys for elections that didn’t take place and was never held responsible.” This alludes to the ghost election in Poland from May. This news had gone memetic, going so far as Minister Sasin’s name being ironically used as a dimensionless unit of 70 million (think Avogadro’s number). You’ll discover the same theme in #2, #3, #5, and others.

6 is “CT {Constitutional Tribunal}, you focking botch, stop repressing my abortion”. Just a day before, the Polish constitutional court (whose current legality is disputed at best) has decreed a ban on almost all legal abortion in Poland, giving rise to the biggest street protests in decades.

Not all is political: 4 is “Why study for the exam if we’re not gonna survive until November anyway?”. I hope whoever wrote this is alive and well.

Corollary? Give people a text field, and they’ll use it to express themselves: politically or otherwise.

In fact, I have taken the liberty of chiming in. Shortly after, I altered the thank-you page (which used to just say “thanks for guessing”) to proudly display one of the emblems of the Women’s Strike, along with a link to a crowdfounding campaign for an NGO that supports women needing abortion.

Inside out

I’m not much of a DevOps person, so I deployed it the quick and dirty way, not caring about scalability or performance. The maxim “make it as simple as possible” permeates the setup.

I just started a REPL within a screen session on the tiny Scaleway C1 server that also hosts this blog and some of my other personal stuff. I launched a Jetty server within it, and set up a nginx proxy. And that’s pretty much it. I liberally tinker with the app’s state in “production,” evaluating all kinds of expressions when I feel like it.

Code changes are deployed by git pulling new developments and doing (require 'koronalotek.core :reload) in the REPL.

Someone tried a SQL injection attack. This is doomed to fail because there’s no SQL involved. In fact, there’s no database at all. The entire state is kept in an in-memory atom and periodically synced out to an EDN file. In addition, state is reset and archived daily at the time of announcing winners. (I’ve added the archiving after forgetting it on one occasion – hence the lack of data for 19 October.)

I also don’t yet have a mechanism of automatically pulling in the Ministry of Health’s data. Every morning, I spend two minutes checking if there’s excess automatic votes, removing them if any, and then filling in the blanks:

(new-data! #inst "2020-11-08T10:30+01:00" 24785)

For all the violations of good practices in this setup, it has worked out surprisingly well so far. I’ve resorted to removing automated votes a handful of times, and blacklisting IPs of voting bots in the nginx setup twice, but otherwise it’s been a low-maintenance toy. People seem to be willing to have fun, and I’m just not interfering.

Takeaways

You should call on your country’s authorities to exert pressure on the Polish government to respect women’s choices and stop actively repressing them.
Give people a text field, and they’ll use it to express themselves.
Release early, release often.

De-mainstream YouTube? Bad idea

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 07 Nov 2020 18:00:00 GMT

Seriously. Bad idea. In good faith, but bad.

Day of Automation

trv.rs — Sat, 07 Nov 2020 11:00:00 GMT

I thought I'd record the automations that I've written that I use in a normal workday… I started a thread on automators.fm for this, hopefully someone else will take this up and I'll learn some neat things from y'all!

My alarm wakes me up… depends on the day there's different alarms (that's an automation, right?)

I step on my (formerly "smart" but now broken!) scale and use a shortcut to log weight to Apple Health. It then uses Charty to display a custom graph showing me how far I have to go til my goal weight.

Today was a run day… so I grabbed my running gear and drove out to the route I run.

I have a master "headspace" shortcut for all the things on my phone… I choose "Workout" and it runs the workout I have scheduled for that day.

After my run, it's time to have my quiet time: I run my Quiet Time headspace on my phone, it launches Dark Noise to a sound I only play during quiet time, and opens the app that controls the camera in my daughter's room so I can see when she's stirring and swoop her up before she wakes my wife.

I have several recurring to-dos and lists to help me to plan my day, write the Monday blog post for work, etc.

Once at my desk, I use Hyper to open a few apps and clean out a few emails.

Then I used Headspace to enter meeting mode for standup.

Each Monday, I presently write a mini-blog post at work in our wiki.

I use Alfred's text expansion feature to generate the post title (,,puweekly becomes UXWEEKLY (2020.43).
I build the post by pulling the links from a "workspace" in Drafts using the links I'd saved to drafts using the Hammerspoon snip tool I built.

Today happens to have lots of meetings…

A quick script I wrote in Hammerspoon tied to my Hyper key allows me to focus the tab with the google meet. I use it constantly to help me unmute.
I have Autolayout rules that put the window with the meeting tab on the monitor that has the camera.
Headspace is tracking the time spent in meetings to Toggl Track.
Headspace is also blocking distracting applications like Reeder, so that I can stay focused.
Hyper is letting me log little notes into Drafts.
I do have some Drafts automations for tying to a meeting note… so that it get's saved into my Zettelkasten in the right format, but I don't use that too often.
Hyper is letting me push to-dos to Things inbox.
I open a design file, and my QMK custom layout tricks let me use my keyboard as a macropad, changing tabs, zooming, etc.

Meetings are over! Time to go to lunch… I run shutdown headspace that stops tracking, clears the blocked apps, and closes all apps so that I have a fresh chance at being focused when I get back.

More of the same as before lunch… I do take a break, running my "Rest/Play" Headspace so that Reeder and some other distractions are unblocked. I can now set a timer in my headspace by prefixing integers with :, so I use that to for 15 minutes. play :15.

My energy is really low… so I run the "Shallow" time headspace to unblock some distractions, but it also loads up all the work to-dos tagged $Low. Sure enough, I find some good ankle-biter tasks to work on. I'm encouraged and energized by the momentum to get into some better, deeper work.

At the end of the day, I run through my review checklist to review and plan the next day.

I open my Headspace scheduling automation to put to-dos on the calendar for tomorrow.

The day done, I run Shutdown again to stop timer and close apps for a fresh focused start tomorrow.

This was a fun exercise... it was both super interesting to see just how much of my tooling is totally custom to me, and way more satisfying to see that I've already written about and documented a lot of it. In the days since I recorded this, I've realized even more automations that I haven't covered... more iOS stuff, lots of dotfiles automations to keep my computers synced and current... a lot more to share!

Temporal Transparency Update

swyx's site RSS Feed — Sat, 07 Nov 2020 05:04:37 GMT

Our 9th transparency update describing how we are shipping the upcoming Authentication feature

Demystifying malloc

River Gillis [riv.dev] — Sat, 07 Nov 2020 04:29:20 GMT

A journey through the implementation of malloc.

The Man Who Solved the Market: How Jim Simons Launched the Quant Revolution

🦉 olshansky 🦁 — Sat, 07 Nov 2020 02:00:00 GMT

Olshansky's review of The Man Who Solved the Market: How Jim Simons Launched the Quant Revolution by Gregory Zuckerman

Stranded • Art

Muffin Man — Sat, 07 Nov 2020 02:00:00 GMT

Created:	November 2020
Size:	42x23cm
Paper:	Fabriano Bristol 250gsm
Pens:	Pigma Micron, Sakura Gelly Rolls

Stranded • Art

Muffin Man — Sat, 07 Nov 2020 02:00:00 GMT

Created:	November 2020
Size:	42x30cm
Paper:	Fabriano Bristol 250gsm
Pens:	Pigma Micron, Uni-ball Signo UM-153, Mechanical pencil

The Hidden Cost of Streaming Music: is it really IN music?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 06 Nov 2020 20:00:00 GMT

Me, I am not sure.

Naked Emperors in Tech

swyx's site RSS Feed — Fri, 06 Nov 2020 15:01:55 GMT

Some things we often repeat as truth just aren't. We should call bullshit more often.

New ways to generate ideas online

I Learnt — Fri, 06 Nov 2020 14:00:00 GMT

There are a lot of sites that claim to be idea generators. Most of them turn out to be just a button you click that slings together a bunch of random words that are meant to then become a prompt to generate ideas.

One-Button Workout Shortcut

trv.rs — Fri, 06 Nov 2020 11:02:00 GMT

Any area of your life where there are regular patterns or discipline is ripe for automation... like exercise!

Alarming developments...

I’ve going to f3birmingham.com workouts, and I’ve got a shortcut to remind me to log the workout to Strava. I don’t go every time, so I just figured out a way to automate it: fire the automation only when the alarm I set to attend in the morning goes off.

The automation in the “Do” block looks like:

One button

On all the other days, I used to have a "choose one" shortcut that let me pick what workout I wanted to do from a list. I was using the iOS14 "choose from Shortcut folder" action, so I have separate Shortcuts for each type of workout I do.

Last night I realized that I could make life even easier on my sleepy self... I have a consistent schedule for my workouts, so if I know in advance what workout I'm going to do in the morning, I can program that in!

So I wrote a new workout shortcut:

I have a shortcut that builds that dictionary in Data Jar to “set my schedule”

That means if I want to tune a particular workout, I just change its shortcut in that Workouts folder… I’ll probably make one for stretching or yoga and just add it, and I can change my schedule!

This means that when I wake up, I hit workout and the app opens, playlist starts, all that jazz.

It's easy.

On mornings when I commit to F3, my phone will automatically remind me to log my workout.

On all other mornings, I hit one button and it does exactly what I want. I'm basically using iOS Shortcuts to do a User Experience style experiment on my morning routine. 💪

Lang. Identification in Code Mixed Text - Lit. Review 3

Playground — Fri, 06 Nov 2020 02:00:00 GMT

I am reviewing the literature available on language identification for multilingual documents, focusing on Indic languages. I’ll try to cover it in chronological order, but there might be a few misses here and there. After a decent coverage of the research, I expect to have enough understanding to discuss the challenges present in this task in a code-switched setting and its importance.

When Joe Biden becomes the next president this site will beep

ferrucc.io — Fri, 06 Nov 2020 00:20:05 GMT

As requested by Hunter Walk

Data sourced from the NYT

Just because it isn't Closed, Doesn't Mean It Is Socialism

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 05 Nov 2020 20:00:00 GMT

Is it so hard to see it, with software or anything else?

Rise of the Bandits

one mikro2nd — Thu, 05 Nov 2020 15:19:38 GMT

One of the most potent signs of a disintegrating state is the rise of independent militias and the ruling government’s inability/unwillingness to eliminate them. Now read today’s DM lead article: Radical document by ‘retired generals’ spurs Gwede Mantashe into action. Huh. Who woulda thunk. Retired generals want to stage a military coup. Supplize. We already have a variety of part-time militias, some of mysterious provenance, some pretty obvious. There are the metropoles’ various Police “forces”, the so-called Red Ants (who is behind them, anyway?

Mining Instagram location IDs

Geography & Coding — Wed, 04 Nov 2020 21:37:39 GMT

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{
 "Jena": "673710",
 "Jena-Zentrum": "1006125655",
 "Plan B Boulderhalle Jena": "613227777",
 "Kassablanca Jena": "5752477",
 "Kulturarena Jena": "574057843",
 "University of Jena": "251848169",
 "Paradiespark Para - Jena Paradies": "228419929",
 "Ernst-Abbe-Sportfeld": "326665551",
 ...
}

How to quickly mine location IDs from Instagram’s location explorer.

Let's read Libra Shrugged

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 04 Nov 2020 20:00:00 GMT

This seems like the next book I (and you too) should read.

Questions to ask yourself before you buy

I Learnt — Wed, 04 Nov 2020 14:00:00 GMT

Sometimes it is very easy to buy things just for the sake of it. We get that rush when either we bring the item home or it is delivered. It is easy to form an addiction to that feeling.

Using CloudWatch for On-Premise Ubuntu Servers

Greg Herlein — Wed, 04 Nov 2020 10:00:01 GMT

Can I replace a local Prometheus with AWS CloudWatch? And include syslog monitoring as well? Yes, I can. But it was a PITA to set up. Here’s how.

How To Improve Your Luck

JonPaulUritis.com — Wed, 04 Nov 2020 02:00:00 GMT

In September 1928 a bacteriologist name Alexander Fleming came back from a "holiday" (vacation) to find a set of petri dishes which had been "necessarily exposed to the air" had become contaminated by various micro-organisms. One of those petri dishes had grown a mold that appeared to be inhibiting the growth of a colony of Staphylococcus aureus. (You can read the original paper here)

To quote Dr. Fleming: “When I woke up just after dawn on September 28, 1928, I certainly didn’t plan to revolutionize all medicine by discovering the world’s first antibiotic, or bacteria killer."

You've probably heard this story before. However, there is a second part to the penicillin story that is told a lot less.

What happened following the discovery on the part of Dr. Fleming was practically nothing. He published a paper on his findings, and that was pretty much it. Sure he did some more research in the area, but he ultimately concluded that his discovery wouldn't be useful for treating infection and wasn't worth pursuing further. Effectivity his part in the story stopped right after publication.

I want to pause to let that sink in for a second.

Not many people realize that it is entirely possible that the publication could have been the end of the story... or worse it might not have even made publication. If that were the case no one would would have benefited from perhaps the greatest life saving discovery in the 20th century.

What did happen (a decade after publication) was that an Oxford Scientist named Dr. Howard Florey was reading back issues of The Journal of Experimental Pathology and ran into Fleming's article. It interested him enough to try and reproduce the work, eventually culminating in a streptococcus study of mice in 1940 with Dr. Ernst Chain & Dr. Norman Heatley. That study produced verifiable results that the production of antibiotics were indeed possible (though engineering efforts were required to make mass production viable).

How narratives shape our perception

Dr. Fleming's story is clearly in the genre of "dumb luck" (supposedly he even used the most famous phrase in all of science: "Huh. That's funny..."). Sure, he was a trained bacteriologist, and he had been working on infectious disease studies for a long time, but the mold literally fell out of the air perfectly into his petri dish.

Dr. Florey's is on the other side of the "narrative" coin. He built a top lab at Oxford through hustle and grit. According to a PBS article on Florey he was "a master at extracting research grants from tight-fisted bureaucrats and an absolute wizard at administering a large laboratory filled with talented but quirky scientists." His "discovery" of the Fleming study was the result of searching, learning, and accumulating resources to be able to leverage.

These two versions of the "luck narrative" form a dichotomy for the way people like to look at luck. Either 1) serendipity is out of our control and "the gods will gift who they deem worthy" or 2) we can make our own luck through hard work, and determination.

What is luck anyway?

So the following is what is called the "Probability Mass Function" (one way of defining "luck"):

$$ f(x,n,p) = \frac{!x}{n!(x-n)!}p^n(1-p)^x-n $$

# Defined
p = probablility of success
x = is the number of attempts
n = the number of winning attempts / successful attempts

To be "lucky" in the mathematical sense is where an agent achieves a successful (or unsuccessful) outcome that exceeds its probabilistic expectation given an environment. (link)

"Luck", in the colloquial sense, is the result of chance. It's a surprise, an unexpected event. You have "bad" luck when bad (unexpected) things happen to you or "good" luck when good things happen to you. Luck in this sense is all around us, impacts us all, with the only difference being how much it effects us.

[This colloquial luck has powerful psychological effects too... Higher socioeconomic status is correlated with lower belief in luck and vice-versa (e.g. research). ]

How can one improve their luck?

So how can you improve your luck?

...Well, you can't. Thanks for reading my article! (just kidding)

That is to say, you can't improve your luck in a "mathematical sense." Luck in that sense is actually random... otherwise it wouldn't be luck.

"Luck" in the colloquial sense however is arguably more about narrative than probability. The magnitude of the outcomes in "colloquial luck" create the perceived "luckiness." Receiving an "asymmetric payoff" makes you appear to be more lucky. So if you are interested in improving your luck in the colloquial sense you can accomplish this by exposing yourself to environments that have outsized payouts.

...Ergo the process of "Improving your luck" is simple. Marry together what we know about mathematical luck and colloquial luck.

Start by finding environments that can provide outsized payouts. I cannot stress enough how important it is to FIND THE RIGHT ENVIRONMENT, as it will basically be the single largest influence on your probability of success. Searching for these sorts of environments takes work. You need to figure out as accurately as possible what your probability of success will be, so that you can commit to working in that environment... which is no easy task.

After you have chosen the right environment you want to search for ways to improve the probability of being successful and execute them. More often than not the simplest way to improve the probability of a successful outcome is just to increase the number of attempts. But similarly, there are often ways to decrease the total number of attempts needed to be successful- basically influence your environment in a way that is favorable to you.

And of course the last piece of the puzzle is to figure out ways to increase the narrative value when you are successful. Being perceived as being lucky is a powerful attribute and can help you improve your "luck" for future opportunities.

Hundreds of millions of lives saved...

When looking back at Dr. Howard Florey, his search through back issues of the Journal of Experimental Pathology was effectively his way of choosing the environment he wanted to work in. He was beginning in a place that already had some level of success and he knew he could increase the chances of success by starting with something that was already somewhat successful. The work of Alexander Fleming wasn't just a starting place- it reduced the total number of attempts it would take to be successful... all Florey had to do was put the work in and there was a good chance that luck would find him. The final piece of the puzzle was that Florey was choosing to work on something that had an outsized payout or impact.

...If you want it distilled, here's your recipe for improving your luck:

Select what you work on carefully.
Make sure that you can be successful and that the payouts are outsized.
Try to find ways to reduce what it takes to be successful.
Be focused and work hard.

Fight or flight? Breathe instead

Alexander's Blog Blog — Wed, 04 Nov 2020 02:00:00 GMT

How to achieve the best attitude at work with integrity and self-awareness

A random list of ten Python standard library modules you must learn know

ThatXliner's Blog — Wed, 04 Nov 2020 02:00:00 GMT

This is a *very short* ~~blog post~~ list of 10 Python standard library modules (that I think) you **must know** (as of python3.9).

Specular Realms founded.

Specular Realms — Wed, 04 Nov 2020 00:39:45 GMT

With blood, sweat and tears, this company is finally off the ground. Now let the money start rolling in baby because I have the combined business acumen of the underpants gnomes combined with Sam the Eagle.

The post Specular Realms founded. appeared first on Specular Realms.

A quick portrait of digital gerrymandering

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 03 Nov 2020 20:00:00 GMT

Why TODAY? Well, why NOT?

Nationality as a Service

Non-Human Party — Tue, 03 Nov 2020 17:25:05 GMT

As a sentient being, thinking about your own existence and your place in the world; you must’ve asked yourself what runs the world. Money! That’s what it used to be, but now, it’s undeniable that software has to be included in the answer too.

Exposing the beauty in a problem

I Learnt — Tue, 03 Nov 2020 14:00:00 GMT

I was recently reminded of this great quote:

“When I’m working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong.” - R. Buckminster Fuller

Guidelines for choosing React Libraries

Max Rozen — Tue, 03 Nov 2020 11:48:48 GMT

There are a *lot* of React libraries out there. So how do you pick which one to use in your app? This article provides guidelines for choosing.

Single file applications

Posts on rand[om] — Tue, 03 Nov 2020 02:00:00 GMT

Storytelling is part of the human essence. Stories have let humans survive until today, they became the medium to move information between individuals.

Programming can be considered another form of moving information. With different levels of abstraction, programming becomes a medium to transfer information between humans and computers.

A programming paradigm mixing those two concepts is literate programming. According to its definition:

Literate programming is a methodology that combines a programming language with a documentation language, thereby making programs more robust, more portable, more easily maintained, and arguably more fun to write than programs that are written only in a high-level language. The main idea is to treat a program as a piece of literature, addressed to human beings rather than to a computer. ¹

Uvinji

Ben Overmyer's Site — Tue, 03 Nov 2020 02:00:00 GMT

As soon as the pre-expansion patch launched, I fired up World of Warcraft with the intent to create my new main character and begin my adventure anew.

Well, that first night there was a bit of a hiccup, as everyone else had the same idea. It took a lot longer to unlock the Vulpera allied race than I thought it would, as there was a quest chain for me to complete and it was very buggy. On top of that, I kept getting logged out. Eventually, I gave up for the night and logged back in the next day. Everything went smoothly after that.

I created Uvinji, my Vulpera Shaman, and logged in. I expected to start at the new new-player island, but instead it started me off at level 10 and skipped all of that. I was a little disappointed, but oh well.

After a little bit of running around Orgrimmar, I went on my way to Northrend to do the Wrath of the Lich King content. It was fun, at first, especially because I started in an area that I'd never actually done before. I made my way through all the quests in the Borean Tundra and ended up in Dragonblight.

At this point, I'd leveled up to 31. With the level cap being 50 now, that was quite a bit of progress. However, at this point, I got bored of the grind. The content was familiar, and honestly, I hate the Scourge and all of the undead-filled areas. They're very monotone in color and boring to level through. So, I used my one-time Shadowlands level-up-to-50 token on Uvinji to power him all the way up, and went through the "you're a brand new 50" tutorial.

I'm looking forward to the pre-expansion event that starts next Tuesday. Here's hoping it's more exciting than the last time the Scourge event happened...

On the Storing of Dates

The David Brownman Blog — Tue, 03 Nov 2020 02:00:00 GMT

In all of my personal record keeping, I use a slightly unusual method of tracking the date. Learn how I do it and what problems it solves

Read the whole thing.

DKIM: Show Your Privates

rya.nc — Mon, 02 Nov 2020 22:02:00 GMT

If you’re like most people, there’s a good chance that it’s been years since you’ve sent an email that wasn’t cryptographically signed. You don’t use PGP, you say? Well, even if you are not signing your email, your provider is almost certainly doing it for you. Plausible deniability has been tossed aside in the name of stopping spam, but it doesn’t have to be.

On Being a Postdoctoral Researcher

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Mon, 02 Nov 2020 21:00:00 GMT

This is one of the posts I wish I had available when I was pondering options¹ after my Ph.D. So if you, dear reader, are in the same position, I wrote this just for you.

Postdocs or postdoctoral researchers are arguably the middle children of academia—not getting the attention Ph.D. students should be getting while also not counting as faculty, making them ineligible for numerous grants and other programmes. Many academics therefore see this time as something that is to be endured until the ’next step’ finally beckons, i.e. they become faculty members. I want to offer another perspective here, viz. one of cherishing this time.

Here’s why: as a postdoc in a good lab,² you are in a perfect position. First, you enjoy a certain freedom to pursue your own research directions, create your own own research agenda, and in general just play around with whatever tickles your fancy. This is more freedom than you probably had as a Ph.D. student, and it can be extremely liberating (so much so that it might make you dizzy at first). Second, you can also assume more responsibilities! You can serve as a mentor to Ph.D. students and undergrads. You can learn how to supervise them and how to manage projects. This is a privilege, but it also carries a certain weight—suddenly, you are supposed to be the experienced person in the room. The saving grace of this situation is that you are still in a protected position! You are not responsible for running a whole lab; you are not responsible for acquiring more funding for all your employees; you are not responsible for ensuring future collaborations.

This is why I think that a postdoctoral position can be an excellent one, having just the right amount of responsibilities and competencies. Or, to paraphrase the inimitable Terry Pratchett:

It is a very useful rank for the right person. The optimum balance of power and responsibility.

Therefore, enjoy and cherish this time as long as it lasts. And if you are still on the fence whether you should look for such a position: consider the postdoc to be a ‘taste test’ for academia, directing research, and mentorship. That way—regardless of what your personal next step should be—you will be able to make an informed decision.

What you should not do is treat your postdoctoral time as a ‘Ph.D. 2.0.’ By this I mean that you should not just continue your research as a ‘lone wolf.’ Treat this time as a period of growth; try out different directions and see where they will take you in the end. And most of all: enjoy yourself.

Good luck with this incredible adventure, until next time!

I had even less of a foggy idea of academia than I have now. ↩︎
Meaning one that is run by a supervisor who cares about their employees. I concede that there are some places you do not want to work at. ↩︎

Here is my hippocratic oath

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 02 Nov 2020 20:00:00 GMT

You should take one too. It’s NOT about medicine.

Fractional talent as an alternative pool of resources

I Learnt — Mon, 02 Nov 2020 14:00:00 GMT

At the startup I work for we used to have a full time HR person. These were in the days when the expectation was that we would have explosive growth and the company would “at least double in size every few months”. Unfortunately that is not the way it played out - we are still making good progress but have much more realistic growth plans.

Books and Links: October

trv.rs — Mon, 02 Nov 2020 12:03:00 GMT

This year is flying by... I can't believe how quickly this month has come and gone. Grateful for a lot of beautiful days spent outside with those that I love.

I did get some writing done this month... I have adjusted my system to where I have a big folder of wild, half-written idea sentences, and an area where I continue to edit and develop those ideas, and I think that's unblocked me a little bit when it comes to writing.

I do owe many of you an update to my Simple Markdown Zettelkasten post... honestly I started a transition between two styles of ZK and kinda got stuck. Fortunately, both have stable IDs in the content of the file, so it's still usable... it just offends my data aesthetic. 🤷‍♂️

Books

I've been catching up on some web reading and some writing, so I haven't done as much reading this month.

Currently reading: Knowing God, Whiskey Tango Foxtrot, Programmer's Introduction to Mathematics.

Links

Interesting Thoughts

Before the Web: The 1980s Dream of a Free and Borderless Virtual World (Pt.1) - YouTube

Part one of a four-part documentary series about the cypherpunk movement of the 1990s, which led to the creation of bitcoin, WikiLeaks, and more.

Originally linked on Marginal Revolution, I enjoyed this dive into the philosophical underpinnings of the early cypherpunk folk… makes me want to read Cryptonomicon again.

(And reinstall PGP or something!)

Before the Web: The 1980s Dream of a Free and Borderless Virtual World (Pt.1) - YouTube

America in Decay: The Sources of Political Disfunction

What was arguably one of the most important changes in American public policy came about not because Congress as representative of the American people voted for it but because private individuals litigated through the court system to change the rules. […] There is virtually no other liberal democracy that proceeds in this fashion.

I'm not a giant Fukuyama fan, but his depiction of "adversarial legalism" is interesting.

America in Decay: The Sources of Political Disfunction, Francis Fukuyama, 2014

The Recursive Universe

Since its inception in 1970, Conway’s Game of Life has developed a cult following of researchers, engineers, and hobbyists, pushing each other to construct increasingly elaborate “machines” from pixels on a screen. These investigations have resulted in a lengthy taxonomy of motifs, reactions, and mechanisms, as well as engineering principals and design abstractions. These days you can spend a long time on YouTube exploring the seemingly impossible things people are designing in the “simple” Life universe.

I've been in love with this simple puzzle toy ever since learning about it… it seems every five years when I check back in on it there are new crazy nano machines being built in it.

The Recursive Universe

This isn't a spaceship, it's a LEGO brain-teaser | The Brothers Brick | The Brothers Brick

This is an amazing behind the scenes look at a master builder… my brain is still boggled at how complicated that wing is.

Nick builds his creations as an iterative process–he’ll work on the same design for months, going through multiple versions and refining the design until everything fits just the way he wants. Usually, he’ll do this without regard for color, and only once he has the design down pat does he rebuild it in the final colors. Nick has published a few images of this ship giving us a look under the hood. First up, let’s look at the wing without the final colors applied, which makes it easier to distinguish the individual elements. One of the key elements you can spot here is that strange slope with 5 studs from the 90s (all those yellow parts in the image below). Incorporating its unique geometry makes for a fantastic starting point if you’re looking to give your LEGO brain a workout.

As my friend Andy B. put it best: "this is the greatest example of the idea that constraints breed creativity i've seen in a LONG time."

How does this...

become this? 🤯

This isn't a spaceship, it's a LEGO brain-teaser | The Brothers Brick | The Brothers Brick

Right Now -- A Giant Ant Short. - YouTube

I've thought about this video a lot. The central question keeps bouncing around my head.

Right Now -- A Giant Ant Short. - YouTube

Productivity

Weekly Planning

Shoutout to Seth for pointing me at this awesome blog. I'm a fan.

Regardless of how this manifests in your system, it's vital to document and review your thoughts, ideas, and notes. Don't disregard them to gather dust in the corner of the closet (of your mind?).

Great write up of a mature planning process.

Weekly Planning

Ankit Pansari on Twitter:

One of my favorite browsers hack is to use the .new domain in the address bar to jump straight into the action. https://t.co/N5LpCIF7VX -> creates a new Google Calendar Event No need to visit the app, find the right menu, and then start an activity. More on this👇 https://t.co/aIEoYEMGql" / Twitter

This thread was worth it just for "meet.new." 🙌

Ankit Pansari on Twitter

Code

This page is a truly naked, brutalist html quine.

👏👏👏

I have loved quines since school, this is a pretty cool example of one.

This page is a truly naked, brutalist html quine.

Focus management and inert | CSS-Tricks

Focus management is the practice of coordinating what can and cannot receive focus events. It is one of the trickier things to do in front-end development, but it is important for making websites and web apps accessible.

Focus management and inert | CSS-Tricks

Functional Programming For The Rest of Us

It's true that FP articles and papers are hard to understand, but they don't have to be. The reasons for the knowledge gap are purely historical. There is nothing inherently hard about FP concepts. Consider this article "an accessible guide to FP", a bridge from our imperative minds into the world of FP. Grab a coffee and keep on reading. With any luck your coworkers will start making fun of you for your FP comments in no time.

I know this has been floating around for a while, but I'm re-reading it and enjoying it.

Functional Programming For The Rest of Us

Coding with voice dictation using Talon Voice · Josh W Comeau

Earlier this year, I developed Cubital Tunnel Syndrome, a repetitive-strain injury, in both of my elbows. As a result, I pretty much can't use a mouse or keyboard; after a few minutes, I get a burning pain shooting down my arms. Even if I try to limit my computer usage to 60-second bursts, I wind up inadvertently making the situation worse.

As you might imagine, this was a pretty big deal; as a software developer, my entire career is based on being able to use a keyboard!

After many failed attempts at solving the problem with physiotherapy, ergonomics, and prescription medications, I've found a solution that allows me to be productive without risking further nerve damage. I now work almost exclusively using a microphone and an eye-tracker.

Coding with voice dictation using Talon Voice · Josh W Comeau

The State of CSS Survey

Flexbox, Grid, Multi-Column… To say nothing of whole new paradigms like CSS-in-JS.

So after the success of our annual State Of JavaScript survey, we’ve decided to take on the world of styles and selectors to help identify the latests trends… and try to bridge the ever-growing “great front-end divide”!

I loved that at the end of this survey it gave me a list of things I marked that I didn't know with a link to learn more… that's awesome.

The State of CSS Survey

The Erlangelist - To spawn, or not to spawn?

But before starting, since this is going to be a long article, I want to immediately share my main points:

Use functions and modules to separate thought concerns. Use processes to separate runtime concerns. Do not use processes (not even agents) to separate thought concerns.

Using a blackjack game, the author explains an interesting idea that feels more like pure mathematics than anything else. I think this will unblock me in Advent of Code!

The Erlangelist - To spawn, or not to spawn?

The Kite Runner

Arun Mani — Mon, 02 Nov 2020 02:00:00 GMT

Beautiful narration of a culture, traditions and difficulties.

A Practical Analysis of the Ledger Phishing Email

SerHack - Security Researcher — Mon, 02 Nov 2020 02:00:00 GMT

Attempts at phishing and social engineering attacks have increased exponentially as of late, especially towards users who are active in the cryptocurrency space. Recently, there has been much discussion about a cunning attempt by phishers against Ledger and its users. -Continue to read the article on SerHack.me

Remove (deinit) a Git submodule

Blog by Jade Michael Thornton — Sun, 01 Nov 2020 17:00:00 GMT

A quick reminder of how to remove a Git submodule, because I always forget

Fruit Plate

utf9k — Sun, 01 Nov 2020 13:00:00 GMT

A minor tangent about... I don't remember, GTD and fruit?

RSS from private multireddits

voussoir.net/writing — Sun, 01 Nov 2020 09:01:56 GMT

Back to writing

RSS from private multireddits

The RSS URL for a multireddit is https://www.reddit.com/user/username/m/multiname/new.rss. But, putting this into your RSS client without making the multi public will yield a 404.

On reddit.com/prefs/feeds you'll find some RSS links that use a static token in the URL to act as the authentication, so you don't have to provide login credentials in your client. It turns out, the token is the same for all of the feeds, and you can use it on the multireddit feed too.

Just use https://www.reddit.com/user/username/m/multiname/new.rss?feed=XXXXXXX&user=username and you're good to go.

Since a single multireddit can aggregate up to 100 subreddits, you can save yourself and reddit up to 99 web requests per refresh by doing it this way.

View this document's history

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

Zettelkasten bookmarks

xenodium.com @alvaro — Sun, 01 Nov 2020 02:00:00 GMT

My first time using rust

Imran's Blog — Sun, 01 Nov 2020 02:00:00 GMT

Replacing one of my shell scripts with a compiled binary using rust.

Solutions: What are you solving for in life?

A Matter of Perspective — Sun, 01 Nov 2020 02:00:00 GMT

👈 Back to all conditions

The second condition required to unlock change is our Solutions to our current life-problems.

Solving for the wrong type of problem can be very frustrating. We are often drawn to the “idea” of a life-style or challenge that is not yet ours to solve. Instead of focusing on where we’re at, we get caught up with shiny objects, distractions, and are lead down the garden path. We lose track of the real issues in our lives and instead choose to focus on other things that appear sexier or more gratifying.

Are you really ready?

How does one truly know whether what they’re focusing on is the “right” thing or not? A good indicator is if you’re currently in balance with your existing life-conditions.

For example, take an employee who is performing remarkably at work. Their clients shower them in constant praise, are continuously promoted, and everyone in the office enjoys their company. They learn from everyone and spend years getting to know the ins and outs of their industry, business, and money. One day, this employee might think to themselves, “Gee, I think I know enough about this to run my own shop!”. After talking about it with their partner (check supporting structure), they decide to start a business on the side. They grow it until they can run it full time and enter a whole new perspective. (A Blue to Orange transition)

In contrast, how would this play out if the employee was terrible with money, didn’t understand the market, and was too junior? It doesn’t matter how many business courses they take, life-coaches they hire, or books they read – they’re not ready. Ultimately, they will fail in their endeavors if they don’t go back and sort out the prerequisites. (Orange cannot be supported by an unhealthy Blue).

If we haven’t solved the problems of where we’re at, we don’t have the supporting structure to jump into the “next thing.”

Are there enough resources to move on?

Let’s look at another example. While travelling, I meet many individuals claiming to be “changing/helping the world,” yet they can’t afford the bill when we go to dinner.

How can you expect to help the world when you can’t help yourself? The world is littered with well-intentioned but broke hippies. Ensure your own house is in order before fixing others.

When a new perspective or set of problems is presented to us, a great deal of energy is invested in understanding and dealing with them. As we mature in this perspective, we (hopefully) get better at solving them (how often do you think about tying your shoelaces these days?) and use fewer resources, time, and energy to solve them.

For example, when you start a new job, there are, imaginably, many modern systems, people, processes, technical words, etc. to learn. Over time these things become familiar to you, and what requires a whole day to complete in your first month may take you a few minutes down the road. This means that there is an excess of energy.

It is in this excess that we find ourselves on stable ground. Only now may we safely stand up and look beyond where we are.

If we were still doing a lousy job (not solving our current problems), there would be no energy to think about other things because we’re tied up in what we’re doing.

Revisit what you think you know

There seems to be this “great rush” to modern life. Grow faster, work faster, learn quicker, go further, more, MORE! All well and good in some situations, but often this leads us to skip over some things. The issue with this is we can sometimes be carrying around unhealthy solutions that we “found” long ago. Re-addressing the problems in Spiral, starting from the base, can strengthen the foundation in more complex places (“higher” colors).

We may test ourselves on these:

The Purple Problem: Do you feel that you truly belong in this world, are loved and accepted, and are comfortable being yourself?
The Red Problem: Are you comfortable standing up for yourself, expressing, and finding your own way (when needed) in life?
The Blue Problem: Are you able to let your own needs go for the group, stay disciplined, and truly sacrifice now for the future?
The Orange Problem: Have you solved the money/success/power problem? Are you able to throw yourself at a problem and never give up, despite fierce competition, incredible odds, debilitating loneliness, and come out victorious? Do you have the external resources to move forward?
The Green Problem: Are you able to let go of material desire, status, and ego? Do you value all life forms? Are you able to encourage diversity, love, and harmony? Are you warm of spirit and words to yourself and others?

Slow down and take stock

We all need work. All of us would have elements in each of these perspectives that could do with a little love and kindness. This is where the power of community and environment helps us, as we are incapable of detecting our blindspots.

So next time you feel friction, come back to these problems. Ask yourself if the desire for something else blinds you, or are you working at where you’re at? This is a hard question answer sometimes, but a necessary one.

Going back to support our foundations transforms us into healthier, stronger, more flexible people/companies/countries/etc. It’s perfectly ok to admit we’re insecure and work to heal it. There is no shame in discovering that we don’t know ourselves or have trouble doing what we really want in life (many of us are like this). Consistency, discipline, and dedication are hard virtues to manifest. Don’t beat yourself up. Just keep going.

Keep living, learning, and growing. Don’t let other people’s ideas of what you should be doing deter you. They’re on their own journey, and no two roads are the same.

But most of all, if you want to change, be honest with yourself about where you are and what you’re doing. You’ll thank yourself in the long run.

👈 Back to all conditions

5 Z shell features that will encourage you to switch from Bash

Jason Eckert's Website and Blog — Sun, 01 Nov 2020 02:00:00 GMT

The command line interface on your system is provided by a shell. And while there are dozens of different shells available to choose from, nearly every UNIX and Linux system over the past two decades has used the feature-rich Bash shell as the default login shell.

Although the Bash shell still remains the most common login shell used on UNIX and Linux systems, macOS and a few Linux distributions (e.g. Kali) have switched their default shell to the Z shell (zsh), and this trend is likely to continue. The Z shell has been around for three decades, and is the second most popular shell next to Bash.

What Does It Take To Resolve A Hostname

Venam's Blog — Patrick Louis (Lebanon) — Sun, 01 Nov 2020 00:00:00 GMT

Resolving hostnames (DNS?) might not seem like it, but it's complicated. Let's take a moment to see if we can at least demystify what happens on the client side instead of seeing it as a big tangled mess of configurations, libraries, and tools.

Habitat 2020 - Coliving during COVID

JonLuca's Blog — Sat, 31 Oct 2020 23:06:30 GMT

A look back at living in a coliving community during the first year of COVID-19.

Robotic Blacksmiths? Yeah!

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 31 Oct 2020 20:00:00 GMT

When hammer and anvil meet bits. Possibly, everywhere.

Masking in Deep Reinforcement Learning

Boring Guy — Sat, 31 Oct 2020 17:20:06 GMT

Introduction I worked on an environment where specific actions are not available at every timestep $t$ when I started deep reinforcement learning. Let’s illustrate the concept of impossible or unavailable action concretely: Suppose you want to develop an agent to play Mario Kart. Next, assume that the agent has an empty inventory (no banana 🍌 or anything). The agent can’t execute the action “use the object in the inventory”. Limiting the agent to a meaningful choice of actions will enable it to explore in a smarter way and output a better policy.

Committing time to stand out from the crowd

I Learnt — Sat, 31 Oct 2020 14:00:00 GMT

There is a famous quote that asks how you eat an elephant and the answer is one bite at a time. It can be very daunting when attempting something new especially if it is potentially a large task. However it is impossible if you don’t take the first step.

Wallace Falls

Sam Hooke — Sat, 31 Oct 2020 05:20:00 GMT

No trip report yet.

Pylons.

Bridge.

Golden.

Up.

Straight.

Waterfall.

Distance.

Democracy is an achievement, not a recipe

nutcroft — Sat, 31 Oct 2020 02:00:00 GMT

There is—thankfully—consid erable collo quy on the quality of our world’s democracies. People talk about whether democracy is actually a good idea while others claim it’s just the least worse. In this text I will attempt to convince you that democracy is not only a good idea, but actually a great one. All this with a transcendental shift in our perception of democracy:

Democracy is not a system; it is an achievement.

Unsystemacy

We can’t have democracy if we do not think in democratic principles. It’s like agile when developing software. You can’t switch to agile by adding a standup and a retro—it’s not a recipe¹. You can only be agile if you have the agile mindset. In the same way, you can’t be a democracy just by having your leaders elected². You can only have democracy if you include everyone in the actual decision making processes. When venturing democracy, there is a constant need for self-reflection on whether everyone is actually involved. This requires a continuous, active examination of societal processes, an everlasting reworking of the methods of decision making. That's core to the democratic mindset.

Recipe status

It's evident in the global society that we view democracy as a set of rules that once applied, all societal desires will be aligned and non-zero sum benefits will unequivocally emerge. One piece of this evidence is the common remark: democracy has failed us; or: democracy doesn’t work. But that’s not the case. There is a limit to the range of influence of any social structure. Nothing will force all our interests to be aligned—only our agency. If not this, then what's left is searching for loopholes in extraneously established laws with the purpose of making one’s life easier at the expense of the whole society's betterment.

That's not to say this is an argument against social structures. On the contrary, it's one in favour. The disparity in this text's thesis is that these structures should be created directly from the people they are designed to apply to, not from a handful of elected officials.

Against chain of agency

Democracy, by definition, is when people themselves have the power to make political and societal decisions. It was, historically, the dialectic antithesis to oligarchy: when power resides with the few. Indirect, or representative democracy, the current prevalent version of democracy, is not democracy, but rather an oligarchy.

In representative democracy, politician (i.e. one who discusses and decides on societal issues) is a career. Whereas, in democracy, everyone is a politician and is expected to actively engage with any societal issues at hand. That is in addition to their job³.

Representative democracy politicians aim to convince people to vote for them via superficially dramatic campaigns, rally-style theatrical performances, and by employing fanfare. There are no real societal plans being discussed⁴ between the politicians and the people, just statements and ambiguous responses—approval or rejection—from the public.

However, discussion is a core feature of the democratic framework. How can a team make a collective decision if there is no discussion on the issue? In other words, how can a people make a democratic decision if there are no processes to discuss issues?

Across the whole wide spectrum of actionable politics, in representative democracy a citizen makes, on average, two political decision every four years⁵. As mentioned, achieving democracy means people exert the governing power. Nevertheless, half a political decision per year is not nearly enough for people to be considered actively involved in the decision making process.

As essence

Before we argue against the shortcomings of direct democracy, there is a case to be made for the essential quality of democracy as a societal framework⁶. Why is democracy important?

Proposition 1: People value autonomy more than life itself. Indications are spread across history, ranging from the motto of Greece to the ethos of Breaking Bad.
Proposition 2: Sharing our lives with other people (friends, family, strangers) is part of the essence of our humanity.
Proposition 3: It follows from 1 and 2 that being autonomous together with other people is vital.

Democracy is the actualisation of this: making collective decisions on how we live. Undoubtedly, fundamental. I would go as far as to say that there is no point in life without this sport of endeavouring democracy. If not for it, our life would be either an autonomous loneliness or an enslaved collectivity.

In defence of

Let's assume Fred, our imaginary assistant for the demonstration of the argumentative against the shortcomings of direct democracy.

Fred: “Well, that’s all nice and stuff, but we can’t have direct democracy because people are not well educated to have opinions on significant matters and/or they will make horrible decisions. See Brexit, for example.”

Fred, allow me to disagree. For one, there are solid arguments to be made in favour of Brexit, regardless of Brexiteers basing their vote on the silly ones.

It is true that people will be unknowledgeable on a variety of issues that they will be asked to give opinions on. However, during an honest discourse where all voices are given equal ground and are heard in the same level, it is an argument of human rationality and aligned interests that people are going to behave responsibly⁷.

Another argument to be made here is that there are, and have been through recent history, multiple political leaders the intelligence of whom is highly questionable. Yet, we survived. Therefore, I have little fear that the consequences of a potential lack of intelligence will be destructive.

Finally, freedom is more important than making the best decision—it’s more important than life itself, for many! Let people decide for themselves, even if it kills them. How much more different would it be from the present, when elected leaders decide for people, and it still kills them⁸?

Fred: “So you’re saying instead of Trump ordering the police to kill people, people will accidentally kill themselves and that’s ok. Great stuff! You know what, direct democracy is never going to work because people don’t have time to talk politics all day. Not to mention that many don't even care about politics. They have jobs and are busy with their actual life. Direct democracy was only good when it was invented, when they had slaves to work the fields, in ancient Athens.”

Come on, Fred, you know the first part is a straw man.

Here's the thing about people's time. All these decisions already are part of their busy life. Politics and societal issues unavoidably affect everyone and the only other path is lack of action—being a puppet while others decide for your life and the civic institutions that eventually you, too, will encounter. Is that not a core part of everyone's life? It seems fundamental to find the time for it.

There are many things to say about all this people's busyness and the concept of bullshit jobs is one of them. The work ethic that is currently permeating the global society is one that associates work with self-worth⁹. David Graeber describes Bullshit Jobs as meaningless jobs that exist only because it's shameful to not work. Hence, we had to invent jobs such as door attendants and airline desk staff who calm passengers whose bags do not arrive to keep up with the demand for virtuous suffering.

In regards to the argument of people not being interested in politics, I see that more of an aftereffect of the complete disaster that modern politics has been¹⁰, rather than an inherent characteristic of people not concerned about how they live. Ultimately, many may not care, but at least those who do will have the capacity to act.

Fred: “Well, you didn’t convince me at all. We have this system for so long now; it's not perfect—nothing is—but it's so much better than all the previous ones¹¹. We're not going to make such an inconsiderate change and abandon it for something evidently not working.”

Today, I learned that up until 2006¹² one could take liquids on a plane. It sounds like a magical time, but it was only fourteen years ago. Something so established and familiar didn’t exist until very recently. The point I want to make is that changes seem scary but only from afar. In the end, this world is all about risk taking and change.

Another point about radical change is technology. It has evolved enormously during the last twenty years, not to mention the last hundred. Who could ever imagine in 1920, that humanity would be so intimately enmeshed in a global cybernetic network?

Yet, on the other side, how much have our societal and political systems change since then? It’s only fitting that a change of an equivalent radical degree is waiting to happen on how our societies work.

There is some disagreement over this but here's a book that expands on agile requiring the appropriate mindset. ↩
Elected is many times better than not elected, yet still not good enough. ↩
In Ancient Athens, they didn't need to do that since slaves would do everyone's job, for free. In the modern world, until robots can do all jobs, I think it’s important that we do both products/services and politics. ↩
The worst version of this absence of real discussion is attacking the opponent’s credibility, the one of the lowest quality of the techniques of persuasion. ↩
Two elections, one for the local authority and one for the country-level authority. This is an average of western countries, but I assume it can also vary, as I'm not aware of all European political systems. Notable exception is Switzerland, in which multiple referendums take place and other direct democracy processes occur frequently. ↩
I'm consciously avoiding the term political system here. ↩
There are also indications that people will behave as you expect them to. I expect them to make serious, responsible decisions that will affect the whole. ↩
Trump prompted the police to not be too nice to the people protesting for rudimentary justice. ↩
Max Weber coined the term in his book The Protestant Ethic and the Spirit of Capitalism. Protestant work ethic is the principle that hard work and discipline are a righteous person's driving ideals. ↩
It is neither accidental nor satisfactory that young smart people allow themselves to become absorbed by the realms of academia or industry rather than politics. ↩
This is another iteration of the famous Churchill quote that helps us accept the inadequacy of the status quo: "Democracy is the worst form of government, except for all the others." ↩
I learned that from this very interesting video on security by Oliver Thorn. ↩

2020–10–31: New p-boot released, new pre-built/updated kernels

megi's PinePhone Development Log — Sat, 31 Oct 2020 01:00:00 GMT

Gotta cut wood

TOGoS's Project Log — Fri, 30 Oct 2020 20:13:37 GMT

Project Cybersin

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 30 Oct 2020 20:00:00 GMT

We rarely look to older technologies for inspiration. That’s bad.

CCEC 2020 3-Minute Pitch [Video]

David Schlachter — Fri, 30 Oct 2020 18:15:00 GMT

Summary of my Masters research at the 2020 Canadian Chemical Engineering Conference. First-place in the 3-Minute Pitch competition.

Why the Westminster Reference Bible is the best all-around bible you can buy

Honza Pokorný — Fri, 30 Oct 2020 18:00:00 GMT

I will argue that the Westminster Reference Bible in black calfskin published by the Trinitarian Bible Society (TBS) is the best all-around printed bible that you can buy today in the English language.

I will not be considering digital bibles, or bibles in languages other than English. And by all-around, I will mean all-purpose. The bible should work well in as many contexts as possible. There are specialty bibles that will work better in some contexts; however, I will focus on having a single bible that can do everything: preaching, personal and group study, casual reading, hospital visits, public and family worship.

There are two parts to this evaluation: the translation, and the printed volume itself.

The translation

The Westminster Reference Bible uses the Authorized Version, more commonly known as the King James Version (KJV). The KJV was first published in 1611, and has since enjoyed great popularity. It has altered the course of the English language as many of its phrases have entered into common speech.

The Received Text

The KJV was translated from the Reformation-era text of Scripture. Unlike most modern English translations of the Bible, the KJV uses the Received Text, as opposed to the Critical Text as its underlying source. The Received Text is the providentially preserved Word of God, without spot or blemish. This means that the KJV doesn’t need to be updated when textual-critical scholarship decides that new manuscript evidence warrants a change of the wording of the Holy Scriptures. The KJV presents in English what the apostles wrote in Greek. The KJV unashamedly and proudly includes all of the text of Scripture, and doesn’t neglect any of the contested passages, like Mark 16:9-20 or John 7:53-8:11. The KJV as a translation is driven by the ideals of the Reformation.

Transliteration

The KJV employs an interesting strategy when transliterating proper names to English. The Bible was original written in Hebrew (and Aramaic), and Greek, and those languages don’t use the Latin alphabet. Each proper name must be transliterated to an English form. This is common to all translations. Παῦλος becomes Paul. But the way that the KJV goes about it is different. Many proper names are rendered differently in the Old Testament, and the New Testament. For example, the prophet Isaiah is called Isaiah in the Old Testament, but Esaias in the New Testament (e.g. Matthew 3:3). This becomes important in passages like Hebrews 4:8 where Jesus is said to give rest. The KJV uses the word Jesus instead of Joshua as many modern translations do. This shows the reader that the two names are the same, and that the context must determine which person is being spoken of. The KJV gives you the option; modern translations don’t.

Singular and plural personal pronouns

The KJV uses archaic personal pronouns like thou, thee, thine, and ye. This is done in order to signal to the reader whether a singular or a plural form of the verb is being employed in the original languages. Modern English doesn’t have this ability aside from regional colloquialisms like y’all. This enables the reader to obtain a greater understanding of the biblical text.

Void of modern gendered language

The KJV doesn’t suffer from modern inclusive language issues. It doesn’t try to be politically correct. It doesn’t say brothers and sisters where the Greek original says brothers.

It’s final

The KJV hasn’t changed in centuries, and isn’t going to. This is in stark contrast with modern translations which need to be updated every few years.

Generations of Christians have read the KJV, and memorized passages from it, and we can follow in their footsteps. We don’t need to worry that our bibles will become obsolete with the next update. We can buy a bible, and it can last us a lifetime.

The standard version

The KJV is the only translation which can be considered a standard version. Thousands of churches recite the Lord’s prayer using the words of the KJV despite using a modern translation in preaching.

Public domain

The text of the KJV is in the public domain in most of the world, and is freely and readily available. The Word of God is precious.

The book itself

Binding and leather

The book itself is smyth-sewn, and bound in black calfskin. The calfskin presents a happy trade-off between quality, price, and comfort. It’s very durable, feels great to the touch, and isn’t very expensive compared to goatskin. It’s only available in black, the only colour that matters.

Subsidized cost

The cost of producing the Westminster Reference Bible is being subsidized from a large pool of money a generous donor has made available. Thus, one is able to acquire a premium bible without a premium price tag. This shows the integrity of the publisher, and assures us that their primary interest isn’t getting rich.

References

The Westminster Reference Bible is a reference edition which means that the text of Scripture is surrounded by footnotes, and notes. These notes link various places in the bible together for study purposes. This bible has over 200,000 of such references in its margins. A regular reference or study bible might have about 60,000-80,000 references. These references come from John Brown’s Self-Interpreting Bible. Using the reference system, one can explore what else the bible has to say on what is currently being read.

And yet, despite this wealth of information on each page, the text of Holy Scripture isn’t obscured or hard to read. The helps and reference material only comes into view when one desires.

No undermining

Despite being a reference edition, it doesn’t undermine the reader’s confidence in the text. The Westminster Reference Bible doesn’t include comments on the quality of the manuscripts used, let alone call any passages into question.

Even R.L. Allan’s flagship Longprimer has comments that say that a certain reading isn’t supported by the oldest manuscripts.

Verse by verse

The modern trend is to present the text of Scripture in a single-column, paragraphed setting. The Westminster Reference Bible is a verse-by-verse, double-column edition. This means that the volume can be smaller because the two columns are more space efficient. And just like the reference system, it encourages bible study because it puts the verse number front and center.

It may be argued that verse-by-verse double-column bibles are hard to read. Yes, I’d say it’s more challenging than reading a novel. However, slower reading leads greater comprehension.

Obscure words

In addition to its extensive reference system, it also has obsure words explained inline. Since the Westminster Reference Bible uses the text of the KJV, some of the words used won’t be easily understood by the modern reader. To aid reading, the bible includes short explanations of these difficult words right on the page. This includes both archaic words, and so-called false friends (e.g. conversation used to have the sense of today’s behaviour).

Chapter summaries

Unlike modern editions of the bible, the Westminster Reference Bible doesn’t use headings through the text. Instead, it uses chapter summaries. The reader can quickly gain an overview of each chapter when locating a passage. Moreover, these summaries are original to the KJV, and as such are full of Reformed theology. This perhaps most obvious in the Psalms, e.g. the summary of Psalm 110 reads:

A prediction of the kingdom, 4. priesthood, 5. triumphs, 7. and sufferings of Christ.

Or, the Song of Solomon, chapter 1:

The church’s love unto Christ

Chapter summaries are arguably more useful than headings in the text. If nothing else, they don’t introduce uninspired breaks in the text.

Pauline authorship of Hebrews

The Westminster Reference Bible proudly and rightly announces that the author of the book of Hebrews is none other than the Apostle Paul.

Black-letter edition

Unlike many modern-day editions of the KJV, the Westminster Reference Bible is a black-letter edition. This means that all of the words of the Holy Bible are in black in contrast with red-letter editions which render the words of Christ in red. Reformed theology argues that all scripture is given by inspiration of God (2 Tim 3:16), and should therefore be red. And red is of course hard to read.

Italics

The KJV translators felt that certain words not present in the original had to be added in order to translate a passage better. These words they marked in italics. The Westminster Reference Bible respects their wishes, and prints these words in italics. It has been shown that these are somewhat arbitrary and inconsistent, and one shouldn’t rely on them too much; however, they are still useful in reminding the reader that translating a text from one language to another isn’t an exact science, and a one-to-one correspondence isn’t always achievable.

No quotation marks

Modern bibles introduce un-inspired quotation marks into the text to indicate when a person starts speaking. The problem is that sometimes these decisions are wrong, and affect doctrine in negative ways. An example of this can be found in Acts 19. The Westminster Reference Bible omits these quotation marks just like the original text, allowing the diligent student of the Scriptures to advance without distractions.

Epistle to the reader

The printed volume includes the KJV translators’ Epistle to the reader where they explain their motivations. Sadly, it is common today to exclude this wonderful letter from printed editions. This epistle argues against the modern concept called KJV Onlyism.

Reading plan

If this is to be the best all-around bible, a reading plan is a must. It keeps the serious student of the word of God on grounded, and on track. No external tools are necessary. It would make the perfect desert island bible.

Digital font

Unlike R.L. Allan’s Longprimer, the Westminster Reference Bible doesn’t pander to 1950s aesthetics, and it uses a modern digital font. This makes the page crisp and clean.

High view of Scripture

The publisher’s high view of Scripture can be seen in their including three separate pages with quotes from the Bible which highlight the Bible’s purity, and excellence. Not one, but three: Psalm 119:18, Psalm 19:7-11, and 2 Timothy 3:16-17

Explanation of thees and thous

The front matter in the book provides a helpful explanation of how the archaic personal pronouns work.

Reformed theology

This bible is called the Westminster Reference Bible: it’s built on the ideals of the English reformation of the 17th century. The Westminster Reference Bible is as Reformed as the Westminster Confession of Faith.

Another example is that the title page of the Old Testament reads The Old Testament of Jesus Christ. And as mentioned above, many of the Old Testament chapter summaries mention Christ or the church. The whole bible is about him, and the Westminster Reference Bible reminds you of it every now and then.

Sizes

The Westminster Reference Bible comes in three size: large print, compact, and standard. This offers the diligent student of the word of God flexibility: running to the hospital on a pastoral visit? Grab the compact version. Do you have folks in your church with aging eyes? Pick up some large print editions.

Conclusion

If you want a premium bible for not-much money, a reformed bible with high standards, a pure text, and a great set of study features, look no further than the Westminster Reference Bible.

New book, same story.

trv.rs — Fri, 30 Oct 2020 11:23:00 GMT

On the left is a journal I started the day before I asked my future father-in-law for his blessing to marry Sarah. It's been with me every night I was single, two trips to Europe, cross-country flights, camping trips… it's chock full of pictures, love notes, the frustrated scrawl of dark nights of the soul and delighted depictions of sunrises. In it I planned our wedding, prayed for my future children, celebrated every anniversary and birthday. It's scarred, tear-stained, rained on, prayed over, scratched out…

It's probably the most precious physical object I own… and it's full. It is time to pass the baton to the next journal.

The timing could not be better.

Each time I finish a journal I leave the last few pages blank. I then read through the whole thing noting down blessings and important moments in those spare pages. I revisit old prayer requests that have been answered, dreams that were realized, goals met. I will sing of God's goodness in marriage. I will rejoice again at the birth of my children. I will again weep over old sins and conflicts. I will see God's sovereign hand over the past five years compressed into a few pages of my handwritten thoughts.

Journaling isn't just therapeutic. It's an opportunity to glance behind the tapestry of history to see what God has been weaving.

When I started the journal, I stood at a precipice: all I hoped for and held dear was in the hands of a loving and almighty God. In the years since, I have found him to be faithful. And I wrote it down.

I'm starting this new journal tomorrow close to another precipice of history: much that I hope for and hold dear in the balance. I'm grateful that God has ordained this moment to remind me to look back on my little leather ebenezer… to remind me that He has not changed.

New page. Same King. 🙏

Restic backups to B2 without delete

Joseph Price — Fri, 30 Oct 2020 02:00:00 GMT

Update 2023-03-28: Restic now supports limited keys by default, and also recommends the B2 S3 api instead. See here

Original Post:

Since setting up restic to back up to b2 I had embarassingly missunderstood the following UI wording around lifecycle settings:

I had assumed that this meant that all versions of the file would be maintained, no matter what clients did. The B2 docs are much clearer, explaining that these rules are really only automation on your existing files. They're not blocks/holds/locks which can take any action before or affect api calls.

In short: Lifecycle Rules schedule cleaning up version history, they don't prevent you deleting that version history.

One of the capabilities I want from my backup system is that the system backing data up should not also be able to destroy existing backups. (I'm not bothered about it reading previous backups) I had assumed lifecycle rules gave me this, I was wrong, time for a quick fix...

The obvious solution is to restrict the capabilities of the b2 api key during creation, specifically to omit deleteFiles. Unfortunately, restic doesn't currently handle this well, for example during basic backups it requires the ability to delete files in the locks/ directory and will complain loudly. There are a few open issues and existing [PR]https://github.com/restic/restic/pull/2398(https://github.com/restic/restic/pull/2398)'s but nothing yet merged.

Enter rclone

A few years ago rclone gained support for serving storage to restic via it's api and since rclone supports using soft delete by default swapping to this backend is pretty trivial.

One of the best bits is that you can continue to use your existing config, the repository format is unchanged.

The first thing to do is to generate a new key without deleteFiles:

$ b2 authorize-account
Using https://api.backblazeb2.com
Backblaze application key ID: <account key id>
Backblaze application key: 

$ b2 create-key --bucket bucketname keyname listBuckets,listFiles,readFiles,writeFiles
<app key id> <app key secret>

...and then take that output to set up an rclone backend. You can either go through rclone config (See their docs for an example) or just drop a file into place:

$ cat ~/.config/rclone/rclone.conf
[b2]
type = b2
account = <app key id>
key = <app key secret>

Then all that's left is to update your backup script:

e.g. if previously it was
```
b2:bucketname:foldername
```
then now it'll be
```
rclone:b2:bucketname/foldername
```
(Note that the second part (b2) now refers to the name of the rclone backend, not the type. e.g. the bit within the square brackets of rclone.conf config)
Alter the options rclone is called with. Annoyingly, even though rclone uses soft delete in b2 by default, restic sets --b2-hard-delete by default... so add the following option whenever you run restic:
```
-o rclone.args="serve restic --stdio"
```

Back to the lifecycle...

Beyond restic forget/prune to manage old snapshots, we've now got to consider cleaning up soft deleted files. Given my backups are largely additive, forget rarely frees much space so it's not my biggest priority.

rclone provides a handy cleanup command, but it's very tempting to go back to where I began and let Backblaze b2 lifecycles clean them up without any extra scheduling on your own infrastructure.

Obligatory https://restic.readthedocs.io/en/stable/045_working_with_repos.html#checking-integrity-and-consistency reminder.

Linux Upskill Challenge

Heitor's log — Fri, 30 Oct 2020 02:00:00 GMT

My notes about the Linux Upskill Challenge.

Once Upon a Snowman

Code & Visuals — Fri, 30 Oct 2020 02:00:00 GMT

For the holiday season this year, Walt Disney Animation Studios has a small surprise release! Last month the studio announced the release of Once Upon a Snowman, which is now out on Disney+ as of last week. Once Upon a Snowman is a short film that takes place during the events of Frozen (2014) and follows Olaf between when Elsa creates him and when he runs into Anna and Kristoff. The short fills in the gap to show how Olaf initially comes to learn who he is and the hijinks he gets up to along the way.

From a technical perspective, Once Upon a Snowman shares pretty much all of its tooling and tech with Frozen 2, because Once Upon a Snowman was made using the exact same pipeline and toolset version that Frozen 2 was made with. In fact, Once Upon a Snowman was made overlapping with the end of production on Frozen 2 and completed a few months after Frozen 2 and has simply been waiting for release since then. The thinking was that rather than spin down the Frozen 2 pipeline upon completion of the movie, why not keep it going for a bit longer to make a short?

However, one enormous difference between Once Upon a Snowman and Frozen 2 is the time period the story is set in, and therefore the versions of characters and sets that had to be built. Since Once Upon a Snowman is set during the events of Frozen, the versions of Anna, Elsa, and Kristoff have to be the younger versions of themselves in their outfits from the first film. In my Frozen 2 post, I wrote about all of the ways that Frozen 2 is an interesting benchmark for what we are capable of making today using Disney’s Hyperion Renderer [Burley et al. 2018] versus what the studio was capable of back on the original Frozen using RenderMan 17. Once Upon a Snowman is an even more direct comparison between Disney Animation’s modern tech and its 2014-era tech since all of the characters and sets are directly one-for-one matched to the original Frozen.

Because Hyperion uses an evolved version [Burley 2015] of the same Disney BSDF that Frozen used an earlier version of [Burley 2012], porting assets forward from Frozen to the modern pipeline doesn’t require a full rebuild from scratch. The port isn’t just a direct copy either; a considerable amount of remastering and improvement has to be carried out to bring Frozen-era assets up to our modern standard, but the work required is also still much less than starting over completely. Oaken is a great example; the Once Upon a Snowman version of Oaken is recognizably the same Oaken from Frozen, but looking closely reveals a number of upgrades. The Once Upon a Snowman version has vastly more convincing subsurface scattering [Chiang et al. 2016a], various subtle global illumination effects, vastly better hair shading, and even some improved geometric details such as far more detailed fuzz on his sweater:

The impact of the modern-day Zootopia-era fur/hair shader is really apparent on the wolves that are seen in both Frozen and Once Upon a Snowman. In Frozen, these wolves are seen chasing Anna and Kristoff’s sled, and in Once Upon a Snowman we see them shortly after the events of Frozen, this time interacting with Olaf. During Zootopia we learned an enormous amount about making white fur look convincingly fluffy and soft [Burkard et al 2016]; the look comes entirely from high-order multiple scattering. Frozen used a version of our Tangled-era hair shader [Sadeghi et al. 2010] and point-based hair lighting [Montoya-Vozmediano 2010], for which high-order multiple scattering effects were simply impossible to practically achieve, but today using Hyperion and our modern Zootopia-era fur/hair model [Chiang et al. 2016b], this is now easy for us to achieve. The version of the wolves in Once Upon a Snowman also sports a considerably higher curve count and far more advanced groom:

The interesting artistic challenge of porting assets from older shows to the modern pipeline is that they simultaneously need to be upgraded but still look recognizable when compared with the original. They don’t need to look identical, but should look similar enough that they feel the same in the viewer’s memory. One way I’ve heard artists describe it that I really like is: things always look better in our memories than they actually did at the time, and the match we have to aim for is as good as our memories’ retroactively improved version. A great example of this is the establishing shot of Wandering Oaken’s Trading Post; this is one of the closest matching shots between Frozen and Once Upon a Snowman. Looking closely there are a bunch of apparent visual improvements in the Once Upon a Snowman version: the Frozen version is very clearly lacking multi-bounce global illumination between the snow and the structure and a lot of the colors on the structure look simultaneously more muted and higher contrast than one would expect, whereas the Once Upon a Snowman version has richer colors from multi-bounce global illumination, softer light scattering through the snow on the trees, more detailed and richer shadowing, and so on and so forth. However, if you just glance between the two versions, they are definitely recognizably the same place at nearly the same moment in time:

In general, an interesting thing about the shift to fully physically based shading and path-traced rendering is that many effects that previously were extraordinarily difficult to achieve in the old ad-hoc rasterized world are now really easy to achieve [Burley et al. 2017]. Another great example in addition to everything above is Elsa’s ice palace; for the original Frozen, rendering the ice palace’s complex layers of refractive ice was an enormous challenge that took a wide combination of specialized rendering and compositing tricks to achieve [Siegel 2014], but with our modern renderer, all we have to do is assign a standard Disney BSDF material, make it refractive, and hit render, and it all just works. The ice palace is only seen in one shot in Once Upon a Snowman, but even that single shot in the pre-Hyperion world would have taken considerable effort to put together, whereas in the modern workflow it didn’t merit any special consideration at all to achieve.

Normally I’d write that the entire short was rendered using Disney’s Hyperion Renderer, but there’s actually a special quirk about Once Upon a Snowman: the first seven shots are directly edited in from the original Frozen’s final footage, so they predate Hyperion. However, one shot is actually extended from the original Frozen; the third shot in Once Upon a Snowman is much longer than in the original Frozen. For Once Upon a Snowman, this shot was extended to stay on Olaf much longer to shift the story focus from Elsa to Olaf, but had to maintain a consistent look with the surrounding shots pulled directly from Frozen. Just re-rendering this single shot by itself using Hyperion would have made for a bit of a jarring jump between older-style RenderMan 17 visuals, a richer looking shot with modern global illumination using Hyperion, and then back to older-style visuals for a few more shots. However, re-rendering all of the Frozen footage using Hyperion was also not a great solution, since it would have entailed a lot of extraneous extra work just to reproduce footage that we already had. Instead, the right solution was to extend, re-animate, and then re-render the shot in the original Frozen pipeline, which is a much harder thing to do than one might think!

In the six years since the original Frozen’s release, the studio’s hardware standard and software stack has evolved considerably, and RenderMan hasn’t been run in-house for many, many years. Instead of trying to get the original Frozen’s pipeline up and running on the latest hardware and operating system version, the easier approach was just to resurrect a handful of mothballed workstations from the Frozen-era and do everything using the Frozen-era pipeline running on the same operating system version used during Frozen. Essentially we built a little time capsule just to create the extended version of that shot. I wasn’t involved with this effort, but it was great fun to watch it unfold a few offices down the hall from the Hyperion team’s office. Starting with the shot of Elsa’s cloak knocking Olaf down the hill, all shots in the rest of the short are rendered using Hyperion, and if you compare the shot immediately before the first Hyperion shot with the first Hyperion shot, you can see a clear jump in visual quality.

I think it’s pretty interesting and cool how Anna, Elsa, Olaf, Sven, Kristoff, and the rest of the Frozen cast and world have become a sort of de facto benchmark for Hyperion and Disney Animation’s continued animation and graphics tech capabilities over time. The original Frozen marked the studio’s last RenderMan show and therefore represents the farthest the studio was able to push visual quality before path tracing, and since then, Frozen Fever, Olaf’s Frozen Adventure, Frozen 2, and Once Upon a Snowman were all made using successively more capable versions of Hyperion, with all of the characters and sets being ported forward to use the latest tech each time. The history of Frozen’s core characters is also a series of snapshots in the history of modern Disney Animation’s continuously advancing technology and CG production capabilities.

Here are some frames from Once Upon a Snowman from Disney+, presented in no particular order. You can stream Once Upon a Snowman on Disney+; as always I suggest seeing it on the biggest screen you can!

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Nicholas Burkard, Hans Keim, Brian Leach, Sean Palmer, Ernest J. Petti, and Michelle Robinson. 2016. From Armadillo to Zebra: Creating the Diverse Characters and World of Zootopia. In ACM SIGGRAPH 2016 Production Sessions. Article 24.

Brent Burley. 2012. Physically Based Shading at Disney. In ACM SIGGRAPH 2012 Course Notes: Practical Physically-Based Shading in Film and Game Production.

Brent Burley. 2015. Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering. In ACM SIGGRAPH 2015 Course Notes: Physically Based Shading in Theory and Practice.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2017. Recent Advancements in Disney’s Hyperion Renderer. In ACM SIGGRAPH 2017 Course Notes: Path Tracing in Production Part 1. 26-34.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 33.

Matt Jen-Yuan Chiang, Benedikt Bitterli, Chuck Tappan, and Brent Burley. 2016. A Practical and Controllable Hair and Fur Model for Production Path Tracing. Computer Graphics Forum (Proc. of Eurographics) 35, 2 (May 2016), 275-283.

Matt Jen-Yuan Chiang, Peter Kutz, and Brent Burley. 2016. Practical and Controllable Subsurface Scattering for Production Path Tracing. In ACM SIGGRAPH 2016 Talks. Article 49.

Ramón Montoya-Vozmediano. 2010. Point-Based Hair Global Illumination. In ACM SIGGRAPH Asia 2010 Sketches. Article 17.

Iman Sadeghi, Heather Pritchett, Henrik Wann Jensen, and Rasmus Tamstorf. 2010. An Artist Friendly Hair Shading System. ACM Transactions on Graphics (Proc. of SIGGRAPH) 29, 4 (Jul. 2010), Article 56.

Lewis Siegel. 2014. Frozen on Ice: Rendering Frost and Ice on Frozen. In ACM SIGGRAPH 2014 Talks. Article 8.

Just More Color

Ben Overmyer's Site — Fri, 30 Oct 2020 02:00:00 GMT

Today I reworked my site's CSS a bit to add more color.

This is a work in progress. Over time I plan to do more to liven the site up, while still trying to keep the overall size small.

Next, I'm considering whether to add more images to some parts of the site.

Google new easter egg!

ThatXliner's Blog — Fri, 30 Oct 2020 02:00:00 GMT

Just Google "is the earth flat".

Sabertooth Motor Drivers in Julia and Javascript

Macleod Sawyer | Web Portal Feed — Fri, 30 Oct 2020 02:00:00 GMT

Julia & Javascript Libraries to control the Sabertooth motor controller

Simple to use Julia and Javascript APIs for Sabertooth motor controllers (made by Dimension Engineering) - The official API only has support for Python, C#, and Arduino - I needed one for quick prototyping and this was born.

Sabertooth.jl repo can be found here
Sabertooth.js repo can be found here

# Sabertooth.jl - Repo

# Source

module Sabertooth

using SerialPorts
# Motor commands
    const forward_m1 = 0x00
    const reverse_m1 = 0x01
    const forward_m2 = 0x04
    const reverse_m2 = 0x05
    const forward_mixed = 0x08
    const reverse_mixed = 0x09
    const right_mixed = 0x0A
    const left_mixed = 0x0B
    const ramp = 0x10
    
# Modify the serial port configs as needed
    const portname = "/dev/ttyACM0"
    const baudrate = 115200 #Default baudrate for Sabertooth (115200)
    const address = 128 # The address of the Sabertooth. By default, this is 128.
    const mask = 127

s = SerialPort(portname, baudrate) # Instantiate the serial port

# sendCommand(command, data): Core motor controller function, this sends an update out to the motor controller hardware, all other functions should call this function instead of rewriting. 
    function sendCommand(command, data)
        checksum = ((address + command + data) & 127) # Calculate the required checksum with the bitwise operator for the bytesize (mask)
        newUpdate = [UInt8(address), UInt8(command), UInt8(data), UInt8(checksum)] # Create an array buffer of bytes to send to the motor controller hardware 
        
        write(s, newUpdate) # Write to the motor controller hardware
    end


    
# drive(motor, speed): This controls one motor at a time
     function drive(motor, speed)
        # Reverse commands are equal to forward + 1
        cmds = [forward_m1, forward_m2] 

        try
            command = cmds[motor]
        catch
            @warn "Drive error, invalid motor number: $motor"
        end

        if speed < 0 
            speed = -speed
            command = command + 1 
        end
        if speed > 100 
            @warn "Drive error, invalid speed: $speed"
        end

        calculatedspeed = round(Int, *(127, speed) / 100)
        sendCommand(command, calculatedspeed)
    end
# stop(): Stops both motors
    function stop() 
        drive(1, 0)
        drive(2, 0)
    end
# driveBoth(speed_m1, speed_m2):
    function driveBoth(speed_m1, speed_m2)
        drive(1, speed_m1)
        drive(2, speed_m2)
    end
# driveTank(speed)
    function driveTank(speed)
        drive(1, speed)
        drive(2, speed)
    end

# Usage


// drive(number, speed) :: number: 1-2 :: speed: -100 to 100 
saber.drive(1, 50)
saber.drive(2, -75)

// driveBoth(speed_m1, speed_m2)
saber.driveBoth(50, 80)

// driveTank(speed) :: speed: -100 to 100
saber.driveTank(50)

// Stop
saber.stop()

# Sabertooth.js (Javascript) - Repo

# Install

npm install sabertoothjs

# Usage


const { Sabertooth } = require('sabertoothjs');

// Sabertooth(port, baudrate, address) (default: 128)
var saber = new Sabertooth('/dev/ttyACM0', 115200);

// drive(number, speed) :: number: 1-2 :: speed: -100 to 100 
saber.drive(1, 50)
saber.drive(2, -75)

// driveBoth(speed_m1, speed_m2)
saber.driveBoth(50, 80)

// driveTank(speed) :: speed: -100 to 100
saber.driveTank(50)

// Stop
saber.stop()

Unevenness at the edges

Bytepawn - Marton Trencseni — Fri, 30 Oct 2020 01:00:00 GMT

Sometimes we look at the top performers in a field and see obviously uneven representations of groups (gender, ethnicity, etc). There a multitude of factors that can lead to it, such as unfair bias in access to opportunities. Here I will show one unintuitive mathematical effect that can contribute to such unevenness in the case of normal distributions.

COVID19 accelerates dispersion of private health data

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 29 Oct 2020 18:00:00 GMT

Here is something unsuprising, but NEVER repeated enough.

How to get the real PHP memory usage on macOS

Swag Industries — Thu, 29 Oct 2020 13:02:25 GMT

PHP provides the function memory_get_usage(), this is very helpful whenever you find your code using a lot of RAM. But this method has a significant limitation: it only counts memory used by your code directly and nothing out of the zend engine.

The problem is that you may

Mastering TorchScript: Tracing vs Scripting, Device Pinning, Direct Graph Modification

Machine Learning Articles on paulbridger.com — Thu, 29 Oct 2020 08:43:23 GMT

TorchScript is one of the most important parts of the Pytorch ecosystem, allowing portable, efficient and nearly seamless deployment. With just a few lines of torch.jit code and some simple model changes you can export an asset that runs anywhere libtorch does. It’s an important toolset to master if you want to run your models outside the lab at high efficiency. This article is a collection of topics going beyond the basics of your first export.

Problem Solving

denvaar's website — Thu, 29 Oct 2020 07:40:09 GMT

Going meta - Working through a programming problem to understand problem solving techniques.

APEX 20.2: improve Dropzone UX with CSS

Philipp Hartenfeller Blog RSS Feed — Thu, 29 Oct 2020 02:00:00 GMT

Great News: APEX 20.2 is finally out and adds a Dropzone component. For me, it does not look as intuitive as the Plug-In did, so I tried to improve the UX of it with a few lines of CSS.

Notes on Serverless GraphQL with AWS AppSync

swyx's site RSS Feed — Thu, 29 Oct 2020 01:28:50 GMT

My Notes on Slobodan Stojanovic's Serverless GraphQL with AppSync talk

Users Council for breaking up web giants?

Syd Norpol — Thu, 29 Oct 2020 01:26:55 GMT

One day thinking about privacy issues lead me to the idea applying of works councils, but just for users.

What is a Works Council? #

In some countries, a works council can be formed if a company has reached a certain amount of employees. These employees then get to vote for representatives that get stake within the company for representing worker rights. This was needed to counteract company interest over their employees basic needs.

How could this work for users? #

Well, why shouldn’t we extend something like this for users?

The idea is basically, if a web company reaches a certain size (by users/countries/…) - users somehow magically get to initialize the process for electing a users council.

This council would then become more included within organizational structure, would get special insights/space within the organization (including seeing more what happens behind the curtains, …) and get to decide certain decisions within that company that would then hopefully lead to more user oriented needs over profits and data mining.

Why? #

So that’s it. Very simple idea, which will most likely never work out. Yet, I do believe we really have to keep thinking about how we can get back control over the web giants.

2020–10–29: New p-boot features – reboot to FEL/eMMC, 60FPS fix

megi's PinePhone Development Log — Thu, 29 Oct 2020 01:00:00 GMT

Horizon Europe does the right thing

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 28 Oct 2020 20:00:00 GMT

Who says the EU is always bad???

Building Local Legends

Mike Kasberg's Blog — Wed, 28 Oct 2020 16:00:00 GMT

In late 2019, Strava’s product development team began forming ideas about a new way to compete on Strava. Strava segments have always provided a way to recognize athletic achievement through a form of racing. Now, we wanted to also recognize the grit that goes into training. Those athletes who are out on the trail every day, rain or shine, striving for more. Local Legends is the result of these ideas, recognizing the athlete who’s completed a segment the most times in the last 90 days. I was lucky enough to play an important role in building this feature, and in this blog post, I want to tell the story of how we developed Local Legends and explore some of the technical challenges we faced along the way.

Battlestation bookmarks

xenodium.com @alvaro — Wed, 28 Oct 2020 02:00:00 GMT

Let's Fix the News

Tilting at Windmills — Wed, 28 Oct 2020 02:00:00 GMT

As another US presidential election draws near, the brokenness of the media is becoming ever more glaring. It's not exactly a new phenomenon…

On Forever Projects

Ben Overmyer's Site — Wed, 28 Oct 2020 02:00:00 GMT

I just finished reading a blog post entitled The Forever Project by Scott Turner of Here Dragons Abound. It made me reflect on Iron Arachne and what I'm trying to do with it.

Lately I've been taking a break from Iron Arachne, and while on that break, I've found that I lack motivation or interest in continuing it. But this blog post made me reconsider that. It's not that I lack motivation or interest in Iron Arachne, but rather that I've been too focused on developing the site as a tool for others to use and not near enough on what I originally started it for.

Iron Arachne is my "forever project," as the article calls it. It's an endless creative exploration meant to delve into many, many different topics at the drop of a hat. It's not meant to be complete, or bug-free, or even useful. I had forgotten that.

In the context of this revelation, I'm taking a look at what Iron Arachne is currently and where I'd like to take it. I definitely don't want to do another rewrite. However, I think I would like to start exploring more random concepts. To that end, I'm going to follow some of the advice from Scott and write a bunch of new issues in GitLab that are just ideas of things to try - small ones, medium ones, big ones, and pie-in-the-sky ones.

I also will need to reinstate my weekly ritual of coding on Sunday mornings, but I think I'll broaden the scope of that a bit to be "work on Iron Arachne" instead of coding. That means anything that falls within that context is fair game - so, I could code, or I could do artwork, or read about geology, or write issues, or write a blog post, or anything else that fits the mold.

Now that I'm getting this down in writing, I'm finally getting excited about Iron Arachne again. That's a tremendous sign that I'm on the right track.

How free is the internet in 2020?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 27 Oct 2020 20:00:00 GMT

Less than before. Don’t let the pandemic hide that.

Why We Will Never Have Enough Software Developers

Who is Nnamdi? — Tue, 27 Oct 2020 19:55:02 GMT

Developers are dropping out of the profession in large numbers

Differences Between Camera Sensors

bastibe.de — Tue, 27 Oct 2020 12:00:00 GMT

get sRGB coordinates here: https://www.babelcolor.com/index_htm_files/RGB%20Coordinates%20of%20the%20Macbeth%20ColorChecker.pdf

Most cameras have the option to capture raw images, i.e. un-processed image data right from the image sensor. In theory, these images are pure physical measurements of light, and should therefore be very comparable between cameras. But are they? To investigate, I took a raw picture of a color target with each of my five cameras, and compared their output.

Capturing accurate colors is a surprisingly intricate matter, as the color depends on the spectrum of the illumination, the reflective spectrum of the colored object, and the color filters in the camera. I normalized the illumination by taking pictures on an overcast day outside¹, and used a standard color target with 24 colored patches of a certified color.

The resulting colored spectra are captured on the camera sensor by photon counters behind three color filters “red”, “green”, and “blue”², which differ from camera to camera in their spectral sensitivity. The recorded colors then get projected on a computer screen with another set of “red”, “green”, and “blue” LEDs of some different spectral makeup, or printed in three or more inks on paper. And this is then seen with eyes of yet another set of “red”, “green”, and “blue” retinal cells. Let's leave it at color is complicated.

At any rate, I took pictures at base ISO in raw, white-balanced and exposed for the third grey patch, and adjusted in saturation on the red patch. Here are the colors on the color checker and the recorded colors of my cameras:

The image shows the 24 colored patches of the color target³. Within each patch, five rectangles show the recorded color of (in reading order) the Pentax Q7, the Ricoh GR, the Panasonic LX100, the Fujifilm X-T2, and the Google Pixel 2.

Due to my exposure and white balance calibration, the third grey patch is completely equalized and shows no difference between the cameras and the target. The “Light Skin” patch (second on the first row) is also very close to equal, which is an important sanity check for my measurement procedure, as skin tones are optimized by all cameras. Seeing that the skin color patch is one of the most similar hopefully indicates that I did not do anything gravely wrong.

The last row shows greys, which are very similar across cameras (excluding the white patch for now). This indicates that RAW values indeed correspond to photon counts, with little additional processing. I have no idea why the white patch is off. Perhaps due to specular reflections on the white patch? Color is complicated.

As for the other colors, the Panasonic LX100 (center) and the Fujifilm X-T2 (bottom left) seem to record somewhat more accurate colors than the other three cameras. The Pentax Q7 (top left) and Ricoh GR (top right) seem comparatively weak across the board. The Panasonic (center) is great in darker blues and reds and greens, but somewhat weaker in yellows and pastels. The Fujifilm (bottom left) is very good in almost all colors, with pastels somewhat weaker than darker colors. The Google Pixel 2 (bottom right) is better in reds and greens and yellows than in blues and oranges. Pastel colors in general seem weaker than darker colors, which might be due to the same process that affected the white patch as well.

I think I learned something from this experiment: There are indeed color differences between cameras even when shooting RAW, but they are relatively minor. Whenever I see larger differences in my pictures, they are likely (easily correctible) white balance differences, and not sensor differences. When in doubt however, my Panasonic LX100 and Fujifilm X-T2 do capture more lifelike colors than my other cameras.

any kind of artificial light is comparatively terrible for color reproduction!↩
in quotes, because colors are three-channel information, and we're talking about single channels here.↩
encoded as sRGB, which might or might not be displayed correctly on your screen, depending on your OS, your browser, your settings, your room's illumination, and your screen. Color is complicated.↩

Download your podcasts

voussoir.net/writing — Tue, 27 Oct 2020 10:57:05 GMT

Back to writing

Download your podcasts

Introduction
Downloading
Tagging
Cutting ads
Listen & enjoy

Introduction (§)

Over the past year or two, more and more podcasts have been moving to Spotify and pulling their material from their other outlets. This is because Spotify wants to pump up their subscriber counts and is offering cash incentives to podcasters for exclusivity. The only situation in which this can possibly be interpreted as a benefit for the listener is when the exclusivity deal literally saves the podcast from going bust, and even this has caveats. Any other messaging that depicts Spotify-exclusivity in a positive light is pure spin and marketing, as the deal invariably removes control and choice from the listener. You can ignore the "oh, but Spotify's not that bad" crowd, because that's not the point. The point is that being corralled into Spotify is bad, and the same would apply to any other exclusive distribution point no matter how technically good it is.

Not to mention, as with any other material on the Internet, every day that passes is a day your favorite show might disappear for any number of reasons. You should also download your favorite Youtube videos and forum threads, too, but let's save that for another article.

Podcasts are often distributed as 128 kbps mp3 or thereabouts. At this bitrate, you can fit 100 hours of audio in 5.5 GiB. Just do it.

Downloading (§)

I am actually a bad ambassador for this kind of thing, because I like writing my own scrapers and downloaders, which most people won't want to do.

Let's start here: the podcast culture has traditionally used RSS for publishing episodes. Until recently, I never had a use for RSS, but now I see it as an important open format in the fight to keep data out of walled gardens like Spotify.

At this point, you can either search the Internet for "RSS podcast downloader" and find a pre-existing software solution to this problem, or you can continue reading and see how I do it. As you'll find, the nature of RSS does often require getting dirty and putting in some extra labor.

With no background experience in RSS readers, I tried QuiteRSS and have been happy with it. QuiteRSS uses SQLite to store the feeds and news items. With a tool like sqlitebrowser, or just the command line sqlite3, it's then trivial to get the published date and enclosure_url fields which I then throw into my own threaded_dl.py by preparing a file that looks like:

https://example.com/episode1.mp3 Podcastname 2015-10-31.mp3
https://example.com/episode2.mp3 Podcastname 2016-12-25.mp3
https://example.com/episode3.mp3 Podcastname 2017-04-01.mp3
https://example.com/episode4.mp3 Podcastname 2018-05-05.mp3

and calling threaded_dl links.txt 3.

To be clear, I only do this for the initial download of the historical posts. After that, when new episodes are released, I just right-click & save file.

However, you'll find that many or most RSS feeds don't contain the entire history of the show. Squarespace, for example, only publishes the 300 most recent items in an RSS feed. Why? Because that's the number Apple uses and apparently it's easier to follow Apple's lead than to realize that RSS is an open format for a variety of consumers. I have seen other feeds that show only 10 items, which surely must be a result of poor configuration [1].

One thing you can try is searching the Wayback machine for the RSS url. If you're lucky, it will have been scraped in the past often enough so as to contain the entire history, though perhaps in piecemeal. You could download the snapshots, combine them into a single xml file, serve it out of a webserver, and temporarily point your RSS client at localhost so it ingests all those historical posts before pointing it back at the real url. Convoluted, perhaps, but I'd rather do this than inject the posts into the db myself. Then, you can pull the enclosures out of QuiteRSS's database (or from the xml file you just made).

But, in the worst case scenario, you'll find that Wayback doesn't have archives of the feed you want. At this point I write a web scraper in Python to find all the episode dates and links, or I just hit the CTRL+C, CTRL+V gym for twenty minutes, copying each link from the website manually. You may also try Xenu's Link Sleuth or HTTrack to see if they can find all of the MP3 urls automatically. I figure it's an effort I'll only have to do once as long as RSS keeps getting new posts after that.

If the podcaster has already taken the Faustian bargain, you can try using spotifeed.timdorr.com to pull Spotify as RSS, but this will give very limited history and only serves to stem the bleeding.

[1] Looks like the market has an opening for an "RSS rebroadcasting service" that archives feeds and gives clients the whole history every time.

Tagging (§)

Some podcasters add metadata to their distribution files, others don't. Most don't. At least not to my tastes. I recommend MP3Tag for tagging and renaming the files.

Tagging podcasts is a little different than tagging music. Firstly, podcasters sometimes release "bonus episodes" or "0.5" episodes, which cause track numbers to deviate from the episode number as listed in the title. Secondly, music files can usually be named artist - album - track.mp3 without much issue, but podcast titles are often much longer and contain unsafe characters like :, ?, /, etc. So, to keep things simple, I just use podcastname - releasedate.mp3 as the filename.

If the mp3 files don't have the ALBUM or YEAR field set, you can use MP3Tag's "filename to tag" feature to fix that.

MP3Tag is great for batch renaming. In your MP3Tag actions, create a Format value action with the parameters Field=_FILENAME, Format string=%album% - %year%.

Clean up all the files' tags to your heart's content, then select them and run your rename action.

Cutting ads (§)

I don't listen to ads. I recommend mp3DirectCut for cutting them out. If you're going to listen to the podcasts on your PC anyway, you can kill two birds with one stone by listening to the entire podcast within mp3DC and cutting the ads as you go. If you just want to kill one bird at a time, you can skim through the file with the arrow keys and page up / page down. Most people speak differently when they read an ad, or record the ad in a separate take. You can hear these difference as you skim even when jumping 10-30 seconds at a time. Listen for differences in intonation, room noise, or long segments of a single speaker instead of the show's usual back-and-forth dialogue. You will get faster as you get familiar with the podcaster's ad spacing and speech habits.

It is important to use a tool that does lossless copying, which means it doesn't re-encode the audio when saving the new file. If you use Audacity, for example, you are loading the mp3 into a wave in memory at the beginning, then encoding that wave back to mp3 at the end. Re-encoding will cause a slight loss in quality, though I admit it will not usually be noticeable. mp3DC copies the mp3 data directly. This comes with the limitation that you can only cut at frame boundaries, but that's ok because mp3 frames are usually ~26ms long, unlike video formats where keyframes might be many seconds apart.

Listen & enjoy (§)

The final step is to rest easy in the knowledge that no one can take the show away from you at this point. You can take as much time as you need to listen through it, and it'll always be right where you left it. As it should be.

If you do not have the tools or resources to perform these steps, I wouldn't mind doing it for you once or twice. Send me an email via writing@voussoir.net.

View this document's history

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

Emacs: chaining org babel blocks

xenodium.com @alvaro — Tue, 27 Oct 2020 02:00:00 GMT

Recently wanted to chain org babel blocks. That is, aggregate separate source blocks and execute as one combined block.

I wanted the chaining primarily driven through header arguments as follows:

#+name: block-0
#+begin_src swift
  print("hello 0")
#+end_src

#+name: block-1
#+begin_src swift :include block-0
  print("hello 1")
#+end_src

#+RESULTS: block-1
: hello 0
: hello 1

I didn't find the above syntax and behaviour supported out of the box (or didn't search hard enough?). Fortunately, this is our beloved and malleable editor, so we can always bend it our way! Wasn't quite sure how to go about it, so I looked at other babel packages for inspiration. ob-async was great for that.

Turns out, advicing org-babel-execute-src-block did the job:

(defun adviced:org-babel-execute-src-block (&optional orig-fun arg info params)
  (let ((body (nth 1 info))
        (include (assoc :include (nth 2 info)))
        (named-blocks (org-element-map (org-element-parse-buffer)
                          'src-block (lambda (item)
                                       (when (org-element-property :name item)
                                         (cons (org-element-property :name item)
                                               item))))))
    (while include
      (unless (cdr include)
        (user-error ":include without value" (cdr include)))
      (unless (assoc (cdr include) named-blocks)
        (user-error "source block \"%s\" not found" (cdr include)))
      (setq body (concat (org-element-property :value (cdr (assoc (cdr include) named-blocks)))
                         body))
      (setf (nth 1 info) body)
      (setq include (assoc :include
                           (org-babel-parse-header-arguments
                            (org-element-property :parameters (cdr (assoc (cdr include) named-blocks)))))))
    (funcall orig-fun arg info params)))

(advice-add 'org-babel-execute-src-block :around 'adviced:org-babel-execute-src-block)

Before I built my own support, I did find that noweb got me most of what I needed, but required sprinkling blocks with placeholder references.

Combining :noweb and :prologue would have been a great match, if only prologue did expand the noweb reference. I'm sure there's an alternative I'm missing. Either way, it was fun to poke at babel blocks and build my own chaining support.

The X in MDX

Rodrigo Pombo — Tue, 27 Oct 2020 02:00:00 GMT

— at NextJS Conf 2020

Youtube-dl git repo in ipfs

Random Notes by agilob — Tue, 27 Oct 2020 02:00:00 GMT

Following this simple tutorial I put youtube-dl on IPFS. Git repo ends on commit 48c5663c5f7dd9ecc4720f7c1522627665197939. You can see the git files using any gateway under hash QmQ8rwm3guU76oSsfQpo3rYfW91MXhmZ4jZLqKpHVTR4uE. and clone ipfs repo using git command using git clone https://ipfs.io/ipfs/QmQ8rwm3guU76oSsfQpo3rYfW91MXhmZ4jZLqKpHVTR4uE :)

The Social Dilemma

Lambda Land — Tue, 27 Oct 2020 02:00:00 GMT

I just finished watching The Social Dilemma, and here’s my hot take: The Social Dilemma is an emotive, accessible introduction to problems that, without exaggeration, pose an existential threat to life as we know it. If you can, watch it.

Nextdoor is not messy. It's just too fast

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 27 Oct 2020 01:50:00 GMT

It is. Trust me.

TrueNAS Configuration and Maintenance

Kevin Deldycke — Tue, 27 Oct 2020 01:00:00 GMT

A collection of personal notes on the setup, configuration and maintenance of a home-office TrueNAS appliance.

🖥 Hardware ¶

I dedicated a whole post on building the machine , in which you’ll get the detailed bill of material, cost breakdown and parts selection process.

Reset BMC password ¶

If you have one of …

Emotional Budgeting

trv.rs — Mon, 26 Oct 2020 23:28:00 GMT

A wise steward budgets resources with limited quantities because humans tend to spend all their resources on the immediate problem.

Budgeting is twofold: an acknowledgement that a resource is finite and a plan predetermining priorities. When a decision arrives we can say yes to the best and no to what is lesser.

I have been budgeting my money for about five years now. I've budgeted my time for three. Deep Work taught me that willpower or focus is a similarly limited resource¹.

I have come to realize that emotional strength is a limited quantity as well. A long day of stressful interactions and decisions reduces my ability to delight in and engage with my family at the end of the day. In quarantimes, when I am working from home and living at work, the symptoms of emotional drain have been greater. Without the natural rhythms of hallway banter my experience with work can become a constant eight hour stream of possibly stressful interactions. Even on a great day, the weight of potential problems can be enough to deplete my emotional energy. I leave my home office exhausted more often than I'd like to admit.

I should take a moment to say: "emotional budgeting" is a phrase that I've been using in my head. There may be existing books and articles using a different name, or someone using this name for a differenct concept. I like the phrase because I believe in a stewardship model of resource management, and while I applaud aspects of the "self-care" movement, I find the phrasing and movement to be too self-focused for my faith.

What I refer to as my "emotional energy" may be simply "empathy." It is the limited resource that allows us to listen intently to others, to concern ourselves with how others think, and give us the power to place ourselves in their shoes. When you have a difficult conversation with someone about a point of conflict and you end up drained, lethargic… unwilling even to commit to light-weight, pleasant conversation… that's what I define as "low emotional energy."

To fight this drain I took the same approach as with any budgeting problem: I started by tracking my "expenses" and "income," I predetermined my priorities, and I continue to adjust and learn.

Tracking

Journaling

To track emotional energy I employed a simple journaling and review practice. Just noting in a journal or text-file that "difficult conversation about Project Y, feeling tired" is sufficient to teach us what topics drain us.²

I still practice bullet journaling. As I go through the day I list out meetings, projects, and events. At the end of the day… I'll typically prepend a smiley based on how different notes made me feel.³ One trend I've noticed is that whatever mood I start the day in is typically the mood that will define the entire day. This is a powerful insight… and really speaks to the importance of starting my day in scripture and prayer before I grab my phone. If you start tracking your mood with an open mind you may discover similar patterns. There are many similar tools, and many wiser folks with counsel on this.

Budget Busters

Returning to the metaphor of financial budgeting: When my wife and I started budgeting, we programmed into our budget all the categories and recurring transactions we could think of. We were surprised by all the things we forgot to consider… tires wear out, registering our cars, that wedding gift, etc. Over time, we learned to count on all the "budget busters" and make allowances.

Two of my emotional "budget busters":

Career Review season. Whether the reviews are trending good or bad "review season" still is a vulnerable time with the potential for stressful conversations. So I've learned to build into my schedule more "supply" of emotional energy when review season comes around, and to forgive myself for being a little drained.
Social Media "doom-scrolling." It doesn't matter how good the day is, if I drift into the infinity pool of sadness that is social media around election season… I'll lose a lot of energy. I'm sure I'm not the only one.

Predetermine Your Priorities

Once I was tracking my energy I realized I was losing more energy in my workday than I thought. I'm going to assume that most people expend the most emotional energy into what is immediately in front of them… which is work. I therefore assume most folks would need to re-prioritize towards their health, friends, and family.

You may need to take some time with your tracking information and determine what area of your life is lacking the emotional attention that it deserves. If you have a trusted friend or family member they can help you determine which relationships or aspects of your life need more attention.

There are different ways to represent this new priority, but simply knowing it is sufficient as we move into the next phase of adjusting and learning.

Adjusting and Learning

Having learned a little bit about myself through tracking my emotional energy and how it's affected by "budget busters," and having rekindled desire to give my little family the best of my emotional energy⁴, I started devising strategies.

The strategies that have helped me can be largely grouped into two categories: "refilling the tank" to give me more energy, and "limiting the leaks" to help me maintain energy.⁵

Refilling the Tank

Through my journaling, I've learned that if I have certain meetings back to back (especially in the morning!), I'll be pretty drained by lunch and have very little energy going into the evening. I'm still able to work… I just have less "words" to share with my family. 😓 Therefore I've started intentionally planning to refill the tank.

Here are some examples:

Gratitude

A very important strategy for staying positive throughout the day is logging any and all positive interactions for which I am grateful:

a colleague shares some encouraging words.
my daughter brings me a crayon-scrawled gift.
quiet time is particularly fruitful.
lunch is delicious.

All of these are great things to remember! At the end of the day, I'm looking into the next day with more positivity if the log of the previous day is filled with blessings… otherwise I tend to remember only the bad. This is a high-value activity that also lines up with the need to track your emotional state, so definitely try this.

Rest and Play

At the office I'd automatically refill the tank by visiting with colleagues, playing ping pong, taking walks. Perversely, while working at home I feel guilty enjoying the same activities! Intentionally giving myself permission to leave my desk for a restful activity has been critical to having a balanced day with emotional energy left at the end.

Being on "work premises" allowed me to embrace restful pauses in the workday while not feeling like I was "abandoning my post." Leaving my desk at home feels like treachery? I don't think this thought pattern is healthy, and I'm trying to fight it. Worse still it affects the way I treat my "coworkers" at home. At the office I happily invest in relationships because I am "at work." If a colleague needs me to help them move a desk I happily drop what I'm doing… however at home I feel guilty for leaving my keyboard to help my wife move a dresser! It's irrational.

My ultimate contribution to work is focused effort resulting in positive progress… not keeping my butt in seat for the whole day. If I allow myself to spend time on the most important people in my life, (i.e. step away to have a dance-party with my daughter for fifteen minutes) I'll wind up with more energy to expend on work and family.

Intentional Conversation

In a remote work situation there is a lot of responding to task-based communication: meetings, email, chat … but not much opportunity for informal visiting. Without the intermittent chit-chat colleague relationships can warp into viewing people as a list of their demands. We forget to check on other people instead of the projects they owe us.

I've found intentionally engaging in "just visit" time to be a great reminder of why I like working with such nice and interesting folks. It relieves some of the emotional drain by rekindling friendships. I no longer just dread your list of demands… I want to continue our conversations and see how you are doing.

Limit the Draining Interactions

This strategy is a little harder. Depending on your role at work you may not have much control over your schedule.

Thanks to journaling I have learned to predict if a day could be an emotional "budget buster." If I have any control of my schedule, I'll try to insert some rest time or use one of the other techniques above to recover.

A simple tactic: I try schedule date night on a different day than the "heavy" day. It's just wise: if I'm going to prioritize listening and loving on my wife I should choose to select an evening where I'll have as much as possible to give.

Intentional Practice

While I'm honestly not sure how to do this perfectly… I have been attempting to deliberately practice this skill and get better at it.

When I realize that I've had a "draining" day, this is a great opportunity to practice my listening and not default to my selfish relaxation habits. As a christian, the command to love on my wife is not relaxed simply because I had a difficult day at work. 😝 One mini-habit that has helped me is to quickly and quietly set an intention for the next interaction or meeting: how do I want the next person to perceive me? I choose my demeanor and consider how I can serve that person. It's been helpful to me and rescued me from defaulting to whatever my tired face depicts.

This is all something I'm still thinking about and working on… so I'd appreciate any thoughts you have about it. I highly doubt that this is a new idea… there are probably whole books and genres dedicated to this.

If you have any thoughts or research, I'd love to hear about it.

Cal Newport. Deep Work. Grand Central Publishing, 2016. A large part of being an effective worker is limiting the distracting environmental leaks that reduce my daily supply of willpower, and spending it first on the work that most deserves my best focus. Over the past year I've sought to increase my supply of focus by deliberate practice and decrease the amount of distraction. My full review from 2019. ↩
To identify distractions from focus, Cal Newport recommended simply noting the sources of distraction on a sticky note, forgiving ourselves, and trying again. Very similar thing. ↩
I actually have several systems for tracking my emotional energy both on a daily, weekly, and monthly level… but start small and learn from yourself. Everyone is different. ↩
I'm going to focus on my particular journey of trying to refocus on my family in a stressful work season, but pretty much any self-care advice could apply here as long as it aligns with your priorities. ↩
You could basically use the same categories that Shawn Blanc uses for margin: increasing ability and decreasing commitment. It's same thing. ↩

Found-Money Startups

swyx's site RSS Feed — Mon, 26 Oct 2020 17:02:59 GMT

'I will find free money for you' is one of my favorite business models. Here's a quick definition of Found-Money Startups and a short list I've been keeping.

How complex is modern web development

Omar Abid - Personal Blog — Mon, 26 Oct 2020 02:00:00 GMT

Visualizing the complexity of modern web development

Waking Up: A Guide to Spirituality Without Religion

Apurva Shukla's RSS Feed — Sun, 25 Oct 2020 09:00:00 GMT

⭐ ⭐ ⭐ ⭐ A great introduction to spirituality and meditation for a Western audience, best suited towards those who have been conditioned to…

Migrating Data When You Never Erase History

jorin.me — Sun, 25 Oct 2020 08:00:00 GMT

Let's explore the considerations necessary when evolving temporal data over time.

Emacs: quote wrap all in region

xenodium.com @alvaro — Sun, 25 Oct 2020 03:00:00 GMT

As I find myself moving more shell commands into Emacs interactive commands to create a Swift package/project, enrich dired's featureset, or search/play Music (macOS), I often need to take a single space-separated string, make an elisp list of strings, and feed it to functions like process-lines. No biggie, but I thought it'd be a fun little function to write: take the region and wrap all items in quotes. As a bonus, made it toggable.

(defun ar/toggle-quote-wrap-all-in-region (beg end)
  "Toggle wrapping all items in region with double quotes."
  (interactive (list (mark) (point)))
  (unless (region-active-p)
    (user-error "no region to wrap"))
  (let ((deactivate-mark nil)
        (replacement (string-join
                      (mapcar (lambda (item)
                                (if (string-match-p "^\".*\"$" item)
                                    (string-trim item "\"" "\"")
                                  (format "\"%s\"" item)))
                              (split-string (buffer-substring beg end)))
                      " ")))
    (delete-region beg end)
    (insert replacement)))

Communication with Blender via Python RPC

ciesie.com — Sun, 25 Oct 2020 03:00:00 GMT

My current 3D graphics setup revolves around Blender, ZBrush and, less often, Marmoset Toolbag 3. Transferring the meshes between ZBrush and Blender is the constant dance of exporting and importing (yes, I know about GoB but it’s not good enough for me). I strongly dislike this back and forth. I feel it adds a lot of friction, throws me off and wastes a lot of time. That makes me experiment with inter process communication. This is one of those experiments.

Messages from afar

A Matter of Perspective — Sun, 25 Oct 2020 03:00:00 GMT

Herein lies a collection of poems that came to me one evening. I do not claim to have written them, more that I was fortunate enough for them to pass through me.

May you reflect on these as I did when they arose.

Teacher on a sinking ship

Surrender.
Surrender to the crashing waves.
And let each breath be a prayer.
Breathe in this world, and breathe out god.
Let go!
Let in the darkness,
Let out the light!
Our fragile minds delight,
For that which is, on this good night.

Who are you?

I am the teacher that is learning,
The constructor of your destruction,
And the seed in your womb.
I am the silence in the crowd,
And the voice that grows ever softer as you wander into this true world.
Are you listening?
My name cannot be spoken.

From whence it came

Give in to the endless river that flows within us all.
Drink from its sweet waters, and sail!
Fear not of destination,
Or biting hesitation,
For all rivers travel rightly to the source.

Standard ML in 2020

Notes on software development — Sun, 25 Oct 2020 03:00:00 GMT

Incredibly, Standard ML implementations are still actively developed. MLton, Poly/ML, MLKit, SML# and SML/NJ are the most prominent. Discussion on the future direction of Standard ML remains healthy as well.

And somehow OCaml's lesser known cousin still beats out OCaml for multicore threading support (in Poly/ML).

While MLton hasn't merged with MultiMLton or RTMLton to support multicore, a new fork of MLton with parallelism is pretty far along and in active development at CMU.

A commentor shared Manticore, another implementation with parallelism support in active development at UChicago.

Furthermore, the last few years have welcomed some entirely new implementations. WebML, by a prominent open source hacker, is written in Rust and compiles Standard ML to WebAssembly. SOSML is an interpreter written in TypeScript by former students of Saarland University. It features a nifty online IDE.

A commenter shared SomewhatML, an actively developing compiler for Standard ML written in Rust.

There have also been some new experimental spins on Standard ML in the last few years. Morel is an interpreter with some nice syntax extensions written in Java by the author of Apache Calcite. And Bright ML is a spin on Standard ML and OCaml written in Standard ML (and using the abandoned Moscow ML compiler of all implementations).

So if you're looking for an easy intro to the ML family of languages, I still recommend the simplicity and performance of Standard ML and its small but definitely, surprisingly, not dead community. :)

Additional resources:

Are you using Standard ML? Let me know how/why!

Standard ML implementations are still in active development! There have even been some interesting new implementations pop up in the last few years.https://t.co/6kOcMKVfQa
— Phil Eaton (@phil_eaton) October 25, 2020

The story of udemy-dl

Gaganpreet Arora — Sun, 25 Oct 2020 03:00:00 GMT

The recent takedown of the youtube-dl project resurfaced memories of a similar project, udemy-dl, that I worked on years ago. I also had to abandon the project due to a DMCA notice. In light of what happened to youtube-dl this felt like the right time to write about it.

Beginnings #

Back in 2013 when I was starting off as a software consultant, I landed a project to build a desktop GUI application. I found a Udemy course for Python GUI development using Qt. After I started the course I ran into an obstacle. My internet was awful. It was slow (4 Mb/s, if I remember correctly) and disconnected frequently making streaming videos a painful experience.

Culture wars:for minorities, but still relevant

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 25 Oct 2020 02:54:00 GMT

Point is: do you know WHY they are relevant?

A Practical Guide to Gaussian Mixture Models

Luke Salamone's Blog — Sun, 25 Oct 2020 02:10:29 GMT



Are you studying machine learning and want to know more about Gaussian Mixture Models? You’ve come to the right place. I have found other online resources to be difficult to approach and/or lacking crucial details. Here I will try to explain GMMs in plain language.



Managing Python Environments
Luke Salamone's Blog — Sun, 25 Oct 2020 01:45:41 GMT
Need to switch between python versions often? Use pyenv.
Installing pyenv
# install pyenv
curl https://pyenv.run | bash

# check pyenv install location
which pyenv

Install another python version
# see a list of available python versions
pyenv install --list

# check installed python versions
pyenv versions

# installs python 3.7.5
pyenv install 3.7.5

Switch python versions
# use python 3.7.5 everywhere on your machine
pyenv global 3.7.5

# use python 3.7.5 in current directory
pyenv local 3.7.5

# use python 3.7.5 in current shell session
pyenv shell 3.7.5



How much specification does a developer need?
Lucas F. Costa - Blog — Sat, 24 Oct 2020 15:00:00 GMT
Specifications work in two ways: they inform developers what needs to be built, and they constrain the problem space so that tasks become approachable.

Without clear specifications, developers may b...


Disposability - the Missing ility
Greg Herlein — Sat, 24 Oct 2020 11:00:01 GMT
You will get the most bang for the buck from cloud by destroying it.


Unsubscribe Me: How to Disconnect from Internet Chaos
Mike Kasberg's Blog — Sat, 24 Oct 2020 05:15:00 GMT
The Social Dilemma came out about a month
ago and got a lot of people thinking about the influence the internet and social
media have on their daily lives. I’m a programmer, and I’ve worked for an
internet advertising company, so none of this was new information to me. But for
a lot of the American public, watching The Social Dilemma was an eye-opening
experience into the ways that internet companies try to track and manipulate
people. Many of my friends used words like “unsettling” and “eerie” to describe
how they felt after seeing the film, but most of them also weren’t really sure
what to do about it. They weren’t sure how to make positive changes in their own
lives.

So in this blog post, I want to explore some simple changes that I’ve made, and
that you might think about making too. I’ve been doing some of these things for
years and I’ve started some much more recently, but with these issues at the
front of my mind recently, and I think I have a reasonable approach to at least
heading in the right direction. The Social Dilemma website provides its own
Take Action page, and most of
the advice there isn’t bad as a starting point. Most of my own suggestions are
minimally invasive and relatively simple. I haven’t completely stopped using
social media, and I wouldn’t necessarily recommend that as a solution, unless
you want to of course. I think it’s important to be pragmatic about what you can
do in a way that still allows you to maximise the personal value you get from
online services.

Your Time is Valuable to Companies

When people ask me about the movie, I always say that I think it’s a really
well-done visualization of the competition for your time. Companies want your
attention because your attention is valuable. This isn’t new. In the 80s and
90s, you saw ads on televisions, billboards, newspapers, and magazines.
Companies want you to notice them so they can sell you a product. And for the
most part, people have been pretty happy to make that trade. Who among us
doesn’t get excited for a 20% off coupon at your favorite store?

In addition to straight-up ads, your time is valuable to companies in other
ways. Have you ever used a free app that didn’t have ads? Instagram didn’t have
ads before late 2013, for
example. So why did Instagram care about your time if they weren’t showing you
ads? Your time is valuable to companies even if they’re not directly advertising
to you. Instagram is valuable as a company, in part, because of the size of its
user base. Apps with a big user base are valuable because they can monetize that
user base in a variety of different ways – from showing ads to selling data or
getting you hooked on a service they’ll charge for.

The Competition for Your Time

So apps want your time and they’re willing to compete for it. But they’re not
the only thing that’s competing for your time. Think about all your interactions
with media in a typical day. Maybe you listen to a podcast in the morning or on
the way to work. Does the podcast have ads? What about the app you use to play
it? Early in the workday, you notice a few notifications from your favorite
social apps. A couple likes, maybe a comment. Also, a notification from one of
your restaurant apps that they’re running a promotion today. Later, your phone
rings. It’s a spam call. You also notice a couple spam text messages that you
delete. In the afternoon, you check your email. Out of all the unread emails in
your inbox, you delete about half without reading them. When you get home, you
turn on your Fire TV where you see ads for a bunch of TV shows or movies. Then
you open the Netflix app where they’re promoting today’s top new shows. Everyone
is competing for your attention all the time.



Ads aren’t new, but the intensity of the competition for your attention and time
is higher than it’s ever been before. It’s in companies’ interest to have as
much of your attention as they can, but it’s in your interest to give them the
minimal amount of attention necessary to get the value you want from them.
Companies are ruthless and aggressive in their pursuit of your attention, and
they’ll win the battle if you don’t put in any effort. Fortunately, there are
some low effort solutions that can produce very beneficial results, and my
approach is focused on this.

Optimize Your Digital Life

Email

Companies love email. Sending emails for marketing purposes is a billion-dollar
industry. For anyone who wants to promote something to you, email is a way for
them to get your attention. Your email address itself is valuable to many
companies because it allows them to (possibly) show you ads when they otherwise
wouldn’t be able to show something to you. (This is why companies will do things
like giveaways in exchange for your email address.) They will hire marketing
professionals whose job is to write a clickbait subject line to get you to open
the email so they can show you their ad. And they will track when you open the
email so they know if it’s working. (Images are used to track views, which is
one reason Gmail blocks images from unknown senders by default.) Unfortunately,
spam is a hard problem to solve, and there isn’t a perfect solution. But there
are some steps you can consider that might help reduce the way email intrudes
into your life.


  Unsubscribe from anything you don’t really need. If you’re not sure, it’s
probably best to unsubscribe. You don’t really need to be bothered by your
college alumni newsletter, and you don’t really need to be bothered by that
mailing list you signed up for 2 years ago. In fact, you probably don’t need
any sort of regular communication except perhaps from an organization you
actually interact with on a very regular basis.
  Try to avoid giving out your email address. It’s really not worth it for
whatever one-time promotion you’re getting. Or perhaps use a second email just
for spam.
  Turn off all your email notifications. Email is asynchronous
communication. It should not ever send you a notification to interrupt
whatever you were doing. Not on your phone, not on your computer. Check your
email on your own schedule at your leisure.
  Keep an eye out for better email services. Hey is trying
to make email better and they have some interesting solutions to the spam
problem. Maybe they will help, or maybe the industry as a whole will come up
with some new solutions based on their ideas. We can be hopeful!


Social Media

Like email, social media is another place companies fight for your attention.
You want to log on to Facebook to see your friends pictures, companies want to
show you their ads instead. Some of this is obvious – like an advertisement in
your news feed. Sometimes, it’s less obvious – like an athlete you follow
posting a photo with a big logo in it or a band you like posting about a show in
a far-away city. Whatever the case, it’s likely that you’re not getting the most
value from whatever social media platform you’re trying to use.


  Unfollow things you don’t want to see. You don’t necessarily have to
“unfriend” someone to “unfollow” them (depending on the platform). But if
you’re following things you’re not interested in (whether it’s people, movies,
companies, or brands), you’re just giving away space in your news feed that
doesn’t even count as an ad.
  Like email, disable all notifications. For 99% of the people in the world,
the ideal way to use social media is on your own schedule, like email. You
don’t need to be notified on your phone or computer when a friend posts or even
replies to a comment. It will be there in your feed when you decide you want to
look at it.
  Consider using Facebook
Container for
Firefox. It prevents Facebook from tracking your activity across many
non-Facebook websites.
  Be aware of the ways you might be targeted by advertisers. When you share
information with Facebook (likes, interests, your profile page), they can use
that information to show you specific ads. This isn’t necessarily the worst
thing in the world, but it’s good to be aware that they’re using the info you
give them to do this.
  Consider ditching social media completely. Do you really need to use Facebook
and Instagram, or can you just email your friends some pictures from your last
trip instead?


Other Apps

Apps are the new email. Shortly after email was born, companies began figuring
out how to use it to advertise. Now, companies are figuring out how to advertise
on phone apps. Most of the time, “ads” on phone apps come in the form of
notifications. Have you ever installed a new app and been surprised about all
the notifications you start getting? “Welcome to so-and-so, let us show you
around.” “5 tips to get started quickly.” “Add this to your account.” “We
noticed you didn’t log in yesterday.” “Act now for a 10% discount.” You don’t
need any of this. Notifications on your phone make a noise to interrupt you.
They cut off the real-world conversation you might be having, or they break your
train of thought. As such, you should only get notifications for things you’d
actually want to interrupt you. Namely, things that require immediate action.
A phone call. Perhaps a text message. Maybe meeting reminders from your calendar.
Of all the apps on your phone that can send you notifications, I bet about 90%
of them can wait ‘til later. I think it’s one of the biggest failures of UX
design on both Android and iOS that notifications are enabled and noisy by
default. Instead, they should probably be opt-in per app (off by default), and
silent by default (or even have a better mechanism for a “notification inbox”
that can be dealt with at your leisure). Regardless, there are some things you
can do that make a really big difference here.


  Disable notifications for any app that doesn’t need to interrupt you. On both
Android and iOS, it’s possible to remove the app’s notification permission.
Any time an app sends you a notification that you don’t care about, do this.
  If you can’t disable all notifications for an app, see if you can turn off all
but the type you care about. For example, maybe you want to be notified of
direct messages on Twitter, but you can still turn off all other notification
types for that app.
  
    If you can’t disable a notification (for whatever reason), would it make sense
to silence it? Silencing a notification means no vibration or noise, but you
can check it at your convenience in the notification bar. You should be able to
do this for any app from the system settings.

    
  
  Can you just uninstall the app? Sometimes that’s an easier solution for
badly-behaved apps than trying to figure out the notification permissions.
Your phone is your space, don’t let companies/apps intrude into it who don’t
need to be there.
  Do you need to limit your screen time? It’s pretty well documented that people
will spend hours using phone apps because of the dopamine hit the apps
provide. Like many things, it’s probably fine to use apps in moderation, but be
wary of becoming sucked in. Maybe talk to someone in real life or go outside
instead. And if you need help breaking the habit,
iOS and
Android come with tools to monitor
and limit the time you spend on your phone.


Internet Ads

As long as people have been using the internet, there have been ads on it. These
days, ads have become much more sophisticated. Large companies – including
Facebook and Google – gather data about nearly every website you visit. Even if
it’s the website of your local news station or a blog you like to read, it’s
likely that at least one big company knows about that visit, and they use this
information primarily to figure out what your interests are. This is why, at
times, some websites seem to have an uncanny ability to advertise whatever you
were just searching for or shopping for. It’s probably impossible to get rid of
100% of ads on the internet, but there are some simple things you can do to
protect your data and reduce the number of ads you see.


  Use Firefox instead of Chrome.
Chrome is owned by Google, a large advertising company. Firefox will protect
you from things like FLoC that Google is integrating with Chrome.
  Use an ad blocker browser extension. I use uBlock
Origin and I think it’s the best because it’s open
source and not controlled by a company.
  Google collects info about what you type into the search box, so consider
using a search engine other than Google and changing your browser’s default.
DuckDuckGo is a good alternative to try.
  If you’re tech-savvy, install a Pi-hole on your home
network. This is the best generic ad-blocking solution I’ve found.
Installation and configuration is somewhat tricky (you need to understand some
basics about your home network), but there are guides available that make it
easy enough to do.


Protect Your Data

If you’re worried about protecting your data online, you’re probably also
worried about geting hacked. One of the most common ways someone might hack you
is from a major data breach. Let’s say you have an account on LinkedIn. And
let’s suppose LinkedIn gets
hacked.
Maybe you’re not worried about this - what’s a thief going to do, steal your
resume??? Well, one of the problems with large data breaches is that people can
use that data to figure out the password you used on LinkedIn. And then they can
try to use that password with your email on hundreds of other sites. Amazon,
your bank, your email, Facebook, the list goes on. So how do you protect your
data online?


  Use a password manager. A password manager helps
you use a different password on every site, so a hack on one site doesn’t
compromise your information on all the others.
  Use a strong master password that’s never been used anywhere else. There are
two pretty good options for making up a password:
    
      Think of a sentence with at least 14 words. Maybe a favorite quote. (But not
something you say all the time!) Use the first letter of each word. Even
better if you include capitalization and punctuation in the sentence.
      Generate a random Passphrase.
    
  


Who’s Paying?

If there’s one important thing to remember from this whole article, it’s that
nothing is free, even on the internet. If you can’t figure out who’s paying
for something, it’s you. You’re exchanging your time (or the ability of
companies to show you ads) for the use of a service. Many people have come to
expect free services on the internet (like email, Facebook, Twitter, YouTube,
etc.). But I think people are finally beginning to realize that they don’t want
to trade their time and attention to companies in the way that they have in the
past. So I think it’s likely that we’ll see a shift in the coming years, where
more online services do away with ads and data tracking and will charge a
nominal fee for their use, and many people will be happy to pay this fee to have
their privacy protected and be free from ads. If you’re paying a company for
their service, the company can focus on building things in a way that benefits
you, the user, rather than looking for ways to benefit their advertisers.


CSS Box Model
Dylan Paulus' Blog — Sat, 24 Oct 2020 03:00:00 GMT
CSS is one of those technologies that has a low barrier to entry (good thing 🎉), but because of this sometimes how it works can seem like magic. It's easy to get started writing CSS, so we dive head first into it. Sometimes we smash our face into the concrete with frustration. Why won't my text move over yonder? Where did that scroll bar come from? How do I center this junk?
This is a series I'm starting called, "CSS: The Important Stuff". The goal is to take a dive into the mechanics of CSS so we can get a better intuition when styling and positioning elements. In part one we'll take a look at Box Model; the underlying layout of the web.
What is the Box Model?
It comes with great danger to myself to tell you that the web is made of boxes. Yes, it's boxes all the way down. Take a look at this website—open the Element Inspector in the browser's dev tools and add * { border: 1px solid red; }. Look at all those fine boxes.

  

The browser uses boxes to layout and render our websites. Within each box there are four properties: content, padding, border, and margin. Boxes are like onions, they have layers. Each property defines the size of our box (ie. width and height), styling, and how content around the box flows.

Content is what we're usually thinking of when we create HTML elements. Text, images, video, etc. This is at the center of the box
Padding gives space around the edges of the content. Separating the content from the border
Borders are used to separate boxes from each other—as seen in our red border example above
Margin is non-stylistic spacing around the box. This is useful to separate boxes from each other and position boxes


Why does this matter?
Dimensions
There are different ways to determine the width and height of a box. Is the box's dimensions determined by all its layers (margin, border, padding, content)? Or is it determined by the content?
The browser made a choice in box-sizing and the default is content-box. When we give our element a width and height, lets say .box { width: 100px; height: 100; } , the default behavior with content-box is to give the content layer the width and height of 100px. But, the box still has extra layers with padding and border. We would assume if we gave the element of 100px that it would be 100px, but the browser needs to add the values of padding and border on top of the content to get its "rendered dimensions".
Given the styling on our moon icon below, what would you expect the width and height to be on the box?
.moon {
  box-sizing: content-box;
  width: 100px;
  height: 100px;
  padding: 10px;
  border: 1px solid #FFF;
}


That's easy it's 100px... wait, 122px, what? How did we get 122px? Remember when we gave the width and height of 100px it only applies to the content layer. We also applied a padding of 10px on each side of the box and a 1px border around it. If we were to calculate the width of the box, we start with 100px, add 10px for the left padding, add 10px for the right padding, add 1px for the left border, and finally 1px for the right border.
$$

100px + 10px + 10px + 1px + 1px = 122px

$$
Seems a little counter intuitive—if I set my box to have a 100px width then it should be 100px darn-it! Luckily, there are more ways for the browser to calculate a box's size. When I think of the size of a box, I think of border-box . Think of the layers of a box, and when assigning box-sizing think of the value as being cascading. When an element has content-box that means the browser determines the size of a box starting from content and adding any child layers. Since there are no children to content width/height only applies to content. When using border-box, width/height is set for the border, padding, and content.
Let's take a look at the same moon icon, but apply border-box instead.
.moon {
  box-sizing: border-box;
  width: 100px;
  height: 100px;
  padding: 10px;
  border: 1px solid #FFF;
}


What we see is something quite different, the width and height of the moon is now a hard 100x100. This isn't magic though, the missing dimensions have to come from somewhere. Using border-box the content layer gets shrunk to make room for padding and border. With content-box our width was guaranteed to be 100px, but what is the width of our content now with border-box?
$$

100px - 10px - 10px - 1px - 1px = 78px

$$
To me border-box is easier to intuit and reason around, and becomes my default, but if the width and height of the content matters more than the visible box itself, use content-box.


References

Introduction to the CSS basic box model
The box model



SQL Injection Overview
Thomas Countz — Sat, 24 Oct 2020 03:00:00 GMT


An SQL Injection occurs when untrusted input is used directly in the construction of an SQL query. This attack is commonly executed by introducing a meta character (such as a comment) into a data plane in such a way that allows an attacker to add commands to the control plane. Essentially, when building a SQL query from user input, an attacker can insert SQL instructions that cause the application to behave in unintended ways.


  SQL Injection flaws are introduced when software developers create dynamic database queries that include user supplied input.

  —OWASP SQL Injection Prevention Cheat Sheet


Authentication Form Example

Let’s take a login form that requires a user enter a username and password to log in.



In the happy-path case, a user enters in their details, presses “Login,” and a request is sent to a server where we take what the user enters, search the database for that user, and then log that user into our web application.

# SQL Injection Vulnerable Pseudocode
# -- snip --

let user = ORM.query(
  "SELECT * 
    FROM users 
    WHERE username = '#{params['username']}'
    AND password = '#{params['password']}';"
).execute()[0];

# -- snip -- 

log_in_user(user)


Using a tool like SQL Fiddle, you can see how a query like this will behave in PostgreSQL 9.6. Here’s is the schema and seed data for our example:

CREATE TABLE users(
   id        INT PRIMARY KEY   NOT NULL,
   username  VARCHAR           NOT NULL,
   password  VARCHAR           NOT NULL
);

INSERT INTO users (id, username, password) 
VALUES (1, 'sarah', 'good-password');

INSERT INTO users (id, username, password) 
VALUES (2, 'thomas', 'password123');

INSERT INTO users (id, username, password) 
VALUES (3, 'patrice', md5('palm-kumquat-futon-padden'));



After the user submits the username:

thomas


and password:

password123


the resulting query will look like this:

SELECT * 
FROM users 
WHERE username='thomas' AND password='password123';


and the following results will be returned:

| id | username |    password |
|----|----------|-------------|
|  2 |  tcountz | password123 |


As expected, our backend code then takes this result, and calls the log_in_user() function which authorizes the user to access certain parts of the application.

Authentication Attack

The vulnerability here will allow an attacker to log in as any user.

Because user input is interpolated directly into the SQL query, we can have the server execute arbitrary SQL statements. Let’s take a look at an example that would allow us to log into any user, given we have their username.

After the user submits the username:

patrice'; --


and a blank (or arbitrary) password, the resulting query looks like this

SELECT * 
FROM users 
WHERE username='patrice'; --' AND password='<anything>'


and the results:

| id | username |                         password |
|----|----------|----------------------------------|
|  3 |  patrice | 26e9053a783f364d949b4e400dd2f68c |


Now, our application, again, takes the 0th results (this time the user patrice) and logs them in.

What Happened?

Even though we hashed Patrice’s password, our query was vulnerable to interpreting the ', ;, and -- PostgreSQL meta characters.

Firstly, the single quote: ', ends the string (or VARCHAR) that we’re searching for; in this case the username patrice. Next, the semi-colon ; represents the end of the SQL statement. Finally, the -- represents a comment and tells PostgreSQL to ignore everything that comes after.

This effectively makes our query look like this:

SELECT * 
FROM users 
WHERE username='patrice';


Which, as we’ve seen, and as we expect, returns the patrice user and logs them in. Now our attacker has been authenticated and has access to Patrice’s account!

This combination: '; --, and others like it, show up often in SQL injection attacks and it works by prematurely ending a SQL statement.

Data Destruction Attack

In the example above, the 0th row of the results returned from the query will be passed into the log_in_user() function, but the scope of this attack vector isn’t limited to logging in.

As an example of how we can attack the server to execute any SQL, take this example where we destroy the users table.

If we enter a username of:

'; DROP TABLE users; --


and a blank (or arbitrary) password, the resulting query looks like this

SELECT * 
FROM users 
WHERE username=''; DROP TABLE users; --' AND password='<anything>'


The user-facing effect of this query might not tell us exactly what has happened, but a developer might see something like this show up in the logs:

ERROR: relation "users" does not exist


What Happened?

Similar to the first attack, we’ve cut the original query short and this time, we’ve injected our own query to drop the users table.

Prevention

Parameterized Queries


  The use of prepared statements with variable binding (aka parameterized queries) is how all developers should first be taught how to write database queries.

  Parameterized queries force the developer to first define all the SQL code, and then pass in each parameter to the query later. This coding style allows the database to distinguish between code and data, regardless of what user input is supplied.

  — OWASP SQL Injection Prevention Cheat Sheet


Our code was vulnerable because we use string interpolation to build an SQL statement directly from user input. Instead, we should “parameterize” our query by using whatever tools our language gives us to separate the data plane (input) from the control plane (SQL). This is the idea of using variable binding (placing user input into a type of variable) with prepared statements (the rest of the SQL that we don’t want the user to be able to alter).

The way to code this varies depending on the language you’re working with, so check out the OWASP SQL Injection Prevention Cheat Sheet section on parameterized queries.

Other Defenses

Another defense against SQL injection are stored procedures, which are predefined SQL statements stored in the data table. These procedures can have parameters and can effectively be similar to constructs from different languages. Read more here.

To read more about allow-listing or escaping user input, see the rest of the OWASP SQL Injection Prevention Cheat Sheet.

Next Steps


  Research SQL injection prevention for your ORM/database
    
      e.g. https://guides.rubyonrails.org/security.html#sql-injection
    
  
  Aim to separate queries and data (parameterized queries)
    
      e.g. https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Query_Parameterization_Cheat_Sheet.md
    
  
  Consider SQL Injection when reviewing code


Code Review


  Aim to validate user input by testing type, length, format, and range.
  Avoid building SQL statements directly from user input.
  Implement multiple layers of validation.
  Avoid concatenating user input that is not validated; this is the primary point of entry for script injection.
  You should review all code that calls execute(), exe(), and any SQL calls or commands that can call out outside resources or the command line.


Example Code Snippets

Ruby/ActiveRecord

# SQL Injection Vulnerable Ruby/ActiveRecord
# -- snip --

user = User.where(
    "username = #{params[:username]} AND " \
    "password = #{params[:password]}"
).first

# -- snip --


# SQL Injection Safe Ruby/ActiveRecord
# -- snip --

user = User.where(
    username: params[:username], password: params[:password]
).first

# -- snip --


Elixir/Ecto

# SQL Injection Vulnerable Elixir/Ecto
# -- snip --

query = """
  SELECT *
  FROM users
  WHERE username = \'#{params["username"]}\'
  AND password = \'#{params["pasword"]}\';
"""

user = Ecto.Adapters.SQL.query!(
    MyApp.Repo, query, []
)[:rows][0]

# -- snip --


# SQL Injection Safe Elixir/Ecto
# -- snip --

query = """
  SELECT * 
  FROM users 
  WHERE username = $1 
  AND password = $2;
"""

user = Ecto.Adapters.SQL.query!(
    MyApp.Repo, query, params["username"], params["password"]
)[:rows][0]

# -- snip --


Java

// SQL Injection Vulnerable Java
// -- snip --

String username = request.getParameter("username");
String password = request.getParameter("password");

String query = "SELECT * FROM users WHERE username = "
    + username + " AND password = " + password + ";";

Statement statement = connection.createStatement();
Object user = statement.executeQuery(query).getObject(0);

// -- snip --


// SQL Injection Safe Java
// -- snip --

String username = request.getParameter("username");
String password = request.getParameter("password");

String query = "SELECT * FROM users WHERE username = ? AND password = ?";
PreparedStatement pstmt = connection.prepareStatement(query);
pstmt.setString( 1, username);
pstmt.setString( 2, password);

Object user = pstmt.executeQuery().getObject(0);

// -- snip --


Resources


  https://github.com/OWASP/railsgoat/wiki/R5-A1-SQL-Injection-Concatentation
  https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html
  https://www.websec.ca/kb/sql_injection
  https://owasp.org/www-community/attacks/SQL_Injection
  https://www.websec.ca/kb/sql_injection





Angrynomics, and my two cents to fix it
Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 23 Oct 2020 21:00:00 GMT
Don’t know angrynomics yet? You really should.


Running FastAPI applications in production
Posts by Petr Stribny — Fri, 23 Oct 2020 19:06:27 GMT



On Earth We're Briefly Gorgeous
Apurva Shukla's RSS Feed — Fri, 23 Oct 2020 14:57:08 GMT
⭐ ⭐ ⭐ ⭐ Vuong is someone who uses words in a way that I haven’t seen before. His words are raw, loving, violent, kind, hurt, political, but…


GitHub Actions - Copy to S3 on Commit to Master
Greg Herlein — Fri, 23 Oct 2020 11:00:01 GMT
GitHub Actions can be for more than CI/CD.  You can also use them to just simply publish to S3.


Broken Femur, Mutual Care, ROI
Yasha Solutions — Fri, 23 Oct 2020 03:00:00 GMT
Margaret Mead was a controversial anthropologist from the 60’s in america. One thing not controversial though, and maybe that has made more famous recently than any of her research or publications - is the story of a short annecdote, which has become viral all over the internet.
The story goes like this:
 “A student once asked anthropologist Margaret Mead, “What is the earliest sign of civilization?” The student expected her to say a clay pot, a grinding stone, or maybe a weapon.


Lang. Identification in Code Mixed Text - Lit. Review 2
Playground — Fri, 23 Oct 2020 03:00:00 GMT
I am reviewing the literature available on language identification for multilingual documents, focusing on Indic languages. I’ll try to cover it in chronological order, but there might be a few misses here and there. After a decent coverage of the research, I expect to have enough understanding to discuss the challenges present in this task in a code-switched setting and its importance.


Rewilding half the planet? OK, if WE do it
Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 22 Oct 2020 21:00:00 GMT
Where “WE” means ALL humans, of course.


30-year Illusion of a Conscious Effort
THINK@RJ — Thu, 22 Oct 2020 17:44:00 GMT
I turned 32 yesterday and I wanted to do this last year but Transcelestial's Series A raise kept me busy. 
Below is a non-comprehensive list of elements from my personal framework that I have built over the years. It's a glimpse of how I think, what


Golang Unikernel Is Reality!
Greg Herlein — Thu, 22 Oct 2020 10:00:01 GMT
Back in January of 2018 I blogged on LinkedIn asking the question if golang would enable development of micro-kernels.


COVID-19's Impact on IPv6 Traffic
IvyMike.dev — Thu, 22 Oct 2020 10:00:00 GMT
COVID-19 has had a significant impact on IPv6 traffic.
Among all of the other crazy impacts that COVID-19 has had in 2020, I did not expect COVID-19 to change the fundamental protocol mix of packets on the internet.  But it did!  Read on to find out how.
IPv6 primer
A …


Versioning large files in git with DVC
Posts by Petr Stribny — Wed, 21 Oct 2020 13:06:27 GMT



Internet centralization is inevitable, and may even be good
Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 21 Oct 2020 07:00:00 GMT
Here is something about the “logic” of the Internet that we should not ignore.


How to Improve Interface Responsiveness With Web Workers
Danny Guo — Wed, 21 Oct 2020 03:00:00 GMT



Adapting types to fix SQLAlchemy's "can't adapt type" error
Gaganpreet Arora — Wed, 21 Oct 2020 03:00:00 GMT
I have recently been using Pydantic a lot, which is pretty cool since it lets me specify concrete types on my previously untyped data.
Until I ran into an error when I was inserting data stored in a Pydantic model to my postgres database using SQLAlchemy.
Problem #
I have a Pydantic model like this to define my incoming data from a request:


My Wireless Internet Setup
Adriano Caloiaro's personal blog — Wed, 21 Oct 2020 03:00:00 GMT
If you’re reading this, you’re probably familiar with the fact that I’m permanently roaming, and I’d be foolish not to acknowledge the importance of my wireless setup in making that possible. Getting – and staying – connected while permanently roaming is nothing like paying the local cable cabal to get your house or apartment connected.
Remember the last show you binged on Netflix over the weekend? That consumed 20x most mobile users’ monthly data allotment, but you never had to think about data because when you’re wired you generally don’t pay for consumption. How many hours of video conferencing do you do every day? That’ll be ~1GB/hr, please.
Let me make one thing clear from the outset: I’m by no means an expert in wireless technology, and there may be many better ways to get connected than the one(s) I’ve chosen. So if you spot a flaw or have suggestions for me, be sure to get in touch and let me know. I’ll keep this post up to date as reasonably possible as I get more feedback and make changes in the future.
I also feel compelled to call out “Unlimited Data” plans before we get started. For us mere mortals who are not grandfathered into legacy plans: unlimited data plans no longer exist. You may find a fly-by-night company offering unlimited data plans for a short time, but you’ll probably soon realize that they’re doing it in violation of their agreement with the upstream provider when they get shut down. In the best case scenario, you’ll need to get a new SIM card if they’re able to remain in operation and switch to a new upstream provider. These fly-by-nights are more commonly known as MVNOs (Mobile Virtual Network Operators). They’re the likes of Cricket Wireless, Boost Mobile, etc. These companies are piggybacked on AT&T, Verizon, T-Mobile, and Sprint. As a result, they’re subject to the true operator’s rules and the terms of their agreement.
Figuring all this out was a bit of a journey for me and I’ve had a ton of conversations with friends and acquaintances about the details. So I figured it was time to finally capture the details.

Choices
Between equipment and service providers, there’s no shortage of choices to be made early in the wireless setup journey. I’ll try to capture a non-exhaustive list of pros and cons for a few options before getting into the choices I’ve personally made, and the motivation for having made them. However, be sure to consider your data needs and access patterns before taking my opinions and choices as gospel, because if you won’t be working or streaming media – your choice might be quite simple: cell phone hotspot.
Equipment
Equipment is the most technical aspect of getting yourself connected while roaming. For the time being, I’ll forego talking at all about dedicated antennas, because that’s a topic that is both very deep, and one I cannot speak very intelligently about. However, if you want a serious setup, you’ll want to buy a dedicated antenna; trust me.
Cell Phone Hotspot
This is the most obvious choice if you’re just dipping your toes in the roaming waters. It doesn’t require any equipment or service that you don’t already have, and is actually quite fast and versatile. If cell phone antennas were as good as external, dedicated antennas, and the wireless cabals would stop pretending that anyone who wants to hotspot from their phone is some kind deviant, phone hotspots would be a fantastic option. And for many people, they still can be.
Pros

No new equipment or service required
Cheap

Cons

Cell phone antennas don’t have enough gain to pick up the weak radio signals that whisper across BLM (Bureau of Land management), National Forest, and other public lands. “No bars” will quickly render those lands uninhabitable.
The wireless cabals often place very low “Hotspot” data caps on their “Unlimited” plans. While on-device data may be relatively unlimited, you’ll find that hotspot data is capped at 5-15GB.

Signal Boosters
Signal boosters can be great, especially if you’re using your cell phone as a hotspot. Boosters can range from tens of dollars to several hundred. These can be thought of as cell signal re-transmitters; they make a weak signal stronger – which is why they’re a good complement to a cell phone hotspot. I’ve never owned one, so be aware that my opinion is of limited value here.
Pros

Usually cheaper than dedicated wireless routers.
Can be used in conjunction with your existing equipment (cell phone)

Cons

Don’t provide connectivity to non-cell devices (laptops, tablets, smart TVs, etc.)
The range over which they amplify cell signal can be quite small. Sometimes less than a 10 feet.
Not all allow for external antennas, so their gain (ability to pick up signal) and placement can be limited.

Wireless Routers
Wireless routers are the most versatile option. And generally the most expensive.
Pros

Usually provide a WiFi network, allowing you to bring non-cell devices online.
Have external antennas, allowing for optimal placement/gain.
Can be purchased directly from your wireless provider or 3rd party.

Cons

Many wireless providers only officially support their own “hotspot” devices. E.g. “Nighthawks” and “MIFIs”
Can be quite expensive

Wireless Providers
This section could become very long. So instead of providing all the details here, I’ll point you to the Mobile Internet Resource Center, which is an invaluable resource for understanding the current mobile service provider landscape. Things change fast when it comes to providers, so by the time you read this there may be a new kid on the block. I used Mobile Internet Resource Center extensively when I was researching both gear and providers.
Quick Summary
Instead of getting into the details here, I’ll leave you with the most important thing that you should take away:
Redundant service providers is essential
I cannot stress that enough. If you can afford it, and you know you’ll move locations often – have at lease two service providers at all times. Whether it’s Verzion, Sprint, T-Mobile, or AT&T – have at least two. If you can afford more, get more than two; especially if you don’t have a direct relationship with the provider and one is at risk of going out of business (i.e. you’re connecting through an MVNO).
Throttling and Deprioritization
You’ll need to know the difference between throttling and deprioritization before purchasing a data plan. Every wireless provider engages in one, if not both of these two practices, and the difference between them are stark.
Throttling
Providers may refer to throttling by some other name, but the details are generally the same.
Scenario:

You have an “unlimited” data plan with some fine print that says after X gigabytes of usage, you’ll be throttled to 2G speed.


You’re half way through the month and you’ve already consumed X gigabytes of data. You try to load https://reuters.com to see what insanity the US cheeto-in-chief has been spewing, but the page won’t load.

Now it’s important to know how fast “2G speed” is. Almost all providers throttle to 2G speed when they throttle, so make no mistake: 2G SPEED IS WAY TOO SLOW
You’re not getting anything done at 2G speed. Specifically, it’s 128Kbps, or an ISDN line, or ~2x the speed of your childhood AOL dialup connection. Scientifically, it’s the speed at which you test a web page in your browser and you give up saying: “Nope, no internet here”. So if you’re getting throttled, you’re neither playing nor working on the interwebs. And let me be clear, if you’re getting throttled to 128Kbps/2G speed, you’re lucky if you see that kind of speed. You’re probably running at 1995 56K dialup at best.
Throttling to 2G speed on a 4G data plan should be illegal. That’s not hyperbole.
Deprioritization
This is the lesser of two evils.
Scenario:

You have an “unlimited” data plan with some fine print that says after X gigabytes of usage, you’ll be deprioritized.


Someone who hasn’t exceeded their soft cap needs to download a file and the cell tower you’re on is already overloaded. Your Netflix stream gets a little stuttery.

This one’s a bit more ambiguous. But in my experience, and in the places I tend to stay, it’s hard to tell the difference between mediocre signal and deprioritization. Basically, your traffic gets put behind everyone else’s traffic. If you’re somewhere rural, you’re probably not going to notice because service already sucks. That’s the reality.

My Setup
If you look at the pros/cons above, you’ll probably have guessed my choice. I decided to purchase a LTE Router with an omnidirectional antenna, and service provided by Verizon and AT&T.
LTE Router
I’m specifically calling out that I have a LTE router and not 5G because in 2020 it’s relevant to note that we’re on the verge of a nationwide 5G rollout. There are two reasons I chose 4G LTE:

5G is far from rolled out nation-wide and coverage in rural areas hardly exists.
5G in the high frequencies does not penetrate obstacles or refract well. Making it non-ideal for hilly and wooded areas.

My router is a Sierra Wireless RV55. It’s not cheap, but it’s reliable and rugged as hell.
Purchase Motivation:

It’s LTE-A Pro (CAT 12), supporting up to 600Mbps/150Mbps (DL/UL) [But let’s be real, you’ll never see that kind of speed on any provider. I think I wasted my money on this feature]
Supports Ethernet for when it’s available
Supports dual-mode WiFi. I have two 9dBi WiFi antennas attached to it – one can pick up WiFi, and the other can transmit it. This way, my Kindle, Apple TV, laptops, etc. don’t need to be reconfigured to use a new WiFi provider if I’m on campground WiFi. It sees the new WiFi connection as one they’re already familiar with: roam.
Has two SIM slots. This means I can switch between Verizon and AT&T automatically without having to replace any SIM cards. Granted, network switching is slow because it has to reload network-specific firmware any time it switches network.
Supports both AC and DC power. Living in an RV this is huge. The router is wired directly into the RV’s 12V DC power, drawing less than an amp of power. Not running a DC->AC inverter is key to saving battery.

Antenna(s)
I actually have both directional (9 dBi) and omnidirectional (6 dBi) antennas. Now you’ve seen me talk about both gain and this dBi nonsense. Gain is measured in dBi. The higher the number, the more signal you can grab out of the air. 6 dBi is decent, 9 dBi is pretty good. A good “Yagi” antenna gain can be > 10 dBi.
I use my omnidirectional antenna almost 100% of the time because I generally have no idea where the tower is and would rather not spend 20 minutes pointing my directional antenna all over creation trying to find a tower when I arrive somewhere new. Not when there’s only 3 dBi difference between my omni/directional antennas. If 3 dBi is the difference between me being connected and disconnected – chances are, I’m not going to get much done on that connection anyway. It’s going to be volatile. Temperature changes will affect it, humidity will affect it; maybe even star alignment. I don’t know.
My antenna is a Proxicast Pro-Gain 4G/LTE MIMO Antenna. That this antenna is a MIMO antenna is what helps me take advantage of the LTE-A tech in my router. Am I getting 600Mbps download speed? Not a chance! But it does allow the router to bond together the signal from two different channels into a more reliable connection. Neat.
Signal Tester
I’ll come back to this one. I have a signal tester, but I’m not looking at it and can’t remember what brand/model it is. These are great if you want to do some recon in a new area. Bring your signal tester and the antenna that you use with your device to get an idea of whether or not the signal is viable. I won’t bet my life on the results I get from my tester, but it’s a good gut check if I don’t want to pull the rig into a new location blind.
Wireless Providers
As I already let on earlier, I have two redundant providers: AT&T is my primary and Verizon is my backup. At one point, I had T-Mobile provided by a MVNO called OTR Mobile. That gravy train was great while it lasted: $80/mo for a truly unlimited plan. It lasted all of four months before (I’m guessing), T-Mobile terminated the relationship for violation of contract.
AT&T
My AT&T service is a bit unique, and I acknowledge that this option is not available to everyone because it requires one to have either a business of their own, or a business willing to open a line on their behalf.
My AT&T service is a “unlimited” 12Mbps Mobile Broadband plan for ~$150/mo (after taxes). This is as near unlimited as I believe possible today, as my data is simply “deprioritized” after 75GB. I’ve not been able to notice deprioritization. See the Throttling & Deprioritization for more details on what this means.
UPDATE: I now have AT&T’s 50Mbps plan for ~$200/mo. Deprioritization may begin at 125GB. See AT&T’s Wireless Broadband Plans here.
Verizon
My backup Verizon service is through US Mobile. It’s “Unlimited”, but they can throttle after 50GB. Granted, I’ve never noticed any throttling because it’s always fairly slow. It’s reliable though, and acceptably slow.
UPDATE: US Mobile now throttles at 50GB rather than deprioritizes. As I’ve mentioned, being throttled renders this plan useless, which is why this plan is only a backup plan.

Conclusions
I’ve glossed over a lot of details and yet this post is still too long. My apologies. If there’s anything I can help clarify, or details you think I may be able to reveal, please feel free to reach out to me. I can probably feign some degree of competence in providing answers. Also, please let me know if I’ve gotten anything wrong. I researched a lot of service providers and equipment throughout this process, so if there’s a router, booster, or providers you have questions about – I may know a thing or two.
Cheers


Medical Artificial Intelligence depends on where you live
Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 20 Oct 2020 21:00:00 GMT
Especially if you live in the right places.


Moderating a 220k Developer Community
swyx's site RSS Feed — Tue, 20 Oct 2020 20:19:41 GMT
Reflections from two-and-a bit years helping to run /r/Reactjs


Is There a Better Way?
Philipe Fatio — Tue, 20 Oct 2020 15:15:56 GMT
The question to keep asking yourself when using your tools


Understanding when to use useMemo
Max Rozen — Tue, 20 Oct 2020 12:48:48 GMT
If you've just learned what useCallback is, you might be wondering why on earth useMemo exists, and why we don't use it everywhere.


Building Solution Architecture Teams - A Software Approach
Greg Herlein — Tue, 20 Oct 2020 11:00:01 GMT
What actually is Solution Architecture?  How do you build and manage an Architecture team?  Can you take a software approach to that?


Defining Data Intuition
blog.harterrt.com — Tue, 20 Oct 2020 10:00:00 GMT
Last week, one of my peers asked me to explain what I meant by "Data Intuition",
and I realized I really didn't have a good definition.
That's a problem! I refer to data intuition all the time!
Data intuition is one of the three skills I interview new data scientists …


Advertising is broken, and stock market-like regulations won't fix it
Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 19 Oct 2020 21:00:00 GMT
Just like “breaking up Facebook”. It can’t work.


Startup prerequisites: Part 1
Perspective by Ankit Agarwal — Mon, 19 Oct 2020 13:29:31 GMT
Learning from my journey as an entrepreneur..


No, C++ still isn't cutting it.
Dave's Data — Mon, 19 Oct 2020 00:31:18 GMT
 A recent blog post asked "Does C++ still deserve its bad rap?"
While that title presumes the answer (after all, a "bad rap" implies that the judgement is undeserved), I think C++'s reputation for making it very difficult to achieve security or memory safety or thread-safety remains well-deserved, even though it's gotten a lot better over the years. I mean that: The c++-17 version of that program is years better than it would have been with c++0x, and it let a careful programmer write a nice program.
The post used a simple multi-file word count as an example:  Count all words in all ".txt" files in the current directory and all of its subdirectories, where words were defined as things matching the regexp "([a-z]{2,})" in a case-insensitive manner.
First, I'll walk through the example from that blog post and identify some remaining bugs.  And then we'll build up a (IMO) better version in Rust that's faster, safer, and more concise.  Let's start with the body that processes the files and adds the counts to a hash table:
A few things immediately come to mind when examining this code from a safety perspective.  First, it uses a filesystem::recursive_directory_iterator to identify the files in the current directory, testing their types and names before processing them.  It then opens the file using std::ifstream.
From the way I phrased this, you're probably already guessing that this is a TTCTTOU eror -- time-to-check-to-time-of-use. The program validates that the entry is a regular file, but then later opens it, and assumes that the result of the check holds. We should ask ourselves:
What happens if the file has been deleted between the directory listing and the open?
What happens if the file has been replaced by, e.g., a pipe or other not regular_file?
For the second case, it's pretty clear that the program will try to open it and operate on it; this is therefore a bug with respect to the intent of the programmer. Is it a huge bug in this context? Not at all, but analogous bugs have led to serious security problems.
For the first case, I wasn't sure - and I bet many other C++ programmers aren't either. The answer is that the std::ifstream is safe, and will simply behave in the same way as if EOF was reached, but I wasn't certain about that until I googled it a lot and wrote a test program to verify it. Accidental correctness is better than being wrong, but we should strive for more in our programs.

Finally, the way it's written isn't very conducive to future multithreading. That's wasn't the point of the exercise, but I think it's worth thinking about as many modern data processing programs take an evolutionary path from single-threaded to multi-threaded. Constructs like this invite threading bugs:
(1) It separates the computation of a safe size range from the use of that range;  (2) it destructively modifies the word_array for no reason other than to make it easy to print the first 10 items.  And adding multithreading to things in C++ still remains somewhat painful.

Here are some reasons I'm coming to prefer Rust as a systems language. It's not remarkably shorter - the Python or shell version of all this can be expressed in a few lines of code!  But it's a bit shorter, and it's more clear where corners have been cut or errors properly handled:

use anyhow::Result;
use lazysort::SortedBy;
use std::collections::HashMap;
use std::io::prelude::*;

fn scanfiles() -> Result<()> {
    let mut wordcounts = HashMap::new();

    let words = regex::Regex::new(r"([a-zA-Z]{2,})")?;
    for file in globwalk::glob("*.txt")?
        .filter_map(Result::ok)
        .filter_map(|dirent| std::fs::File::open(dirent.path()).ok())
        .filter(|file| {
            file.metadata()
                .and_then(|md| Ok(md.is_file()))
                .unwrap_or(false)
        })
    {
        let reader = std::io::BufReader::new(file);
        for line in reader.lines().filter_map(Result::ok) {
            for word in words.captures_iter(&line) {
                let w = word[0].to_lowercase();
                *wordcounts.entry(w).or_insert(0) += 1;
            }
        }
    }
    let words_sorted = wordcounts.iter().sorted_by(|a, b| b.1.cmp(a.1));
    for kv in words_sorted.take(10) {
        println!("{} {}", kv.0, kv.1);
    }
    Ok(())
}

fn main() {
    if let Err(e) = scanfiles() {
        println!("Error:  Something unexpected happened: {:#?}", e);
    }
}


Ok, that's 39 lines of code, but not counting the Cargo.toml file, which is another 4 of non-boilerplate (the four dependencies, which are fairly analogous to the #include lines in the C++ version, so should be counted).

A little shorter.  But what I like about the Rust version is that there's a lot less guessing about the error handling.  Did the file fail to open?  We know it's skipped, because the filter_map will discard that file if the result of the open is not ok.  We know errors in globwalk or constructing the regex are handled, because the ? will cause the function to return an error if they return an error.  The more functional .take(N) idiom is a little more idiot-proof than the equivalent code in the C++ version, as we know it returns early if there are fewer than N items.

While modern C++ lets a careful programmer write a good program, it still lets a less-careful programmer do things in ways that create errors. Rust has less of that: it will nag you to cover corner cases more carefully.

Rust doesn't make it magically easier to avoid that TTCTTOU problem - it would have been just as easy to write the code using std::fs::metadata(path) in the same way it was in the C++ version.  But it does improve my confidence that were that bug present, the code would have handled the "file doesn't exist" problem. But it still would have opened a special file or directory with aplomb.

It's also very easy to convert to a multithreaded program:  Just turn the for x in ... loop into a for_each, and use the Rayon crate .par_bridge() to cause the function in for_each to be executed in parallel.

    globwalk::glob("*.txt")?
        .filter_map(Result::ok)
        .filter_map(|dirent| std::fs::File::open(dirent.path()).ok())
        .filter(|file| {
            file.metadata()
                .and_then(|md| Ok(md.is_file()))
                .unwrap_or(false)
        })
        .par_bridge()
        .for_each(|file| {

Of course, that won't compile, because we've forgotten to use any locking on the hash table into which we're inserting counts:

error[E0596]: cannot borrow `wordcounts` as mutable, as it is a captured variable in a `Fn` closure
  --> src/main.rs:25:22
   |
25 |                     *wordcounts.entry(w).or_insert(0) += 1;
   |                      ^^^^^^^^^^ cannot borrow as mutable


So then we either switch it to a parallel hash table, or we guard it with a mutex. I'll take the simple approach of locking the table before using it, and do just enough optimization to make it faster than the single-threaded version. In this case, I parse each line into lowercased-words, which I store in a vector, before locking the hash table and then bulk-inserting the line's worth of words:

use anyhow::Result;
use lazysort::SortedBy;
use rayon::prelude::*;                                                 // NEW
use std::collections::HashMap;
use std::io::prelude::*;

fn scanfiles() -> Result<()> {
    let mut wordcounts = HashMap::new();
    let wordcounts_locked = std::sync::Mutex::new(&mut wordcounts);     // NEW

    let words = regex::Regex::new(r"([a-zA-Z]{2,})")?;
    globwalk::glob("*.txt")?
        .filter_map(Result::ok)
        .filter_map(|dirent| std::fs::File::open(dirent.path()).ok())
        .filter(|file| {
            file.metadata()
                .and_then(|md| Ok(md.is_file()))
                .unwrap_or(false)
        })
        .par_bridge()                                                   // NEW
        .for_each(|file| {                                              // CHANGED
            let reader = std::io::BufReader::new(file);
            for line in reader.lines().filter_map(Result::ok) {
                let wordlist: Vec<String> = words                       // NEW DESIGN
                    .captures_iter(&line)
                    .map(|w| w[0].to_lowercase())
                    .collect();
                let mut wordcounts = wordcounts_locked.lock().unwrap();
                for w in wordlist {
                    *wordcounts.entry(w).or_insert(0) += 1;
                }
            }
        });

    let words_sorted = wordcounts.iter().sorted_by(|a, b| b.1.cmp(a.1));
    for kv in words_sorted.take(10) {
        println!("{} {}", kv.1, kv.0);
    }
    Ok(())
}

fn main() {
    if let Err(e) = scanfiles() {
        println!("Error:  Something unexpected happened: {:#?}", e);
    }
}



Not bad - about ten lines of code changed to make a basic parallel version of the code.

There are some advantages to the C++ version that reflect the immaturity of the Rust standard library.  It was able to easily use a partial_sort to reduce the amount of work done sorting by counts; I had to find the lazysort crate, which isn't part of the standard library, which does the same thing.  

Both are fast; depending on the C++ compiler, the rust one may or may not be faster.  (Updated:  More benchmarks, prompted by @cb321 on hacker news, and switched the benchmark to something easily downloaded:  Charles Dickens "A Tale of Two Cities")

   - I7-7920HQ, MacOS
       Rust version:  90ms    (rustc 1.47.0 (18bf6b4f0 2020-10-07))
       C++ version:  340ms   (Apple clang version 12.0.0 (clang-1200.0.32.2))

   - Xeon(R) Gold 6130  (skylake, Ubuntu 20.04)
      Rust:  76ms  (rustc-1.47.0)
      C++:  60ms   (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)), -O3, no PGO

With a lot of larger files, the parallel version can wordcount four 2mb sample files in 0.307 seconds using a quad-core CPU - not quite linear scaling, but not bad for a few minutes of tweaking.  The single-threaded C++ version, meanwhile, is stuck taking about 3.55 seconds on MacOS.

Despite its shorter length, the Rust version probably took me longer to write than the C++ version would have.  I had to spend more time figuring out what functions returned so I could handle their possible return types, and had to spend a bit more time googling for things like the filesystem metadata, probably because I'm newer to the language. But making it correct and making it parallel took less time, even though I'm less experienced in the language. I'll take that tradeoff in a lot of what I do.

-------
Update:  Some good feedback from burntsushi about the unnecessary use of globwalk introducing a lot of extra dependencies.  This is a good point about the risks of making it too easy to use random crates off of github -- or, perhaps, about the dangers inherent in having a less-fully-fledged stdlib.  An updated version of the code that's faster to compile is:

use anyhow::Result;
use lazysort::SortedBy;
use rayon::prelude::*;
use std::collections::HashMap;
use std::io::prelude::*;

fn scanfiles() -> Result<()> {
    let mut wordcounts = HashMap::new();
    let wordcounts_locked = std::sync::Mutex::new(&mut wordcounts);

    let words = regex::Regex::new(r"([a-zA-Z]{2,})")?;
    walkdir::WalkDir::new(".")
        .into_iter()
        .filter_map(Result::ok)
        .filter(|dirent| dirent.path().extension().and_then(std::ffi::OsStr::to_str) == Some("txt"))
        .filter_map(|dirent| std::fs::File::open(dirent.path()).ok())
        .filter(|file| {
            file.metadata()
                .and_then(|md| Ok(md.is_file()))
                .unwrap_or(false)
        })
        .par_bridge()
        .for_each(|file| {
            let reader = std::io::BufReader::new(file);
            for line in reader.lines().filter_map(Result::ok) {
                let wordlist: Vec<String> = words
                    .captures_iter(&line)
                    .map(|w| w[0].to_lowercase())
                    .collect();
                let mut wordcounts = wordcounts_locked.lock().unwrap();
                for w in wordlist {
                    *wordcounts.entry(w).or_insert(0) += 1;
                }
            }
        });

    let words_sorted = wordcounts.iter().sorted_by(|a, b| b.1.cmp(a.1));
    for kv in words_sorted.take(10) {
        println!("{} {}", kv.1, kv.0);
    }
    Ok(())
}

fn main() {
    if let Err(e) = scanfiles() {
        println!("Error:  Something unexpected happened: {:#?}", e);
    }
}




Firefly III monthly summary emails
David Schlachter — Sun, 18 Oct 2020 19:59:00 GMT
Self-hosted budgeting software firefly-iii doesn't have an email reports feature. I missed this from Mint and re-implemented it with the API.


Tolerating two years of a pandemic in no easy (but some manageable) steps
Dave's Data — Sun, 18 Oct 2020 16:28:03 GMT
First off, a reminder:  Don't listen to random or not-so-random people like me when it comes to predictions about the spread and danger of COVID-19.  Listen to the epidemiologists.  I suggest starting with:


The Imperial College London team, who've done groundbreaking work on previous coronaviruses including SARS and MERS.  It's an experienced, all-star team.
Trevor Bedford's twitter feed is full of great stuff tracing the path of the coronavirus in Seattle.
Adam Kucharski et al. at the London School of Hygiene & Tropical Medicine;
The work from Christian Althaus's group at the Institute for Social and Preventative Medicine
Helen Branswell's reporting on the subject is accurate and lucid.

My goal here is to lay out the case that:  (1)  It's worth doing our collective best at minimizing social contact for a while; and that (2)  It won't be as bad in the long run, due to a combination of social and technical factors.  In other words, we'll get through this, but minimizing the pain requires us to be smart now.  I'm writing this up because I found myself repeating the same fragments of this discussion in various social media, and it seemed like it would make more sense to put it all together.



Background:  Social distancing, isolation, and all the rest can save a lot of lives.  Don't take my word for it, read the ICL report.



But when you read that report, you'll notice something interesting.  It's possible that when we try to finish the big part of social distancing, the virus just returns nearly as bad as before:



Figure 3A reprinted from the ICL report.

I'm going to argue that we'll get through that, because of substantial changes we'll be able to make over the medium term to handle it.  Some of this is taken from a series of tweets from me a week ago, is further bolstered by tweets from Trevor Bedford, and includes some material beyond that.





Why will this get better, and when?

Advance 1:  Testing

Over the coming weeks to months, we're going to make reasonably good progress in increasing the ease of testing.  The US got off to a terrible start on this, which has crippled our ability to use an isolation-based strategy, but eventually, things will catch up.  We'll see several improvements:




Testing phase 0:  Insufficient testing to even measure a substantial fraction of cases.  It's reasonable to estimate that as of a week ago, the US had 10x more cases than we were able to measure.  Just a week ago, the state of Pennsylvania had tested 252 people, identifying 47 positives.  Today, we've tested 4137 people, finding 371 positives.  To put that in context, the state has about 12 million residents, so we've tested about 1 in 2900 people.  At the outset, testing latency was about 3 days to get results.  Some tests now have the latency down to a matter of hours.




Testing phase 1:  Sufficient testing to measure the true prevalence of cases and test all reasonable possibilities.   We're probably going to reach this phase while we're still in a "lockdown" state, so it won't make things easier, but it's a necessary step to being able to exit complete lockdown.  Without accurately understanding the local and national prevalence of the disease, we can't make good decisions.




Future advance:  Serum antibody tests to tell if someone has already had the disease.  This may be particularly important for health care workers, food chain workers, etc.  We don't yet have hard data on how immunity behaves with COVID-19, but it's reasonable to think there will be some for at least a limited period of time.  Once you've had it, you're probably clear to go back to work.  There may be people who had low-symptomatic or asymptomatic cases who we can clear but didn't test (see above), so a serum antibody test that can measure whether you've been previously infected can be valuable for helping repopulate the workforce, particularly for some of the highest-risk jobs.  [Example:  Health care workers are at substantially higher risk of contracting the virus].




Future advance:  Widespread, cheap testing akin to that in Singapore and South Korea.  When people with nearly any symptoms can get tested, it may become possible to engage in much more careful isolation of confirmed cases, while allowing more limited interaction between the healthy, combined with aggressive contact tracing.  Trevor Bedford's tweets laid this out well, but South Korea's success in containment is a good example of this, keeping in mind that there is still very substantial restriction on contact there:









Future advance:  Cheap testing at home.  If it's possible to test every member of your family daily, we can probably let people go out on their business much more than we can now.  This obviously requires substantial cost-reduction and throughput improvement of the testing, and seems farther off than the previous two.




tl;dr:  As testing gets better, we can relax some of the social restrictions.  Not all of them, but it opens a middle ground between "don't leave your house" and "go hang out with 5000 people from around the globe at a conference in Boston."  Getting to such a middle ground is important for the livelihood of local businesses, particularly those that can't change to remote or delivery models.  Resuming, e.g., operating a house cleaning business at the cost of requiring daily coronavirus tests may be feasible, particularly if the tests were free and could be done at home.



Want more?  See this Wired interview with Larry Brilliant, an epidemiologist who helped eradicate smallpox.

Advance 2:  Supplies

One of the reasons we need to flatten the curve is that we have limited supplies of hospital beds, medical personnel, personal protective equipment, and ventilators.  Over time, we can begin to address some of these issues.




It takes a while to ramp up manufacturing, even of seemingly simple things like masks.  For critically-needed supplies such as masks/PPE and ventilators, we should expect to see two increases in our manufacturing capacity:




Increase 1:  Increase the throughput of existing lines by increasing the duty cycle.  By hiring more workers (assuming that the inputs are available), some manufacturers will be able to increase throughput by working additional shifts, up to the point that equipment is running full-time.  (Full time, of course, is limited by downtime needed to maintain the equipment, so very little actually runs at a 100% duty cycle.)  For example, Medtronic announced that they were moving to 24/7 production of ventilators at their existing facility in Ireland.  Similar increases are happening with mask manufacturing and the other equipment components we need.  This is the fastest way to increase production, but it's limited:  Very few lines were probably running at less than shift per day (~30% of capacity), so we won't get more than a 3x increase in production this way.  But that still helps.




Increase 2:  Add more manufacturing lines.  Depending on the complexity of the process, building a new line could take from weeks to many months.  In some cases we can convert existing lines:  Some sewing facilities in China have rapidly switched from clothing to masks, but there are bottlenecks farther up the line, such as the melt-blown fabric fabricating machines that may take five to six months to build.  While Hanes has rapidly turned around to manufacture masks, they're only able to make three-ply cotton masks, which are less effective than conventional surgical masks.  Ventilator manufacturer Ventec estimates they could go to 5x ventilator production in 3-4 months.  If we ballpark that increasing global critical supply production capacity by a factor of ten will take a year, we're probably in the right range.




The return on this investment is somewhat limited, because it also takes trained personnel to operate ventilators, but we get good advantages from them nonetheless:


Sufficient PPE for health care workers increases the number of available (i.e., not out sick or quarantined) skilled workers;
Reducing the chances of health care workers spreading the disease to themselves or vulnerable populations is an important part of mitigating the ICU cost of coronavirus cases;
Having sufficient masks of any sort to create a general culture of mask-wearing in the US is likely to reduce transmission, allowing more going out-and-about at the same transmission rate, particularly in high-density areas such as New York.


This doesn't buy us unfettered movement, but it lets us tolerate a higher active number of cases at any given time -- i.e., means we don't have to flatten the curve quite as much to achieve the same savings of life.  Not having to flatten as much later means that we don't have to impose as many restrictions on people's lives and ability to make a living.

Advance 3:  Treatment

The more COVID-19 cases we treat (particularly when we have some breathing room to be deliberate about it), the more we learn about how to do better with future cases.  This too will come in several phases:




Phase 1:  Best practices.  Easy to develop and fast to communicate, improvements in best practices for management of patients with COVID-19 can help improve the survival rate and reduce hospitalization times.  There's no silver bullet here, but it's cheap.  Examples of this include the development of specialized drive-through testing, which minimizes patient-to-patient and patient-to-provider contact.  More technical examples include the WHO's continually-updated best practices for clinical management of COVID-19 cases.




Phase 2:  Pharmaceutical interventions.  There's a lot of breathless discussion out there about whether or not particular drugs or drug combinations can help reduce the extent of symptoms of COVID-19 (or, similarly, the back-and-forth about whether NSAIDs are harmful).  The reality is that it's very early for all of these and that it will take substantial more study before we have solid answers.  An example is the ongoing evaluation of chloroquine combination therapy.  We don't have anything yet, but we're likely to find things that help.  This help likely won't be a panacea either - consider that the most effective antiviral we have against influenza decreases symptoms by about a day - but every bit helps in combination with the other bits.




Phase 3:  A vaccine.  Depending on the efficacy and side effect profile of the vaccine, this could be the end of the discussion, or it could simply massively reduce the cost and deaths associated with the virus.  But most experts concur this is 1-2 years out.




Treatment has the same effect as fixing supply limitations:  We can tolerate more active cases at a time, i.e., we can lower the cost to people's lives and jobs while keeping as many people from dying.

Advance 4:  Societal Changes that Reduce Contact


There's a lot of important face-to-face interaction in society, but there's a lot we can get rid of, and we're about to have several months of experimenting with what we can and can't change easily.  As a warning, this is the least-baked of everything I'm saying here.




Take, as an example, three models of grocery shopping:




Model 1:  Conventional US shopping.  Each customer goes to the grocery store weekly (or whenever).  At the grocery store, that customer comes into contact with many of the workers present in the store, as well as the other people shopping at the same time.




Model 2:  The Instacart Model.  A specialized delivery service sends shopper/drivers to the store who do the shopping and deliver groceries to the customer.  An instacart driver can probably do in the range of 5 shopping trips per day.  Over the course of a week, then, this shopper/driver reduces the number of contacts that store employees experience by a factor of 25 (5 replaced shoppers per day).  A typical grocery store has in the range of two thousand customers each week, putting every employee in contact with hundreds of different people.  Moving to model 2 reduces that substantially, but there are still a large number of different shoppers entering the store each day.




Model 3:  Store shoppers shop, drivers drive:  In this model, a customer order is picked and packed by employees of the store itself.  The store employees then transfer the packed groceries to a driver (who never enters the store and can pick up the groceries from a distance).  The driver drops the groceries at the customer's house.  In this model, drivers never have customer or employee contact, and thus never act as a disease conduit from other people to store employees or to customers.  Store employees still come into contact with each other regularly, but it's far better to have regular contact with the same smaller group than to have a much larger contact graph.




Other areas may go completely online.  Consider that today we have the NASDAQ, an entirely electronic exchange, and we have the NYSE, where traders -- until three days ago -- would physically cluster in a cramped environment.  The NYSE has temporarily gone electronic-only (which probably won't change much), and I suspect it won't go back.




This great collective experiment we're doing in remote-everything is likely to yield other changes that let us continue to get things done with reduced chances for viral transmission and/or travel.  When this is all done, I'm not going to fly to another in-person program committee meeting to referee scientific papers.  I'm done with it - I love my colleagues and love the opportunities to chat research, but it's unnecessary travel and by the end of this we'll have perfected on-line paper discussions and program committee meetings.




I don't want to paint too happy a picture for this.  There are changes that are going to be absolutely destructive to people's jobs and income -- unemployment claims are skyrocketing, and industries such as hospitality are going to be very, very deeply hurt, and these are things that won't be fixed by modest social changes.  That's going to take concerted government response at a scale we haven't seen since the great depression.  There are other people for whom the critical nature of their jobs, from food production to healthcare, will require them to continue working with other people and suffer an increased risk of catching coronavirus, and we we need to ensure there's support in place for them.  But there are things that will improve life for all of us as we make it through this pandemic, and we get the most benefit from them if we can minimize the spread of the disease for a period of months. 




As more people have passed through the disease and gain immunity, as we roll out more and more testing, as we reduce the case fatality rate through better interventions, and as we reduce the amount of "low-importance" social contact, we'll be able to increase the amount of high-value social contact we allow ourselves.  But with the extent of spread in the US already, this is likely to be a game we play out over months to two years, with possible seasonal ebbs and flows.  But we'll come through it better-equipped to handle future pandemics (and reduce the chances of them going pandemic).





Updates on factors that will ease living with a coronavirus pandemic
Dave's Data — Sun, 18 Oct 2020 16:27:56 GMT
Three weeks ago, I posted about things that will help us loosen restrictions as we move forward through the coronavirus pandemic.  In this post, I'm following up on a few of those in the form of a progress tracker.


Testing

PCR viral tests




March 23, 2020:  58,500 tests/day

Apr 14, 2020:  145,000 tests/day.




Source:  https://covidtracking.com/data/us-daily

Taken as the average of a 5 day window centered around the date of concern.


SARS-CoV-2 tests per day







As with all things, test availability is subject to multiple bottlenecks.  It takes swabs, PCR reagents, PPE for the people doing the tests, machines on which to run tests, trained personnel, etc.  The manufacturing and supply of all of these can take time to spin up. We're not yet to the point where we can test all likely cases, but we're running about 2.4x as many tests as we were three weeks ago.




This is not enough.  Estimates vary widely, but one Harvard study suggests we may need in the millions of tests per day range.  Others put the numbers in the high hundreds of thousands per day, but no matter what, we need at least to go 5x where we are now.




Serum antibody tests




Serum antibody tests are starting to roll out.  The NIH has started a study to quantify the true prevalence of cases.  Several commercial test providers have created tests and they are available (at high cost) in some areas in limited quantities.  They're not yet playing a big role but I expect that they'll begin to within a month.  As an example, UW Virology in Seattle is hoping to begin 1000 tests/day next week.  Stanford is testing 500 per day now.




As a warning, of course, the false positive and false negative rates on these tests are too high for them to be used alone for purposes such as "certify that I am immune to covid-19".  For example, the Cellex test is somewhere in the 95% range, which means that for every 100 people who have never had coronavirus, the test will tell 5 of them it has.  That's higher than the expected fraction of people in the US who have been infected thus far, which means that if the test says you've had it you ... might have had it.  So for the coming month, at least, the value of these tests is primarily to help understand the spread (and infection fatality rate) of the disease better, but that understanding is critical to crafting policy on how to respond to it.  But for individual benefit, more improvements and more data sources are needed.




Supplies

This is hard to quantify;  data is limited and messy.  Some anecdotes but not overall numbers:


3m says it has doubled N95 production to 100 million / month, with 35M/month in the US.
Together with smaller manufacturers, that has us up to about 50M per month.
But DHHS estimates we need 300M masks / month.


Melt-blown poly continues to be one of the key bottlenecks, which as I noted a few weeks ago, may take several months to ramp up supply.  N95s are not the only masks we need, and other manufacturers are bottlenecked by other components, such as elastic, when making ASTM level 1 and 2 masks (standard procedure masks).





Relations between the US and China are an important factor in mask availability, which may hopefully be improving for medical supply shipments.




This is all a long-winded way of saying we're better, we're not better enough, and it's going to be at least several months before we're up to speed on this one, but it's helped by...




Progress in reusing masks



There have been several independent efforts to start reusing masks, and these are entering real deployment.  For more information here, see https://www.n95decon.org/.  As one example, see Battelle's 80k masks/day (theoretical) decontamination system.  There are a few of these in deployment now, so we might expect them to reduce N95 mask demand by a million or two masks per month.  As they ramp up production (and these are fairly easy to build), they could conceivably reduce demand by something like 50M masks per month by the end of May.  That's equivalent to doubling our current mask production, which is a great start.  Other hospital systems are deploying UV or heat-based mask sanitization.  But we need all of these and more to meet demand, and it won't happen overnight.




(Calculation note:  I've de-rated the decontamination system throughput to 25k masks/day sustained for this calculation to be conservative about the effective duty cycle and the fact that it's likely hard to keep the system perfectly balanced as far as masks available to clean at all times.  In theory the 60 machines ordered for HHS could top 100M mask reuses/month).




But in total, this means that over the course of 3 months, we'll have increased "effective mask production" fourfold, with more increases possible in the subsequent 3 months.

Treatment

Three weeks is basically no time in terms of drug development and testing with humans, so very little new to say here.  Testing in humans takes time, and even with a lot of emergency leeway, it still takes time.  There are more (preliminary and small) studies coming in about hydroxychloroquine combinations that make it clear that it's no wonder drug;  it's still up in the air if it helps, but every week that passes we get more data.  Data about Remdesivir is still coming in, but is patchy and many of the studies lack a control group, which makes it very hard to say much with certainty.  There are more than a dozen drugs in various stages of development as either vaccines or therapies.  A lot of activity, which is promising for future results, but I don't pretend to have a crystal ball.

Societal Changes that Reduce Contact

This section should probably be called:  Go go human adaptability!  There have been lots of promising advances here.




No-contact shopping:  Retailers and delivery services continue to try to optimize for more and more store-packed or delivered grocery service.  As with everything, improvements here require process optimizations, infrastructure changes, and hiring more people, all of which will take time.  Instacart reports that orders have quadrupled from a year ago, and says it has doubled its workforce from 200k to 350k in the last two weeks.  Amazon is reported to have hired 100,000 workers and is trying to hire another 75,000.  Many of the platforms are improving their process:  Instacart has added a "fast and flexible" delivery option to help smooth demand and keep drivers busy.  Amazon is reducing in-person store hours at some locations to facilitate more delivery.



Even in-person shopping is getting on the ball.  Some grocery stores are installing plexiglass barriers to reduce contact between customers and (high contact fan-out) cashiers.  Many are upping cleaning schedules, providing sanitizer, and requiring masks.  (Of course, this also requires an increase in the manufacturing of sanitizer, but that's happening too).




Masks for all:  There's a growing consensus that widespread mask wearing can help reduce the spread of coronavirus.  (There's also a lot of fighting and nitpicking about it, but this seems to be the median opinion).  The CDC has updated its guidelines to suggest that everyone wear a mask when out in public.  New York now has a mandatory mask law "when social distancing is not possible."  These changes are just coming online, so we can't see their effects yet, but this seems likely to start making progress by a few weeks from now.




Doin' it online:  More states are beefing up their vote-by-mail efforts.  Some are not.  Doctors have been getting increasingly online with providing telemedicine services, and for some visits, this appears to be working very well.  This may have a long-lasting effect in reducing the amount we all go to the doctor's office to share germs with each other.  We do need primary care visits to resume, however, because data also shows that vaccination prescriptions are down right now as people can't get physically to their doctors.




Tl;dr:  We've made measurable progress for three weeks, but there's a lot more to go before we're near getting things back to normal.  I'll try to update this again in a few weeks unless some better site comes online for tracking these things.


Speeding up fuzzing rust with shared initialization
Dave's Data — Sun, 18 Oct 2020 16:27:50 GMT
Having ported the Pi searcher to Rust, I've started exploring fuzzing frameworks to help find bugs.  I settled on Rust's cargo-fuzz using the libfuzzer backend, since I'm familiar with libfuzzer from using it on TensorFlow.



Naturally, since the Pi Searcher had been comparatively stable for 7 years in Go, and the translation from Go to Rust was fairly straightforward... I introduced a bug that my tests didn't find, and the fuzzer found over dinner:







Whoopsie.  That bug arose because of an error in handling the boundary condition of searching for just "9" using the index (which doesn't happen normally because the search function uses a simple sequential search for short strings):





Here, num_digits is 200 million, but they're stored compressed, so the bug was using the size of the file (100 million bytes), which caused the binary search to return a bad range.  Fuzzing is awesome, and adapting the fuzzer to the Pi searcher was straightforward:





But one thing that bugged me:  Fuzzing was a little slow.



Why was it slow?  Every time the fuzz function is run with a new input, it had to instantiate a new PiSearcher.  Each PiSearcher opens and memory maps using mmap two files:  100MB of compressed digits of Pi and 800MB of a suffix array index into those bytes.  This adds a bunch of unnecessary overhead compared to the relatively fast searching code.



Libfuzzer's documentation suggests that initialization should use a statically initialized global object.  But this is Rust.  One does not simply willy-nilly toss around globals.  Some quick Googling didn't turn up much useful, so I put it aside, as this was merely a speed optimization.



Fortunately, the most recent "this week in Rust" blog pointed to Paul Kernfeld's very useful Guide to Global Data in Rust.  Which, in turn, linked to the once_cell crate.  Just what the fuzzer ordered!  A quick rewrite of the fuzzing target yielded:









Which allows the (read-only) PiSearcher object to be reused across fuzzing calls.  Simple and easy.



How much did this help?



> cargo fuzz run coretest



Debug build (default):

Old version (new mmap per function call):  304 tests/second.

New version (statically initialized PiSeacher):  340 tests/second.



10% is nice and all, but that's not a particularly large gain.  But wait - that's very slow compared to the normal speed of the Pi searcher.  Ahh, yes:  cargo fuzz builds debug mode by default.  Let's try release, keeping in mind that coverage may not be quite as good (but with the speed gains, it's probably worth it):



> cargo fuzz run coretest --release --jobs 4



Release build, 4 jobs:


Old version:  10800

New version    26050


Not bad - 10% in debug mode and almost 2.5x faster in release mode with multiple jobs.  Thanks, once_cell!







Moving the Pi Searcher from Go to Rust
Dave's Data — Sun, 18 Oct 2020 16:27:44 GMT
Seven years ago (almost to the date), I wrote about my experience moving the Pi Searcher from C++ to Go.  In that article, I mentioned that while the core searching code got slower, the system as a whole sped up by avoiding the per-request fork overhead of the earlier CGI-based code.



This year I moved the Pi Searcher to Rust, hoping to get the best of both worlds.  The tl;dr is that it worked and gave me and end-to-end throughput increase of at least 22%.  But not entirely in the place I'd expected!  The code is basically a transliteration of the Go to Rust, except for moving from Go's internal net/http server to actix-web for Rust.  The algorithm is the same:  sequential search for short search strings, and using a suffix array for longer strings.


Results
Code length isn't too different.  I was a little sloppy in writing the Rust version, so I don't think a comparison is entirely fair, but they're within 50% of each other and would probably be closer if I structured the Rust better.


Core search times using the suffix array
The core search didn't change by much, as timed on my 2017 i7 Macbook Pro once the cache was fully warmed:



Go Version:  8.1 microseconds/search

Rust version:  6.1 microseconds/search



No big deal.  And not surprising - this time is bounded by the binary search / indirect memory reference time.  With 200M positions in the file, searching takes about log2(200M) = 27 steps.  Each of those steps involves two dependent memory lookups, and at about 100ns per memory lookup, that's 27*2*100 = 5.4 microseconds of memory time.  That's obviously a handwavy calculation that ignores real memory timings and the fact that the first few lookups will hit in cache, but it suggests that both bits of code are mostly memory-bound.


Direct access over HTTP to the JSON results
What about direct reqs/second, measured using ab -c 2 -n 1000?  Measurements on the Pi searcher server (an n1 small GCP instance):



Rust

Requests per second:    6657.61 [#/sec] (mean)

Time per request:       0.300 [ms] (mean)

Time per request:       0.150 [ms] (mean, across all concurrent requests)



Go

Requests per second:    6074.71 [#/sec] (mean)

Time per request:       0.329 [ms] (mean)

Time per request:       0.165 [ms] (mean, across all concurrent requests)



So far, that doesn't look all that better, and certainly not like it was worth rewriting.  But the pi searcher doesn't exist in a vacuum - the backend is accessed through an Apache reverse proxy.


Access via the Apache reverse proxy
Go

Requests per second:    2811.57 [#/sec] (mean)

Time per request:       0.711 [ms] (mean)

Time per request:       0.356 [ms] (mean, across all concurrent requests)




Rust



Requests per second:    3445.73 [#/sec] (mean)

Time per request:       0.580 [ms] (mean)

Time per request:       0.290 [ms] (mean, across all concurrent requests)






Ah-ha!  We finally have a speedup.  And what's probably happening is that the speedup is a little bigger than it looks here, but we're bottlenecked in our measurements by doing the measurements on the same machine.  With the server running both the ab benchmarker, the Apache proxy, and the pi searcher, it's hard to find huge speedups.  I'd probably be better off moving to nginx but that's for another year.

Observations
The process of rewriting wouldn't have been all that painful for someone familiar with actix-web.  As a newbie, I spent a lot of time fighting with it to be able to use its derived parsers for requests while still being able to handle errors myself in some cases.  I still don't like the solution I came up with, where there are two analogous-but-different functions to serve GETs and POSTs:







(The "legacy" bit is because the pi searcher exports both a modern JSON-based API and a legacy interface that looks like the old CGI interface.)  I'm sure there are better ways to accomplish what I was trying to do:  I'm not a Rust expert by any means;  I used this rewrite as a way to dive in a little to see what creating web services was like in Rust so that I could better evaluate the costs/benefits for real work projects.



Rust's binary_search_by_key isn't designed as well for suffix array use.  Actually, it's designed and implemented just fine, but the documentation notes:


 If there are multiple matches, then any one of the matches could be returned. 
This is an entirely sane decision for an API designer to make, because it gives them freedom to change the implementation in the future.  Unfortunately, as an API user, it means I can't use it to find the full range of results in a suffix array, which one does by finding the first and last matching elements.  I had to re-implement binary search within my code, which, fortunately, was about 15 lines of very easy-to-read code.  This was a place that Rust really shined:







 I've internalized the -1, 0, 1 mapping of C-style comparison functions such as Go's, but I really liked here that it was an explicit type, not an integer.



The 22% speedup is probably under-measuring, but it's also the number I care about:  the actual throughput my server can sustain.



Things that were painful:  Getting the routing and whatnot set up with actix-web took a lot of work.  The documentation isn't comprehensive for weird usecases like mine (which are, admittedly, stupid -- I'm supporting a legacy interface on both GET/POST and a JSON-ish interface that also stupidly takes both GET/POST).  Sorting through the twists of using #[get(...)] vs explicitly routing things .route(web::get().to(serve_legacy_get) took a while to get right.  Mixing my own "in-band" error handling with Actix's error handling was also a pain.  It's clear that the framework is designed for supporting modern API development usecases, and that's not unreasonable.



I miss having explicitly named multiple return values.  The Search function returns a tuple of (position, count), both of which are of type usize,  and it's nice to have the semantics documented explicitly through return parameter naming in Go.



Things that were awesome:  Automatic request parsing into a struct (and automatic marshaling of the struct back to json) was very nice.  I'd implemented the JSON marshaling as a wrapper in the Go version, but it was automatic in Rust.

Rust has better support for some functional forms, so, for example, finding the "next search result" (i.e., the element that appears first after position p) was elegant: 






I do love that compared to the Go version:







One thing I had been worried about was mmap support, but this was mostly a no-op.  The only issue was that I had to recalculate the "mapped as uint32" version in functions where I used it (which is basically nothing), because one doesn't store two pointers to the same thing in a struct in Rust:




let idxu32: &[u32] = bytemuck::cast_slice(&self.pi_index_map);


No biggie and if I did it in more than two places it'd be an easy helper function.


Wrapup

A 22% speedup isn't bad, and is probably under-measured because of the other overhead in the system that should also be removed (hi, Apache, I'm looking at you).  But it was a good experience, and I'm pleased with the results:  this is code that I've poked at on and off for quite a while, and getting a nontrivial speed boost without adding any real complexity or extra lines of code was quite welcome.  There are aspects of the Rust ecosystem that remain frustrating -- error handling and documentation primary among them, and Actix is fast and helpful but has a lot of sharp edges -- but it's matured drastically since the last time(s) I've taken a peek at it.






Tolerating the Pandemic #3
Dave's Data — Sun, 18 Oct 2020 16:27:30 GMT
In two previous posts, I discussed some ongoing processes that will help us tolerate the pandemic better, and an update 3 weeks after the first.
It's been two months, which is enough time for more progress to have happened.  There hasn't been enough, and some things have had less progress than I've hoped for, but we've also seen a bit more progress in other areas.  Let's stick with the same framework from before:  Testing, PPE/Supplies, Treatments, and Societal changes to reduce contact.

Testing

In PCR testing, a picture is worth a thousand words, such as this one taken from Johns Hopkins:

We've gone from 58k tests/day, through 145k tests/day, up to about 450k tests/day.  It took us about a month to triple and another two months to triple again.  That's good, but it's still not at the levels that the Harvard study recommend.  It is, however, about halfway there, and we can probably get to where we need to be in another five months if the federal and state governments fund it.  Unfortunately, the harvard number is probably excessively conservative.  Newer research suggests that testing may need to happen as often as weekly to effectively suppress transmission within a population;  we won't reach those levels for a lot longer.  (But we have other means of suppressing transmission, below).  The more testing and tracing the better -- every time we catch a case, we help prevent them from infecting more people, which drives down the effective reproductive rate.

Serum antibody tests are now out in force, and some of them, such as the Abbott test, are showing very high selectivity and specificity.  They're helping drive better understandings of how much of the population has been infected and clearing up earlier divergent hypotheses about this (answer: in most places, the numbers are at or under 5%, which means we're very far from any sort of herd immunity).  We're not yet to the point of understanding immunity and the tests well enough to trust them as immunity certificates, but it's getting closer.

Supplies

n95 masks:  3M claims to be ramping up from the previous 35m masks/month domestically to 95m masks/month, but this is on a timeline that may take until the end of 2020.  The current phase of expansion is only expected to start producing more masks about ... now.  There's still a general mask shortage, and some of the decontamination efforts are scaling back their claims a little (e.g., Battelle's decontamination system, which they hoped would allow 20 reuses per mask, is looking perhaps like it will handle three reuses).  There's still a lot of benefit there, and hospitals are finding ways to use other respirators, but we still need more.  By the end of the year we'll probably be close to what we need.

surgical masks:  I don't have data yet on the production rates of ASTM level 1 and 2 masks.  Will try to update;  there are some folks at CMU attempting to quantify this.  Regardless, you still can't buy them in the store, so that's at least one sign. :-)

Cloth masks:  Evidence continues to pile up for the effectiveness of widespread cloth mask wearing as a way to reduce the chances of transmitting the virus.  More on that below.

Treatment

Some success stories!  In the previous update, I just handwaved that Remdesivir was possibly maybe helpful.  It's now part of the standard of care, having been shown to reduce hospitalization time (but the reduction in mortality was not statistically significant).  But two months is enough time to run a lot of studies, and the UK's large RECOVERY study completed an analysis of the use of the steroid dexamethasone in severely-ill COVID-19 patients, finding that it could cut mortality in ventilated patients by up to 1/3.  It's not a silver bullet, and isn't appropriate for use in less seriously-ill patients or the earlier stages of the illness (it suppresses the immune system), but this is a fantastic treatment improvement that can cut the overall fatality rate.

There are continuing trials of other antivirals and treatments, but as before, these can take a long time.  It's still reasonable to expect a few more treatment improvements that will reduce the fatality rate, but hold on to your hats:  The huge reduction in fatalities from dexamethasone is unusual and not something we should expect to stumble upon monthly.

In vaccines, there are now three vaccines in phase-3 trials , and from all signs it seems there's reason to be optimistic that we may have a vaccine in limited production by the end of 2021.  If we're lucky, front-line workers may have something even earlier, but we're still looking at least a year, and possibly longer, before things start to make a difference.  But this remains impressively fast and hopeful-looking progress for a vaccine.

Societal Changes & Knowledge About Transmission

I've sneakily retitled this section, because part of the improvements we get are from being able to be more and more selective about what we close, reopen, or change, and understanding the dynamics of SARS-CoV-2 transmission helps us do those things better.

Some of the most hopeful news is the increasing piles of evidence that non-medical masks and staying outside are effective, low-cost interventions to reduce spread.  It's hard to do randomized trials where people don't know if they're wearing a mask, but the correlations remain strong, and events such as the recent protests (outdoors + most people masked) did not seem to result in case spikes, compared to other reopening-related events (indoor dining/bars) that have been traced to case spikes.

Delivery services have fully ramped up and wait times for, e.g., Instacart and Amazon Prime Now and other services are back near to baseline in many locations.

Business are struggling to figure out how to safely allow employees to come back to work, and this isn't going to be a fast change.  Guidelines for schools and daycares are unclear, but I'm hopeful that as we gain more evidence about the spread in children, an update in two months will note that we now have effective policies for how to resume these things (comparatively) safely.  We'll hopefully be able to learn from Japan's experience reopening their schools, though keeping in mind that mask wearing is much more normalized there and baseline transmission rates have been lower.

We're nowhere near being back to normal.  But we've made continued progress in knocking down the reproductive rate, and that will let us have better choices in balancing daily activities and risk.  Now, of course, it will be necessary to actually make good choices, but that's a different post...




Tolerating the Pandemic #4: The Ventilation Edition
Dave's Data — Sun, 18 Oct 2020 16:27:22 GMT
 This is #4 in the series of how we can tolerate the pandemic in a lot of slow, steady progress and hard work.  As a quick recap, the first post laid out a bit of a roadmap (March 23, 2020); the second (April 17, 2020) and third (June 26, 2020) noted updates on the categories described in the first.
In the two months since my previous post, one of the most exciting advances has been in improved understanding of the transmission dynamics of SARS-CoV-2, particularly with respect to aerosol-like transmission.  An Aug 11 New York Times article has a good summary of some of the research in this area.  Tl;dr:  Ballistic (large droplet) transmission is probably still a big factor, and is helped quite a bit by masks.  Aerosol transmission is a big contributor particularly to superspreader events (and particularly indoors).  Cloth or surgical masks help but are not a silver bullet, particularly in crowded, confined spaces.
Why is this exciting? Because it helps us bring more tools to bear against transmission. It furthers the (already very strong) case for the outdoors being substantially safer than being indoors; it reinforces the importance of masks as part of a source control strategy; and it suggests that improvements to ventilation and good (HEPA or MERV-13 or higher) filtration also have a role to play in reducing the number of people who get infected from a transmitting individual.
For more on this, I suggest starting with the Harvard Healthy Buildings report from June and this slide deck from CU Boulder.  As a taste of some of the great things in there, this slide, analyzing the infection probability of a known real super-spreader event (an indoor choir practice) while changing different components of the scenario, such as increasing ventilation. Or just moving the darn thing outside, which would have approximately solved the problem.  (The advantage of being outside was already clear as of the previous update in June, but more evidence and understanding of why is good.)
With that said, let's get back to the framework from the previous posts: Testing, PPE/Supplies, Treatments, and Societal changes to reduce contact. I'll fold ventilation improvements into the latter, since that's where I'd previously discussed things like plexiglass barriers in retail.
Testing
I'm going to just incorporate covidtracking's graph here, since it's now nicer than mine [source]

So that's .. not what I'd like to see, but we have to be careful reading that. In update #3 (June), we were at 450k tests/day. As of now the US is up to about 700k.  Better than June, but case counts are also still higher than they were in June;  test positive rates are only just now getting back down to where they were. The good news is that we're testing a lot; the bad news is that with cases so high our testing isn't anywhere near sufficient to keep cases low.


PPE
N95 mask production continues to ramp up about as expected.  3M says it's on track to produce 95 million per month by October in the US, and 2B annually for the global market.  In our previous update, 3m was trying to hit 50M/month by July, which they seem to have.  Recall that DHS estimates we need about 300M masks/month in the US, but 3M isn't the only supplier. We're basically on track ramping up our PPE capacity, but it also still may be until the end of 2020 that we've really gotten this solved.
Anecdotally, N95 masks are showing up occasionally on Amazon now (before getting re-restricted for healthcare use only). Reusable options are emerging. But there are still few high-quality surgical masks (ASTM level 1-3 certified) available for the general public, which would be an improvement over the high variance in quality of uncertified masks.
The Purell Indicator
I've been monitoring availability of Purell as one interesting, informal indicator of the availability of hand sanitizer. In the last two weeks, it's become available again. There's been plenty of other-brand hand sanitizer available for the last two months, but as one of the most well-known brands, this is a good sign that manufacturing is catching up with demand.
Treatment Improvements
The big news two months ago was the outcome of the dexamethasone study; that remains part of the standard of care. Overall, studies of the infection fatality rate (IFR) suggest that it is lower now than in earlier phases of the epidemic, which is likely due to a combination of factors: treatment improvements, less overloading of hospitals, and better protection of highly-vulnerable populations such as nursing homes.  But within this hides an important benefit: It's better to get SARS-CoV-2 today than it was in February.  And it's likely to continue to improve as we run through additional options for improving treatment.

There are now approximately 10 vaccine candidates in phase III trials. (I say approximately, because two of them are technically approved, but have been approved prior to their phase III trial, which means that their use will effectively be a phase III, albeit a potentially poorly controlled one).  Prospects for the vaccine remain good:
There are candidates based upon several different technologies.
Results appear promising that infection yields protection for at least several months for most people.
The different technologies part is important. Several of the candidate vaccines are based upon an adenovirus vector, which has the drawback that prior infection by the same or similar adenovirus may reduce the vaccine's effectiveness. This appears to be the case, for example, for the CanSino Ad5-vectored vaccine, which otherwise appears promising and is now in Phase III. This is in no way a fatal flaw for the Ad-based vaccines, but it means we may need alternatives. Fortunately, several of the other candidates are mRNA-based vaccines, such as that by Moderna in the US, which are likely to have entirely different failure patterns: we've never yet taken an mRNA vaccine to approval. But if one of these vaccines works, it probably won't fail for the same people for whom Ad5-vectored vaccines fail. And a few candidates, such as the vaccine from Wuhan Institute of Biological Products / Sinopharm, are inactivated virus vaccine, presenting a third mechanism to play with.

It's important to remember that no vaccine is likely to be perfect, and the deployment of a vaccine is not next week. Even once manufacturing is ramped up, several of the vaccines require two doses, potentially separated by several weeks. But progress remains good.

I'm not going to say much about convalescent plasma. Too much stupid politics going around about it, and it's still awaiting more randomized trial results. It's unlikely to be a silver bullet either, of course: making it requires people who're sick, it's expensive and rare, and its efficacy is as yet unknown. More scalable technologies, such as monoclonal antibodies, are still in development & testing.
Societal and Environmental Changes
As noted above, this is where I'm most excited this month. Ventilation improvements to existing buildings can be expensive, but they pay dividends for years in healthier, more productive occupants, so upgrading as part of a pandemic-reduction strategy may not be a net cost increase as much as an unexpected cost shift to, well, now. This is something we can and should be pursuing aggressively, but we'll see how much traction it gets. I'd love to find someone/where that tracks ventilation system upgrades!

A large concern now is school reopening. We're just seeing the leading edge of it, with universities such as UNC reopening and then backtracking on it. Many k-12 schools have delayed their reopening plans as cases in the US remain well above levels where opening would be safe. School closures have been widely regarded as one of the more effective NPIs (non-pharmaceutical interventions), both because they reduce the chances of children spreading the disease, but also because they reduce parents' mobility.





Svelte for Sites, React for Apps
swyx's site RSS Feed — Sun, 18 Oct 2020 04:08:02 GMT
Why we should stop trying to use the same tool for different jobs.


Emacs: org block complete and edit
xenodium.com @alvaro — Sun, 18 Oct 2020 03:00:00 GMT
I quickly got used to Emacs org block company completion. I did, however, almost always found myself running org-edit-special immediately after inserting completion. I use C-c ' for that. That's all fine, but it just felt redundant.
Why not automatically edit the source block in corresponding major mode after completion? I think I can also get used to that!

Or maybe the automatic approach is too eager? There's also a middle ground: ask immediately after.

Or maybe I don't want either in the end? Time will tell, but I now have all three options available:
(defcustom company-org-block-edit-mode 'auto
  "Customize whether edit mode, post completion was inserted."
  :type '(choice
          (const :tag "nil: no edit after insertion" nil)
          (const :tag "prompt: ask before edit" prompt)
          (const :tag "auto edit, no prompt" auto)))

The new option is now in the company-org-block snippet with my latest config.


The 6 Conditions of Change
A Matter of Perspective — Sun, 18 Oct 2020 03:00:00 GMT
👇 Skip to the conditions 👇

Potential: How flexible are you? 👉
Solutions: What are you solving for in life? 👉
Dissonance: Your tolerance for change
Barriers: What confronts you? (Coming Soon)
Insight: What can the past teach you? (Coming Soon)
Environment: Who is helping you transition? (Coming Soon)


Dear reader,
Have you ever spent what felt like an eon forcing everything you’ve got into trying to change something?
Maybe it is the endless attempts of diets or quitting smoking.
Perhaps, you’re trying to become a hard-working, responsible citizen that everybody else seems to have worked out how to be. Instead, you’re halfway through your first degree and just quit your job to travel to South America.
Are you trying to rally a team together at work but feel like you don’t have enough support?
Can you relate to me when I say, “I’ll change one of these days,” Or are you more the type to respond with, “Change? I’m open to anything, really.”
Do you feel like you’re reaching for something, but everything around you is pulling you down? You can’t get a handle on the world around you, and you just want, so badly, to arrive at the end?

Removing the friction
Changing things is complicated. No matter how big or small, adjusting our current reality requires great dedication and energy. This is a series exploring the 6 necessary conditions that need to be present in order for change to occur.
Imagine these conditions as six keys we need to unlock the door of real change. If one of these conditions is not met, the door remains closed. If more than one is not met…we may even regress.
With all conditions present, the door opens, and we can move easily through the passage. Change will live within us and act upon us effortlessly.
The 6 conditions of Change
Behold! The 6 Keys, initially outlined by Clare Graves and his students Don Beck and Edward Cowen. This series will address the high-level concepts, providing illustrations, because…who doesn’t like pictures 😁? If you’re interested in reading further, I would encourage you to pick up a copy of their book for some bedtime reading.

  
    
      
    
  
  
    
      
    
  
  
    
      
    
  
  
    
      
    
  
  
    
      
    
  
  
    
      
    
  



Potential: How flexible are you?
A Matter of Perspective — Sun, 18 Oct 2020 03:00:00 GMT
👈 Back to all conditions
Our first condition of change is Potential. Not everyone is open to change, you see? Ever heard the phrase, you can lead a horse to water, but you can’t make it drink?
As the world moves to more chaos, and complexity rises like our sea (😕), we move toward a system of many different needs, values, and aspirations. It would make sense for us to be able to move between these right?
Potential is divided up into three “states”, depending on how “open” you are to other ways of thinking (or color systems).

Open: Healthiest. Readiness to accept new modes of being
Arrested: Caught up with the current way of life, and the Barriers present (more on Barriers later)
‌Closed: Unhealthiest. Blocked entirely by life conditions, and lacks flexibility

Open: “Going with the flow”

To be “open” in terms of one’s potential, means there is flexibility for more complex levels of functioning to happen.
For you psychology buffs out there, this could correlate to the “Openness” of the Big 5.
Being open is the healthiest form, with the individual/organization/country adapting and shifting as the surrounding world changes. In our image above, we jump on a boat to escape the rising tide, able to move wherever we please.
Someone who is open does not hold a sharp picture (aka: a strong frame). They deal well with Barriers, as they are able to move outside the current way of thought, dodge things, or think about how to turn the situation around.
Open people/entities have great listening skills, are non-judgmental, tolerant of differences, and lack close-mindedness. Generally, they’re pretty bad-ass people 😎.

Open:

Healthy, flexible, and anticipates change. Adapts when needed

The power of “openness” comes from it’s anticipation that things will change. There is an “elasticity” between perspectives, and they’ll often engage in multiple colors.
This is the ideal place for us to all be, but it takes a LOT of work, and is something we must constantly be vigilant on. Lest we become…
Arrested: “Don’t rock the boat”

For most of us, this is the default mode (unless we work really hard to be otherwise). Change can occur, but only if our Barriers are overcome. Also, we need more Dissonance, as we become more “entrenched” into our current way of life (not a problem for Open).
Arrested people/companies/countries may also lack Insights to explain what is happening. As a result, an Arrested entity will make excuses for the status quo, reluctant to change things. They will come to grips with the Barriers, as opposed to working with, changing, or transmuting them in Open.
If you catch yourself saying:

“Let’s do what we can with what we have”
“I’ll change one of these days
“Stick with the tried and true”
“I’ve come to peace with it now”

There may be a chance you’re a touch arrested in how you’re approaching the situation.

Identifying the Barriers, teaming up with others, and finding innovative ways to approach your problems could help increase your openness.

If we chose to remain arrested, expect little movement towards more complex thinkings. Efforts will instead be placed toward refining and working harder/smarter, versus a fundamental shift of thinking and approach where things are more open.
New life conditions (org changes at work, new tax laws, natural disasters, breakups, etc) pose much more disruption for someone who is Arrested. They generate more anxiety, and uncertainty, as the “fixed” way of life is forced to change.
To extend our metaphor of the image above, the house on the island is quite stationary. Although we are fairing better than the drowning, Closed, figure, when the sea eventually rises we will be envious of the “S.S. Open”. Unless we re-address the fundamental relationship to our problems, we’ll just keep building higher, and higher, eventually ending up in…
Closed: “Life sucks, and it’s NOT my fault”

The Barriers are high in a Closed state of mind. If we ignore the warning signs, and slip from Beta into the Gamma Trap, they will grow higher still. It’s not a great place to be.
There are a number of places a “Closed” perspective may come from:

Historical trauma that created incredible, negative meaning towards something
Refusal to identify and address the issues one is experiencing before it’s “too late”
Strong social/parental conditioning (passing on fears and beliefs to children is sadly very common)
Physical ailments or damage (unfortunately preventing the individual to progress developmentally)

What little flexibility we had in Arrested is now completely gone. The confusion, and uncertainty of Arrested give way to anger, hatred, panic, violence, suicide, depression, and a range of other nasties. Someone who is Closed feels threatened by just about everything that’s not “my perspective”, and will fight tooth and nail to protect the failing way of life.
If we observe, again, our metaphor, the individual has sand-bagged himself on a sinking island. Refusing refuge from the “Open Boat,” and too late to build a house, he remains trapped in his condition until he either changes, or drowns.
Grim, much 💀?
The danger of Closed minded thinking stems from it’s inability to see the Barriers. Much like the power that Openness receives from it’s flexibility, the opposite is true for Closed thinking. Closed perspectives are often exhibited by an overactive, single color that feels threatened by introduced circumstances.
Here is what to watch for in yourself and others to identify a Closed perspective:

“My way is the only way a rational person would think”
“This is all there is”
“I can’t think of any way to solve this”
“It’s done for”
“Everyone else is wrong”
“Why can’t you see my view?”
Constant doubt (“Do you really love me?”)
Insatiability
Every idea washed with the same color
Irritable quickly under stress
Completes things to extremes
Builds a shell to hide in


Closed:

Inability to see Barriers, shuts down, avoids all change, and fights to stay put.

A Closed perspective will usually collapse in on itself, unless supported by others inflicted with the same thinking. Groups of closed-mindedness can clump together, dig trenches, and try to “take on the world”. Take, for example, how the Samurai of Japans dying feudal era resisted against western ideas. Instead of adapting, integrating, and merging into the modern world, they fought to the last man, and are no more.

Open for Business: Using “Potential”
What do we get when we assess the “potential” of some thing, or some one?
The greatest benefit is we may now identify the probability that it/we/they can change.
If they are Open, then the probability is high.
If we are Arrested, the Barriers need to be identified that are creating walls, and the Dissonance volume may need to be turned up (make them sweat it 😓).
If it’s closed, then deep thought should be given to whether it’s worth trying to change it or not. This may sound drastic, however if Potential is low (Closed), and the fortifications are up, it may be more trouble than it’s worth.
Once we can read the Potential, we may also assess the required energy to shift the system. This can be critical at an organizational level to determine if resources should be placed in re-directing the current way of things, or investing energy into something else.
Potential will also give us insight into the stress produced from changes (side-effects). We additionally learn how much can be changed, and in what ways, and to what degree.

Potential is just one of the Six Conditions of Change. It is arguably one of the most important condition needed for enabling a shift, and provides valuable information on how receptive and flexible we/they/you are.
So next time, before jumping into that new exercise routine, company structure, or country, have a look at your own Potential. Are you Closed in some areas, and Open in others? Where could you be more flexible? What is holding you Arrested, limiting your options?
(Next up, we’ll take a look at Solutions)[/articles/6-conditions-of-change/solutions/], and how focusing on where we’re at can make all the difference.
👈 Back to all conditions


BEBEN X8 Review
Danny Guo — Sun, 18 Oct 2020 03:00:00 GMT



Yahoo groups is finally closing, and I am SO happy
Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 17 Oct 2020 21:00:00 GMT
Why? Because it may (just may) help many people see the light.


Why does package-lock.json exist, and how does it work?
Lucas F. Costa - Blog — Sat, 17 Oct 2020 15:00:00 GMT
When writing JavaScript applications, you describe its dependencies using a `package.json` file. This file contains all of your applications direct dependencies and their versions.

If you're buildin...


Mac and Chrome HotKeys
Greg Herlein — Sat, 17 Oct 2020 11:00:01 GMT
A simple hotkey extension in Chrome that makes my life easier.


Object Detection at 1840 FPS with TorchScript, TensorRT and DeepStream
Machine Learning Articles on paulbridger.com — Sat, 17 Oct 2020 09:43:23 GMT
In this article we take performance of the SSD300 model even further, leaving Python behind and moving towards true production deployment technologies: TorchScript, TensorRT and DeepStream. We also identify and understand several limitations in Nvidia’s DeepStream framework, and then remove them by modifying how the nvinfer element works.


AxiDraw Lego camera mount
Muffin Man — Sat, 17 Oct 2020 03:00:00 GMT
I already built a mount for my Raspberry Pi camera, but it was 3d printed. Meaning that it has one fixed position. I used it a couple of times, but I wasn't super satisfied. Then I stumbled on these images, and thought it was a great idea to use Lego to build a new, more versatile mount.
A friend of mine had a bunch of spare Legos laying around, so he gave me two bags full of Legos to play around. I wasn't sure where to start, especially because I wanted to make both height and rotation adjustable. After fiddling with the blocks for about an hour, I came up with this:

It is quite simple, but the secret is in the blue part. Because it can be connected with the black "rail" I
glued on to the plotterI used double sided mounting tape, as it is stable enough and easy to remove if I want to

This allows me to change the height of the mount, by inserting the blue part into the different hole. And to make things even better, the blue part behaves as an axis, and the whole mount can be rotated around it. Once mounted, it looks like this:

And finally I shot a short test video plotting my logo:

If you want to see pen plots of my generative art, check my Instagram account.


Distribute ALL THE THINGS!
Q. Marchi: A Blog — Sat, 17 Oct 2020 03:00:00 GMT
Bad Ideas Are Born out of Necessity
So recently, I got a new Dell R720xd (from TechMikeNY) to compliment my Dell C2100 that I’ve had for a few years.
The new server was promptly christened “Irminsul” (following the C2100 as “Yggdrasil”, a “World Tree”),
and begun a full disk test to make sure everything was spotless. Unfortunately, a disk ended 99.97% with bad sectors, so a quick email to
Mike (Seriously, shout out to the team over there.) and we were off to the races.


October 2020 Projects
Venam's Blog — Patrick Louis (Lebanon) — Sat, 17 Oct 2020 00:00:00 GMT
Seven long and perilous months have gone by since my previous article, what feels like an eternity, and yet feels like a day — Nothing and everything has happened. All I can add to the situation in my country, that I've already drawn countless times, is that my expectations weren't fulfilled. Indeed, after a governmental void and a horrific explosion engulfing a tremendous part of the capital, I'm not sure any words can express the conflicting feelings and anger I have. Sentimentalities aside, let's get to what I've been up to.


How to extract plain text from an HTML page in Python
Posts by Petr Stribny — Fri, 16 Oct 2020 22:49:14 GMT



Always beware of mathwashing!
Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 16 Oct 2020 21:00:00 GMT
You still don’t know how mathwashing may hurt you? Keep reading, then.


Intermedia Workshop Notes 1999-2003
mbutler — Fri, 16 Oct 2020 18:46:09 GMT
Matthew T. Butler Intermedia Workshop notes University of Iowa 1999-2003 Intermedia Writing explores language from a visual arts perspective. It examines the relationship between word and image as well as word as image. It asks, “How can we write with video, with paint, with our bodies? How does language function in the artist’s studio?” Intermedia […]


Lenses Matter
bastibe.de — Fri, 16 Oct 2020 15:14:00 GMT
A common trope in discussions about cameras on the internet is: “sensor size trumps all”, and as a corollary, “smartphone cameras suck”. Which is obviously false to anyone who has ever taken a good picture on a smartphone (with its tiny sensor).
The argument, however, goes like this: Bigger pixels¹ can capture a greater dynamic range, and bigger sensors are made from bigger pixels. Therefore, and here is the dangerous leap, cameras with bigger sensors take better pictures.
But I currently have a bit of time on my hands, and way too many cameras, so let's try it out, shall we?
My five contestants are:



A Google Pixel 2, with a 12.2 MP 5.75 × 4.32 mm “1/2.55 in” sensor from 2017, and a 3.8 mm f/1.8 lens (27 mm f/13 equivalent, crop 7.14),
A Pentax Q7, with a 12 MP 7.44 × 5.58 mm “1/1.7 in” sensor from 2013, and a 5-15 mm f/2.8-4.5 lens (24-70 mm f/13-22 equivalent, crop 4.76),
A Panasonic LX100, with a 13 MP, cropped 17.3 × 13 mm “micro four thirds” sensor from 2014, and a 11-34 mm f/1.7-2.8 lens (24-75 mm f/3.8-6 equivalent, crop 2.2),
A Ricoh GR), with a 16 MP 23.7 × 15.7 mm “APS-C” sensor from 2013, and a 28 mm f/2.8 lens (27 mm f/4.2 equivalent, crop 1.5),
A Fujifilm X-T2, with a 24 MP 23.6 × 15.6 mm “APS-C” sensor from 2016, and a 18-135 mm f/3.5-5.6 lens (27-200 mm f/5.25-8.4 equivalent, crop 1.5).

So I went for a walk around the neighborhood, and took a few pictures with a bag full of cameras. In particular, I was looking for a high dynamic range scene, in the form of a sunset. All pictures were taken at a low ISO² and underexposed by two stops to account for the backlighting. All images were taken at 27 mm (equivalent³) focal length.
Measurements predict that the dynamic range and noise levels of these cameras should get better as the sensor size increases.



  
    
      
      Google Pixel 2
    
    
      
      Pentax Q7
    
    
      
      Panasonic LX100
    
    
      
      Ricoh GR
    
    
      
      Fujifilm X-T2
    
  


The first scene was taken essentially in auto mode, and adjusted in Darktable to match roughly in tonality (exposure adjustments of ±0.25EV), but not color. In general, all cameras were able to capture the entire dynamic range of this scene, from the brightest almost-white area around the sun, to the darkest patches in the shade near the waterline.
This, right off the bat, is the most important result of this experiment: Most scenes do not have enough dynamic range to matter.
Differences appear in the noise levels, which are fairly constant across all cameras, with only the X-T2 being noticeably less noisy than the rest. However, this advantage of the X-T2 is not one of sensor size (the Ricoh GR has a same-size sensor), but simply of being two-generations newer.
The largest difference between these photos is instead in flare: The Pixel 2, Pentax Q7 and Ricoh GR are flaring across the entire image, while the LX100 and X-T2 are far more controlled. This is probably because the Pixel 2 and Ricoh GR have collected dust and scratches from my pockets, and the Pentax Q7 lens being somewhat mediocre. The LX100 and X-T2 are in a better condition, and a better design, too. All lenses also show some amount of colorful lens flares to the bottom-right of the sun, particularly in the LX100 and Pentax Q7, and strange, colored rings around the edge of the Ricoh GR's frame.



  
    
      
      Google Pixel 2
    
    
      
      Pentax Q7
    
    
      
      Panasonic LX100
    
    
      
      Ricoh GR
    
    
      
      Fujifilm X-T2
    
  


To look into these differences between lenses in more detail, the second scene was taken from a slightly different spot, with the lenses stopped down (if possible). This image includes two bright sun spots, one from the sun itself, and one in the reflection on the water.
Notably, the Pixel 2, like most smartphones, can not change its aperture. And it produces a bright red flare towards the left edge of the image. The Pentax Q7 again flares dramatically and colorful, and my Ricoh GR's lens is probably beyond cleaning at this point, judging from the massive flare it produces. Again, the standout pictures here are the X-T2 and LX100, with well-defined sun stars and comparatively little flaring. (Note the red dots in the X-T2: these are the phase-detection pixels on the X-T2's sensor reflecting light back into the lens and back on the image.)



  
    
      
      Google Pixel 2
    
    
      
      Pentax Q7
    
    
      
      Panasonic LX100
    
    
      
      Ricoh GR
    
    
      
      Fujifilm X-T2
    
  


Here is an enlargement of the dark patch at the right edge of the second scene, to judge noise levels and detail in the shadow. Perhaps surprisingly, the differences between these images are not big at all. The X-T2 clearly wins in terms of both shadow detail and noise levels, but by a rather small margin, and only really visible when enlarged. The Pentax Q7 resolves almost no detail in the shadow, but neither does the Ricoh GR. So sensor size does not seem to affect this round.



  
    
      
      Google Pixel 2
    
    
      
      Pentax Q7
    
    
      
      Panasonic LX100
    
    
      
      Ricoh GR
    
    
      
      Fujifilm X-T2
    
  


This third scene was shot one day later, with settings optimized for optimal image quality: Base ISO, aperture one stop closed from wide open, focused on a high-contrast part of the scene. I processed these images with white balance from a grey card, then edited exposure, contrast, and saturation to match across images. Chromatic aberrations and vignetting were corrected using a profile or by hand. No sharpening was applied, but the highest-quality deinterlacing was selected (AMaZe for Bayer sensors, Markensteijn 3-pass for X-Trans sensors).
At this magnification, even my 24in/4K screen only barely hints at differences in sharpness. Which implies that prints up to A3 should be possible from all of these cameras without much difference in perceived sharpness.



  
    
      
      Google Pixel 2
    
    
      
      Pentax Q7
    
    
      
      Panasonic LX100
    
    
      
      Ricoh GR
    
    
      
      Fujifilm X-T2
    
  


If we crop into these images, differences in detail resolution indeed become apparent. The Pentax Q7 is definitely the least detailed, on account of its soft lens. The Pixel 2 and LX100 show a very similar amount of details, which is a testament to how good modern smart phone cameras truly have become. While the two cameras with the biggest sensors, the Ricoh GR and the X-T2, do resolve visibly more detail, the differences are not dramatic.
So does sensor size trump all? From these experiments, I wouldn't say so. Essentially, all of these cameras took decent images. The X-T2 clearly was better than the rest, but not hugely so, while being much bigger and more expensive and newer.
While the measured dynamic range differences surely exist, they did not manifest even in a sunrise, so I'm inclined to discount their importance. This is, frankly, not what I expected. The miniscule differences in resolution were equally unexpected.
What did however make a big difference was lens quality. Flare control, aperture range, and focal range play a huge role in the resulting image quality. Zooming in (and stopping down) are still the most effective way of extracting more detail from a scene. And having a camera with a secure grip that is quick to operate helps, too. I think I am growing out of sensor size snobbery, is what I'm saying.


technically, “sensor elements”, not “picture elements”, i.e. “sensels”↩
lower ISO → less noise.↩
the focal length a lens on a 35 mm film camera would need to cover the same angle of view.↩




Talking to Strangers: What We Should Know About the People We Don’t Know
Apurva Shukla's RSS Feed — Fri, 16 Oct 2020 10:00:00 GMT
⭐ ⭐ ⭐ ⭐ ⭐ You’re complex, why can’t we think of strangers the same way? Throughout a series of case studies Gladwell attempts to answer why…


Flexbile Python use in Blender
ciesie.com — Fri, 16 Oct 2020 03:00:00 GMT
This post is about using Python in Blender. It’s not a tutorial. It’s more of an inspiration
to learn and experiment with Python.
Imagine you work on a piece. The first stage is very experimental. You look for what works and what
doesn’t. At some point the piece and the process take shape. Some things become repetitive. It’s
very useful to notice those patterns early and optimize them.
Lets say I’m working in ZBrush and I want to render in Blender. The geometry moves between
those two pretty often. Exporting and importing takes quite a few clicks. All that clicking adds
up and nibbles at my, that day’s, energy.


Simple video buffering loader in React
Muffin Man — Fri, 16 Oct 2020 03:00:00 GMT
For multiple projects, I had to add a simple video component with
play/pause buttonAlways include at least basic video controls for accessibility
and a buffering loader. It is not hard to detect the buffering state, but it can be tricky to get everything right.
Therefore, I created a simple component which I now copy from project to project with slight style adjustments. Jump to the code if you are not interested in how it's made.
Demo #
Here you can see the final version (it might be easier to see the functionality if you open it in a separate tab).

How it works #
The solution is simple and relies on four video events:

waiting - playback has stopped because of a temporary lack of data
pause - playback has been paused
play - playback has begun
playing - playback is ready to start after having been paused or delayed due to lack of data

We are going to use waiting event to set our loading flag to true and all other events to set it to false.
If the video is already playing, and buffering happens, we'll get events fired in this order:
"waiting"
"playing"

This is all fine and dandy, but if the video is paused, and we play it things get a little messy.
Now, if buffering happens, we'll get these events fired:
"play"
"playing"
"waiting"
"playing"

All of these events are fired in a very quick succession and alter the state too fast leading to some bad UX. The secret is to add a short debounce time to avoid it. What I found to work best is 200ms for waiting event and 50ms for all others.
Code #
I'm using React with hooks, but the same thing can be easily ported to React class or vanilla JavaScript.
import React, { useEffect, useState, useRef } from 'react';

const PLAYING_DEBOUNCE_TIME = 50;
const WAITING_DEBOUNCE_TIME = 200;

const Video = ({ src, ...props }) => {
  const [isPlaying, setIsPlaying] = useState(false);
  const [isWaiting, setIsWaiting] = useState(false);

  const isWaitingTimeout = useRef(null);
  const isPlayingTimeout = useRef(null);

  const videoElementRef = useRef();

  useEffect(() => {
    if (!videoElementRef.current) {
      return;
    }

    const waitingHandler = () => {
      clearTimeout(isWaitingTimeout.current);

      isWaitingTimeout.current = setTimeout(() => {
        setIsWaiting(true);
      }, WAITING_DEBOUNCE_TIME);
    };

    const playHandler = () => {
      clearTimeout(isWaitingTimeout.current);
      clearTimeout(isPlayingTimeout.current);

      isPlayingTimeout.current = setTimeout(() => {
        setIsPlaying(true);
        setIsWaiting(false);
      }, PLAYING_DEBOUNCE_TIME);
    };

    const pauseHandler = () => {
      clearTimeout(isWaitingTimeout.current);
      clearTimeout(isPlayingTimeout.current);

      isPlayingTimeout.current = setTimeout(() => {
        setIsPlaying(false);
        setIsWaiting(false);
      }, PLAYING_DEBOUNCE_TIME);
    };

    const element = videoElementRef.current;

    element.addEventListener("waiting", waitingHandler);
    element.addEventListener("play", playHandler);
    element.addEventListener("playing", playHandler);
    element.addEventListener("pause", pauseHandler);

    // clean up
    return () => {
      clearTimeout(isWaitingTimeout.current);
      clearTimeout(isPlayingTimeout.current);

      element.removeEventListener("waiting", waitingHandler);
      element.removeEventListener("play", playHandler);
      element.removeEventListener("playing", playHandler);
      element.removeEventListener("pause", pauseHandler);
    };
  }, [videoElementRef]);

  const handlePlayPauseClick = () => {
    if (videoElementRef.current) {
      if (isPlaying) {
        videoElementRef.current.pause();
      } else {
        videoElementRef.current.play();
      }
    }
  };

  return (
    <div className="SimpleVideo">
      <video {...props} ref={videoElementRef} src={src} className="SimpleVideo-video" />

      <button onClick={handlePlayPauseClick} className="SimpleVideo-playPause">
        {isPlaying ? "Pause" : "Play"}
        {isWaiting && <span className="SimpleVideo-loader">Buffering</span>}
      </button>
    </div>
  );
};



Let's Peek: A tale of finding "Waypoint"
Caius Theory — Thu, 15 Oct 2020 22:00:00 GMT
Following a product launch at work earlier this year, I theorised if someone was watching the published lists of SSL Certificates they could potentially sneak a peak at things before they were publicised. Probably far too much noise to monitor continuously, but as a potential hint towards naming of things with a more targeted search it might be useful. Sites like https://crt.sh/ and https://censys.io/certificates make these logs searchable and queryable.
Fast forward to this week, where at HashiConf Digital HashiCorp are announcing two new products, which they’ve been teasing for a month or so. Watching Boundary get announced in the HashiConf opening keynote I then wondered what the second project might be called.
I’ve spent a chunk of the last month looking at various HashiCorp documentation for their projects, and I noticed they have a pattern recently of using <name>project.io as the product websites. The newly announced Boundary also fits this pattern.
🤔 Could I figure out the second product name 24 hours before public release? Amazingly, yes! 🎉
Searching at random for all certificates issued for *project.io was probably going to be a bit futile, so to narrow the search space slightly I started by looking at when boundaryproject.io had its certificate issued, and who by. The list of things I spotted were:

Common name is “boundaryproject.io”
Issued by LetsEncrypt (no real surprise there)
Issued on 2020-09-23
Leaf certificate
Not yet expired (still trusted)
No alternate names in the certificate

Loading up https://censys.io/certificates and building a query for this, resulted in a regexp lookup against the common name, and an issued at date range of 10 days, just before and a week after the boundary certificate issued date.
parsed.subject.common_name:/[a-z]+project\.io/ AND
parsed.issuer.organization.raw:"Let's Encrypt" AND
parsed.validity.start:["2020-09-20" TO "2020-09-30"] AND
tags.raw:"leaf" AND
tags.raw:"trusted"

(Run the search yourself)
Searching brought back a couple of pages of results, I scanned them by eye and copied out the ones that only had the single name in the certificate which resulted in the following shortlist:

boundaryproject.io
essenceproject.io
lumiereproject.io
techproject.io
udproject.io
vesselproject.io
waypointproject.io

We already know about Boundary, so the fact I found it in our list suggests the query might have captured the new product site too. Loading all these sites in a web browser showed some had password protection on them (ooh!) and some just plain didn’t load (ooh!), and some others were blatently other things (boo!). Removing the latter ones left us with a much shorter list:

essenceproject.io
udproject.io¹
waypointproject.io

All domains on the internet have to point somewhere, using DNS records. On a hunch I looked up a couple of the existing HashiCorp websites to see if they happened to all point at the same IP Address(es).
$ host boundaryproject.io
boundaryproject.io has address 76.76.21.21
$ host nomadproject.io
nomadproject.io has address 76.76.21.21
$ host hashicorp.com | head -1
hashicorp.com has address 76.76.21.21
Ah ha, now I wonder if any of the shortlist also points to 76.76.21.21 🤔²
$ host essenceproject.io | head -1
essenceproject.io has address 198.185.159.145
$ host udproject.io | head -1
udproject.io has address 137.74.116.3
$ host waypointproject.io
waypointproject.io has address 76.76.21.21
🎉 Excellent, https://waypointproject.io was a password protected site pointed at HashiCorp’s IP address 🎉
I then wondered if I could verify this somehow ahead of waiting for the second keynote. I firstly tweeted about it but didn’t name Waypoint explicitly, just hid “way” and “point” in the tweet. I got a reply from @ksatirli which suggested it was correct (and then later @mitchellh confirmed it.³)
HashiCorp also does a lot in public, and all the source code and related materials are on GitHub so perhaps some of their commit messages or marketing sites will contain reference to Waypoint. One github search later across their organisation: https://github.com/search?q=org%3Ahashicorp+waypoint&type=issues and I’d discovered a commit in the newly-public hashicorp/boundary-ui repo which references Waypoint: 346f76404

chore: tweak colors to match waypoint and for a11y

Good enough for me, now to wait and see what the project is for. Given it’s now all announced and live, you can just visit https://waypointproject.io to find out! (It’s so much cooler/useful than I’d hoped for.)




I so hope whoever registered this was going for UDP in the name, rather than UD Project. ↩︎


I’m a massive fan of IP address related quirks. Facebook’s IPv6 address contains face:b00c for example. A nice repeating 76.76.21.21 is almost IPv4 art somehow. ↩︎


Secrets are more fun when they are kept secret. 🥳 ↩︎





Websoft giants are doing great
Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 15 Oct 2020 21:00:00 GMT
Do you know HOW great?


Étude in C minor
zserge's blog — Thu, 15 Oct 2020 03:00:00 GMT
Let me start with a picture.
int main() {float f ;char c;int d,o;while(scanf( "%d%c%d ",&d,&c,&o) >0){c&=31;for(f=!(c>>4)*55,c=(c*8/5+8)%12+o* 12-24;c--;f*=1.0595); for(d=16e3/d;d--; putchar(d*f *.032)) ;}} I don’t really draw well, so I used formatted C code instead. That was supposed to be a triangular sound wave written in C. In fact, this very C code plays two-octave melodies written in text notation from stdin.
It’s only 160 bytes, fits into a modern-day tweet, and the reason it’s there is to show how simple it is to create minimal music with code in any programming language, not just the special languages like CSound, ChucK, or SonicPi.


Using dynamic SQL query in Redshift database
Rasik Jain - Senior Architect | Generative AI & Full-Stack Engineering — Thu, 15 Oct 2020 03:00:00 GMT
While working with a client, we had a requirement to perform bulk insert/update using Retool table and Redshift database. For this situation, We had to loop through the table records and execute a dynamic SQL query in Redshift.
Redshift database supports execution of dynamic SQL with the help of Prepared Statements or Stored Procedures.
Prepared Statements We use prepared statements when we want to execute dynamic SQL queries directly without a stored procedure.


Hacktoberfest 2020: Government edition
Daniel Immke — Thu, 15 Oct 2020 03:00:00 GMT
Focusing entirely on government contributions this year.


A foreigner's opinion on Walt Disney's debts, or bailouts
Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 14 Oct 2020 21:00:00 GMT
Disney is too big for something else, not failure.


Future of Copyright & Competition Law in Media
Essays by Matthew Eric Bassett — Wed, 14 Oct 2020 19:20:00 GMT



Class Unloading in Layered Java Applications
Gunnar Morling — Wed, 14 Oct 2020 18:45:00 GMT

Table of Contents

Do We Really Need Plug-ins?
Plug-ins in Layered Java Applications
Class Unloading in Practice
If Things Go Wrong
Learning More



Layers are sort of the secret sauce of the Java platform module system (JPMS):
by providing fine-grained control over how individual JPMS modules and their classes are loaded by the JVM,
they enable advanced usages like loading multiple versions of a given module, or dynamically adding and removing modules at application runtime.


The Layrry API and launcher provides a small plug-in API based on top of layers,
which for instance can be used to dynamically add plug-ins contributing new views and widgets to a running JavaFX application.
If such plug-in gets removed from the application again,
all its classes need to be unloaded by the JVM, avoiding an ever-increasing memory consumption if for instance a plug-in gets updated multiple times.


In this blog post I’m going to explore how to ensure classes from removed plug-in layers are unloaded in a timely manner,
and how to find the culprit in case some class fails to be unloaded.



HashMap 使用与分析
张可的博客 — Wed, 14 Oct 2020 18:16:27 GMT
HashMap
HashMap 实现了 Map 接口，用于存储键值对，与 Hashtable 不同的是，HashMap 允许 null 元素。

HashMap 具有两个性能相关的参数：初始容量(initial capacity)和负载因子(load factor)， 容量是指 HashMap 桶（buckets）的容量，初始容量即 HashMap 在创建时桶的默认大小。

负载因子默认是 0.75，默认初始容量是 16，初始容量如果设置过大会导致遍历时间变长，但可以降低 resize 的次数，需要结合场景自己权衡。

在容量达到阈值后会进行扩容，扩容后容量为当前的两倍。

另外设置的初始容量并不代表实际数组的初始大小，而是会根据设定值找到一个最近的 2 的次幂当做初始容量。而因为每次扩容都是两倍，所以也保证了数组长度一直是 2 的次幂。

HashMap 是非线程安全的，可以通过 Collections 类获取同步的包装对象：
Map m = Collections.synchronizedMap(new HashMap(...));


HashMap 的迭代器被设计为 fail-fast 的，如果迭代器创建完成后 HashMap 的结构被修改，迭代器将会抛出ConcurrentModificationException异常。

HashMap 采用数组+链表/红黑树的数据结构，当链表长度大于等于 8 时链表将转换为红黑树。

源码
HashMap 的源码相比较之前的集合要多很多，主要是因为 1.8 之后引入了红黑树，很多代码都是对红黑树的操作。

构造器
我们先看看初始化：
public HashMap(int initialCapacity, float loadFactor) {
    if (initialCapacity < 0)
        throw new IllegalArgumentException("Illegal initial capacity: " +
                initialCapacity);
    if (initialCapacity > MAXIMUM_CAPACITY)
        initialCapacity = MAXIMUM_CAPACITY;
    if (loadFactor <= 0 || Float.isNaN(loadFactor))
        throw new IllegalArgumentException("Illegal load factor: " +
                loadFactor);
    this.loadFactor = loadFactor;
    this.threshold = tableSizeFor(initialCapacity);
}
static final int tableSizeFor(int cap) {
    int n = cap - 1;
    n |= n >>> 1;
    n |= n >>> 2;
    n |= n >>> 4;
    n |= n >>> 8;
    n |= n >>> 16;
    return (n < 0) ? 1 : (n >= MAXIMUM_CAPACITY) ? MAXIMUM_CAPACITY : n + 1;
}

构造器中除了对两个入参做了合法性检测之外就是调用了tableSizeFor方法计算扩容阈值，而这个值也会被当成数组的初始大小。tableSizeFor方法的一系列位运算骚操作即可保证返回值是最接近给定参数的一个 2 的次幂。

put(K, V)
public V put(K key, V value) {
    return putVal(hash(key), key, value, false, true);
}
static final int hash(Object key) {
    int h;
    return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16);
}
final V putVal(int hash, K key, V value, boolean onlyIfAbsent,
               boolean evict) {
    Node<K,V>[] tab; Node<K,V> p; int n, i;
    if ((tab = table) == null || (n = tab.length) == 0)
        // 初始化数组
        n = (tab = resize()).length;
    if ((p = tab[i = (n - 1) & hash]) == null)
        // 通过哈希值计算该 key 的数组索引并判断该位置是否已经存在数据，
        // 没有则创建一个新的链表
        tab[i] = newNode(hash, key, value, null);
    else {
        // 该索引已存在数据：链表或红黑树
        Node<K,V> e; K k;
        if (p.hash == hash &&
                ((k = p.key) == key || (key != null && key.equals(k))))
            // key 已存在，后面会直接更新 value
            e = p;
        else if (p instanceof TreeNode)
            // key 不存在，且当前索引位置的元素为红黑树
            // 则将该键值对插入红黑树中
            e = ((TreeNode<K,V>)p).putTreeVal(this, tab, hash, key, value);
        else {
            // key 不存在，且当前索引位置的元素为链表
            for (int binCount = 0; ; ++binCount) {
                if ((e = p.next) == null) {
                    // 创建链表节点并插入到链表末尾
                    p.next = newNode(hash, key, value, null);
                    if (binCount >= TREEIFY_THRESHOLD - 1) // -1 for 1st
                        // 链表长度大于阈值后开始将链表转换成红黑树
                        treeifyBin(tab, hash);
                    break;
                }
                if (e.hash == hash &&
                        ((k = e.key) == key || (key != null && key.equals(k))))
                    break;
                p = e;
            }
        }
        if (e != null) { // existing mapping for key
            V oldValue = e.value;
            if (!onlyIfAbsent || oldValue == null)
                // 更新 value
                e.value = value;
            afterNodeAccess(e);
            return oldValue;
        }
    }
    ++modCount;
    if (++size > threshold)
        // 长度大于阈值后进行 resize 操作
        resize();
    afterNodeInsertion(evict);
    return null;
}

内容主要在putVal中，通过哈希值计算索引的算法也很简单：
// n 为数组的长度
(n - 1) & hash

因为之前保证过数组长度永远是 2 的次幂，所以对n - 1进行 & 运算相当于求余，从而保证索引在数组内。

再将元素添加到链表中后会判断是否需要转换成红黑树，需要的话则会调用treeifyBin转换，最后还会调用resize方法进行扩容。

treeifyBin(Node<K,V>[], int)
该方法用于将链表转换成红黑树。
final void treeifyBin(Node<K,V>[] tab, int hash) {
    int n, index; Node<K,V> e;
    if (tab == null || (n = tab.length) < MIN_TREEIFY_CAPACITY)
        resize();
    else if ((e = tab[index = (n - 1) & hash]) != null) {
        TreeNode<K,V> hd = null, tl = null;
        do {
            // 链表节点转换成树节点
            TreeNode<K,V> p = replacementTreeNode(e, null);
            if (tl == null)
                hd = p;
            else {
                p.prev = tl;
                tl.next = p;
            }
            tl = p;
        } while ((e = e.next) != null);
        if ((tab[index] = hd) != null)
            hd.treeify(tab);
    }
}

遍历链表所有节点，将其转换成树节点，此时只是把链表节点转换成了树节点，还没有称为红黑树，等到节点全部转换完成后调用hd.treeify(tab)方法才会调整为红黑树。

resize()
当数组长度超出阈值后会进行 resize 操作，该操作会创建一个两倍大小的数组，并对数据进行重新分配。
final Node<K,V>[] resize() {
    Node<K,V>[] oldTab = table;
    int oldCap = (oldTab == null) ? 0 : oldTab.length;
    int oldThr = threshold;
    // 初始化新的数组大小与新的阈值
    int newCap, newThr = 0;
    if (oldCap > 0) {
        if (oldCap >= MAXIMUM_CAPACITY) {
            threshold = Integer.MAX_VALUE;
            return oldTab; // 超过最大值则不进行扩容
        }
        else if ((newCap = oldCap << 1) < MAXIMUM_CAPACITY &&
                 oldCap >= DEFAULT_INITIAL_CAPACITY)
            newThr = oldThr << 1; // 扩容为原来的两倍
    }
    else if (oldThr > 0) // initial capacity was placed in threshold
        newCap = oldThr;
    else {               // zero initial threshold signifies using defaults
        newCap = DEFAULT_INITIAL_CAPACITY;
        newThr = (int)(DEFAULT_LOAD_FACTOR * DEFAULT_INITIAL_CAPACITY);
    }
    if (newThr == 0) {
        float ft = (float)newCap * loadFactor;
        newThr = (newCap < MAXIMUM_CAPACITY && ft < (float)MAXIMUM_CAPACITY ?
                  (int)ft : Integer.MAX_VALUE);
    }
    threshold = newThr;
    // 创建新的数组
    @SuppressWarnings({"rawtypes","unchecked"})
        Node<K,V>[] newTab = (Node<K,V>[])new Node[newCap];
    table = newTab;
    if (oldTab != null) {
        // 将旧数组中的数据迁移到新的数组中
        for (int j = 0; j < oldCap; ++j) {
            Node<K,V> e;
            if ((e = oldTab[j]) != null) {
                oldTab[j] = null;// 删除数组中的引用，防止内存泄漏
                if (e.next == null)
                    newTab[e.hash & (newCap - 1)] = e;// 链表只有一个元素直接存入新的数组即可
                else if (e instanceof TreeNode)
                    ((TreeNode<K,V>)e).split(this, newTab, j, oldCap); // 将红黑树中移入新的数组
                else { // preserve order
                    // 将链表从旧数组移入到新的数组
                    Node<K,V> loHead = null, loTail = null;
                    Node<K,V> hiHead = null, hiTail = null;
                    Node<K,V> next;
                    do {
                        next = e.next;
                        if ((e.hash & oldCap) == 0) {
                            if (loTail == null)
                                loHead = e;
                            else
                                loTail.next = e;
                            loTail = e;
                        }
                        else {
                            if (hiTail == null)
                                hiHead = e;
                            else
                                hiTail.next = e;
                            hiTail = e;
                        }
                    } while ((e = next) != null);
                    if (loTail != null) {
                        loTail.next = null;
                        newTab[j] = loHead;
                    }
                    if (hiTail != null) {
                        hiTail.next = null;
                        newTab[j + oldCap] = hiHead;
                    }
                }
            }
        }
    }
    return newTab;
}



De-Agonizing Performance Self-Evaluations
IvyMike.dev — Wed, 14 Oct 2020 10:00:00 GMT
When I worked at medium-to-large companies, I used to dread the yearly "performance self-assessments".  Not that I didn't want a raise, but writing about my own performance was agonizing.  It reminded me of writing college entrance essays, another torturous process for me.
But at some point, I figured out the …


What's "good" code and does it matter?
ntietz.com blog - technically a blog — Wed, 14 Oct 2020 03:00:00 GMT
I take pride in my work and in writing good code, and it's important sometimes to take a step back and ask: what does that even mean? And does it matter?
At a high level, "good code" is code that is suitable for its purpose and achieves its goals. That definition is pretty lacking, though, I think. You can write some very very hacky prototypes that achieve their goals—proving out an idea—while also being pretty objectively bad code. But objectively, by what measure?

This is where we get back into what it means for code to really be suitable for its purpose. Code is a living thing. It is written, edited, read, and used, in order of increasing frequency: most code will be used far more often than it is edited, read far more often than it is edited, and edited far more often than it's newly written. This means that for code to be good, it has to support these activities. It has to do its job well when it's used. It has to be able to be read. It has to be able to be updated.
A great deal has been written already on how code can do its job well. The short summary here is that it has to do what is expected (per the spec, if you are fortunate enough to have one) and have few defects. On the non-functional side, it has to also do what's expected in a reasonable amount of time, reliably. It doesn't matter how free of bugs your program is if it literally never terminates. And sometimes a "reasonable amount of time" might actually be a floor on the time for things like bcrypt, which we want to make reasonably slow.
Supporting reading and editing go hand-in-hand, because they are core parts of maintaining a codebase. You cannot really edit a codebase confidently if you cannot read it and understand what it's doing, and you cannot fix bugs or add features if you cannot edit it confidently. While tests are a big portion of this, they are distinct from the quality of the code under test. In an ideal world, they add assurance, but the code itself should have a clear design that presents itself. It should be designed from the outset to be extensible.

But does that really capture what we're doing day-to-day? I'm a software engineer, not a computer programmer. While I take pride in writing good code, my job is not to produce good code but to effectively solve problems, usually using code. In practice, engineering means you have to make tradeoffs.
When you're trying to solve a problem, but you're not sure exactly how to solve it, you reach for prototypes and proofs of concept. These will be sufficient to test an idea and validate the approach, but you can cut a lot of corners on them. The code doesn't look good, it's almost certainly not maintainable long-term. But is this a good engineering decision? In a lot of cases, yes! It's the right tradeoff to make.
Similarly, you can write the absolute best code you have ever created for that shiny new feature, but... realistically, you're probably working on it in a business, and realistically, improving that quality to make it super readable and super extensible won't deliver value to the business. It really depends on how much the code will be extended and read, and it's also a tradeoff between time now (for a startup burning cash, time right now is in very short supply!) and time later (once you get profitable or take another infusion of cash from rich suckers venture capitalists, you can afford to rewrite things).
This comes back in a lot of decisions you have to make as a software engineer. If you design a super-scalable system that can handle all the traffic you will need three years from now... well, that growth will probably never materialize, because you did not spend that time developing your product right now. It's often a better decision to write something that works okay for now, and refactor/rewrite later when you need to scale up.

So, does writing good code matter?
It does, to an extent. Your code has to be good enough to do its job, which is usually to deliver value and to optimize more for the here and now than down the road, scale that might never materialize. If your code does its job well enough and it can be (maybe somewhat painfully) maintained and updated for a couple of years, well, by the time those two years are gone you may well have rewritten it anyway! If you'd spent twice as long at the outset writing your magnus opus, that time would have been wasted.
Your code can't be a dumpster fire. It probably shouldn't be the Mona Lisa, either. Striking a balance is an important facet of engineering for all things, ranging from the quality of your code (good, but not too good!) to how much scale to handle (enough, but don't over-engineer it!) to how much coffee to drink (just kidding, never too much coffee).


Telegraf: dynamically adding custom tags
Srijan Choudhary, all posts — Wed, 14 Oct 2020 03:00:00 GMT
Adding a custom tag to data coming in from an input plugin for telegraf


Car design will improve forever
Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 13 Oct 2020 21:00:00 GMT
But only if it has those two features.


Stop useEffect from running on every render with useCallback
Max Rozen — Tue, 13 Oct 2020 12:48:48 GMT
Did ESLint tell you to add a function to your dependency array, and now you're getting infinite re-renders? Let's learn how useEffect can help.


3D Printing Garden Cage Hinges
IvyMike.dev — Tue, 13 Oct 2020 10:00:00 GMT
My girlfriend's COVID-19 hobby is gardening--broccoli, greens, carrots, radishes, you name it.  The only problem has been the rascally neighborhood squirrels and raccoons, who insist on digging in the raised garden beds, ruining her crops.  A few weeks ago, with a friend's help, we built PVC cages as protection. PVC …


Follow up: Intentional Documentation
blog.harterrt.com — Tue, 13 Oct 2020 04:00:00 GMT
Last week I presented the idea of 
Intentional Documentation
to Mozilla's data science team.
Here's a link to the slides.
The rest of this post is a transcription of what I shared with the team
(give or take).

In Q4, I'm trying to build a set of trainings
to help …


Setting up Raspberry Pi 3 with PiHole
Heitor's log — Tue, 13 Oct 2020 03:00:00 GMT
How to setup PiHole on a Raspberry Pi.


Nokia Composer in 512 bytes
zserge's blog — Tue, 13 Oct 2020 03:00:00 GMT
Those who ever owned an old Nokia phone like 3310 or 3210 might still remember its wonderful ability to compose your own ringtones straight on the phone’s keyboard. By arranging notes and pauses you could end up with a popular tune beeped out of the phone’s speaker, and furthermore, you could share it with your friends! If you missed that era, here’s how it looked like:
Unimpressed? Well, trust me, it was really cool back then especially if you are into music.


Another Amazon problem: its returns do not return
Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 12 Oct 2020 21:00:00 GMT
But it’s not only Amazon.


AmpliBox - a Self Hosted File Storage App with AWS Amplify
swyx's site RSS Feed — Mon, 12 Oct 2020 18:35:04 GMT
A demo using AWS S3 (Simple Storage Service) using Amplify Storage, Svelte, and Tailwind CSS


Hashtable 使用与分析
张可的博客 — Mon, 12 Oct 2020 18:17:27 GMT
Hashtable
Hashtable 实现了 Map 接口，用于存储键值对，禁止 null 元素。

用作键的对象必须保证hashCode与equals的可用性。

Hashtable 是线程安全的，但一般来说，如果不需要考虑线程安全问题可以使用 HashMap 作为替代，如果需要线程安全的高并发哈希表可以使用 ConcurrentHashMap，总的来说就是，一般不需要使用 HashTable。

Hashtable 具有两个性能相关的参数：初始容量(initial capacity)和负载因子(load factor)， 容量是指 HashTable 桶（buckets）的容量，初始容量即 HashTable 在创建时桶的默认大小。在哈希冲突的情况下，每个桶会通过链表存储多条数据。

负载因子默认是 0.75，一般来说该值是最佳值。

当 HashTable 容量超出负载因子时将会进行 rehash 操作，该操作会对所有数据的位置进行重新分配，是耗时操作。

HashTable 的迭代器被设计为 fail-fast 的，如果迭代器创建完成后 Hashtable 的结构被修改，迭代器将会抛出ConcurrentModificationException异常。但 Hashtable 本身的get及put方法并不是 fail-fast 的。

但迭代器也并不能保证准确的 fail-fast，只会尽最大努力的抛出异常。

源码
Hashtable 存储的是键值对，在内部会将键值封装成一个Entry<K,V>对象，我们先看看这个类：
private static class Entry<K,V> implements Map.Entry<K,V> {
    final int hash;
    final K key;
    V value;
    Entry<K,V> next;

    public boolean equals(Object o) {
        if (!(o instanceof Map.Entry))
            return false;
        Map.Entry<?,?> e = (Map.Entry<?,?>)o;

        return (key==null ? e.getKey()==null : key.equals(e.getKey())) &&
                (value==null ? e.getValue()==null : value.equals(e.getValue()));
    }

    public int hashCode() {
        return hash ^ Objects.hashCode(value);
    }
}

Entry 类主要是对键值对的封装，其中hash是键的hashCode，next表示下一个节点，我们上面也说过了，每个位置可能会存储多条数据，而存储方式就是链表。

我们再看看类中定义的几个主要的参数：
private transient Entry<?,?>[] table;
private transient int count;//当前Hashtable的实际大小
private int threshold;//扩容阈值 = capacity * loadFactor
private float loadFactor;//负载因子，默认0.75

负载因子可以用过构造器指定，threshold会在初始化时被设置，每次扩容也会更新。

table数组则是 Hashtable 的底层数据结构，用于存储数据，也就是上面说的桶，而数组的每个元素都是个链表。

put(K,V)
再来通过一个典型的 put 方法看看其内部实现：
public synchronized V put(K key, V value) {
    // Make sure the value is not null
    if (value == null) {
        throw new NullPointerException();
    }

    // Makes sure the key is not already in the hashtable.
    Hashtable.Entry<?,?> tab[] = table;
    int hash = key.hashCode();
    int index = (hash & 0x7FFFFFFF) % tab.length;
    @SuppressWarnings("unchecked")
    Entry<K,V> entry = (Entry<K,V>)tab[index];
    for(; entry != null ; entry = entry.next) {
        if ((entry.hash == hash) && entry.key.equals(key)) {
            V old = entry.value;
            entry.value = value;
            return old;
        }
    }

    addEntry(hash, key, value, index);
    return null;
}

我们上面说过需要保证键的hashCode与equals的可用性，这里就可以看到原因，计算数组位置的依据就是hashCode。

通过键来计算对应数组索引的算法很简单：
int index = (hash & 0x7FFFFFFF) % tab.length;

0x7FFFFFFF是Integer.MAX_VALUE的值，按位与这一步是为了防止索引值超出整型范围，然后根据数组长度求余可以将索引控制在数组范围内。

下面的 for 循环是用来搜索 HashTable 中是否已经包含当前键对象，如果已经包含则直接更新value的值。

否则调用addEntry方法将该键值对添加到 HashTable 中去。

addEntry(int, K, V, int)
private void addEntry(int hash, K key, V value, int index) {
    modCount++;// 尽可能保证 fail-fast

    Hashtable.Entry<?,?> tab[] = table;
    if (count >= threshold) {
        // Rehash the table if the threshold is exceeded
        rehash();

        tab = table;
        hash = key.hashCode();
        index = (hash & 0x7FFFFFFF) % tab.length;
    }

    // Creates the new entry.
    @SuppressWarnings("unchecked")
    Entry<K,V> e = (Entry<K,V>) tab[index];
    tab[index] = new Entry<>(hash, key, value, e);
    count++;
}

首先判断当前数组是否已达到扩容阈值，达到则先进行 rehash 操作，该操作会重新创建一个更大的数组，并重新分配所有的数据。

rehash 操作完成后就可以将键值对信息组装成 Entry 并添加到table中。

rehash()
rehash 操作时耗时操作，主要负责两个任务，一是扩容，二是重新分配数据位置。
protected void rehash() {
    int oldCapacity = table.length;
    Entry<?,?>[] oldMap = table;

    // overflow-conscious code
    int newCapacity = (oldCapacity << 1) + 1;
    if (newCapacity - MAX_ARRAY_SIZE > 0) {
        if (oldCapacity == MAX_ARRAY_SIZE)
            // Keep running with MAX_ARRAY_SIZE buckets
            return;
        newCapacity = MAX_ARRAY_SIZE;
    }
    Entry<?,?>[] newMap = new Entry<?,?>[newCapacity];

    modCount++;
    threshold = (int)Math.min(newCapacity * loadFactor, MAX_ARRAY_SIZE + 1);
    table = newMap;

    for (int i = oldCapacity ; i-- > 0 ;) {
        for (Entry<K,V> old = (Entry<K,V>)oldMap[i]; old != null ; ) {
            Entry<K,V> e = old;
            old = old.next;

            int index = (e.hash & 0x7FFFFFFF) % newCapacity;
            e.next = (Entry<K,V>)newMap[index];
            newMap[index] = e;
        }
    }
}

每次的扩容大小是(oldCapacity << 1) + 1也就是原来长度的两倍 + 1。

如果长度已达到最大值，则不会进行扩容，而且在原来长度的基础上继续添加，此时数组上链表的长度会越来越长，扩容完成后会遍历所有数据重新分配位置。

get(Object)
通过get方法获取对应键的值，没有则返回null。
public synchronized V get(Object key) {
    Entry<?,?> tab[] = table;
    int hash = key.hashCode();
    int index = (hash & 0x7FFFFFFF) % tab.length;
    for (Entry<?,?> e = tab[index] ; e != null ; e = e.next) {
        if ((e.hash == hash) && e.key.equals(key)) {
            return (V)e.value;
        }
    }
    return null;
}

先通过哈希值获取到数组的索引，然后遍历该索引处的链表，直到找到哈希值相同且equals方法返回true的值，然后返回该值，或者返回null。


Fix Steam 'Corrupt Update Files'
Blog by Jade Michael Thornton — Mon, 12 Oct 2020 18:00:00 GMT
How to fix the annoying error when downloading or updating some Steam games, which reports the error 'Corrupt Update Files'


Return on Data: When governments fail to make surveillance useful
Nikhil's blog — Mon, 12 Oct 2020 05:34:02 GMT
There is this common refrain about how companies have data about you and could do bad things. Yet governments often pass surveillance laws that are much worse, and by definition, you can’t “vote with your wallet” for them. So in a world where I’m already giving data to third parties, it is useful to ask what I get in return.
If we are talking about corporations, we get ads! Yes, but we also get:


Notes on Time Management from a Dying Professor
swyx's site RSS Feed — Mon, 12 Oct 2020 03:10:32 GMT
Notes on Randy Pausch's lecture on Time Management


The wonders of a proxy auto-config (PAC) file
Imran's Blog — Mon, 12 Oct 2020 03:00:00 GMT
How I improved my work from home setup using a proxy auto-config (PAC) file.


Phishing: Persuading People through a Voice Synthesized by AI - Part 1
SerHack - Security Researcher — Mon, 12 Oct 2020 03:00:00 GMT
I get between 10 to 20 emails per day and, usually, more than half of them are phishing attempts. Phishing attempts are a type of email scam that typically aims to obtain personal data of the victim, including username and password for email accounts and popular services (Gmail, Hotmail, mail.ru, Twitter, Amazon, etc.), through pages forged ad hoc to simulate login sections. -Continue to read the article on SerHack.me


Untitled • Art
Muffin Man — Mon, 12 Oct 2020 03:00:00 GMT
One plot ended up being smaller because I screwed up cutting it.
Created: October 2020
Size: 15x15cm and 13x13cm
Paper: Fabriano Bristol 250gsm
Pens: Molotow Blackliner, Koi Coloring Brush, Mechanical pencil



Hello World
Q. Marchi: A Blog — Mon, 12 Oct 2020 03:00:00 GMT
Decided to setup a blog to track some of the things that I wanted to work on, and maybe hold myself accountable.
Stack:

Engine: Hugo
Hosting: GitLab (Pages)

It’s actually surprising how easy it is to use the WebIDE inside of Gitlab. Means I don’t have to maintain an actual work environment,
though, I may end up to just keep using Gitpod instead as state is maintained inside…
Who knows what lays ahead…


Why I love teaching online
Jason Eckert's Website and Blog — Mon, 12 Oct 2020 03:00:00 GMT

When the pandemic hit earlier this year, schools around the globe quickly embraced online teaching to continue their curriculum. While online teaching wasn’t entirely new in this space, it certainly wasn’t the norm. This meant that many teachers had to adapt quickly, as these 2 funny videos illustrate:

https://www.youtube.com/watch?v=CCe5PaeAeew
https://www.youtube.com/watch?v=Zy_y9yOrgxk

However, I’ve loved every part of this transition because I absolutely love teaching tech courses online.
Grant you, I’ve had some practice prior to 2020, as well as a general interest in how the Web could be used for education. Back in the early 2000s, I designed and wrote all of the online Linux curriculum for Caldera (including the HTML and Perl scripts that hosted it). It was an early stab at online education that had as many faults as it did benefits, but I learned a great deal during the process. Years later, I got to refine what I learned during my online computer science courses at Nova Southeastern, as well as the Train-the-Trainer courses I taught for CompTIA. By this time, technology had caught up to the needs of education, and online education was quite mature with a set of common best practices and technologies that everyone implemented.


Circular Gradients for the HTML5 Canvas
Posts on Evert Timberg — Mon, 12 Oct 2020 01:19:32 GMT
Circular gradients are commonly used in pie or doughnut charts to change the colour of segment as the angle around the center of the chart changes. In many non web platforms, the ability to generate these gradients is provided as part of the standard APIs. For example, Android provides a SweepGradient. In the browsers, only Linear and Radial gradients are provided. Thus, if we want to a circular gradient, we’ll need to create our own.


Billing pipeline: a critical time sensitive system
Kevin Deldycke — Mon, 12 Oct 2020 01:00:00 GMT

 This is an answer to
 
  Antoine Veuiller
 
 ’s article, an
 
  Introduction to Flaky Tests by Example
 
 , in which he shares his experience on tackling unstable tests.





 And I’m the author of the flaky tests he’s talking about! 😬





 
  Story time!
   ¶
  
 
 
  I started designing and implementing Scaleway’s billing pipeline …


2020–10–12: Backlight changes when swithcing between USB-C power modes
megi's PinePhone Development Log — Mon, 12 Oct 2020 01:00:00 GMT



[Computing] The different kinds of authentication protocols
devever.net/~hl — Sun, 11 Oct 2020 21:45:00 GMT
There are almost as many authentication protocols out there as there are
application protocols, making it a confusing landscape. Probably most confusing
of all is that attention rarely is drawn to the fact that there are many
different kinds of authentication protocol, which seek to fill completely
different roles.


Trying Different Camera Setups
Navan's Archive — Sun, 11 Oct 2020 19:12:00 GMT
Comparison of different cameras setups for using as a webcam and tutorials for the same.


ExoTracker Issues - Abandoning the grid
nyanpasu64's blog — Sun, 11 Oct 2020 05:16:00 GMT
Trackers have decades of design, with interlocking features and design decisions, many based on the assumption that every event is quantized on a grid:

You don't need lines above events, since it's obvious which row the event is in.

In regular trackers, events are treated as taking up height. Most events are triggered when the cursor enters them, but pattern-jump effects are triggered when the cursor exits them.


Empty grid cells have dashes in them, to indicate an empty slot belonging to a subcolumn (note/instrument/volume/effect...) and row. (I refer to channels as columns, for historical reasons and because Renoise uses the same terminology.)

ExoTracker's central feature is that events/notes no longer have to be quantized to a grid, but are stored as rational numbers (fractions) in terms of "beats". Events are mapped onto rows using a zoom level (rows per beat), and changing the zoom level causes notes to fall onto or off gridlines.
(tl;dr: Skip forwards to "Per-digit cursors" where the problems start.)

  
  In ExoTracker, each subcolumn has a different background color. The instrument subcolumn has two digits plus padding. The cursor is located in the instrument subcolumn, and takes up both digits plus padding.

As a result, many tracker conventions must be adapted to work with off-grid notes.

Closely spaced notes/events can cause text to draw over other text. I programmed lower events to erase the text drawn by upper events.
The "events have height" model breaks down when off-grid events exist and you can change the zoom level. So I draw a line above every note, representing the instant in time the event occurs at (which I find easier to reason about than 1-row-tall events).

Initially, for each event, I drew a line across all subcolumns (even empty ones), but that became confusing when combined with closely spaced events erasing text in only occupied subcolumns (not empty ones).
So now only occupied subcolumns have lines.


Pressing Delete 4 times will no longer delete all notes in the next 4 rows, because it won't erase events between rows.

I felt that the alternative, where pressing Delete also deletes notes in between rows, would be more confusing and inflexible for users.


So I made "cursor step" both snap to the grid and off-grid notes... which is actually rather janky and surprising.

The issue is that my code snaps to all events in the channel, not just events with a non-empty value in the cursor X position's subcolumn. Changing this would complicate the code.
To alleviate this, I turned off off-grid event snapping, and added a mode where pressing Delete (or entering a value) steps to the next event, regardless of the cursor's X subcolumn. This is useful for deleting many events, adding or changing instruments, etc.


Trackers usually have gaps between note/event/volume/effect, so I add padding around each subcolumn.
If empty grid cells are drawn with dashes, then on-grid notes will erase the dash. But it's not clear if off-grid or triplet notes should erase the dashes or not. I chose to replace dashes with horizontal gridlines. But they don't indicate subcolumns, so I add colored backgrounds.

One alternative would be to draw dashes before drawing events, and simply erase dashes whenever text overlaps them. This looks fine with on-grid events, but with off-grid events, you can end up with partial dashes which might look a bit ugly.

In retrospect, this was probably a better idea, since colored backgrounds proved to be a disaster.





🔗
Per-digit cursors
Everything was more-or-less working, until I added per-digit cursors and the setup broke down. Suddenly the cursor width became highly inconsistent; the cursor is wider when you place it in the leftmost or rightmost digit of a subcolumn.

  
  The effect subcolumn has three positions, or one character plus two digits. The cursor is wider in the left and right subcolumn, and narrower in the center.

So I can make the cursor narrower... now there's a gap around the cursor. And several people have said the gap looks very weird, so this isn't a good solution.

  
  There is a gap between the left border of the subcolumn's background, and the cursor's left boundary. The same issue occurs with the right of each subcolumn.


  
  There is a gap between the note subcolumn's boundaries and the cursor's boundaries.

This is actually what FamiTracker does too, but it doesn't look ugly because there aren't background stripes.

  
  I screen-recorded all cursor positions of FamiTracker into an animated GIF, and overlayed the frames into a single screenshot. There are gaps to the left and right of each subcolumn (note, instrument, volume, and effect). The composite screenshot is ugly.

So do I make the stripes narrower?

  
  The background color is confined to the width filled by text and selectable by the cursor. There are gray stripes around each subcolumn. It's still ugly.

Get rid of the dividers too?

  
  Now the subcolumn dividers are gone, replaced by gray stripes. It looks weird to have middle gaps wider than side gaps.

So I can make them the same width... but then either the note lines overlap, or they have gaps, or the widths are inconsistent. This will cause problems with mouse handling as well.

  
  The gray lines are rearranged, such that the width between subcolumns equals the width between a subcolumn and a channel divider. The note lines overlap.

🔗
A solution?
At this point, I feel that background stripes to indicate subcolumn boundaries, and cursors which occupy digits within a multi-digit subcolumn, are simply incompatible.
If I want to keep that latter feature, I could remove background stripes and replace them with dashes, which are erased whenever text overlaps them. This looks fine with on-grid events, but with off-grid events, you can end up with partial dashes which might look a bit ugly.


Emacs: create a Swift package/project
xenodium.com @alvaro — Sun, 11 Oct 2020 03:00:00 GMT
Been playing around with Swift Package Manager (SPM). Creating a new Swift package (ie. project) is pretty simple.
To create a library package, we can use the following:
swift package init --type library

Alternatively, to create a command-line utility use:
swift package init --type executable

Turns out, there are a few options: empty, library, executable, system-module, manifest.
With a little elisp, we can write a completing function to quickly generate a Swift package/project without the need to drop to the shell.
Bonus: I won't have to look up SPM options if I ever forget them.

(defun ar/swift-package-init ()
  "Execute `swift package init', with optional name and completing type."
  (interactive)
  (let* ((name (read-string "name (default): "))
         (type (completing-read
                "project type: "
                ;; Splits "--type empty|library|executable|system-module|manifest"
                (split-string
                 (nth 1 (split-string
                         (string-trim
                          (seq-find
                           (lambda (line)
                             (string-match "--type" line))
                           (process-lines "swift" "package" "init" "--help")))
                         "   "))
                 "|")))
         (command (format "swift package init --type %s" type)))
    (unless (string-empty-p name)
      (append command "--name " name))
    (shell-command command))
  (dired default-directory)
  (revert-buffer))



Setting up my blog
Imran's Blog — Sun, 11 Oct 2020 03:00:00 GMT
I wanted to use Docusaurus to setup my blog because I might be using it for work and wanted to test it out.
I am also a fan of the fact that it's a static site builder, which means I can use it with a free static site hosting solution.
I decided to go with GitLab Pages because I am already using GitLab and it lets me use a custom domain.

    #

Setting up the GitLab project
This was pretty straight forward. I followed this guide.
First I created a simple enough html page.
echo 'This is me testing stuff.' > index.html
Followed by a .gitlab-ci.yml.
image: alpine:3.12.0

workflow:
  rules:
    - if: "$CI_COMMIT_BRANCH"

pages:
  script:
    - mkdir public
    - cp index.html public/index.html
  artifacts:
    paths:
      - public
  rules:
    - if: '$CI_COMMIT_BRANCH == "master"'
Then another quick guide for setting up a custom domain + SSL certificate.
And voilà!


    #

Using Docusaurus
Like most modern JavaScript projects, to actually use Docusaurus I needed to first install an initializer that would setup a project for me.
The install docs mentioned using npx to install and use the initializer.
This was my first time hearing about npx and I wasn't too keen on installing yet another JavaScript tool on my system.
yarn add @docusaurus/init@next
node node_modules/@docusaurus/init/bin/index.js init my-blog classic
This seemed to work well enough, the sole issue was the Docusaurus files were now in a dir called my-blog rather than my project root.
First I needed to cleanup the install files then move everything in my-blog a directory up.
rm -rf node_modules package.json yarn.lock
mv my-blog/* .
rm -rf my-blog
yarn start now showed me the default Docusaurus project page. At this point all that I had to do was to follow the Blog-only mode guide and customize to my needs.
One annoying thing is that the doc directory has to exist with a single .md file in it to prevent Docusaurus from failing to start.

    #

Updating .gitlab-ci.yml
Updated my CI configuration to use a node image and to cache node_modules/yarn cache
image: node:14.13.1-alpine3.12

workflow:
  rules:
    - if: "$CI_COMMIT_BRANCH"

cache:
  key: "$CI_PROJECT_ID"
  paths:
    - node_modules/
    - .yarn/

pages:
  script:
    - yarn config set cache-folder .yarn
    - yarn install
    - yarn build --out-dir public
  artifacts:
    paths:
      - public
  rules:
    - if: '$CI_COMMIT_BRANCH == "master"'


World smallest office suite
zserge's blog — Sun, 11 Oct 2020 03:00:00 GMT
We are all familiar with a traditional office suite - a word processor, a spreadsheet, a presentation program, maybe a diagramming or note-taking app. We have seen it all in Microsoft Office and Google Docs. Those are really powerful and large. But what would be the most minimal amount of code required to build an office suite?
platform Obviously, our office suite won’t be a desktop GUI app - those require plenty of code and efforts to build one.


Arrays, vectors and slices in Rust
Sun, 11 Oct 2020 03:00:00 GMT
Introduction
In this post I will introduce you to arrays, vectors and slices in Rust. Programmers coming from C or C++ will already be familiar with arrays and vectors, but because of Rust's focus on safety there are some differences from their unsafe language counterparts. Slices, on the other hand, will entirely be a new, albeit a very useful concept.
Arrays
Arrays are one of the first data types beginner programmers learn. An array is a collection of elements of the same type allocated in a contiguous memory block. For example, if you allocate an array like this:
let array: [i32; 4] = [42, 10, 5, 2];
Then all the i32 integers are allocated next to each other on the stack:

In Rust, an array's size is part of the type. For example, this code will not compile:
//error:expected an array with a fixed size of 4 elements,
//found one with 3 elements
let array: [i32; 4] = [0, 1, 2];
Rust's strictness also prevents problems like array to pointer decay in C/C++:
//C++ code
#include <iostream>

using namespace std;

//Looks can be deceiving: arr is not a pointer
//to an array of 5 integers. It has decayed to
//a pointer to an integer.
void print_array_size(int (*arr)[5]) {
    //prints 8 (the size of a pointer)
    cout << "Array size in print_array_size function: " << sizeof(arr) << endl;
}

int main()
{
    int arr[5] = {1, 2, 3, 4, 5};
    //prints 20 (size of 5 4-byte integers)
    cout << "Array size in main function: " << sizeof(arr) << endl;
    print_array_size(&arr);
    return 0;
}
The print_array_size function prints 8 instead of the expected 20 (5 integers of 4 bytes) because arr has decayed from a pointer to an array of 5 integers to just a pointer to an integer.  Similar code in Rust does the right thing:
use std::mem::size_of_val;

fn print_array_size(arr: [i32; 5]) {
    //prints 20
    println!("Array size in print_array_size function: {}", size_of_val(&arr));
}

fn main() {
    let arr: [i32; 5] = [1, 2, 3, 4, 5];
    //print 20
    println!("Array size in main function: {}", size_of_val(&arr));
    print_array_size(arr);
}
Another difference between an array in C/C++ and Rust is that accessing elements in Rust does bounds checking. For example, in the following C++ code, we try to access 5th element in an array of size 3. This produces undefined behaviour:
#include <iostream>

using namespace std;

int main()
{
    int arr[3] = {1, 2, 3};
    const auto index = 5;
    //arr[index] is undefined behaviour
    cout << "Integer at index " << index << ": " << arr[index] << endl;
    return 0;
}
While similar code in Rust is a panic:
fn main() {
    let arr: [i32; 3] = [1, 2, 3];
    let index = 5;
    //arr[index] panics with the following message:
    //index out of bounds: the len is 3 but the index is 5
    println!("Integer at index {}: {}", index, arr[index]);
}
You might wonder, how is the Rust version better than the C++ version? Well, because the C++ version exhibits undefined behaviour, it gives a no holds barred license to the compiler to do anything in the name of optimizations. In the worst case, this can leak information to an attacker.
The Rust version, in contrast, will always panic. Moreover, because the process terminates due to the panic, a programmer is more likely to notice and fix this bug. In contrast, C++ sweeps the problem under the rug and the process could carry on as if nothing had happened. I will take a Rust panic any day over a C/C++ undefined behaviour.
Vectors
The big limitation of arrays is that they are fixed in size. In contrast, vectors can grow at runtime:
fn main() {
    //There are three elements in the vector initially
    let mut v: Vec<i32> = vec![1, 2, 3];
    //prints 3
    println!("v has {} elements", v.len());
    //but you can add more at runtime
    v.push(4);
    v.push(5);
    //prints 5
    println!("v has {} elements", v.len());
}
How does a vector allow dynamic growth? Internally, a vector keeps all the elements in an array allocated on the heap. When a new element is pushed, the vector checks if there is still some capacity left in the array. If not, the vector allocates a bigger array, copies all the elements to the new array and deallocates the old array. This can be seen in the following code:
fn main() {
    let mut v: Vec<i32> = vec![1, 2, 3, 4];
    //prints 4
    println!("v's capacity is {}", v.capacity());
    println!("Address of v's first element: {:p}", &v[0]);//{:p} prints the address
    v.push(5);
    //prints 8
    println!("v's capacity is {}", v.capacity());
    println!("Address of v's first element: {:p}", &v[0]);
}
Initially the capacity of the v's backing array is 4:

A new element is then pushed onto the vector. This makes the vector copy all the elements to a new backing array of capacity 8:

The program also prints the address of the first element of the array before and after pushing a new element onto the vector. Both these printed addresses will be different from each other. This change in addresses is clear evidence of a new array of size 8 being allocated behind the scenes.

    Note
    Note
    
If you do not see a different address after pushing more elements onto a vector, it might be because the allocator had enough space at the end of the original buffer such that the new and the old buffers have the same starting address. Try pushing more elements and you will see a different address. Read about C library function realloc to understand how this might happen.
    

Slices
Slices act like temporary views into an array or a vector. For example if you have an array:
let arr: [i32; 4] = [10, 20, 30, 40];
You can create a slice containing second and third elements like this:
let s = &arr[1..3];
The [1..3] syntax creates a range from index 1 (inclusive)  to 3 (exclusive). If you omit the first number in the range ([..3]) it defaults to zero and if you omit the last number ([1..]) it defaults to the length of the array. If you print the elements in the [1..3] slice, you get 20 and 30:
//prints 20
println!("First element in slice: {:}", s[0]);
//prints 30
println!("Second element in slice: {:}", s[1]);
But if you try to access an element outside the range of the slice, it will panic:
//panics: index out of bounds
println!("Third element in slice: {:}", s[2]);
But how does the slice know that it has only two elements? That's because a slice is not simply a pointer to the array, it also carries around the number of elements of the slice in an additional length field.

    Note
    Note
    
        A pointer with some additional data besides just the address of the pointed to object is called a fat pointer. Slices are not the only kind of fat pointer in Rust. Trait objects, for example, carry a vtable pointer in addition to the pointer to an object.
    

For example, if you create a slice to a vector:
let v: Vec<i32> = vec![1, 2, 3, 4];
let s = &v[1..3];
Then in addition to a pointer to the second element in v's buffer, s also has an 8 byte length field with value 2:

The presence of the length field can also be seen in the following code in which the size of a slice(&[i32]) is 16 bytes (8 for the buffer pointer and 8 for the length field):
use std::mem::size_of;

fn main() {
    //prints 8
    println!("Size of a reference to an i32: {:}", size_of::<&i32>());
    //print 16
    println!("Size of a slice: {:}", size_of::<&[i32]>());
}
Slices of arrays are similar, but instead of the buffer pointer pointing to a buffer on the heap, it points to the array on the stack.
Since slices borrow from the underlying data structure, all the usual borrowing rules apply. For example, this code is rejected by the compiler:
fn main() {
    let mut v: Vec<i32> = vec![1, 2, 3, 4];
    let s = &v[..];
    v.push(5);
    println!("First element in slice: {:}", s[0]);
}
Why? Because when the slice is created, it points to the first element of the vector's backing buffer and as a new element is pushed onto the vector, it allocates a new buffer and the old buffer is deallocated. This leaves the slice pointing to an invalid memory address, which if accessed would have lead to undefined behaviour. Rust has saved you from disaster again.

    Note
    Note
    
Since slices can be created from both arrays and vectors, they are a very powerful abstraction. Hence for arguments in functions, the default choice should be to accept a slice instead of an array or a vector. In fact many functions like len, is_empty etc. work on slices instead of on vectors or arrays.
    

Conclusion
Arrays and vectors being one of the first few data structures that new programmers learn, it is no surprise that Rust too has a solid support for them. But as we saw, Rust's safety guarantees do not allow programmers to abuse these fundamental data types. Slices are a novel concept in Rust but since they are such a useful abstraction, you will find them used pervasively in any Rust codebase.


my first blog post
ThatXliner's Blog — Sun, 11 Oct 2020 03:00:00 GMT
Hello! this is a test!


The Cycle of Life
A Matter of Perspective — Sun, 11 Oct 2020 03:00:00 GMT
Or, why life gives you lemons just when you think you’ve got it figured out 🍋
Things seem to go round and round don’t they? Have you ever noticed that we seem to be thrown into the midst of these crazy situations (a new job, country, or a deadly pandemic that forces us to stay home and rewatch old Friends episodes for the 17th time)? And just about when we think we’ve got a handle on things, life (like the big bully it is) comes and kicks over our sandcastle.
The good times and the bad: Cycles Baby!
Life has ups and downs, for sure. We all know this intrinsically, but seem to forget it. When things are sunny, we turn a blind eye to the blinking fuel light. When things are crap, we fall into a hopeless pit of self-deprecation and “Nietzsche-esque” pessimism. Are we doing something wrong? Why do things turn sour when they’re going oh-so-right? (Hello, America 👋). Turns out there’s a mechanic to this, like many natural processes.

Introducing The Cycle of Life! (And no, in case you’re wondering, this has nothing to do with the awkward “birds and bee’s” science lesson you got in 10th grade).

Let’s break this down into 5 stages:

Alpha Wave: (AKA: 😎 I got this life thing sorted!)
Beta Condition: (AKA: 😫 Please God, Not again)
Shortcut – Reform option: (AKA: 😅 Whoah! That was close!)
Gamma Trap: (AKA: 🔥 Living Hell 😡)
Delta Surge: (AKA: 😃 Woo! We did it! Wait… what’s this?)

We go around and around and around… That’s life 🤷‍♀️.
1. The Alpha Wave:
😎 I got this life thing sorted!
Ahh, life is bliss, right? The Alpha Wave is when we are “in sync” with what’s going on. In other words, our internal matches our external.
As an individual in Alpha, we’ve “got it together”. We’re winning at life. Go you. This might look like someone who has a good job, in a happy relationship and is at peace with where they are. There’s no gloomy rainclouds or fly’s in the butter here.
For companies and organizations, this is a fantastic time for growth. The service/product/team/vision is matched to the niche, and everything is running like a well-oiled machine.
At the societal/governmental level, this is a wonderful time for all involved. The needs of the citizens are met by opportunities and resources.
Humanity is in a state of equilibrium…
Happily ever after. Right…Guys?

Until it’s not.

Perhaps you can recall a time when, from your paradise, the storm clouds began to gather on the distant horizon.
“Fear not ye happy citizens! We have food and wine plenty!” Our leaders may call.
“Those are problems for when I’m older!” You might claim, unwilling to accept that the peace is yet again about to be disturbed.

Leaving Alpha for Beta:

Our world appears to be stable, but it’s not.

In true Spiral fashion, the solutions for where we are now create more problems. Harder ones. Ain’t life a bitch?

2. The Beta Condition
😫 Please God, Not again

Soon enough, the crack of thunder bellows from above, beckoning change and new types of problems. The Beta Condition is a time of uncertainty, questioning, frustration, and doubt.
“Things just aren’t like they used to be!” Cries the disgruntled hippy, “These Yuppies are knocking down all the trees for shiny condos, and killing our culture! THEY’RE CULTURE KILLERS!”
Perhaps you’ve found yourself thinking, “Something is wrong…but what is it?” For example:

A strained marriage
Organization is beginning to lose its vital talent and productivity
That “Netflix and Chill” friend you have has stopped responding to your texts
People are moving out of the country to look for better work

Oh, what are we to do! More of the same, perhaps? Indeed, this is the mentality many of us adopt when the winds of change begin to howl. After all, it worked so well back in Alpha. Right?
Wrong. We’ll often say things like:

“Let’s just focus more”
“Work harder and smarter”
“We’ve got to purge the negativity”

And so, we put more and more energy into the sinking ship of the old Alpha way of life (system).

For those studying Spiral Dynamics, here is a little excerpt from Spiral Dynamics - Mastering Values - Leadership and Changes:
(If none of this makes any sense, you can skip this ⏩)
Beige:
Habitat can no longer support human bands as food becomes more difficult to find.
Purple:
Offerings and rituals fail to fend off evil spirits as tribal order beings to collapse.
Red:
Have/have-not contract begins to erode as the powerful haves are ‘weighed in the balance and found wanting’.
Blue:
The Truth no longer guarantees order, and the future is masked as doubt. Skepticism and new options appear.
Orange:
The “good life” is frayed and tarnished by consumptiveness—the search for inner peace flourishes.
Green:
Warm, human aspirations wear thin as the realities of complex societal problems, and limited resources are laid bare.
Yellow:
Reliance on individual choices and freedom fails to provide the cooperative action necessary for overall survival.
With the good ol’ days behind us, we have two choices. Sit here and pout (which will undoubtedly lead to bigger issues) or look for new solutions.

3. Shortcut – Reform option
😅 Whoah! That was close!

Have you ever almost stood on a snake before someone yelled and screamed at you to watch out? Having a knowledgable environment can save your life. Literally.
As the Beta Conditions grow louder, and the wind blows stronger, a savvy mentor can mean the difference between losing everything and making safe passage to the promised land. Someone who’s “been and done it all before” (and lived to tell the tale) can point out the tiny trap door that briefly presents itself—a pathway out of the Gamma Trap and into the way of solving life’s new problems.
The name of the game here is “pro-active” rather than “re-active”
Some examples might include:

A concerned family holding an “intervention” for someone with alcohol problems
The CEOs personal coach, who whispers that he needs to look after his staff, lest they find greener pastures
A wisened business owner, offering advice to the young and reckless entrepreneur

The Reform option lies somewhere on the road to the Gamma Trap, as things stop working. Take it, and you may pass unscathed (mostly) to the next cycle (or color). Ignore it… and you’re in for a rough ride.

If you are unsure of where you might be in the cycle, join our Slack community and ask! We’re here to assist with your existentialism.
😫 HAAAALP!


4. Gamma Trap
🔥 Living Hell 😡

Oh, dear. We were warned. But we didn’t listen. And now look where we are. The shortcut is gone, the ship has sunk, the storm is here, and we’re kinda screwed.
The Gamma Trap is a state of anger, hopelessness, and desperate revolution. We become trapped by the barriers we’ve constructed by ignoring the new problems. Often, we enter this stage if we become too attached to a way of life (or color), and refuse to do anything different.
The frustration and confusion of the Beta Conditions give way to anger and hostility. The scary thing is:

Once we’re in The Gamma Trap, there is no turning back to how it used to be, yet no place to go.


According to Don Beck and Edward Cowen, “the pain penetrates into the depth of the soul”. Even scarier is this trap can become an entire lifestyle. This is serious business, folks.
Have you ever caught yourself saying or doing any of the following?

“Damn the torpedoes, full speed ahead!”
Curling up in a fetal position, preparing one’s self for imminent death
Sinking into a righteous crusade (My Truth is the only Truth)
Engaging in some shady shit (blackmail, bribery, lying, etc.)
Arrogantly questioning everyone else’s motives

Jokes aside, many of us (the author is shuddering at their own examples) posses elements of our lives where we’re stuck in this trap. We refuse to acknowledge there’s a problem with US and instead blame the rest of the world.

If you find yourself in this position (and many of us are if we’re honest with ourselves), please consult your local Spiral Wizard, psycho-therapist, shaman, or philosopher.
Did we mention we have a Slack channel?

5. Delta Surge
😃 Woo! We did it! Wait… what’s this?

And we’re out! The barriers of Gamma are broken, and we are free of our problems! Grab the champaign, we are liberated from the –
Hold your horse 🐴, here be dragons.

Breaking free of the barriers in Gamma, is not the same as reaching the next Alpha.

I know it’s tempting. But the yeasty excitement and rapid changes we’re experiencing can quickly disappear. Sometimes, people even return back to Gamma once the party is over!
Slow down, take stock, and tread carefully. The promised land is close, but the old behaviors may still persist for some.
6. The New Alpha
🥰 Life is sweet, once again

At long last, the world returns to a steady state. The storm has passed, the enemy has retreated, our company is in harmony, and we once again feel at peace.
This is the time to reflect, rebuild, and discover our new powers. The problems we faced are long behind us. We are presented with unexplored wonders, abilities, and opportunities that we unlocked with our perspective shift.
Many believe we have reached the pinnacle. They claim the ultimate stage of human development is here to stay. Beware! By now, we know that this is just the eye of the storm.
Enjoy the balance while it lasts. It may be months, years, or decades. But one thing is for certain, life is a cycle. Like that annoying telemarketer that never stops calling, the storm will return. Keep your heart in the present moment, but your eyes to the horizon.
Cultivate the company of those who have walked your path before you. Their stories are worth their weight in gold.
Be honest with yourself about your problems when they present themselves. It will save you pain down the road.
But most of all, remember to enjoy the ride! An open heart, mind, and eyes will allow us to sail smoothly around the storm. Stay vigilant, readers!


These concepts were taken from the wise works of Clare Graves, Don Beck, and Edward Cowen. You can read more about it all here, in their book.


Obsession Log: Tactile Switches
trv.rs — Sun, 11 Oct 2020 00:53:00 GMT

I currently type on MX Kailh Speed Copper and Gchocs from Gheavy Industries.

The feeling of mechanical switches is a deeply personal, subjective choice. For many, it's a fun hobby pursuit of some kind of typing experience, whether that's a sound, a tactile feel when the key hits… but if you are unwilling to spend lots of money outfitting boards¹ it can be a bewildering and frustrating endeavor.
Because everyone assumes everyone wants the same thing, they forget to disclose what qualities they are looking for.² I make my switch decisions on a couple, unscientific assumptions:

I am optimizing for speed and reducing RSI.
I believe that the minimum amount of finger travel means optimum speed and minimizes impact.
Having some kind of "signal" at the switch actuation point helps me minimize finger travel.

I originally tried to avoid "bottoming out"³ because Tom told me to. Later, I was trying to type more quietly on my already loud MX Blue 60% board. I discovered that if I could avoid bottoming out, 70% of the noise was diminished! By the time I had taught myself to type this way, some people couldn't believe I was typing on a "noisy" mechanical keyboard. I also noticed that my typing had become faster… although this could have been the result of deliberate practice.
When I first became interested in mechanical keyboards, it was pretty easy to find people who were interested in the same things: it was simply people who bought tactile switches. This has shifted in the last few years. Many of the popular tactile switches out there at the moment have a much stronger tactile moment than the venerable MX Brown and MX Clear. This stronger tactile moment is mostly for "feel," rather than providing a signal to "stop pressing" to the typist. This strong tactile moment requires a lot of force to overcome… therefore the finger is almost certainly going to travel the full length of the switch travel and "bottom out," producing a sound that others don't like. It is getting hard to find a switch that doesn't prioritize this ridiculously strong tactile moment.⁴
I know that there are non-MX switches that seem to appeal to people with the same sort of typing goals... there are Matias, ALPS, and Topre switches come to mind. I never wound up venturing down that path, so I can't speak to those. I have typed on a Topre board before, and I did like it, although at the time I didn't like it as much as the MX Clears that I had.
Switches
Owned and Liked:
MX Blue
The classic. I really should order some more to occasionally throw on a hotswap board. The audible sound is lovely (and only happens on the travel down, unlike the clickbar options out there!) and provides a tactile moment that is sufficient to teach your brain to move to the next switch.
If you don't have a good typing style and a loud case, it can get your neighbors upset at you though.
MX Clear
The best non-blue option of the Cherry switches, IMHO. Heavier, but with a decided tactile moment that is strong enough to teach your fingers, but not too strong so that you burst through it and fall all the way to the bottom.
Zealio 67g v1
When I built my first custom, I was just going to put MX Clears on it, but I decided to try the fancy switches Reddit raved over.
Oh man.
These are still really nice switches. They are essentially MX Clear with a lighter spring and a cleaned up stem, apparently a mass-produced version of a frankenspring known as the "ergo clear."
Sadly, they don't make these anymore. The current version of Zealios don't have the same feeling.
Originative MOD-M Tactile
The original version of this article detailed me buying around 200 switches of different varieties in order to try and find a worthy heir to my beloved Zealios V1. (Some of the losers in that race are below...)
The Originative MOD-M tactile is a pretty close substitute to the Zealios V1... and comes with a similar expensive price tag. They are both designed to be luxury "ergo clear" style switches. If you really want to try Zealios V1, this is the closest that I've been able to find on the market at the moment, although the "feel" of the switch is rougher.
Given that they are so expensive, I can't say that I recommend them at the moment.
20g Gchocs from Gheavy Industries
My current favorite choc switch is linear. With a crazy low spring weight, you can't even really rest your hands on the keys for fear of typing. That being said, it's a very unique typing experience, and feels very fast and very easy. Being as the choc travel distance is so low, I tend to treat the light slap of the key at the bottom as the actuation point, and it feels relatively similar to a good MX tactile. It's weird, but it's good.
Kailh Speed Copper
My current favorite MX Switch. Very high actuation point. Not terribly expensive. I owe this one to Arley McBain, he was right.
The Speed Copper has an extremely high actuation point, very early in travel. The spring experience is relatively light up until a definite tactile moment, then it gets quite heavy afterwards. This "cushion" really helps with not bottoming out, and doesn't feel bad at all.
I've been typing on them 100% for the two weeks since I got them, comparing them back and forth with the Zealio V1 I have left and the Originative MOD-Ms I had saved to put on the Atreus. The MOD-Ms are going back in the bag to be sold. 🤷‍♂️
Owned and Abandoned:
MX Brown
The tactile moment is just not pronounced enough for me to really tell that it's there.
Zilent 67g v2
Zeal knows how to make pleasant switches. They are freaky silent, smooth and nice... I just don't like the spring/tactile experience. It starts out heavy, hits the tactile moment, then gets too light so that you crash right through to the bottom. In addition, the bump is very sharp feeling. I liked the "roundness" of the MX Clear bump, it taught your finger that it was approaching actuation, not slapped it as you went past.
NovelKeys Kailh Burnt Orange Chocs
These switches are technically tactile, but honestly the total switch travel of the kailh low-profile "choc" switch is so short that you can't feel it unless you are trying really hard.
The switch's spring is strong enough for my fingers to learn how to type on it without bottoming out, but honestly it's a bit strong for the tiny keyboards on which I tend to put chocs.
Aliaz
These feel just like silenced lubed MX Browns, which is basically what they are. I couldn't feel the tactile moment.
Hako Clear
The bump is basically at the top, and is more of a resistance than an actual bump at the actuation point. I did't like these at all. They are heavy, stiff, and don't really emulate clears or zeals. They almost feel like really really heavy linears. I will say that the force curve does seem to lean towards keeping a person from bottoming out, and I think that with some practice I could keep from bottoming out entirely.
Results
I'm pretty happy with the combination of switches that I have at the moment. I am floating somewhere between 105-115 wpm on quotes on monkey-type.com at the moment.

As with all endeavors... buying things won't make you faster or better, you can achieve much higher speeds than that with an old dell membrane keyboard. Deliberate practice > buying tools... but if you have to buy tools, I'll try to help. :)



Especially before the rise of readily available hot swap PCBs! ↩


I personally don't care so much about spring noise or stem feel… I care about how accurately and easily I can type. ↩


bottoming out is the finger pressing to the bottom of the switch travel. ↩


If you like this feel… more power to you. It's deeply personal, and you live in a golden age of meme switches. I just am looking for something different. ↩





Which word is missing in "The Internet of Plastic Things"?
Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 10 Oct 2020 21:00:00 GMT
The same word that is missing in the Internet of NON-plastic things, of course.


Choice Architecture
swyx's site RSS Feed — Sat, 10 Oct 2020 19:29:19 GMT
Reflection on how both gyms and startups do choice architecture and how we can be more conscious of this


The Mystery Misfire
Shield Digital Design — Sat, 10 Oct 2020 19:00:00 GMT

The dreaded 
    P0171


 code… If you've experienced it, just the mention
likely gives you nightmares. If not - here's the rundown. 
    P0171
 is the
automotive 
    On-Board Diagnostics
 (
    OBD
) code¹ for a 
    lean condition

on the engine. If the engine is running lean, it simply means that the

    air to fuel ratio
 is not optimized, and there's either too much air or
not enough fuel. Before I get too far ahead here, let's set the scene.


The Best Of - 2020 Edition
smcleod.net — Sat, 10 Oct 2020 19:00:00 GMT
Near the end of every year I note down a summary of the best apps, hardware & podcasts I’ve enjoyed throughout the year (and often for some time before).
This post has been superseded. You can find the latest version of this post here.
Software and Services

Plex
Things
Pixelmator Pro
Reeder
NextMeeting
Bumpr
Fastmail
1Password
Calibre
Backblaze
iTerm
Gitlab
PFsense
Stay
Onyx
Evernote
Handbrake
Amphetamine
Firefox

Software - Mobile

Plex
Things
Reeder
Overcast
Evernote
MiniHack
Alien Blue
Singal
Prompt

Browser Addons

uBlock Origin
Dencentraleyes

Hardware

LG C9 OLED TV
Das Ultimate 4
Logitech MX Master 3
16’’ Macbook Pro
iPhone 12 Pro
AppleTV 4K
Audioquest DragonFly Red USB DAC
Ultrasone Signature Pro Headphones
Kindle Paperwhite
All Day Socks
Bellroy Wallets

Podcasts

The Skeptics Guide To The Universion (SGU)
You Are Not So Smart
Risky.Biz
RHLSTP
Louis Theroux - Grounded
Hiphop Saved My Life
Geological
99% Invisible
BBC Global News

YouTube Channels

Techmoan
Jim Jefferies - Well I Don’t Know About That!
Salvatore Ganacci
Warped Perception
Smarter Every Day
DTM
First We Feast (HotOnes)
NPR Tiny Desk Concerts
Logic

The Absolute Worst Software I’ve Experienced

💩 Microsoft Office365 💩
💩 Microsoft Teams 💩
💩 Atlassian Jira 💩
💩 Atlassian Bamboo 💩
💩 Atlassian Confluence 💩
💩 McAffee Endpoint Security 💩
💩 Facebook 💩



Digital Spring Cleaning
utf9k — Sat, 10 Oct 2020 14:00:00 GMT
As I work through deleting a bunch of accounts, I note down the various issues that make it harder than it should be


Rethinking Education – 3 talks to watch
Yasha Solutions — Sat, 10 Oct 2020 03:00:00 GMT
If you are not familiar with these 3 speakers and their thesis about education, time to catch up!
Entertaining and well-articulated, they will help you understand where the current education system migh be falling short.
Ken Robinson is more focused on the what, Seth on the why and Sugata Mitra is looking for an alternative approach (hint: it involves the Internet).
Ken Robinson: School kills creativity I am not sure how much Ken Robinson still need an introduction by now, but his was a reknown thinker on creativity and innovation.


How to Build a Plugin for Docusaurus v2
Adriaan Knapen — Sat, 10 Oct 2020 03:00:00 GMT
Cool stuff: This blogpost got featured on the official Docusaurus blog! 🎉
Now, go read on about creating your own Docusaurus plugin 🚀.
 At ClaimR we have been using Docusaurus v2 for our developer and documentation website for a while now. After a couple of months of using it we can confidently say that we’re very happy with our choice.
Docusaurus comes batteries-included, so you immediately get a lot of value, while it is flexible enough to tailor it to your needs if there’s something you like differently.


Installing Ubuntu Linux on the Lenovo ThinkPad E14 Gen 2 (AMD)
Rich Dougherty — Sat, 10 Oct 2020 02:00:00 GMT
A cheap, decent Linux laptop


Focus Budget: Updated Implementation
trv.rs — Sat, 10 Oct 2020 00:54:00 GMT
Some time ago I wrote a blog post about the rough pattern of my weekly planning of a "focus budget." The objective is to estimate the time I have available for important tasks.
Thanks to a Jacob Collins pointing out a cool feature on Things I've started using a new trick and written a new headspace that has dramatically sped up my weekly planning and daily scheduling.
Understanding this new system requires you to know three facts:

If you copy or drag a to-do in Things you can get the plaintext version of that to-do or project.
If you drag any body of text into Fantastical, it creates an event using the current default calendar.
You can set the default calendar for new events in each Calendar Set in Fantastical.

So… armed with these facts, I've written a simple layout "space" using my Headspace module.
You can read the whole thing here, but here's the gist of it.

Open Fantastical and Things, and position them so that I can see the whole week in Fantastical and a list in Things. If it's Sunday, I'm doing my weekly planning, so I show the Upcoming list. Otherwise, I show Today. Both are filtered to my work to-dos.
Switch the Fantastical calendar set to "Focus Planning," a calendar set where the default calendar is a private calendar I use to plan my week.
Once all is in place, I drag todos from my Upcoming view in Things onto my calendar, creating little time blocks! I adjust their length based on how much time I've estimated… and this is a really good way to see if I've even got enough time to do all that I've planned.
When I move away from the space, the tear down function will put Fantastical and Things back to their normal states. It's great!

I really love this pattern. Thanks to the tags on my to-dos, I can see if I have enough time to execute on my daily plan.I have to admit that the little bit o' automation that chooses the right Things list based on the time makes me really happy.
-- setup the space for Headspace
table.insert(config.spaces, &lbrace
-- […]
&rbrace)

config.funcs.focus_budget = &lbrace
  setup = function()

    if os.date("%a") == "Sun" or os.date("%a") == "Sat" then
      hs.urlevent.openURL("things:///show?id=upcoming&filter=%40Proctoru%2CEstimates")
    else
      hs.urlevent.openURL("things:///show?id=today&filter=%40Proctoru%2CEstimates")
    end

    hs.application.launchOrFocusByBundleID('com.culturedcode.ThingsMac')
    hs.application.launchOrFocusByBundleID('com.flexibits.fantastical2.mac')

    local things = hs.application.find('com.culturedcode.ThingsMac')
    local fantastical = hs.application.find('com.flexibits.fantastical2.mac')

    local upcoming = things:focusedWindow()
    upcoming:application():selectMenuItem("Hide Sidebar")

    local cal = fantastical:focusedWindow()
    cal:application():selectMenuItem("Hide Sidebar")
    cal:application():selectMenuItem("Focus Budget")
    cal:application():selectMenuItem("By Week")

    hs.layout.apply(
      &lbrace
        &lbrace"Fantastical", nil, hs.screen.primaryScreen(), hs.layout.left70, 0, 0&rbrace,
        &lbrace"Things", nil, hs.screen.primaryScreen(), hs.layout.right30, 0, 0&rbrace
      &rbrace
    )
  end,
  teardown = function()
    local fantastical = hs.application.find('com.flexibits.fantastical2.mac')
    fantastical:selectMenuItem("Show Sidebar")
    fantastical:selectMenuItem("Daily Focus")
  end
&rbrace



Custom markdown blocks with marked
Qubyte Codes — Sat, 10 Oct 2020 00:45:00 GMT
I use marked to do the markdown rendering for this blog. A recent
feature makes it possible to create custom block types with a little hacking. In
this post I show you how!
I'll be using mathematics (TeX) blocks for my example. The marked setup
code looks like this:
marked.use({
  walkTokens(token) {
    const { type, raw } = token;

    // Modify paragraph blocks beginning and ending with $$.
    if (type === 'paragraph' && raw.startsWith('$$\n') && raw.endsWith('\n$$')) {
      token.type = 'code';
      token.lang = 'mathematics';
      token.text = token.raw.slice(3, -3); // Remove the $$ boundaries.
      token.tokens.length = 0; // Remove child tokens.
    }
  },
  renderer: {
    code(code, language) {
      // Use custom mathematics renderer.
      if (language === 'mathematics') {
        return renderMathematics(code);
      }

      // Use default code renderer.
      return false;
    }
  }
});
I'm passing two things to marked to configure it here. The first is a
token walker function, which is the recent feature which makes
this all possible. It is called for each token, traversing the children of a
token before it progresses to its siblings (so it's sort of depth-first).
The idea is for blocks of text with a $$ above and below them to be handled
as mathematics. To a person this looks like a fenced code block with two dollar
symbols in the place of the three backticks. For example:
Some example text with some mathematics to render below:

$$
a^2 = b^2 + c^2
$$

Some example text below.
It's a common extension to place LaTeX code inside $$ delimited blocks.
Even if you're not familiar, the dollar symbols above and below are a little
like a fenced code block. To marked the block looks like a paragraph. This
means that the token walker will receive some paragraph tokens which need to be
modified.
To know the difference, paragraph tokens are checked by the token walker to see
if they begin with $$\n and end with \n$$. When they do, the block is
modified to look like a code block with a special 'markdown' language. Child
tokens are removed because the content shouldn't be treated as markdown, and the
text property is set by snipping the leading and trailing dollar symbols and
newlines off.
The second part of this trick is in the renderer option. The renderer for code
blocks is modified with special handling for the 'mathematics' language. The
code parameter received by it is the text we set on the token, so it's ready to
be rendered to mathematics. The rendering itself is beyond the scope of this
post, but I use MathJax. When code is of any other language the
custom code renderer returns false to instruct marked to use the default
code rendering behaviour.


Crowdsourced fact checking? Hmm...
Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 09 Oct 2020 21:00:00 GMT
Why not look at pandemics, instead?


On the current state of privacy
Apurva Shukla's RSS Feed — Fri, 09 Oct 2020 08:30:23 GMT
Where are we? It’s 2020. The year so far has been nothing short of a global collapse of order, yet frighteningly the realities we’re facing…


Adding a search tags interface
Home on Pinecoder blog — Fri, 09 Oct 2020 03:00:00 GMT
Introduction In my last attempt to build this app, search organically grew in prominence as I added new features. Originally the app was just date ordered markdown documents, and search was free text with a few added features. In this version, I wanted search to be more central and flexible. Rather than an adjunct on the selected journal, I thought it could be the main driver of what was on screen.


2020-10-09/01
ho.dges.online — Fri, 09 Oct 2020 03:00:00 GMT



2020-10-09/02
ho.dges.online — Fri, 09 Oct 2020 03:00:00 GMT



What makerspaces REALLY (should) make
Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 08 Oct 2020 21:00:00 GMT
Spoiler: it’s not objects.


Email Shenanigans
infrequent oscillations — Thu, 08 Oct 2020 18:33:06 GMT
In which we proxy emails.


Rest in Peace Jimothy
smcleod.net — Thu, 08 Oct 2020 18:20:00 GMT
You magnificent girl, you brought me 14 years of love and happiness
Photos








































A Whisker Away
🦉 olshansky 🦁 — Thu, 08 Oct 2020 16:03:08 GMT
Olshansky's review of A Whisker Away


Upload: Season 1
🦉 olshansky 🦁 — Thu, 08 Oct 2020 16:00:07 GMT
Olshansky's review of Upload: Season 1


Quick Tip: Don't Sweat the Duplicates
trv.rs — Thu, 08 Oct 2020 12:09:00 GMT
Quick tip prompted by Rosemary Orchard in this week's episode of Nested Folders Podcast:

It's ok to capture something more than once.

When I first started using a to-do list I had a bad habit: at the moment of capturing a to-do I wasted time trying to remember whether I had already recorded, usually ending up wandering endlessly through my database to make sure I haven't entered this before.
This friction kept me from focus and from capturing to-dos. I was afraid of "cluttering my system" or reducing the purity of my system. (I'm a data nerd, ok? Consistency is important!)
Now, I capture everything I might want to do, without worrying whether it exists yet. I trust my daily review to place the to-do in the right location, and I trust my weekly review to delete duplicates.
This simple mental shift allows me to capture quickly and effortlessly, returning my precious attention to the task at hand.


How does K-means clustering work?
Luke Salamone's Blog — Thu, 08 Oct 2020 03:39:22 GMT
K-means clustering (not to be confused with K-nearest neighbors) is an unsupervised learning algorithm used for grouping similar points together into clusters.






Algorithm
The basic K-means algorithm is fairly simple and has two steps, repeated until convergence:

assign points to cluster corresponding to closest centroid
update centroid locations to the mean of all points assigned to the associated cluster

The algorithm converges when the centroids stop moving, i.e. no points can be switched between clusters to a closer centroid.


Lang. Identification in Code Mixed Text - Lit. Review 1
Playground — Thu, 08 Oct 2020 03:00:00 GMT
I am reviewing the literature available on language identification for multilingual documents, focusing on Indic languages. I’ll try to cover it in chronological order, but there might be a few misses here and there. After a decent coverage of the research, I expect to have enough understanding to discuss the challenges present in this task in a code-switched setting and its importance.


Contact Tracing remains irrelevant
Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 07 Oct 2020 21:00:00 GMT
Never start from software…


Service as a Service
swyx's site RSS Feed — Wed, 07 Oct 2020 20:16:35 GMT
Before you start offering Software as a Service, perhaps just offer a Service.


The Crisis of the Postmaterial Condition
mbutler — Wed, 07 Oct 2020 15:56:00 GMT
Originally appeared in Infinite Regress In 1983, at the tender age of 8, I had my first real crisis as an artist.  It involved a new-media project I was working on for Mrs. Widenbacher’s class that called for a particular interactive element that I knew would really drive home what I was trying to say.  […]


Surrogation
blog.harterrt.com — Wed, 07 Oct 2020 10:00:00 GMT
A year or so ago, I read
this article
about how Wells Fargo ended up in such a mess.
If you don't remember,
Wells Fargo was opening accounts in their clients' name without their consent
and ended up paying a few hundred million dollars in fines.
Long story short, a …


What is in the box? Ask Airpods..
Perspective by Ankit Agarwal — Wed, 07 Oct 2020 09:54:21 GMT
what makes the new trend of Bluetooth earbuds/pods/earphones click..


General thoughts on product, startups and technology
Perspective by Ankit Agarwal — Tue, 06 Oct 2020 21:07:35 GMT
Welcome to Perspective by Ankit Agarwal by me, Ankit.


Castes will be castes. So...
Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 06 Oct 2020 21:00:00 GMT
Applies to ALL castes, of course.


Enterprise Software Monetization is Fat-Tailed
Who is Nnamdi? — Tue, 06 Oct 2020 18:59:16 GMT
In enterprise software, averages are meaningless. Instead, focus on the tails.


Follow Up
swyx's site RSS Feed — Tue, 06 Oct 2020 13:29:38 GMT
Following up and following through is a well known formula for success. Yet people don't do it. Why?


How the React Hooks ESLint plugin saved me hours debugging useEffect
Max Rozen — Tue, 06 Oct 2020 12:48:48 GMT
useEffect is great for fetching and rendering data, but are you using it correctly? There's an eslint plugin to check!


Don't check emptyness before loops
Blog on blog.gnoack.org — Tue, 06 Oct 2020 08:12:04 GMT
If your plan is to loop over a collection: You shouldn’t check for it being non-empty first.
Don’t do this:
if (!stuff.empty()) {
  for (Item item : stuff) {
    // ...
  }
}

Do this instead:
for (Item item : stuff) {
  // ...
}

Benefits:

Shorter
Less nesting
Same performance

I used to do this too at some point in the past, because I thought
this would have performance benefits, but in reality, the performance
benefit is hardly measurable and generally falls into the category
“premature optimization”.
Even if there is a little extra code happening between the if and
the for, the performance impact needs to be significant for this to
make a difference in most cases, and it’s probably not worth the
additional code convolution.


Mystery Knowledge and Useful Tools
Nikhil's blog — Tue, 06 Oct 2020 03:22:11 GMT
Hillel Wayne has a great newsletter, and one recent post had this observation:
The abstract concept here is knowledge or skills that
You are unlikely to discover on your own, neither through practice and reflection nor by observing others apply it. Once somebody tells you about it, you can easily learn and apply it. Once you can use it, it immediately gives you significant benefits, possibly to the point of raising your expertise level.


Dare to be Good Enough—You'll be Happier for it
Thomas Countz — Tue, 06 Oct 2020 03:00:00 GMT
Originally published on 8th Light’s blog

As I sit down to write, I noticed a tug at the back of my neck—a tension forming as I begin editing my words obsessively. It’s a habit of writing characterized by my attempts at achieving the best possible outcome. “I know that I can write well,” I tell myself, “I’ve just got to strike the right tempo, tone, voice…”

I often have the same sensation when I’m writing code. The giants of our industry, whose shoulders I stand upon, tell me all sorts of things that should be true about my code—the way it reads, the way it’s coupled to other parts of a system, and the amount of complexity it’s allowed to have—it’s all qualified and quantified.

This tension, of balancing all of the small discrete decisions that crop up when implementing a software feature, causes me to try to painstakingly eke out the “correct” way to do things. It’s as if when faced with a choice, I scan through an endless tree of outcomes trying to game out what my best option would be. “Do I extract a shared setup for these tests, or do I duplicate it for each assertion?” “Under what circumstance should I choose the builder pattern over a factory method?” “Should this method be called is_ready or is_prepared?” The list goes on and on.

If this sounds familiar to you, you’re not alone. People who tackle decision-making in this way are called maximizers.


  In order to determine their optimal decision outcome, maximizers feel compelled to examine each and every alternative available, which is often infeasible in reality due to the limitations in human cognition (Roets, et al., 2012).


In theory, maximizers make the best choices, but research shows that, in practice, maximizers aren’t ever sure that the choices they make are the best choices, and it’s this “not knowing” that leads to analysis paralysis, and ultimately, regret.

Satisficers, on the other hand, are happy with whatever “good enough” choice they make:


  Several options may fall within a satisficer‘s threshold for acceptability, providing greater flexibility and latitude in achieving a desired decision outcome. As soon as [they encounter] a good enough option, the satisficer can easily ignore the addition of new choices to the decision domain (Peng, 2013).


Satisficers are pragmatists at their core; the solution that works is the solution they commit to.

This way of decision-making is embedded in the fabric of agile methodologies, which gives us permission to make small incremental changes over time, rather than grasping for the perfect solution from the start. From this, we know that iterative design can help us build tighter feedback loops, deliver value to stakeholders sooner, and ultimately keep our development reactive and responsive to the needs of our customers.

Our projects’ bottom line isn’t the only beneficiary to thinking like a satisficer. Even though the research is still out over whether maximizers or satisficers achieve objectively better decision-making (Peng, 2013), what is clear is that satisficers are happier. In fact, even if a better option occurs to them later, satisficers are less likely to experience regret (Schwartz et al., 2002).

If you find yourself like me, trying to game out the best future outcomes based on limited knowledge and with bounded cognition, remember the ways of the satisficers. Make a good enough decision today, and remain flexible and forgiving in order to make another good enough decision tomorrow. Of course, we all have metaphorical hills on which we may choose to die, but choose your metaphorical battles wisely and dare to be good enough.

References
Roets, A., Schwartz, B., & Guan, Y. (2012). The tyranny of choice: a cross-cultural investigation of satisficing effects on well-being. Judgment and Decision Making.

Peng, Starry. “Maximizing and Satisficing in Decision-Making Dyads.” ScholarlyCommons, 2013, https://repository.upenn.edu/cgi/viewcontent.cgi?referer=&httpsredir=1&article=1101&context=wharton_research_scholars

Schwartz, B., Ward, A., Monterosso, J.,Lyubomirsky, S., White, K., & Lehman, D. R. (2002). Maximizing versus satisficing: happiness is a matter of choice. Journal of personality and social psychology.

Agile Alliance. “What is Incremental Development?”, 2019, https://www.agilealliance.org/glossary/incremental-development


Don't Ignore Your Functions
Daniel Little Dev — Mon, 05 Oct 2020 23:00:00 GMT
Ignoring return values can often be dangerous in subtle ways but you may be so used to doing it that you don't even notice anymore. It's…


Building hsdis for OpenJDK 15
Gunnar Morling — Mon, 05 Oct 2020 21:30:00 GMT

Lately I’ve been fascinated by the possibility to analyse the assembly code emitted by the Java JIT (just-in-time) compiler.
So far I had only looked only into Java class files using javap;
diving into the world of assembly code feels a bit like Alice must have felt when falling down the rabbit whole into wonderland.



An unexpected analysis of our digital economy
Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 05 Oct 2020 21:00:00 GMT
Sometimes, truth is in the strangest places (and times).


Relying on the internet as a business
utf9k — Mon, 05 Oct 2020 14:00:00 GMT
It's wild that you can couple a physical business to digital technology to the extent that one can't operate without the other


iTerm2 Paste Buffers on MacOS/OSX - how to increase size
Greg Herlein — Mon, 05 Oct 2020 11:00:01 GMT
Quick reminder: how to increase the copy-paste buffers size in iTerm2.


Testing Pelican
IvyMike.dev — Mon, 05 Oct 2020 10:00:00 GMT
I'm going to test this puppy out


An Image is Worth 16x16 Words: Transformers for Image Recognition (Paper Explained)
Who is Nnamdi? — Mon, 05 Oct 2020 02:17:13 GMT
Large Transformer trained on large datasets outperform CNN-based architectures and achieve state of the art results on image recognition tasks


2020–10–05: Reboot to FEL, skipping SD card boot, external display fixes
megi's PinePhone Development Log — Mon, 05 Oct 2020 01:00:00 GMT



Corruption Is Attractive!
Venam's Blog — Patrick Louis (Lebanon) — Mon, 05 Oct 2020 00:00:00 GMT
We live in a world that is gradually and incessantly attracted by over-rationality and order. In this article we'll burst the enchanted bubble and embrace corruption and chaos — We're going to discuss the topic of image glitch art.


Sonos is spying on me… (and you)
Gingerlime — Sun, 04 Oct 2020 17:19:40 GMT
I recently decided to get a wireless speaker for our Kitchen. Sonos seems like an obvious choice these days. The sound quality and aesthetics were very appealing. So I ordered a Sonos One SL speaker. In terms of sound quality and looks, I was very pleased. I’m not an audiophile but the sound quality seemed…
The post Sonos is spying on me… (and you) appeared first on Gingerlime.


Enola Holmes
🦉 olshansky 🦁 — Sun, 04 Oct 2020 16:48:18 GMT
Olshansky's review of Enola Holmes


Parasite
🦉 olshansky 🦁 — Sun, 04 Oct 2020 16:46:36 GMT
Olshansky's review of Parasite


Hello internet
blog — Sun, 04 Oct 2020 15:07:48 GMT



Why Tailwind CSS
swyx's site RSS Feed — Sun, 04 Oct 2020 05:48:22 GMT
Why I changed my mind on Tailwind CSS, and why I now consider it the Goldilocks Styling Solution


Improved Ctrl-p/Ctrl-n macOS movement
xenodium.com @alvaro — Sun, 04 Oct 2020 03:00:00 GMT
macOS supports many Emacs bindings (out of the box). You can, for example, press C-p and C-n to move the cursor up and down (whether editing text in Emacs or any other macOS app). Jacob Rus's Customizing the Cocoa Text System offers a more in-depth picture and also shows how to customize global macOS keybindings (via DefaultKeyBinding.dict).
In addition to moving Emacs point (cursor) up/down using C-p/C-n, I've internalized the same bindings to select an option from a list. Good Emacs examples of these are company mode and ivy.
Vertical cursor movement using Emacs bindings works well in most macOS apps, including forms and text boxes in web pages. However, selecting from a completion list doesn't quite work as expected. Although the binding is technically handled, it moves the cursor within the text widget, ignoring the suggested choices.

Atif Afzal's Use emacs key bindings everywhere has a solution for the ignored case. He uses Karabiner Elements to remap c-p and c-n to arrow-up and arrow-down.
It's been roughly a week since I started using the Karabiner remapping, and I've yet to find a case where a web page (or any other macOS app) did not respond to c-p and c-n to move selection from a list.

My ~/.config/karabiner/karabiner.json configuration is as follows:
{
    "global": {
        "check_for_updates_on_startup": true,
        "show_in_menu_bar": true,
        "show_profile_name_in_menu_bar": false
    },
    "profiles": [
        {
            "complex_modifications": {
                "parameters": {
                    "basic.simultaneous_threshold_milliseconds": 50,
                    "basic.to_delayed_action_delay_milliseconds": 500,
                    "basic.to_if_alone_timeout_milliseconds": 1000,
                    "basic.to_if_held_down_threshold_milliseconds": 500,
                    "mouse_motion_to_scroll.speed": 100
                },
                "rules": [
                    {
                        "description": "Ctrl+p/Ctrl+n to arrow up/down",
                        "manipulators": [
                            {
                                "from": {
                                    "key_code": "p",
                                    "modifiers": {
                                        "mandatory": [
                                            "control"
                                        ]
                                    }
                                },
                                "to": [
                                    {
                                        "key_code": "up_arrow"
                                    }
                                ],
                                "conditions": [
                                    {
                                        "type": "frontmost_application_unless",
                                        "bundle_identifiers": [
                                            "^org\\.gnu\\.Emacs"
                                        ]
                                    }
                                ],
                                "type": "basic"
                            },
                            {
                                "from": {
                                    "key_code": "n",
                                    "modifiers": {
                                        "mandatory": [
                                            "control"
                                        ]
                                    }
                                },
                                "to": [
                                    {
                                        "key_code": "down_arrow"
                                    }
                                ],
                                "conditions": [
                                    {
                                        "type": "frontmost_application_unless",
                                        "bundle_identifiers": [
                                            "^org\\.gnu\\.Emacs"
                                        ]
                                    }
                                ],
                                "type": "basic"
                            }
                        ]
                    }
                ]
            },
            "devices": [],
            "fn_function_keys": [],
            "name": "Default profile",
            "parameters": {
                "delay_milliseconds_before_open_device": 1000
            },
            "selected": true,
            "simple_modifications": [],
            "virtual_hid_keyboard": {
                "country_code": 0,
                "mouse_key_xy_scale": 100
            }
        }
    ]
}

Bonus (C-g to exit)
Pressing Esc often dismisses or cancels macOS windows, menus, etc. This is also the case for web pages. As an Emacs user, I'm pretty used to pressing C-g to cancel, quit, or exit things. With that in mind, mapping C-g to Esc is surprisingly useful outside of Emacs. Here's the relevant Karabiner C-g binding for that:
{
    "description": "Ctrl+G to Escape",
    "manipulators": [
        {
            "description": "emacs like escape",
            "from": {
                "key_code": "g",
                "modifiers": {
                    "mandatory": [
                        "left_control"
                    ]
                }
            },
            "to": [
                {
                    "key_code": "escape"
                }
            ],
            "conditions": [
                {
                    "type": "frontmost_application_unless",
                    "bundle_identifiers": [
                        "^org\\.gnu\\.Emacs"
                    ]
                }
            ],
            "conditions": [
                {
                    "type": "frontmost_application_unless",
                    "bundle_identifiers": [
                        "^org\\.gnu\\.Emacs"
                    ]
                }
            ],
            "type": "basic"
        }
    ]
}

ps. Note to self: Lennart's blog has notes on the topic but for linux.
UPDATE: Ensure bindings are only active when Emacs is [not]{.underline} active.


Basmati rice pudding recipe
xenodium.com @alvaro — Sun, 04 Oct 2020 03:00:00 GMT

Combine in a pot

2/3 cup of basmati rice
400 ml of coconut milk
4 cups of milk [^6]
3 tablespoons of honey [^7]
1/4 teaspoon of crushed cardamom seeds [^8]
1/8 teaspoon of salt

Simple. Combine all ingredients in a pot.
Boil and simmer
Bring ingredients to a boil and simmer at low heat for 45 minutes. Stir occasionally.
Mix in butter

1 tablespoon of butter.

Turn stove off, add a tablespoon of butter, and mix in.
Serve warm or cold
After mixing in the butter, the rice pudding is done. You can serve warm or cold.
Garnish (optional)

Pistachios
Cinnamon

Optionally garnish with either pistachios or cinnamon (or both).


A brief history of technology
Jason Eckert's Website and Blog — Sun, 04 Oct 2020 03:00:00 GMT
Technology is a crazy ride, and like that lyric in the Talking Heads song Once in a Lifetime, “…you may ask yourself, well, how did I get here?”
If that’s the case, you’re in luck, because this post is a no-nonsense, give-me-the-big-picture timeline of what the heck happened in the technology landscape these past 50 years. Of course, not every little detail is in there. Just those that are actually important (in my expert opinion, of course ;-) Enjoy!


Artificial Intelligence in Python
Posts by Petr Stribny — Sun, 04 Oct 2020 02:27:26 GMT



Rust 2021
Nikhil's blog — Sat, 03 Oct 2020 22:16:57 GMT
This is a contribution to the Rust 2021 Roadmap Blogs request.
As a developer using Rust in a proprietary, polyglot code base, I want Rust to improve non-Cargo builds My story comes from spending the better part of a year migrating a roughly 700,000 line code base (and several hundred upstream dependencies) to build using Bazel circa 2019. As Rust becomes more popular, a large chunk of developers will want to use it as a complement to existing code in other languages.


The Apple parable, from creativity to hoarding
Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 03 Oct 2020 21:00:00 GMT
No (more) innovation here. Please move along.


Beware the fascist driverless car
Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 02 Oct 2020 21:00:00 GMT
Old problems have an annoying habits to return, wearing different hats.


Artsakh Must Be Independent 🗺️
Ash's Blog — Fri, 02 Oct 2020 03:00:00 GMT
A data-driven argument for why Artsakh must be independent — examining historical claims, demographic evidence, and the indisputable pattern of ethnic violence.


ZFS rename devices in pool to disk ID
Random Notes by agilob — Fri, 02 Oct 2020 03:00:00 GMT
zpool import -d /dev/disk/by-id poolname Converts:
NAME STATE READ WRITE CKSUM poolname ONLINE 0 0 0 mdisk1 ONLINE 0 0 204 mdisk2 ONLINE 0 0 627K mdisk3 ONLINE 0 0 0 to map disks by ID (example):
NAME STATE READ WRITE CKSUM poolname ONLINE 0 0 0 wwn-0x1000000 ONLINE 0 0 204 wwn-0x2000000 ONLINE 0 0 627K wwn-0x3000000 ONLINE 0 0 0


Alien Machinery • Art
Muffin Man — Fri, 02 Oct 2020 03:00:00 GMT
It was generated using the same algorithm as the previous drawing. Metallic pens give it a really nice effect in person.
Created: October 2020
Size: 32x19cm
Paper: Fabriano Black Black 300gsm
Pens: Sakura Gelly Rolls Metallic



Do you realize that you are socially cooling?
Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 01 Oct 2020 21:00:00 GMT
“[Theoretically] well behaved, surely less human”.


Why We Sleep: Unlocking the Power of Sleep and Dreams
Apurva Shukla's RSS Feed — Thu, 01 Oct 2020 14:54:09 GMT
⭐ ⭐ ⭐ ⭐ ⭐ One of the most important books I’ve ever read. It has completely changed my relationship with sleep, and made me fall in love…


Youtube-dl format code cheatsheet
voussoir.net/writing — Thu, 01 Oct 2020 05:25:25 GMT

Back to writing

Youtube-dl format code cheatsheet
Table of contentsVideo formatsAVC only
VP9 only
AV1 only
4320p only
2160p only
1440p only
1080p only
720p only
480p only
360p only
240p only
144p only
60fps only
30fps only
Audio formats
Combined formats
Here are all the youtube format codes I'm aware of.
Please let me know if there are more, or if something here is outdated.
contact@voussoir.net
You can pass them to youtube-dl as --format a+b/c+d/e,
where a+b is your preferred video+audio pair, c+d is your second-best
preference, and e is your last resort.
Don't forget the magic words like "bestvideo", "bestaudio" to round out your selection.
See https://github.com/ytdl-org/youtube-dl/blob/master/README.md#format-selection
There are more formats listed in youtube-dl's source code,
but some of them seem to be obsolete and are no longer returned by -F
even when using the example video IDs listed there. I will only include
codes for which I have an example.
Video formats (§)


id
quality
codec
examples




571
4320p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


272
4320p30
VP9
youtube-dl -F i6fWG4FxDZw


337
2160p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


401
2160p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


305
2160p60
AVC
youtube-dl -F -xNN-bJQ4vI


315
2160p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


266
2160p30
AVC
youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw


313
2160p30
VP9
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


336
1440p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


400
1440p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


304
1440p60
AVC
youtube-dl -F -xNN-bJQ4vI


308
1440p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


264
1440p30
AVC
youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw


271
1440p30
VP9
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


335
1080p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


399
1080p60
AV1
youtube-dl -F 1La4QzGeaaQ


299
1080p60
AVC
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


303
1080p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


137
1080p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


248
1080p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


334
720p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


398
720p60
AV1
youtube-dl -F 1La4QzGeaaQ


298
720p60
AVC
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


302
720p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


136
720p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


247
720p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


333
480p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


397
480p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


135
480p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


244
480p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


332
360p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


396
360p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


134
360p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


243
360p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


331
240p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


395
240p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


133
240p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


242
240p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


330
144p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


394
144p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


160
144p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


278
144p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw



AVC only (§)


id
quality
codec
examples




305
2160p60
AVC
youtube-dl -F -xNN-bJQ4vI


266
2160p30
AVC
youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw


304
1440p60
AVC
youtube-dl -F -xNN-bJQ4vI


264
1440p30
AVC
youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw


299
1080p60
AVC
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


137
1080p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


298
720p60
AVC
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


136
720p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


135
480p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


134
360p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


133
240p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


160
144p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw



VP9 only (§)


id
quality
codec
examples




272
4320p30
VP9
youtube-dl -F i6fWG4FxDZw


337
2160p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


315
2160p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


313
2160p30
VP9
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


336
1440p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


308
1440p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


271
1440p30
VP9
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


335
1080p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


303
1080p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


248
1080p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


334
720p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


302
720p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


247
720p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


333
480p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


244
480p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


332
360p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


243
360p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


331
240p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


242
240p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


330
144p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


278
144p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw



AV1 only (§)


id
quality
codec
examples




571
4320p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


401
2160p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


400
1440p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


399
1080p60
AV1
youtube-dl -F 1La4QzGeaaQ


398
720p60
AV1
youtube-dl -F 1La4QzGeaaQ


397
480p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


396
360p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


395
240p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


394
144p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ



4320p only (§)


id
quality
codec
examples




571
4320p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


272
4320p30
VP9
youtube-dl -F i6fWG4FxDZw



2160p only (§)


id
quality
codec
examples




337
2160p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


305
2160p60
AVC
youtube-dl -F -xNN-bJQ4vI


315
2160p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


401
2160p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


266
2160p30
AVC
youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw


313
2160p30
VP9
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw



1440p only (§)


id
quality
codec
examples




336
1440p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


304
1440p60
AVC
youtube-dl -F -xNN-bJQ4vI


308
1440p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


400
1440p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


264
1440p30
AVC
youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw


271
1440p30
VP9
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw



1080p only (§)


id
quality
codec
examples




335
1080p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


299
1080p60
AVC
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


303
1080p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


399
1080p60
AV1
youtube-dl -F 1La4QzGeaaQ


137
1080p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


248
1080p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw



720p only (§)


id
quality
codec
examples




334
720p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


298
720p60
AVC
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


302
720p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


398
720p60
AV1
youtube-dl -F 1La4QzGeaaQ


22
720p30+128k
AVC+m4a
youtube-dl -F S8Zt6cB_NPU


136
720p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


247
720p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw



480p only (§)


id
quality
codec
examples




333
480p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


135
480p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


244
480p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


397
480p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ



360p only (§)


id
quality
codec
examples




332
360p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


18
360p30+96k
AVC+m4a
youtube-dl -F S8Zt6cB_NPU


134
360p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


243
360p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


396
360p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ



240p only (§)


id
quality
codec
examples




331
240p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


133
240p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


242
240p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


395
240p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ



144p only (§)


id
quality
codec
examples




330
144p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


160
144p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


278
144p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


394
144p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ



60fps only (§)


id
quality
codec
examples




571
4320p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


337
2160p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


305
2160p60
AVC
youtube-dl -F -xNN-bJQ4vI


315
2160p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


401
2160p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


336
1440p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


304
1440p60
AVC
youtube-dl -F -xNN-bJQ4vI


308
1440p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


400
1440p60
AV1
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ


335
1080p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


299
1080p60
AVC
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


303
1080p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


399
1080p60
AV1
youtube-dl -F 1La4QzGeaaQ


334
720p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


298
720p60
AVC
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


302
720p60
VP9
youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI


398
720p60
AV1
youtube-dl -F 1La4QzGeaaQ


333
480p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


332
360p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


331
240p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ


330
144p60 HDR
VP9
youtube-dl -F 1La4QzGeaaQ



30fps only (§)


id
quality
codec
examples




272
4320p30
VP9
youtube-dl -F i6fWG4FxDZw


266
2160p30
AVC
youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw


313
2160p30
VP9
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


264
1440p30
AVC
youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw


271
1440p30
VP9
youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


137
1080p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


248
1080p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


22
720p30+128k
AVC+m4a
youtube-dl -F S8Zt6cB_NPU


136
720p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


247
720p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


135
480p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


244
480p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


397
480p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


18
360p30+96k
AVC+m4a
youtube-dl -F S8Zt6cB_NPU


134
360p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


243
360p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


396
360p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


133
240p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


242
240p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


395
240p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ


160
144p30
AVC
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


278
144p30
VP9
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw


394
144p30
AV1
youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ



Audio formats (§)


id
quality
codec
examples




258
386k
m4a
youtube-dl -F NMANRHz4UAY


256
195k
m4a
youtube-dl -F NMANRHz4UAY


251
160k
Opus
youtube-dl -F S8Zt6cB_NPU


140
128k
m4a
youtube-dl -F S8Zt6cB_NPU


250
70k
Opus
youtube-dl -F S8Zt6cB_NPU


249
50k
Opus
youtube-dl -F S8Zt6cB_NPU



Combined formats (§)


id
quality
codec
examples




22
720p30+128k
AVC+m4a
youtube-dl -F S8Zt6cB_NPU


18
360p30+96k
AVC+m4a
youtube-dl -F S8Zt6cB_NPU



This document is generated programmatically.
sqlite3 :memory: ".read youtubedl_formats.sql" > youtubedl_formats.md


View this document's history

2022-03-25 Add resolution and fps tables.
2020-11-06 Add audio bitrates to 18, 22 quality column.
2020-10-01 Add more formats, examples, and auto-generate the md table aligners.
2020-09-30 Add youtubedl_formats.md.

Contact me: writing@voussoir.net
If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom



The 7 Sins of Turkish Autocracy 🇹🇷
Ash's Blog — Thu, 01 Oct 2020 03:00:00 GMT
Seven stories tracing Turkey's path from secular reforms to neo-Ottoman ambitions — examining crimes unpunished, allies betrayed, and the dangerous pattern repeating today.


A simple CSS ruleset
nutcroft — Thu, 01 Oct 2020 03:00:00 GMT
The content of this article is a set of 8 rules along with a CSS writer's workflow, grown out of developing small web projects. It's designed for simple layouts with minimal JavaScript interactiveness, and its top concern is code readability, rather than DRY.
Ruleset
1. No preprocessors
Using a preprocessor, such as Sass, for a few thousand lines of CSS code is a clear case of overengineering. The scope of a project is the most crucial datum for the preprocessor decision, but what happens in modern web development is that the preprocessor overhead is downplayed because of the omnipresence of the Node.js toolset.
2. No frameworks
Unless one plans to use every Bootstrap component it's not worth adding 160KB of un-gzipped CSS files. In addition to the size tradeoff, the framework buy-in is an important engineering decision as one is potentially adding something that they will have to fight either in a case of over-growing it or in a case of not understanding how it's designed to be extended.
3. Yes CSS reset
Browser implementation inconsistencies¹ are undying. Therefore, while no framework is advised, a CSS reset library is. I've been using normalize.css for the last many years with no issues.
4. No externally loaded fonts
Typography is great and fun. In cases of doubt Practical Typography provides enlightenment. But this is the case against fonts, and especially against Google Fonts², as the only real contender for free web fonts. Our criteria are quality and speed. All OSes have good enough default fonts, thus it's not worth loading a slightly different font and introducing yet another third-party dependency. But maybe one wants a more-than-slightly different font. Then, my argument is that most Google Fonts are not of high quality, and a
professional font might be worth buying.
5. Yes semantic HTML elements
Semantic HTML is great. Not only does it enable one to think through the information architecture of their layout, but also improves the accessibility of the website.
6. Yes CSS variables
CSS variables have good enough browser support for a while now. Colours and margin/padding lengths are ideal usecases for them.
7. Yes namespaces
I've found very useful to overload class names with namespaces, based on page or resource names. For example, if form buttons are different than bare buttons, then those classes will be form-btn-submit. Another recurring example is special styling for the landing page, which I'd namespace as landing-*.
8. Yes rule-local responsive media queries
The question is whether to place all rules for a specific breakpoints all together, or rules for the same classes together across different breakpoints. I vote for the second option, so that when one wants to read or change the rules for e.g. nav, they see all rules for all breakpoints together, rather than jumping to different places of the file for the other breakpoints. Example:
.projects-grid {
    display: grid;
    grid-template-columns: 1fr 1fr 1fr;
}
@media (max-width: 490px) {
  .projects-grid {
    grid-template-columns: 1fr 1fr;
  }
}
@media (max-width: 340px) {
  .projects-grid {
    grid-template-columns: 1fr;
  }
}


Limitations
The limitations of this system lie mostly on compatibility. 
Firstly, preprocessors would automagically fill the blanks with vendor-prefixed properties and polyfills. Without those, some problems might arise.
Secondly, IE11 does not support CSS variables, which we say yes to.
Workflow
Most of the times I'm designing a new webpage with no existing design, I have found that capturing my vague ideas into a concrete design in a vector editor is the best way to start. After that, I focus on writing the HTML. The reason for not writing the HTML and the CSS in parallel lies mostly in the unknown intricacies of the layout that will be followed. If a CSS grid is needed, for example, this is the step to decide that.
Final stage is actually writing the CSS code. I try to group the rules together based on where they are used. Generic selectors like a and h1 go first, then form and their conceptual children such as input, and then namespaces of other resources and pages.
One can find an actual example of all the ideas in action on the sr.ht repo of the mataroa.blog project.




Stephanie Stimac has a interesting example on font rendering incosistencies. Also, the source code comments for normalize.css are enlightening. ↩


Matthew Butterick provides another case, a very interesting critique on Google Fonts, in regards to them not being open-source. ↩





Substitution and Equational Reasoning
λm.me — Thu, 01 Oct 2020 03:00:00 GMT
In a lazy functional language where variables are immutable and effects are explicit, new powerful tools for understanding what programs mean become available - substitution and equational reasoning.


Beware The Fallacy of Composition
swyx's site RSS Feed — Thu, 01 Oct 2020 03:00:00 GMT
Many things that make sense individually don't make sense as a group.


SRE 123
Greg Herlein — Thu, 01 Oct 2020 01:50:08 GMT
Everyone is all on fire about SRE.  But what is it really?  It’s as easy as 1-2-3 if you boil it down to the basics.


How to handle fights in almost any Facebook group
Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 30 Sep 2020 21:00:00 GMT
Have you been fighting over the purpose and content of a Facebook group? Get real!


Books and Links: September
trv.rs — Wed, 30 Sep 2020 19:48:00 GMT
Around here autumn is lurking... the rising wind has the dry rustle of turning leaves, the air just feels crisper, and I'm craving baked goods and campfire smoke.
I am still in limbo with some of my zettelkasten/note-taking work... I'm just going to have to do a post about where I am at the moment and hope that others can build on it. Suffice to say, obsidian is super awesome and influencing a lot of my decisions on that front at the moment.
We are still safe and happy in our little home... learning to find a new steady-state with our increased population count. I've been spending time messing around with iOS14 automations and keyboard stuff... I'll write about it soon.
Books
This month was pretty good for book-reading. I think I read... four books, not including comics (every month I look forward to my new allotment of Hoopla borrows!)
On deck: Knowing God by JI Packer, A Programmer's Introduction to Mathematics by Jeremy Kun, Whiskey Tango Foxtrot by Lynne Black Jr.
Talking to Strangers by Malcolm Gladwell
This is a dark book. In telling the story of a single traffic stop gone wrong, Gladwell meanders through the history of communication, deception, and the incentives of trust and suspicion. I found the discussions of different conversational fallacies interesting as to how they aligned with the Gricean Maxims, but I am conflicted because I felt there was a definite bias to how some of the social science was handled and statistics interpreted. Content warning: Language, graphic depictions of violence and sexual violence.
Redshirts by John Scalzi
I knew going in that I was going to enjoy this one given the premise, but I surprised at how moving and thoughtful it was. It definitely transcends its initial setup to ask some hard questions about life and meaning. I know that's vague, but to say more would be a spoiler. Content warning: Foul language and coarse joking.
The Passage by Justin Cronin
I really enjoyed the character building. For a book ostensibly about virus-vampire apocalypse, it takes its sweet time telling about all the humans and giving everyone a worthy backstory. Very southern-gothic, Stephen King-esque. Against that tapestry of history, the stakes (see what I did there?) of decisions felt rich and meaningful, and Cronin managed to keep that pace up till the bitter end. I don't think I'll continue through the series though. Content warning: Violence, language, threatening situations.
Links
My link saving automation is mostly broken, and I started doing a weekly newsletter at work that has soaked up a lot of my desire to post links. I think I've also just spent more time in books than in Instapaper these days.
Stop doing design system projects

When new designers and developers join forces to evolve your design system it is tempting to start working in projects. You might even create some epics and sprints to move user stories from your backlog in a Scrum environment. But when your design system is not a full-time effort, this will work counter-productive.

I'm still trying to get our design system off the ground, but I had settled on doing it iteratively, bit by bit rather than a "project" all at once. Nice to see that others agree.
Stop doing design system projects
Marc Andreessen On Productivity, Scheduling, Reading Habits, Work, and More

You really have to uplevel and figure out what’s important. I usually take a good hour to look at what I’ve been doing. It’s basically figuring out the threshold for “yes” versus “no”. I try to revise that about once a year.

As a student of effective people, there were a couple good gems in this interview.
Marc Andreessen On Productivity, Scheduling, Reading Habits, Work, and More
PhD Student Reporting What It's Like to Use a Zettelkasten for Classes
I'm still a student of this... reading about the system is more useful than the tools.
PhD Student Reporting What It's Like to Use a Zettelkasten for Classes


Parallel Plate Capacitor Equation - Simulated
Shield Digital Design — Wed, 30 Sep 2020 19:00:00 GMT


The Question



A recent homework problem had us derive the parallel plate
capacitor¹ equation for electrostatics. This gives the
capacitance of a parallel plate structure in terms of the plate area,
distance between the plates and relative permittivity. The equation
for this situation is:

$
C = \epsilon_{R}\epsilon_{0}\frac{A}{d}
$

After solving this problem for the analytical solution, I was curious
to see how closely the result matched a 3D field solver. Using the
Capacitance Extraction mode in Cadence 3D Workbench, I designed a
parametric model of the parallel plate capacitor with variable edge
length, distance and 
    dielectric constant


 (See image above for the
model). 
    Dk
² was varied but the complex component (dissipation factor)
was set to zero to ignore any dielectric losses. The edges were swept
at lengths of 0.1 inch, 1 inch and 10 inch. The distance was swept
with values of 1mil, 10mil and 100mil. Finally, the 
    dielectric
constant
 was swept with values of 1, 10 and 100. This resulted in 27
unique cases, and the analytical result was compared to the extracted
result in each case.


Geo in GL part 1: Flattening Earth
dcz's posts — Wed, 30 Sep 2020 17:00:00 GMT
Geo in GL part 1: Flattening Earth
It's my guilty pleasure to flatten the Earth. To put it on various planar surfaces: pages of paper, scrolls, billboards, screens. Many prominent people attempted that before me: Mercator and Cassini, just to name two.
Yes, I'm talking about maps and projections.
This is the first part of the story of how I used OpenGL shaders to project the Earth onto the flat computer screen. Here I introduce the reader to the basics of projections, write about the scope of my project, and then choose a projection that fits the scope.
Warning: if you believe the Earth is already flat, this blog post won't offer anything interesting for you. You might still be interested in the following parts in the series.
What's the point?
I read someone online say shaders are bad for drawing maps: they aren't precise enough to distinguish distances a meter apart! Of course, I don't believe everything online, so I decided to check it myself, while refreshing my OpenGL knowledge, while playing with maps.
Which brings me to the next point.
I like maps
While there are many representations of the real world, maps feel the most real. Some models turn the real into an unrecognizeable mess of abstract concepts. Maps are elegant, because they copy the real thing, just smaller. They aim to be recognizeable and familiar, so making them is always visually satisfying.
Distortions
Maps are not perfect, though. They are not scale models, and they contain some abstract parts: a mountain represented on a map will be as flat as the paper or screen it's on. If it's marked, it will be either a drawing of a mountain, or a squiggle of height lines.
Most maps are also not globes. Our planet is round, and that means a flat map can't capture it accurately. Like a flattened orange peel, the map will get distorted: there will be tears or stretched places. The different ways to flatten the Earth to fit on a map are called projections.

This is also not a globe. It's just its picture! The picture is called orthographic projection.
The scope
There are multitudes of projections available. To narrow it down, it helps to define what kind of maps are interesting in this project. With that in mind, the initial version of the program will

display only small areas: not more than 100km in size. The smaller the area, the flatter the Earth surface, so distortions are smaller.
Load a GPS track and display it as a squiggle. My GPS tracks are usually only a few hours long, and fit in the 100km size limit.
Display nothing else. I don't want to get carried away and never prove what I set out to prove.

Projection choices
The last big decision was to choose a projection.
WGS84
Normally, we're used to seeing geographical coordinates of a point looking like: "54,5°N 23,7°E". This representation seems obvious. We live on a sphere, and so we use spherical coordinates. However, the prime meridian, and the precise measurement of the Earth axis, among oteher things, need to be standardized, so this representation is called WGS84. Since this is what GPS is using, it would be trivial to just plot the points from track on the screen as they are. However, the results would end up rather unappetizing:

The distortions are huge! Can you see the grid sizes? Places at the equator are much smaller than at the poles, and the poles are extremely stretched sideways too – if you look at the globe, the grid is composed of wedges close to the poles. This would be a very unpleasant map to look at, even on our small scale.
Mercator
The stretching is easily fixable by applying a varying stretching factor across the vertical position. This gives us the Mercator projection:
 Image copyright Strebe, CC-BY-SA 3.0 Unported.
This projection is easy to calculate, and it covers most of the world, so it's very popular on the Internet. Most online maps use it, including OpenStreetMap's own osm.org. However, it's still rather bad: the stretching factor approaches infinity as we approach the poles, so the map doesn't have a top and bottom edge. Instead it's just cut off somewhere. The other problem is that the areas at the equator are still much smaller than near poles.
Something different
It seems that both of those projections are quite bad at some places. But they are also both quite good at certain places: in this case, at the equator. So why don't we choose the best projection for our data?
That's what paper maps do. Unlike general purpose computer map viewers, paper maps are often limited to a small area on the Earth. They can focus on making this area undistorted, and they can ignore all the misshapen mess that's not printed on the paper. I deliberately chose to limit my program's area of interest to a 100km radius to take advantage of this.
For this, we need a projection that's not general like WGS84, or like the Mercator projection, but instead focussed on the area in question. To look at the North Pole in an undistorted way, we could choose the The North Pole Lambert Azimuthal Equal Area projection:

Here, I decided to increase my difficulty: I will not make an assumption about where on the Earth my GPS track is. It could be Europe, the Equator, or the Arctic. Instead, I will try to hit the bullseye and make sure that my projection favors the area I'm displaying, no matter what it is. For that, I need to find a family of projections that I can adjust.
Thankfully, people have come up with plenty of different ways of projecting:









Various projections: Alaska, Mollweide, Winkel I, van der Grinten (I), Gauss-Schreiber Transverse Mercator, Tobler-Mercator, International Map of the World Polyconic, Oblique Cylindrical Equal Area. (Images courtesy PROJ.)
Some of them are tweakable, to name a few: azimuthal equidistant, Cassini, gnomonic.
Most of them preserve shapes over small areas rather well, so the choice is not that important. Nevertheless, I'm not an expert, so it's possible that I'm underestimating distortions. To stay on the safe side, I choose the azimuthal equidistant, because it promises that distances from the central point can be measured with a ruler. I used it previously with an overlaid kilometer grid and it worked well.
Here's the projection in proj form, so you can apply it in QGIS. Don't forget to adjust lat_0 and lon_0 to your area of interest!
+proj=aeqd +lat_0=54 +lon_0=24 +x_0=0 +y_0=0 +a=6371000 +b=6371000 +units=m +no_defs

Next steps
I have the projection, but I still don't know the math behind it, and don't have any code. Stay tuned for the next part!


Fred Roger's Method For Writing Scripts
Josh Thompson — Wed, 30 Sep 2020 16:00:00 GMT
Someone said:


  People think this is silly, but read about Fred rogers’ method for writing a script for his show. The rules aren’t fully applicable to presentations, but the attention to detail and to the Interpretation of the audience is. Don’t use any words carelessly.


I said:


  I’m not normal. I’ve got three articles open right now. Is there a canonical resource you’d recommend, or I should just seat-of-the-pants research it?





I’m going to knock out what I can in 20 minutes, share back with said individual, and calibrate on if they think I learned what I need to learn. Timer starts at 3:17p.

It’s now 3:37.

Conclusion:

I didn’t find anything super specific to how he writes his scripts, but I think there’s a lot to glean from his approach to strongly empathizing with his audience. “strong empathy” feels like a criminal understatement.

I’ve never seen Mr. Rogers. I’m tempted now to find some and watch it. If you’d recommend it, where should I start? Any youtube video overviews I should check out?

10 things Tom Hangs learned by Playing Mr. Rogers

A Beautiful Day In The Neighborhood: 10 Things Tom Hanks Learned By Playing Mr. Rogers (screenrant)

The 10 lessons from the article.


  Be kind, be kind, and be kind
  Every single person you’re talking to should feel like they’re the only person in the world who matters
  The slow, reflective nature of Mr. Rogers’ kindness
  Be the message, don’t preach the message
  Even now, keep looking for the helpers
  The sweater is the key
  Embrace the good and true, over the easy and cynical
  To spread joy is the natural order of things
  Kindness and compassion as ministry
  It’s not just a children’s show


The titles don’t do it justice. Some of these spoke very loudly to me.

###



A few quotes in particular jumped out at me:


  Per the pamphlet, there were nine steps for translating into Freddish:

  
    “State the idea you wish to express as clearly as possible, and in terms preschoolers can understand.” Example: It is dangerous to play in the street.
    “Rephrase in a positive manner,” as in It is good to play where it is safe.
    “Rephrase the idea, bearing in mind that preschoolers cannot yet make subtle distinctions and need to be redirected to authorities they trust.” As in, Ask your parents where it is safe to play.
    “Rephrase your idea to eliminate all elements that could be considered prescriptive, directive, or instructive.” In the example, that’d mean getting rid of “ask”: Your parents will tell you where it is safe to play.`
    “Rephrase any element that suggests certainty.” That’d be “will”: Your parents can tell you where it is safe to play.`
    “Rephrase your idea to eliminate any element that may not apply to all children.” Not all children know their parents, so: Your favorite grown-ups can tell you where it is safe to play.
    “Add a simple motivational idea that gives preschoolers a reason to follow your advice.” Perhaps: Your favorite grown-ups can tell you where it is safe to play. It is good to listen to them.`
    “Rephrase your new statement, repeating the first step.” “Good” represents a value judgment, so: Your favorite grown-ups can tell you where it is safe to play. It is important to try to listen to them.`
    “Rephrase your idea a ﬁnal time, relating it to some phase of development a preschooler can understand.” Maybe: Your favorite grown-ups can tell you where it is safe to play. It is important to try to listen to them, and listening is an important part of growing.`
  



  In working on the show, Rogers interacted extensively with academic researchers. Daniel R. Anderson, a psychologist formerly at the University of Massachusetts who worked as an advisor for the show, remembered a speaking trip to Germany at which some members of an academic audience raised questions about Rogers’s direct approach on television. They were concerned that it could lead to false expectations from children of personal support from a televised figure. Anderson was impressed with the depth of Rogers’s reaction, and with the fact that he went back to production carefully screening scripts for any hint of language that could confuse children in that way.



  As simple as Mister Rogers’ Neighborhood looked and sounded, every detail in it was the product of a tremendously careful, academically informed process.




Now I’m wondering about the financial success of his show.

Wikipedia: Fred Rogers

link

The show was popular:


  by 2016 it was the third-longest running program in PBS history


Ah, here’s how involved he was:


  According to King, the process of putting each episode of Mister Rogers’ Neighborhood together was “painstaking”[59] and Rogers’ contribution to the program was “astounding”. Rogers wrote and edited all the episodes, played the piano and sang for most of the songs, wrote 200 songs and 13 operas, created all the characters (both puppet and human), played most of the major puppet roles, hosted every episode, and produced and approved every detail of the program.


He had two stages to the show:


  n 1975 Rogers stopped producing Mr. Rogers’ Neighborhood to focus on adult programming. […] In 1979, after an almost five-year hiatus, Rogers returned to producing the Neighborhood; King calls the new version “stronger and more sophisticated than ever”


Ahhh…. influence, and storytelling:


  Even though Rogers was not yet nationally known, he was chosen to testify because of his ability to make persuasive arguments and to connect emotionally with his audience. The clip of Rogers’ testimony, which was televised and has since been viewed by millions of people on the internet, helped to secure funding for PBS for many years afterwards.


Lets watch the clip, or queue it up for later at least. He starts to speak at 0:20:



I will watch this video, just not quite right now.

Oh, and he’s from Pittsburgh. I went to school near there, just finished working remotely for a company in Pittsburgh. I loved visiting the city.

Optional Next Steps


  I could read Good Neighbor: The Life and Work of Fred Rogers
  Actually watch some Mr. Rogers clips. I’m tempted to do that anyway, now.
  Any recommended youtube videos?



How to write a Principle
Writing on Adam Craven — Wed, 30 Sep 2020 13:53:24 GMT



Object Detection from 9 FPS to 650 FPS in 6 Steps
Machine Learning Articles on paulbridger.com — Wed, 30 Sep 2020 09:43:23 GMT
Making code run fast on GPUs requires a very different approach to making code run fast on CPUs because the hardware architecture is fundamentally different. Machine learning engineers of all kinds should care about squeezing performance from their models and hardware — not just for production purposes, but also for research and training. In research as in development, a fast iteration loop leads to faster improvement. This article is a practical deep dive into making a specific deep learning model (Nvidia’s SSD300) run fast on a powerful GPU server, but the general principles apply to all GPU programming.


Bullshit Jobs to Basic Income : Redesiging economic rewards
Yasha Solutions — Wed, 30 Sep 2020 03:00:00 GMT
Bullshit Job : not what you think David Graeber has turned what would have happily stayed a colourful expression of our daily lives into an actual concept. In his book Bullshit Jobs - A Theory published fairly recently in 2018, he expands on the concept :
 A bullshit job is one that even the person doing it secretly believes need not, or should not, exist. That if the job, or even the whole industry, were to vanish, either it would make no difference to anyone, or the world might even be a slightly better place.


Untitled • Art
Muffin Man — Wed, 30 Sep 2020 03:00:00 GMT
Be sure to check the timelapse in the gallery below.
Created: September 2020
Size: 52x27cm
Paper: 200gsm
Pens: Sakura Gelly Roll 0.8mm pen



Leading Up to Shadowlands
Ben Overmyer's Site — Wed, 30 Sep 2020 03:00:00 GMT
The Shadowlands expansion for World of Warcraft comes out next month. A lot of things seem to be happening in October this year. I'm opting to focus my gaming time on WoW, given the new expansion.
A month ago, I was playing a lot of WoW. The new expansion offers a lot of opportunities for starting new characters, and I wanted to unlock the Vulpera allied race to try something new. Unlocking that race previously required completing a particular story arc and reaching Exalted status with the Vulpera. I'd finished the story arc, but the reputation grind was quite a slog, and I hadn't come close to finishing it.
Then, I discovered that the pre-patch for Shadowlands was going to remove the reputation grind from unlocking allied races. This completely killed my interest in playing WoW before the pre-patch hits, and I haven't played since, despite keeping my subscription active.
As soon as the pre-patch is live, I'm diving back into the game with a vengeance. My new character will be a Vulpera shaman. I want to play a healer, and the Vulpera shaman's totems are fox-shaped which I love.
The new leveling experience takes you from 1 to 50 (the new pre-expansion cap) in a single expansion's content. I'm going to go through Wrath of the Lich King as my choice, since that remains my favorite expansion so far. It remains to be seen if I will blog about this process or not, but it might be fun to journal my experience. I don't think I'll be able to reach level 50 with my Vulpera before the expansion lands, but that's going to be my goal anyway.
I don't know if pet battle levels will be rewired too, but if they are, I definitely want to try and bring at least one group of pets up to level cap during my leveling. We'll see how that works out.
So yeah. I'll be retiring Exsecratus as my main, which is only the second time I've ever changed mains since I first started playing WoW back in 2004. Here's to new beginnings!


Faster Literal String Matching in Go
Ben E. C. Boyter — Wed, 30 Sep 2020 03:00:00 GMT
TL/DR: I wrote a fast literal string matching library in Go get it here https://github.com/boyter/go-string/
Recently I rebuilt searchcode in Go, as mentioned.
While there was a variety of reasons for this one was that performance in the previous Python version was not great. Please note this was mostly due to my own shortcomings and not the language itself. However I have always used searchcode as a test bed for interesting problems, since it gets enough traffic to verify things at scale, and I wanted to get better at Go.


Pre-training via Paraphrasing (Paper Explained)
Who is Nnamdi? — Wed, 30 Sep 2020 00:03:16 GMT
Transformer model pre-trained on document retrieval and reconstruction performs well on both fine-tuned and zero-shot downstream tasks


Recovering after a root-partition resize gone wrong
zufallsheld — Tue, 29 Sep 2020 23:00:00 GMT
Recently I tried to resize a LVM logical volume with its volume group and the physical disk belonging to the vg.
Because the logical volume contained all partitions except for /boot I knew it would be hard.


I also knew that I had no backup and no snapshot of the …


What Amy Coney Barrett on the Supreme Court would REALLY mean for the future of tech
Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 29 Sep 2020 16:00:00 GMT
Let’s put things in perspective, please.


The Definitive Guide to Commonly Used Words in React
Max Rozen — Tue, 29 Sep 2020 12:48:48 GMT
Do some words in React have you scratching your head? Ref? Stateless component? Let's learn some words!


Intentional Documentation
blog.harterrt.com — Tue, 29 Sep 2020 10:00:00 GMT
Randy Au has a great post on documentation for data scientists here:
Let's get intentional about documentation.
Take a look, it's worth a read.
I've been able to find some decent guides for writing documentation
but they're usually targeted at engineers.
That's a shame. 
Data scientists have significantly different 
constraints …


Adapting Learning Systems To Today's World
Yasha Solutions — Tue, 29 Sep 2020 03:00:00 GMT
Covid-19 has forced us into a reality where colocallity can only be virtual.
While the press, inspired by the promises from some Silicon Valley giants, has tried to forecast how and when humanity would become a remote-first society, nobody came up with the creative scenario where a pandemic would be the trigger for global digital transformation.
In this new reality, schools and universities need to adapat.
Beyond the Zoom-Classroom experience.


2020–09–29: New codec driver in my 5.9 kernel
megi's PinePhone Development Log — Tue, 29 Sep 2020 01:00:00 GMT



Minimum elucidating examples
Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Mon, 28 Sep 2020 22:00:40 GMT
I tend to learn new concepts best by seeing an example of them. This
dates back to when I first started studying mathematics: I felt that I
was able to actually work with a certain definition once I had at
least an intuitive grasp of what is meant. I quickly discovered that
there are at least three kinds of examples. The first kind is trivial,
leading to even more confusion, the second kind is too complex, and the
third kind is actually elucidating. It is the third kind of example that
I am primarily interested in here, so let’s quickly discuss the other
two and move on.
Examples of the first kind are trivial. Suppose you want to illustrate
the concept of addition and state that $1 + 0 = 1$. This is obviously
correct but does not teach you anything because it contains the idea
that $a + 0 = a$ as a confounding property. Likewise, mentioning the
empty set $\emptyset$ as an example of a certain kind of structure is
always good for some laughs,¹ but it does not really help you
understand what is going on.
Examples of the second kind are not more helpful. They are characterised
by a rapid increase in complexity that just bogs you down without really
helping you gain intuition or understanding. This Abstruse Goose comic
provides a marvellous instance.² Other examples include, say, teaching
you about determinants by
letting you invert a $10 \times 10$ matrix, when a smaller dimension
would have been sufficient.
Examples of the third kind help you grow and understand. I call these
examples minimum elucidating examples: they have just the right
amount of complexity to not be wholly trivial, but they are simple
enough to be remembered and understood correctly. For instance, when
learning about group theory, some elucidating examples are the additive
group over the integers, $\mathbb{Z}/2\mathbb{Z}$ (the group with
only two elements), or, to be a little bit more complex, the general
linear group of all real invertible matrices. The utility of such an
elucidating example depends heavily on the context—as a student,
I preferred learning about the ’number groups’ first, but nowadays, my preferred
go-to example is the general linear group, because it is more
abstract and provides a better glimpse into the power of group theory.
Closely related to such elucidating examples are elucidating
non-examples (or counterexamples). In group theory, the natural
numbers $\mathbb{N}$ come to mind: lacking an additive
inverse for every element except $0$, every student can easily grasp
that the additional requirements raise the bar for something to be
called a group, while at the same time illustrating that a much richer
structure is exposed by demanding these properties.
Of particular pedagogical interest are examples that defy our
presuppositions. These are counterexamples to our misplaced intuitions!
For instance, the Cantor set
is a marvellous counterexample in measure theory:³ typically,
students only encounter sets with measure zero that are countable.
This can lead to the wrong impression that ‘measure zero’ and
‘countable’ are somewhat equivalent. The Cantor set destroys this
illusion, as it constitutes a set that is uncountable while having
measure zero! My mind was shattered when I first learned about this, and
since then, the Cantor set has been a cherished counterexample in my
toolbox.
Having switched academic fields for a while now, I sometimes wonder what
minimum elucidating examples we should develop for machine
learning (ML).  What is the smallest useful architecture and data set that demonstrates
issues such as vanishing gradients?
Is there a way to store and disseminate such counterexamples? Would the
aspiring ML researcher find them useful and digest them, or are we still
in the ‘pragmatic phase’ of our field and use whatever works, without
caring too much about the theoretical underpinnings? Time will
tell—and I for one hope to collect as many juicy minimum elucidating
(counter)examples as I possibly can.
Until next time, have fun being exemplary!




Full disclosure: as a cheeky undergraduate, I did this a few times
in oral exams. It is funny but probably less so for the professors who
have to endure that sort of humour all day long. ↩︎


I disagree with the comic on one thing, though: maths textbooks
tend to follow this recipe more often than computer science textbooks,
in my experience. ↩︎


Measure theory deals with measuring subsets of Euclidean space,
assigning them a number that describes their ‘size’; think of a generalisation of
the concepts of area or volume. Measure theory is often invoked
when dealing with problematic solutions or inputs to
functions—essentially, one tries to show that while such degenerate
inputs may exist, they are ’too small’ (in the sense of measure
theory) to worry about. ↩︎





Why did you choose Microsoft? Sorry, don't know
Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 28 Sep 2020 21:00:00 GMT
A sad, sadly unsurprising story about an italian Ministry.


How To Set Up a Ruby Dev Environment on Ubuntu Linux
Mike Kasberg's Blog — Sun, 27 Sep 2020 18:15:00 GMT
Setting up a development environment correctly might not seem like a big deal at
first, but an incorrectly set-up environment can cause a lot of problems down
the road if you’re not careful. In my experience, the internet’s full of
solutions – both good and bad – for environment problems, but often does a poor
job of explaining why a solution’s good or bad. And installing Ruby can be
tricky if you’ve never done it before. So here’s my advice.

Let’s say you want to run a gem – for example, Jekyll.
To run the gem, you’ll have to install Ruby and then install the gem. Rather
than jumping straight to the best solution, I’m first going to look at what
not to do so you can understand what problems this might cause. After seeing
what not do to, we’ll look at a couple better solutions, and we’ll see the
advantages and disadvantages of each. By the end, I’ll show you what I think is
the best way to install Ruby so you can avoid environment headaches.


In a hurry?
Want to go straight to the instructions for the proper way (in my opinion) to install Ruby without reading about what not to do?
Skip ahead


What Not To Do

When you’re a beginner with Ruby or Linux (we’ve all been there) and all you
want to do is run a gem, sudo appears to solve all your problems. You might
come across advice like this:


  
    Install Ruby

    $ sudo apt-get update && sudo apt-get install ruby
    
  
  
    Install the gem

    $ sudo gem install jekyll
    
  
  
    Use the gem. Done.

    $ jekyll --version
jekyll 4.1.1
    
  


While this might appear to solve everything, it can also create a host of new
problems. For example, maybe you’ll need to install some gems with
Bundler a few days later. If you try using sudo to
install gems with Bundler, you’ll probably see a warning like this one:

$ sudo bundle install
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.


Bundler warns you that this isn’t the best way to do things - root will be the
owner of all the files bundler installs, and from that point on you’ll have to
continue using sudo to get gem updates until you fix your file permissions.

But beyond this, installing gems with sudo can actually be dangerous -
particularly since gems can execute scripts when they install. For example, see
killergem. Even if you didn’t install
this malicious gem directly, if one of the gems you try to install even depends
on a malicious or compromised gem, it could be bad news. Let’s look at a better
solution…

A Quick & Simple Solution

So if the above steps are problematic, what can we do that’s safer? Maybe you
just want to run a quick script or gem and you don’t care about managing
specific versions of Ruby. You want a simple solution that works and doesn’t
have permissions problems.


  
    Install Ruby. For simplicity, we’ll use whatever version is in Ubuntu’s
repositories. Using sudo is OK here because we’re using apt-get.

    $ sudo apt-get install ruby
    
  
  
    Install the gem. Since we’re installing a gem (rather than a system package
from apt), we won’t use sudo. Instead, we’ll pass the --user-install flag when
we install our gem.

    $ gem install --user-install jekyll
    

    The --user-install portion is key – it installs the gem in a directory our
user has permissions to. You’ll probably see a warning like this:

    
      WARNING:  You don’t have /home/mike/.gem/ruby/2.7.1/bin in your PATH, gem executables will not run.
    

    That’s OK – we’ll fix it in the next step.
  
  
    As the warning says, ~/.gem/ruby/2.7.1/bin isn’t in our PATH. So let’s put
it there. Ubuntu comes with a ~/.profile file that adds a couple things to our
path by default, and I like to build on that file. Modify it to include these
lines:

    # Add User Ruby Gems
if [ -d "$HOME/.gem/ruby/2.7.1/bin" ]; then
    PATH="$HOME/.gem/ruby/2.7.1/bin:$PATH"
fi
    

    For the changes to take effect, we can either log out and log back in or
simply run source ~/.profile. We can confirm it worked by running which
jekyll, which should return ~/.gem/ruby/2.7.1/bin/jekyll.
  


Done. We’ve installed the jekyll gem without using sudo. In this case, for
simplicity, we’re relying on our OS to provide an easy-to-user Ruby installer.
This is fine, but it does mean we’re stuck with the version of Ruby that our
OS provides. Which is fine for simple needs (like running a few simple scripts
or gems), but might not be ideal if, for example, you’re developing a Ruby
application that requires a specific version and want to upgrade your version of
Ruby independently of your OS. If you’re regularly programming in Ruby, it will
be important for you to install the version of Ruby required by your
application, and possibly install multiple versions of Ruby for multiple
applications or during upgrades.

The Best Solution: How to Install Ruby with rbenv

The best solution for installing Ruby and managing your installation is a Ruby
version manager. Using a version manager allows you to have multiple versions of
Ruby installed at once and select the one you want to use. It also decouples
your Ruby installation from your OS package manager, and provides a solution for
doing everything without sudo. There are a few options for version managers.
RVM is reputable and has a good following, but in my humble
opinion rbenv is simpler and easier to use. To
install rbenv, you can find the most up-to-date official instructions
here, but they can be a
little confusing, and they’ll boil down to something like this:


Remove old Ruby versions first!
It is confusing to have Ruby installed by both apt-get and rbenv on the same system. If you've already installed Ruby with apt-get, you should remove it before continuing. You can do so with sudo apt-get purge ruby.



  
    We’ll need git and curl to run the rbenv installer. Run the following to
make sure they’re installed.

    $ sudo apt-get install git curl
    
  
  
    Run the rbenv installer.

    $ curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash
    
  
  
    Add rbenv to your path by adding it to your ~/.bashrc. You’ll have to log
out and log back in or source ~/.bashrc for the changes to take effect.
(And we’re tackling 2 steps at once by including the rbenv init line
here.)

    $ echo '
if [ -d "$HOME/.rbenv/bin" ]; then
    PATH="$HOME/.rbenv/bin:$PATH"
    eval "$(rbenv init - bash)"
fi' >> ~/.bashrc
    
  
  
    Check to make sure it worked with rbenv-doctor. This will save you some
headaches by printing errors if anything’s not quite right.

    $ curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-doctor | bash
    

    You’ll see something like this, letting you know that rbenv is installed but
no versions of Ruby are installed yet.

    
      Counting installed Ruby versions: none
  There aren’t any Ruby versions installed under `/home/mkasberg/.rbenv/versions’.
  You can install Ruby versions like so: rbenv install 2.7.1
    
  
  
    We’re going to use rbenv to install a version of Ruby, but before we do that
we need to make sure we have the tools to build ruby installed. (If you
skip this step, the installer won’t work and will give you an error message
about what you need to install to compile Ruby. We can save time by just
installing what we need now.)

    $ sudo apt-get install build-essential libssl-dev libyaml-dev zlib1g-dev libffi-dev
    
  
  
    Now, we’re ready to install Ruby. I like to run the help command first to
see what I’m about to do. You should generally install whatever the latest
stable version is. At the time of writing, it’s 2.7.1.

    $ rbenv help install
$ rbenv install --list
$ rbenv install 2.7.1
    
  
  
    At this point, you’re basically done! Again, we can skim the help menu to
see what other commands are available. And it’s a good idea to set the
version we just installed as the global ruby version.

    $ rbenv help
$ rbenv global 2.7.1
$ rbenv version
$ ruby --version
    
  


One thing that’s great about rbenv is it does not tie you in to a specific ruby
version. You can set up local
(per-directory) and shell (per-terminal) versions if you want. Simply rbenv
install x.x.x and then use the correct commands (see the docs) to pick your
version. In addition to this, by default, rbenv will allow you to install
gems to a location that
doesn’t require sudo (with a normal gem install command), which should prevent
a lot of headaches.


How behaviour can shape the culture
I Learnt — Sun, 27 Sep 2020 14:00:00 GMT
Andrew Cherrie is the pastor of Home Church. In a recent sermon entitled “7 lessons from 7 years at Home Church” he gave the following quote:


Solitude and conversations lead to great ideas
I Learnt — Sun, 27 Sep 2020 14:00:00 GMT
There was a tweet recently by Sahil Lavingia that stated


5 TILs about Node.js Fundamentals from the Node.js Design Patterns Book
swyx's site RSS Feed — Sun, 27 Sep 2020 11:48:33 GMT
5 Things I Learned about Node.js Fundamentals from the Node.js Design Patterns Book


Adding images to pdfs (macOS)
xenodium.com @alvaro — Sun, 27 Sep 2020 03:00:00 GMT
The macOS Preview app does a great job inserting signatures to existing pdfs. I was hoping it could overlay images just as easily. Doesn't look like it's possible, without exporting/reimporting to image formats and losing pdf structure. Did I miss something?
In any case, I found formulatepro. Dormant at Google Code Archive, but also checked in to github. With a tiny patch, it builds and runs on Catalina. One can easily insert an image via "File > Place Image…".



DIY bookmarks
xenodium.com @alvaro — Sun, 27 Sep 2020 03:00:00 GMT

Best electrical insulation tape.
I’ve spent the last 3 months building the home office of my dreams.



Armenia, Azerbaijan, Turkey. Who's the Aggressor? ⚔️
Ash's Blog — Sun, 27 Sep 2020 03:00:00 GMT
Three countries, vastly different resources and freedoms. The numbers reveal who's really threatening peace in the region.


Our New Species (Part 2/2)
A Matter of Perspective — Sun, 27 Sep 2020 03:00:00 GMT

This is part two of this story. You can read part one here!

The Age of the Spiritual Machine
Lunch with a “Connected One”
The unbranded restaurant was eerily silent as Harrison climbed the marble steps to meet the host.
“Hi, ugh, I’m here to meet someone?” He asked.
“Right this way please, Mr. Fuller,” the restaurant’s host was dressed in smart trousers, a white dress shirt, and a black bow-tie.
“Wait, how do you know my–“Harrison started.
“Ah, my apologies, sir. The Link allows for easy facial recognition matching from a public dataset. It’s standard practice. I did not mean to invade your privacy” He smiled, but professionally, almost… practiced. His British accent and cool demeanor intimidated Harrison. What was this place?
He lead them down a dark hallway, modern art dripping from the long, polished, concrete walls, which opened out into a circular, cobblestone courtyard. A large tree rose from the center, reaching for the blue sky. The open ceiling shone a cold light on the small restaurant, dotted with exotic plants, polished wooden tables, and suave guests eating in silence. The entire scene gave Harrison the chills.
The host directed Harrison to a table on the opposite side of the circle. A young, attractive man was sitting at the wooden table alone, his eyes closed. He seemed peaceful, almost meditative, as Harrison approached.
Jacob awoke from wherever he was.
“Hello Harrison,” his voice was calm and smooth, “It’s nice to meet you finally.”
Harrison was a little taken aback. He pulled out the chair and sat opposite this strange character.
“You too. This place is–”
“Different to what you’re use to?” Jacob finished.
“Yeah, it’s like; everyone knows me, but I have no idea who they are” Harrison explained. It really was very disconcerting.
Jacob smiled knowingly. He pushed back his styled blonde hair and looked at Harrison with an intense, green-eyed gaze. It seemed to pierce Harrison, almost like he was being searched.
“You’ll get used to it,” he explained, “There are many things you’ll get used to. The Link will show you.”
“Wait, did Jacob know something about my application?” Harrison wondered. The waiter stepped in to fill Harrison’s glass with water.
“So, tell me why we’re here,” Jacob asked, “Your Telegram comments are always amusing. I’m glad we finally made it in person.”
Harrison paused, Jacob’s energy was almost overwhelming. And those eyes…
“Me too. And, well, you mentioned you were linked on the group. I’ve been trying to understand what it’s going to be like. What it’s really like, from someone who gets things, you know?” Harrison explained.
Jacob nodded patiently, “I sense your frustrations. I was the same way when I was on the wait-list. Well, how much do you know?” Jacob’s eyes sparkled with the inquiry.
“Not much, I guess. Just what’s in the news, Reddit, movies, you know. A guy at work is taking over everything.” Harrison tried to hide his disappointment, but Jacob saw through him. He frowned.
“Yeah, that happens.” Jacob looked out over the courtyard, furrowing his brow. He paused for a good minute, and Harrison began to fidget, unwilling to disturb this pensive man.
A Magnifying Glass
“What is it you think the Neuralink is, Harrison? What do you think it does?” Jacob broke the silence returning his gaze to the conversation.
“It connects us. Enables us to engage with technology at a level we’ve never had before,” Harrison attempted, “It makes us smarter, right? The bandwidth problem.”
“Yes, and also no” Jacob mysteriously replied, “Think of the Neuralink like all technology that has come before us. It’s a magnifying glass.
“What did electricity promise us? Planes? Phones? The Internet? They all heralded quantum shifts in how we approached the fundamentals of daily life. The internet gives us the ability, for example, to search for anything, anywhere, anytime—the entire catalog of recorded history and information at our fingertips. And yet, what do most people use it for? Looking at cat videos and pictures of naked women.
“All the internet did was exaggerate behaviors that were already present—a giant magnifying glass over human values. The Neuralink is no different. It doesn’t solve the world’s problems, as much as it enables us to me more… us. And that’s not always a good thing.”
Jacob’s words crashed over Harrison like waves. The waiter returned with two plates, placing them down in front of the pair. Harrison didn’t remember ordering, but he was so caught up in Jacob’s ideas he barely noticed.
“So it accentuates our pre-existing selves?”
“Exactly. Well, for most of us. It changes some and opens up doors. There are those who understand the technology at a deeper level. Those who push it… The Legion” Jacob’s voice quietened, and he looked up from his salad.
“Legion? I thought that was just an internet myth?” Harrison laughed nervously. Was he serious?
“You’re a programmer, right?” Jacob quizzed, “Then, think about developers for a second. They command our world because technology commands our world. The technology that is so useful that you can’t live without it. It’s become ubiquitous. Therefore, developers – or those who own them – have the power because they control the world’s most useful thing.
“Legion is the same thing, but for the intersection of humans and AI. We command it because, like developers, we create it and understand it more than anyone else. Legion is open to anyone who can command themselves and technology in a way that contributes to the whole. These are the people we’re interested in.”

Harrison hadn’t touched his lunch – this was too much. Was this a test?
“So what, you’re part of an elite group of programmers that have Neuralink’s that control AI?” Harrison retorted.
“Not just programmers. And we’re not ‘the elite’. We are doctors, lawyers, politicians, teachers, cleaners, and taxi drivers. We are Legion. And we also don’t control the ‘AI’.” He paused dramatically.
“I don’t understand. What’s the purpose of a ‘Legion’?”
“Most humans view themselves as distinct, as separate from each other and the world—independent controllers of their fate and actions. In reality, this is ludicrous, but our egoic fascination is a powerful lure. Our ego wants to identify with form. We forget that we belong to a larger eco-system and race, and we are more intrinsically interconnected than we give ourselves credit for. The internet has only increased this and the Neuralink even more so.
Listen, let’s take a blood cell from your body. This cell doesn’t claim independence from its siblings, engaging in self-serving behavior to benefit only its ‘Self’. No, it operates in harmony and accordance with the body’s needs, doing what it can to provide. We have forgotten this as humans.
The Link removes the barrier of communication, ego, and privacy. Should you be open to it, the flood gates are flung wide open, and the union of you and everyone is met. It’s truly beautiful. You mentioned you’re a fan of mushrooms, right?”
“Then what about the AI?” Harrison was on the edge of his seat, this morning’s slack messages a distant memory.
“Right. Well, that’s a little more tricky to explain. Look, humans have a way to go in terms of what we were saying earlier. Until you can act as a unified collective, you’re in trouble. The digital space opens possibilities for virtualization and symbiosis with something we’re all feeding. We become one in a sense.” Jacob’s smooth voice was no more than a whisper.
“You say, ‘humans’ like you’re not one of us.” It was unsettling for Harrison. Jacob chuckled.
“Ha! I guess I did. Funny!”, Jacob’s eyes sparkled, but Harrison did not find anything funny. “Harrison, if we are to conquer space, time, and the great mysteries this universe presents, is it possible we may need a new form?”
The question hung in the air, resonating like a struck gong.
“That’s enough for today; I think” Jacob wiped his mouth with his napkin and folded it neatly on the table. “Let’s bounce!”
The waiter collected Harrison’s untouched salad, and the two walked back onto the upmarket street.

“Wait, don’t we have to pay?” Harrison started.
“Don’t worry, I’ve already got us” Jacob exclaimed, tapping his head with a grin, “You’ll get used to it soon enough. But, before I go, are there any other questions you have?”
Harrison’s mind exploded in a million directions, his thoughts of technical-symbiosis, ethics, and wonder scattering like a firework.
“Would you call yourself human?” Harrison blurted. The words felt clumsy, but the curiosity was too much to bear.
“What is human anyway, Harrison?” Jacob flashed a wicked smile and disappeared into a Fetch that had suddenly whipped around the corner. He was gone.
Harrison stood on the steps of their perplexing interaction, consumed by a cocktail of confusion, fear, and excitement. He remembered Alice.
Emergence
The walk home felt longer than any he had taken. Harrison’s phone incessantly buzzed with immediate responsibilities, updates, and tasks – but Harrison was oblivious. Jacob had opened the door to a new world, and Harrison was standing on the precipice.
He knew the world was changing, but Jacob’s illustration presented something else entirely. A new species. A blend of network, consciousness, flesh, and steel. Was this even possible?
“Perhaps I put too much in my tea this morning” Harrison deliberated.
He returned to his small apartment, high above the world. The afternoon’s blend of techno-infused spirituality had rendered Harrison restless and anxious.
Harrison sat down at his desk and forced an attempt at his nightly ritual of coding practice. The algorithmic puzzle glared smugly at him from the code window. The answer was right before him, and faded just out of reach anytime he came to close. Leetcode’s complexity drove his thoughts way, and they wandered back to Jacob.
“Who is Jacob anyway?” Harrison meditated, “I know so little about him.” He made a promise to himself to never go to lunch with Connies he’d met online again. It only seemed to complicate an already chaotic situation.
He opened a new tab and searched for “Legion”. A disappointing array of links splashed on the page. A collection of demons from the New Testament, an old TV series, details on a faction of the Roman Empire’s army. Nothing about Neuralink. Nothing about AI. The DarkWeb shed no answers either.

Who were they? Was it all a joke?
Harrison sighed. Today had been a long one. He looked up at Alice, swimming in her impervious bubble of glass and water. He recalled the main character of a YouTube show once lamenting to his wife, “I wish I were a farmer. I wish I had grown up in a simple life, with simple ambitions, and simple desires. Take away the complexities of this modern life, and hand me a plow. I’d be happier for it”. Indeed, ignorance appeared to be a blissful blessing.
The day’s troubles swirled around the cauldron of Harrison’s brain. He undressed, brushed his teeth, and stared out the window. His reflection gazed back at him, and Jacob’s final question echoing like a lonely, repetitive drum.
What is human?
Crawling into bed, Harrison stole one last glance into his technical world before retreating to his unconscious. There were only two notifications, both emails.

  11:50 pm – From Neuralink
  

  Your wait is over! Next week’s operation


  11:59 pm – From Anonymous
  

  Welcome to Legion



Rick and Morty: Season 4
🦉 olshansky 🦁 — Sat, 26 Sep 2020 22:14:06 GMT
Olshansky's review of Rick and Morty: Season 4


Some evergreen insights from a survey on makers in Italy
Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 26 Sep 2020 21:00:00 GMT
Spoiler: they are not about technology.


Fixing Hugo Social Media Icons in Blackburn
Greg Herlein — Sat, 26 Sep 2020 11:00:04 GMT
The Blackburn theme for Hugo has a bug in how it shows social media icons.  Here’s how to fix it.


Hello Ubuntu - Kinda
Greg Herlein — Sat, 26 Sep 2020 11:00:01 GMT
Debian not good for me on the desktop - back to Ubuntu, lost the snaps.


Raspberry Pi Zero W Fun
Greg Herlein — Sat, 26 Sep 2020 11:00:01 GMT
The Raspberry Pi Zero W is likely the basis of my next few projects.


Creating Notes Site
Prasham H Trivedi — Sat, 26 Sep 2020 09:17:53 GMT
In this article, we will see how I built my notes site using a simple node script and deployed using Github actions.


Industrial Strength Arch Linux
Tyblog — Sat, 26 Sep 2020 09:00:00 GMT

These days, I almost exclusively run Arch Linux in my homelab and personal machines.
Had I the brain cells to spare, I’d try and get NixOS running on ARMv7, but in the meantime, a mix of vanilla x86_64 Arch and Arch Linux ARM is my chosen flavor just to keep things consistent.
I’ve run Arch as my primary server OS for almost a decade now, and although I’m sure some will balk at the idea, I’ve found that the distribution has performed wonderfully for me, even in contrast to traditional “server” distributions like CentOS.
It sounds counterinuitive, but the simple model of Arch Linux has, overall, helped mitigate some maintenance burdens.





Here’s some of best practices I’ve accumulated over the years of personally administering ~20 Arch Linux machines in my homelab.


Prelude: Why Arch?


I don’t want to get into a distribution holy war here; there’s endless forums for that debate.
However, I will get into the reasons why I now default to Arch Linux when provisioning hosts, as many of those reasons naturally flow into why I include certain best practices.



A “rolling release” distribution is a nice methodology for server maintenance.
See this other blog post enumerating why rolling releases make sense within the context of openSUSE.
Want a “version”? Use the Arch Linux Archive to pinpoint a moment in time.
(I still begrudge that CentOS 6’s upgrade path to 7 discourages in-place upgrades)
In contrast to “kitchen sink” distributions, when I do need to bolt on additions to my install, I know exactly how I’ve done it, so fixing problems and building systems isn’t a game of guesswork about how the distribution chose to configure those services.
I’m the one that did it, so I can better maintain it.
The catalog of packages, when you factor in the AUR, is almost complete for most people.
When I can’t find an Arch package, which is very seldom these days, I package it on the AUR and move on - PKGBUILDs aren’t terribly difficult to write.
The aforementioned simplicity of Arch systems is also reflected in the Arch-specific software projects like pacman.
Setting up a private pacman repo within my LAN was so easy I thought I was doing something wrong.




Pacman Hygiene


One of the trickiest aspects of Linux distribution maintenance that I didn’t understand was keeping configuration files up to date.
By that I mean catching and updating files somewhere like /etc/ so that when package foobar upgrades from 1.0 to 2.0 and the default configuration file /etc/foobar.conf adds the setting telemetry = true, you can merge in the new configuration files and optionally set them to desired values.



The Arch Wiki has a good page about this but the most important thing to remember is that, in the vast majority of cases, an interactive process (via a human at a terminal) is needed to judge which parts of your existing configuration file needs to be edited.
I suggest using pacmatic for this.
The pacmatic web site outlines how to use the utility, but in short: pacmatic will a) alert you to any announcements from the Arch Linux project in case there’s news about needed intervention in the course of normal system operations and b) automatically prompt and help update configuration files.
I like this method because it’s closely linked to pacman operations so you can’t forget to update these files and the pacmatic command is a drop-in replacement for pacman.



pacmatic is starting to age a little, but you can still get the same effect by using the pacdiff command from the pacman-contrib package.




AUR Automation


Using Arch for non-trivial use-cases is almost certainly going to lead you towards relying on some AUR packages.



Unlike official packages that are curated by Arch Linux Trusted Users, the AUR has less control surrounding submission of packages.
Validating the legitimacy of a PKGBUILD is always a good idea, and for first-time users, building and installing an AUR package is typically comprised of a) downloading the package snapshot, checking out the PKGBUILD, and building it with makepkg.



That’s all well and good, but if you use a non-trivial number of AUR packages, this process is way too manual.
You really want some automation in place around this, but I’m not talking about a tool like yay that automates the fetching and building for a package’s source.
You need a tool that handles all update steps as part of a schedule so that installing updated AUR packages is inlined as part of the normal -Syu process.



In my opinion, the most reliable option here is aurto (which, in turn, uses aurutils under the hood).
Why should you use something like aurto?



Scheduled updates. aurto comes bundled with systemd timers to check for package updates, so you don’t burn time checking for updates and building packages manually.
Custom repositories. Following the Arch paradigm of building and indexing packages into proper repositories means that you don’t think about pulling in updates, they just happen for you. You can also easily share built packages across a network (which I rely on heavily).
aurutils follows lots of good operational hygiene (for example, building in clean chroots and explicitly trusting maintainers as a build step).



Remember that, like any sort of AUR automation, you’re trusting the maintainer of the packages you track to not bundle malicious code, but that’s a risk you take with any sort of interaction with the AUR.




Snapshotting


If you’re using Linux in 2020, there’s no reason why you can’t be using a modern, advanced, copy-on-write filesystem.
Are you comfortable relying on dkms?
Then use ZFS on Linux.
Honestly, it’s fantastic, and you get copy-on-write, snapshots, compression, and more.
It’s the current best-of-breed filesystem for Linux, in my opinion.
If you want, btrfs can fit a similar role, but I trust ZFS more.



Providing a point-in-time backup before system upgrades becomes extremely easy on these filesystems.
Run a snapshot as a pacman hook before you make changes, and you can rollback to a known good state easily.
There are some ready-made packages out there that stitch together ZFS/btrfs with pacman hooks so you don’t even need to write the automation yourself.



If you’re already snapshotting a filesystem, cheap (as in effort) backups are just a small extra step if you want to replicate the snapshot images somewhere else.
My ZFS and btrfs filesystems get duplicated offsite over simple ssh commands as added disaster recovery insurance, which means that not only do you have system backups saved outside your network, but your whole history of backups if you need to restore from a point in time.
There are actually a wide variety of projects that do this, but personally, I use zfs-snap-manager for my zpools and snazzer for my btrfs volumes.
Both automate retention and incremental backups, which are the tricky parts.



Note: Some of these suggestions eat up disk space.
I’d definitely suggest using filesystem compression to help with that.




Grab Bag


The preceding three points have had the greatest impact on my Arch Linux system maintainability, but there are a few minor mentions just for sake of completeness:



Remember to run pacman -Sc every once in a while.
Those old packages take up space.
A periodic reflector script can be handy in keeping your mirrorlist healthy.

Using aurto and aurutils to host a repository of local AUR packages? Share it on your network easily:

shell
sudo pacmatic -S darkhttpd
sudo systemctl edit darkhttpd


Systemd
[Service]
ExecStart=
ExecStart=/usr/bin/darkhttpd /var/cache/pacman --ipv6 --uid http --gid http --chroot --mimetypes /etc/conf.d/mimetypes


shell
sudo systemctl enable --now darkhttpd



Then just add the following section to your other machines’ pacman.conf:

Conf[Unix]Font used to highlight variable names.
Font used to highlight type and class names.

[aurto]
Server = http://<hostname>/aurto
SigLevel = Optional





Edits: rhysperry111 caught a systemctl typo, and AladW noted I conflated aurto and aurutils in a few places.





Physics textbooks for self-study
diff × weyl — Sat, 26 Sep 2020 07:16:59 GMT
Here are some physics textbooks that I’ve read over the years. Each textbook is rated from 1-5 Diracs () on quality for self-study. Most topics are divided into (basic) and (advanced). Tips for self-study: Shorter is better when it comes to textbooks. The problem with self-study is missing the forest for the trees. Most textbooks … Continue reading "Physics textbooks for self-study"


Multi-repos KISS
Joseph Price — Sat, 26 Sep 2020 03:00:00 GMT
I have never worked in a monorepo professionally.


Europe and Western Civilization in the Modern Age
🦉 olshansky 🦁 — Sat, 26 Sep 2020 03:00:00 GMT
Olshansky's review of Europe and Western Civilization in the Modern Age by Thomas Childers


Making this blog bilingual…?
The Grey Shadow — Fri, 25 Sep 2020 22:35:04 GMT
As you might know, I’m a native Spanish speaker, and this is an English-language blog. It has served me well for over 15 years and counting, and still does. However, most of my writing have been done lately in Spanish. Not because of anything special; they were just extended status updates in Facebook. But somehow...


Against possessionsless utopias
Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 25 Sep 2020 21:00:00 GMT
How can you distinguish utopias from dystopias?


LinkedBlockingDeque 使用与分析
张可的博客 — Fri, 25 Sep 2020 18:16:27 GMT
LinkedBlockingDeque
LinkedBlockingDeque 是基于链表的双端阻塞队列，线程安全，元素不允许为 null。

空间容量最大一般为Integer.MAX_VALUE，如果构造器中指定了最大值则队列长度将会被限制在该值以下。

大部分方法都以固定时间运行，批量操作，例如：remove， removeFirstOccurrence，removeLastOccurrence，contains，iterator.remove()，将以线性时间运行。

LinkedBlockingDeque 是阻塞队列，是指对于一些指定的操作，在插入或者获取队列元素时如果队列状态不允许该操作可能会阻塞住该线程直到队列状态变更为允许操作，这里的阻塞一般有两种情况。

第一种是插入元素时，如果当前队列已满将会进入阻塞状态，一直等到队列有空的位置时再讲该元素插入，该操作可以通过设置超时参数，超时后返回 false 表示操作失败，也可以不设置超时参数一直阻塞，中断后抛出InterruptedException异常。

第二种是读取元素时如果当前队列为空会阻塞住直到队列不为空然后返回元素，同样可以通过设置超时参数。

一般多用于生产者消费者模式。

方法
LinkedBlockingDeque 实现了 BlockingDeque 接口，除了原本双端队列中的方法外还另外提供了一些阻塞操作的方法。每种操作分为阻塞与超时两种，阻塞方法调用将会一直阻塞知道队列可用，中间如果中断则抛出InterruptedException异常。超时方法同样也会阻塞，允许设置超时时间，超时后返回 false 表示操作失败。

下面我们先看下这几个方法。

head操作：


  
    
      类型
      阻塞
      超时
    
  
  
    
      插入
      putFirst(e)
      offerFirst(e, time, unit)
    
    
      移除
      takeFirst()
      pollFirst(time, unit)
    
  


tail操作：


  
    
      类型
      阻塞
      超时
    
  
  
    
      插入
      putLast(e)
      offerLast(e, time, unit)
    
    
      移除
      takeLast()
      pollLast(time, unit)
    
  


源码
我们先看看类中定义的几个重要的属性：
private transient int count;//当前链表长度
private final int capacity;//链表最大长度
final ReentrantLock lock = new ReentrantLock();
private final Condition notEmpty = lock.newCondition();
private final Condition notFull = lock.newCondition();

capacity表示链表的最大长度，默认是Integer.MAX_VALUE，可以在构造器中指定。

下面的lock是可重入锁，用于在读取或修改链表时使用，保证线程安全。

notEmpty表示不为空时的Condition，当不为空时会调用notEmpty.signal()通知被notEmpty阻塞的线程。notFull是未满状态的Condition，用法同上。

类中的方法可分为三类，第一类是只保证线程安全，但不会阻塞住的方法，第二是会阻塞方法，第三是超时方法。下面我们通过几个典型的方法看看是如何实现的。

通过阅读源码发现，不管是上面三种的哪一种，最终都会调用同一个方法先对节点进行操作，只不过每个方法内部有不同的处理，以读取head节点为例，主要有三个典型的方法：

  pollFirst()：保证线程安全，但不会阻塞
  takeFirst()：阻塞方法
  pollFirst(timeout, unit)：超时方法


上面是三个最终都先调用了unlinkFirst()移除head节点，那么我们先看看这个。
private E unlinkFirst() {
    // assert lock.isHeldByCurrentThread();
    Node<E> f = first;
    if (f == null)
        return null;
    Node<E> n = f.next;
    E item = f.item;
    f.item = null;
    f.next = f; // help GC
    first = n;
    if (n == null)
        last = null;
    else
        n.prev = null;
    --count;
    notFull.signal();
    return item;
}

上面代码其实就是一个移除head节点的操作，重点是最后会调用notFull.signal()通知被notFull阻塞的线程，例如我们使用阻塞/超时方法添加元素时（例如putFirst）如果队列已满则会进入阻塞状态，当有其他线程调用了 remove 操作之后此时队列有了空缺的位置，就可以通知阻塞线程继续将元素添加进去。

pollFirst
下面我们再看看pollFirst()是如何实现线程安全的：
public E pollFirst() {
    final ReentrantLock lock = this.lock;
    lock.lock();
    try {
        return unlinkFirst();
    } finally {
        lock.unlock();
    }
}

这个方法其实很简单，先用ReentrantLock锁定之后对链表进行操作，操作完成后在解锁，然后返回数据，操作完成。

takeFirst
takeFirst()大概是阻塞队列中最常用的一个方法，用于移除并返回head元素，队列为空则阻塞调用线程直到可以返回数据。
public E takeFirst() throws InterruptedException {
    final ReentrantLock lock = this.lock;
    lock.lock();
    try {
        E x;
        while ( (x = unlinkFirst()) == null)
            notEmpty.await();
        return x;
    } finally {
        lock.unlock();
    }
}

这里阻塞方式是通过自旋的方式持续从unlinkFirst获取head节点，空则调用notEmpty.await()阻塞，signal之后再重复这一步骤，直到获取到值为止。

另外一点，该方法也会先调用lock.lock()获得锁，然后继续操作，这点我开始比较疑惑，因为该方法会阻塞住，如果获得了锁别的线程不就没法获得锁从而操作队列了吗，查找资料之后才发现，notEmpty.await()方法出于线程调度目的，该锁会被原子释放，线程进入阻塞状态，因此其他线程可以正常获得锁并继续操作。

pollFirst(timeout, unit)
与takeFirst不用的是，pollFirst可以设置一个超时时间，当阻塞时间超过该时间之后将会从阻塞状态中恢复并返回 null。
    public E pollFirst(long timeout, TimeUnit unit)
        throws InterruptedException {
        long nanos = unit.toNanos(timeout);
        final ReentrantLock lock = this.lock;
        lock.lockInterruptibly();
        try {
            E x;
            while ( (x = unlinkFirst()) == null) {
                if (nanos <= 0)
                    return null;
                nanos = notEmpty.awaitNanos(nanos);
            }
            return x;
        } finally {
            lock.unlock();
        }
    }

跟takeFirst()基本类似，不过调用的是notEmpty.awaitNanos(nanos)方法设置超时时间，而且每次循环都会检测一遍是否超时，超时则直接返回 null 值。

还有一个不同点，pollFirst获取锁的方式是调用lock.lockInterruptibly()方法，该方法与lock不同的是，它允许等待线程被中断，中断后直接抛出InterruptedException异常，而调用lock方法等待线程即使中断仍然会继续等待获取锁，只不过获取成功后会中断线程。


The Least Dangerous Game: How (And Why) To Hunt Miners In Eve Echoes
Greater Dan Or Equal To — Fri, 25 Sep 2020 06:24:34 GMT
Eve Echoes is the mobile version of the MMORPG Eve Online.  In it, you play as a Capsuleer, a cloned human that can pilot space ships all around a giant galaxy, mining, hunting pirates, or being a pirate yourself.  If you get blown up, you lose your ship,


How to make a live wallpaper on Android
Salvatore's Blog — Fri, 25 Sep 2020 03:00:00 GMT
Let’s create a very very very basic live wallpaper on Android.

Background on Backgrounds
Android has had live wallpapers since Android 2.1 (Eclair, API 7) which was released in January of 2010 ¹. If you were using Android back then, you might have remembered this gem of a wallpaper.²



Step 0 - Prep an Android project
Create an Android project using the creation wizard. I used all of Android Studio’s default settings with Kotlin, and it went well. If you already have a project lying around, you could probably use that too.



Step 1 - Initialize the WallpaperService
The main class that runs the live wallpaper is the WallpaperService.Engine which is an inner class of WallpaperService.

Start by making an implementation of WallpaperService with an inner class  that implements WallpaperService.Engine. We’ll come back to it later.

class MyWallpaperService : WallpaperService() {
  override fun onCreateEngine(): Engine = WallpaperEngine()

  inner class WallpaperEngine : WallpaperService.Engine() {
  }
}

Step 2 - Modify your various XML files
For your app to provide the phone with your live wallpaper, you’ll need to make a resource for it and declare your WallpaperService in your AndroidManifest.xml file.

Make an XML file in your resources folder main/res/xml/my_wallpaper.xml. It only really needs a <wallpaper /> tag, but this is where you can define a thumbnail, settings activity, and other neat things.

<?xml version="1.0" encoding="utf-8"?>
<wallpaper />

Jump over to your AndroidManifest.xml file. You’ll need to let Android know that you’re using the live wallpaper API.

<?xml version="1.0" encoding="utf-8"?>
<manifest>
  <!--  ... some stuff ...  -->
  
  <uses-feature
      android:name="android.software.live_wallpaper"
      android:required="true" />

  <!--  ... other stuff  -->
</manifest>

In that same Android Manifest, you’ll also need to declare your WallpaperService to give it permissions to do wallpaper-y things.

<?xml version="1.0" encoding="utf-8"?>
<manifest>
  <application>
    <!-- ... some stuff ... -->

    <service
        android:name="MyWallpaperService"
        android:enabled="true"
        android:permission="android.permission.BIND_WALLPAPER">
      <intent-filter>
        <action android:name="android.service.wallpaper.WallpaperService" />
      </intent-filter>

      <meta-data
          android:name="android.service.wallpaper"
          android:resource="@xml/my_wallpaper" />
    </service>

    <!-- ... other stuff ... -->
  </application>
</manifest>

Step 3 - Launch your empty wallpaper
Now that the system knows about your fun (currently blank) wallpaper, it’s time to get it going! To wire it up, you need to start the WallpaperManager.ACTION_CHANGE_LIVE_WALLPAPER action with your WallpaperService. You can do this from within an OnClickListener, or anywhere that can start an Activity.

val intent = Intent(WallpaperManager.ACTION_CHANGE_LIVE_WALLPAPER)
intent.putExtra(
    WallpaperManager.EXTRA_LIVE_WALLPAPER_COMPONENT,
    ComponentName(this, MyWallpaperService::class.java)
)
startActivity(intent)

It should look something like this.



Step 4 - Have fun with ✨art✨

Go back to your WallpaperService and WallpaperService.Engine implementation from Step 1. We’ll just make changes in these classes for the rest of this tutorial.

Solid Background

To make a background, get the canvas, draw on it, and then post the update. In our first example, override the WallpaperService.Engine#onSurfaceCreated method.

class MyWallpaperService : WallpaperService() {
  override fun onCreateEngine(): Engine = WallpaperEngine()

  private inner class WallpaperEngine : WallpaperService.Engine() {

    override fun onSurfaceCreated(holder: SurfaceHolder) {
      val canvas = holder.lockCanvas()
      // ... do ✨art✨ stuff
      holder.unlockCanvasAndPost(canvas)
    }
  }
}

To set everything as one color, draw a Paint object with fill style and color of your choice.

override fun onSurfaceCreated(holder: SurfaceHolder) {
  val canvas = holder.lockCanvas()
  val paint = Paint().apply {
    color = Color.CYAN
    style = Paint.Style.FILL
  }
  canvas.drawPaint(paint)
  holder.unlockCanvasAndPost(canvas)
}



It’s (a)live!

To show the wallpaper updating as we touch our device, we’re going to override the WallpaperService.Engine#onTouchEvent callback. We want to only update when we actually touch our device and not when we also lift our finger up, so we’ll filter for the MotionEvent.ACTION_DOWN action.

class MyWallpaperService : WallpaperService() {
  override fun onCreateEngine(): Engine = WallpaperEngine()

  private inner class WallpaperEngine : WallpaperService.Engine() {

    override fun onTouchEvent(event: MotionEvent?) {
      // on finder press events
      if (event?.action == MotionEvent.ACTION_DOWN) {
        // get the canvas from the Engine or leave
        val canvas = surfaceHolder?.lockCanvas() ?: return
        // ... do ✨art✨ stuff
        // update the surface
        surfaceHolder.unlockCanvasAndPost(canvas)
      }
    }
  }
}

To show the touch events are working, let’s change the color to a random one every time. Here’s a quick and easy way to pick a random color between #000000 and #FFFFFF.

// value between #000000 and #FFFFFF
val randomColor = Random.nextInt(16_777_216)
  // convert it to hex/base 16
  .toString(16)
  // gurantee it's six characters long
  .padStart(6, '0')
  // still need to prefix with a '#',
  // but you get the idea.

And when we put it all together…

class MyWallpaperService : WallpaperService() {
  override fun onCreateEngine(): Engine = WallpaperEngine()

  private inner class WallpaperEngine : WallpaperService.Engine() {

    override fun onTouchEvent(event: MotionEvent?) {
      if (event?.action == MotionEvent.ACTION_DOWN) {
        val canvas = surfaceHolder?.lockCanvas() ?: return

        val paint = Paint().apply {
          val randomColor = Random.nextInt(16_777_216)
            .toString(16)
            .padStart(6, '0')
          color = Color.parseColor("#$randomColor")
          style = Paint.Style.FILL
        }
        canvas.drawPaint(paint)

        surfaceHolder.unlockCanvasAndPost(canvas)
      }
    }
  }
}

… we get a live wallpaper! Here’s a bunch of colors that come up randomly on my first few taps.³



Keep going!

Now that you have a basic live live wallpaper, try things out! Lars Vogel also wrote a nice tutorial which I learned from before writing this. Their post has more on using a Handler and Runnable which will be useful if you want your live wallpaper to move continuously. If you end up making anything (including this example), please record it and share it with me on Twitter.

Thanks for your time, and I hope you enjoyed this post!




  
    
      The Wikipedia article for Android incorrectly lists Android 2.0 (API 5) as the origin of the live wallpaper. However, if you read the release notes for Android 2.0 (API 5) and Android 2.1 (API 7), you’ll see what I’m talking about. ↩
    
    
      I couldn’t find an emulator running Android 2.1, but I did find Nexus Revamped, a strikingly similar wallpaper. ↩
    
    
      To make a tiled image like this, I installed ImageMagick and ran the following montage command: montage folder-with-screenshots/* -tile 7x4 -geometry +40+40 -background none tiled-wallpaper-montage.png ↩
    
  



searchmysite.net update: Seeding and scaling
Michael Ian Lewis — Fri, 25 Sep 2020 03:00:00 GMT
Note: this post is now on blog.searchmysite.net at https://blog.searchmysite.net/posts/searchmysite.net-update-seeding-and-scaling/.

  Introduction
  
    
    Link to heading
  

It has been just over 2 months since I launched https://searchmysite.net. I’ve had some good feedback from the IndieWeb community in that time, and made some key changes as a result, so thought it was time for an update. You may still want to refer to searchmysite.net: Building a simple search for non-commercial websites for the original overview.


Burn Out at the Hydrogen Bar
Macleod Sawyer | Web Portal Feed — Thu, 24 Sep 2020 23:39:00 GMT



ConcurrentLinkedDeque 使用与分析
张可的博客 — Thu, 24 Sep 2020 18:16:27 GMT
ConcurrentLinkedDeque
ConcurrentLinkedDeque 是基于链表的无限双端队列，线程安全，不允许 null 元素。

ConcurrentLinkedDeque 内部通过 CAS 来实现线程同步，一般来说，如果需要使用线程安全的双端队列，那么推荐使用该类。

由于双端队列的特性，该类同样可以当做栈来使用，所以如果需要在并发环境下使用栈，也可以使用该类。

迭代器设计为弱一致性的（weakly consistent），此外还可以通过descendingIterator方法获取一个通过相反方向遍历的迭代器。

虽然跟 LinkedList 一样都是双端队列的链表实现，但由于其并发特性，导致无法简单点的通过计数来确定队列的长度，所以size方法将会以线性时间运行，并且如果在执行期间被其它线程修改可能返回不准确的结果。

对于批量操作，例如：addAll，removeAll，retainAll，containsAll，equals,toArray来说，不能保证这些操作的原子性。

方法

ConcurrentLinkedDeque 作为队列（FIFO）使用时的方法：


  
    
      队列方法
      等效的双端队列方法
    
  
  
    
      add(e)
      addLast(e)
    
    
      offer(e)
      offerLast(e)
    
    
      remove()
      removeFirst()
    
    
      poll()
      pollFirst()
    
    
      element()
      getFirst()
    
    
      peek()
      peekFirst()
    
  


ConcurrentLinkedDeque 作为堆栈（FILO）使用时的方法：


  
    
      堆栈方法
      等效的双端队列方法
    
  
  
    
      push(e)
      addFirst(e)
    
    
      pop(e)
      removeFirst(e)
    
    
      peek()
      peekFirst()
    
  


源码
因为 ConcurrentLinkedDeque 是基于链表的数据结构，我们先看看该类中使用的链表节点 Node 的相关代码。
Node
static final class Node<E> {
    volatile Node<E> prev;
    volatile E item;
    volatile Node<E> next;

    private static final sun.misc.Unsafe UNSAFE;
    private static final long prevOffset;
    private static final long itemOffset;
    private static final long nextOffset;

    static {
        try {
            UNSAFE = sun.misc.Unsafe.getUnsafe();
            Class<?> k = Node.class;
            prevOffset = UNSAFE.objectFieldOffset
                    (k.getDeclaredField("prev"));
            itemOffset = UNSAFE.objectFieldOffset
                    (k.getDeclaredField("item"));
            nextOffset = UNSAFE.objectFieldOffset
                    (k.getDeclaredField("next"));
        } catch (Exception e) {
            throw new Error(e);
        }
    }
}


上半部分代码就是一个标准的 Node 节点的定义，我们主要看静态初始化块中的代码。

首先介绍一下 Unsafe 类，关于这个类的详细信息及使用可以看美团技术的这篇文章，这里大概介绍一下，Unsafe 中提供了一些执行级别低，不安全的方法，例如内存操作以及待会会看到的 CAS，一般来说不推荐使用，因此命名为 Unsafe。

Node 内部类中定义的三个 offset 变量：prevOffset、itemOffset、nextOffset 分别对应 prev、item、next 这三个变量的内存偏移量（相对内存地址），通过这三个偏移量可以拿到实际的内存中的值，这个后面会看到怎么用。

然后静态初始化块中获取了 Unsafe 实例，接着对三个 offset 变量进行初始化。

下面再来看看 Node 中其它几个方法：
Node(E item) { UNSAFE.putObject(this, itemOffset, item); }
boolean casItem(E cmp, E val) {
    return UNSAFE.compareAndSwapObject(this, itemOffset, cmp, val);
}
void lazySetNext(Node<E> val) {
    UNSAFE.putOrderedObject(this, nextOffset, val);
}
boolean casNext(Node<E> cmp, Node<E> val) {
    return UNSAFE.compareAndSwapObject(this, nextOffset, cmp, val);
}
void lazySetPrev(Node<E> val) {
    UNSAFE.putOrderedObject(this, prevOffset, val);
}
boolean casPrev(Node<E> cmp, Node<E> val) {
    return UNSAFE.compareAndSwapObject(this, prevOffset, cmp, val);
}

第一个构造方法意思就是将参数中的对象设置到指定的地址偏移量处。

下面的几个方法分为两类，一类是 cas 开头的操作一类是 lazySet 操作，先说下 cas 操作，例如casItem方法，其作用是将对应的 item 值替换为参数 val 的值，该方法会先对比内存中的值与 cmp 参数值是否相等，相等则替换，返回值表示操作是否完成，这里调用了UNSAFE.compareAndSwapObject方法来保证并发环境下的安全问题。

而lazySetNext方法则是直接将参数 val 设置到内存中，调用了UNSAFE.putOrderedObject方法。

上面我们已经介绍了 Node 类中的代码，其中已经实现了部分多线程同步代码，毕竟对链表的操作本质上都是对节点的操作，虽然其他还有很多方法，但基本都是层层嵌套调用，最终可能都会调用到那几个重点方法上面，下面我们看看这几个重点。

linkFirst
linkFirst方法用于将指定的节点添加到链表头位置，例如addFirst、offerFirst最终都是调用了该方法。
private void linkFirst(E e) {
  checkNotNull(e);
  final Node<E> newNode = new Node<E>(e);
  restartFromHead:
  for (;;)
    for (Node<E> h = head, p = h, q;;) {
      if ((q = p.prev) != null &&
              (q = (p = q).prev) != null)
        // Check for head updates every other hop.
        // If p == q, we are sure to follow head instead.
        p = (h != (h = head)) ? h : q;
      else if (p.next == p) // PREV_TERMINATOR
        continue restartFromHead;
      else {
        // p is first node
        newNode.lazySetNext(p); // CAS piggyback
        if (p.casPrev(null, newNode)) {
          // Successful CAS is the linearization point
          // for e to become an element of this deque,
          // and for newNode to become "live".
          if (p != h) // hop two nodes at a time
            casHead(h, newNode);  // Failure is OK.
          return;
        }
        // Lost CAS race to another thread; re-read prev
      }
    }
}

上面代码先开启了一个死循环用于自旋，第二个循环用于在每次自旋时找到head节点，因为多线程环境下head节点或者说链表随时可能被修改，所以每次都需要重新获取。

找到head节点后，也就是p变量，开始进行替换操作，将newNode设置为head节点。先调用newNode.lazySetNext(p)将原来的head节点设置为newNode的next节点，然后开始 cas 操作，调用p.casPrev(null, newNode)将原本head节点的prev节点设置为newNode。casPrev的返回值表示是否操作成功，操作失败则继续下一次自旋重试，直到操作成功为止。

好了源码就说这么多吧，其他几个方法也都大同小异，大差不差，这个明白了其他的也就明白了。


For non-westerners, it's harder to write in Wikipedia
Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 24 Sep 2020 11:00:00 GMT
Did you already know that?


Skiing bookmarks
xenodium.com @alvaro — Thu, 24 Sep 2020 03:00:00 GMT

7 far-flung European ski resorts - Lonely Planet.



Trigonometry Graphs with Octave
Thoughts by Thomas Derflinger — Thu, 24 Sep 2020 03:00:00 GMT
Octave is a free and open source scientific computing platform. It is especially suited for mathematical operations and has a simple mechanism to plot the data. Trigonometry visualizations are especially powerful with Octave.


Even if tech is addictive, abstinence is no solution
Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 23 Sep 2020 21:00:00 GMT
Especially (but not only) for kids?


A Simple and Flexible Pytorch Video Pipeline
Machine Learning Articles on paulbridger.com — Wed, 23 Sep 2020 19:21:00 GMT
Taking machine learning models into production for video analytics doesn’t have to be hard. A pipeline with reasonable efficiency can be created very quickly just by plugging together the right libraries. In this post we’ll create a video pipeline with a focus on flexibility and simplicity using two main libraries: Gstreamer and Pytorch.


REST Endpoint + CRUD with AWS Lambda and DynamoDB in 2 minutes
swyx's site RSS Feed — Wed, 23 Sep 2020 17:31:01 GMT
A lightning fast overview of everything you need to know to set up a REST endpoint with full CRUD capability with AWS Lambda, DynamoDB and AWS Amplify in 2 minutes.


ArrayDeque 使用与分析
张可的博客 — Wed, 23 Sep 2020 13:00:03 GMT
ArrayDeque
ArrayDeque 是 Java 集合中双端队列的数组实现，双端队列的链表实现（LinkedList）我们在前几篇文章中讲过了。

ArrayDeque 几乎没有容量限制，设计为线程不安全的，禁止 null 元素。

ArrayDeque 作为栈使用时比 Stack 类效率要高，作为队列使用时比 LinkedList 要快。

ArrayDeque 大多数的额操作都在固定时间内运行，例外情况包括 remove，removeFirstOccurrence，removeLastOccurrence，contains，iterator.remove()，和批量操作，这些将以线性时间运行。

iterator同样也被设计为 fail-fast。

方法
ArrayDeque 作为 队列（FIFO） 使用时的方法：


  
    
      队列方法
      等效的双端队列方法
    
  
  
    
      add(e)
      addLast(e)
    
    
      offer(e)
      offerLast(e)
    
    
      remove()
      removeFirst()
    
    
      poll()
      pollFirst()
    
    
      element()
      getFirst()
    
    
      peek()
      peekFirst()
    
  


ArrayDeque 作为 堆栈（FILO） 使用时的方法：


  
    
      堆栈方法
      等效的双端队列方法
    
  
  
    
      push(e)
      addFirst(e)
    
    
      pop(e)
      removeFirst(e)
    
    
      peek()
      peekFirst()
    
  


源码
我们先看看代码中最重要的三个变量：
transient Object[] elements;
transient int head;
transient int tail;

我们上面也说了 ArrayDeque 是双端队列的数组实现，上面代码可看到确实定义了一个用于存储链表数据数组：elements，此外还定义了head及tail用于描述链表在数组中的头尾位置。另外说明一下，elements长度始终未 2 的次方。

这里需要注意一点，用elements存储的是双端队列，head和tail参数表示双端队列的头和尾的索引，但并不意味着head值永远小于tail，当tail移动至数组末尾，但队列长度小于数组时（意味着此时head大于0），再向队列末尾添加数据将会使tail移动至数组的开头。

假设下图为当前的elements数组，黄色方块表示其中有数据。


当我们向队列末尾添加一个元素时，数组变成了下面这样：



目前都是按照我们预期发展的，现在我们来调用poll方法移除队列头部的一个元素，此时elements变成了下图：


这个时候因为我们移除了队列的头部元素，数组的开头已经空下来了一个位置，这时候再向队列末尾追加一个元素。


可以看到，此时head在tail的右侧，ArrayDeque为了不浪费数组空间进行了这样的设计，也不难理解，我们继续向下看。

现在看一下构造器：
public ArrayDeque() {
    elements = new Object[16];
}

public ArrayDeque(int numElements) {
    allocateElements(numElements);
}

public ArrayDeque(Collection<? extends E> c) {
    allocateElements(c.size());
    addAll(c);
}

private static int calculateSize(int numElements) {
    int initialCapacity = MIN_INITIAL_CAPACITY;
    if (numElements >= initialCapacity) {
        initialCapacity = numElements;
        initialCapacity |= (initialCapacity >>>  1);
        initialCapacity |= (initialCapacity >>>  2);
        initialCapacity |= (initialCapacity >>>  4);
        initialCapacity |= (initialCapacity >>>  8);
        initialCapacity |= (initialCapacity >>> 16);
        initialCapacity++;

        if (initialCapacity < 0)   // Too many elements, must back off
            initialCapacity >>>= 1;// Good luck allocating 2 ^ 30 elements
    }
    return initialCapacity;
}

private void allocateElements(int numElements) {
    elements = new Object[calculateSize(numElements)];
}


默认构造器会将elements初始化为一个长度为 16 的数组，另外两个构造器将根据参数的长度来初始化，最终会调用calculateSize方法计算初始长度。

calculateSize方法就比较有意思了，其中用了很多位运算（虽然我们日常用的很少，但 JDK 中随处可见位运算），最终结果就是控制输出的initialCapacity参数保持在 2 的次方。

下面我们看看添加元素时的源码，添加元素到双向链表最后都会进入addLast方法中：

public void addLast(E e) {
    if (e == null)
        throw new NullPointerException();
    elements[tail] = e;
    if ( (tail = (tail + 1) & (elements.length - 1)) == head)
        doubleCapacity();
}

上面代码的意思就是把元素添加到队列末尾，并移动tail的值，根据上面的介绍，tail移动并不会单纯的向后移动，而是会根据当前elements数组的空余长度进行移动，但这里移动tail的代码写的很巧妙：
tail = (tail + 1) & (elements.length - 1)

我们上面说过，因为elements的长度永远是 2 的次方，所以elements.length - 1的二进制位都是 1，对其进行 & 运算相当于对elements.length进行求余，所以下面两个公式是相等的，不过这种关系必须保证elements长度是 2 的次方才成立：
(tail + 1) & (elements.length - 1)
(tail + 1) % elements.length

由于位运算的效率比算数运算要高很多，所以 JDK 中很多类似的应用。

好了，回到上面的addLast方法中去，该方法回判断tail == head是否成立，相等则表示数组已满，满了就进行扩容，也就是调用doubleCapacity()方法。
private void doubleCapacity() {
    assert head == tail;
    int p = head;
    int n = elements.length;
    int r = n - p; // number of elements to the right of p
    int newCapacity = n << 1;
    if (newCapacity < 0)
        throw new IllegalStateException("Sorry, deque too big");
    Object[] a = new Object[newCapacity];
    System.arraycopy(elements, p, a, 0, r);
    System.arraycopy(elements, 0, a, r, p);
    elements = a;
    head = 0;
    tail = n;
}

扩容操作会创建一个新的数组，长度为原数组的两倍，然后通过两次复制操作将原数组复制到新的数组中。

好了，关于 ArrayDeque 就介绍这么多了，其他的方法大体上都是类似的。


Understanding Integration Testing in React
Max Rozen — Wed, 23 Sep 2020 12:48:48 GMT
Moving from Enzyme to React Testing Library is rough. You can't do a lot of things you used to, and there are new best practices. Let's learn them!


Lifesaver tools: Rule converter
The Grey Shadow — Wed, 23 Sep 2020 08:53:59 GMT
Hi again! It’s been a long time. I know… and there’s lot of things to share and get to speed. But that can wait for now. Today I would like to talk about a tool which I classify as a lifesaver: something which really takes a lot of meaningless worries and headaches from your daily...


About Paul Bridger
Machine Learning Articles on paulbridger.com — Wed, 23 Sep 2020 02:14:00 GMT
Hi. I’m Paul, and I like to write code, build products and start companies.
I started out in New Zealand, and I’ve been lucky enough to work with some very special people in many places around the world. I’ve often played a technical co-founder role, leading and growing engineering teams, and taking complex products from conception to production.
Over the last couple of decades I’ve built products in a wide range of industries, from publishing, 3D game development, investment banking, to iOS, big data and machine learning.


Reading algorithms make your world smaller
Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 22 Sep 2020 21:00:00 GMT
Of course. Why should you care of what happens elsewhere?


Notes on Simple Made Easy
Posts on rand[om] — Tue, 22 Sep 2020 03:00:00 GMT
These are my notes on the talks “Simple Made Easy” by Rich Hickey.
Simplicity is a prerequisite for reliability - Dijkstra
Simple:

One role.
One task.
One concept.
One dimension.
Lack of interleaving. (Does not mean there’s only one thing)

Easy:

“Near”. To our understanding, skill/set or capabilities.
Easy is relative.

We can only make reliable those things we can understand.
We can only consider a few things at a time. If a piece is intertwined with another we already have to keep both in mind. It’s combinatorial.


Shoe Dog: A Memoir by the Creator of Nike
🦉 olshansky 🦁 — Tue, 22 Sep 2020 03:00:00 GMT
Olshansky's review of Shoe Dog: A Memoir by the Creator of Nike by Phil Knight


On Blogging Versus Social Media
Ben Overmyer's Site — Tue, 22 Sep 2020 03:00:00 GMT
I read Rubenerd's post about seriousness, and it got me to thinking. Most people these days are content to post on social media. Very few have their own blogs, let alone a dedicated personal website.
But that's not the web I grew up with.
I've had many blogs over the years. They've been powered by different services or software, and had different focuses. But taken as a whole, they've painted a picture remarkably similar to modern social media posts - snapshots of my life at the time. Things that I've been thinking about, things I'm excited about, things I'm angry about.
There are two key differences, though. Blog posts tend to run a lot longer than social media posts, and I fully control the platform on which they appear.
Also, people who read blogs tend to be more willing to read longer content. On social media, the vast majority of users seem to only be interested in reacting to short sentences and images. There's nothing wrong with that, exactly, but it's not the kind of interaction I'm looking for when I write longer content.
The other thing that Rubenerd made me remember is back when I drew cartoon faces for my earliest websites to reflect my mood for any given update. Maybe I should resurrect that practice and give my site a little more playfulness.


Debug
ferrucc.io — Tue, 22 Sep 2020 01:15:44 GMT
Sometimes I come back home from work, stuck on something, no adults to ask help to and a deadline in the morning.
I fall asleep thinking: “Tomorrow is the day I’ll get caught. This will be the one problem I won’t be able to solve. The rest of my life is going to be misery and failure”
Every morning though, with more or less pain things get sorted.
Anyways here’s what you should do when you’re stuck:


Springer Handbook of Robotics (2nd Ed.)
Macleod Sawyer | Web Portal Feed — Mon, 21 Sep 2020 23:08:23 GMT

The second edition of this handbook provides a state-of-the-art overview on the various aspects in the rapidly developing field of robotics. Reaching for the human frontier, robotics is vigorously engaged in the growing challenges of new emerging domains. Interacting, exploring, and working with humans, the new generation of robots will increasingly touch people and their lives. The credible prospect of practical robots among humans is the result of the scientific endeavour of a half a century of robotic developments that established robotics as a modern scientific discipline. The ongoing vibrant expansion and strong growth of the field during the last decade has fueled this second edition of the Springer Handbook of Robotics



Cosmic Calamity (EP)
Macleod Sawyer | Web Portal Feed — Mon, 21 Sep 2020 23:01:49 GMT



Pegboard Mounts for my Anemometer and Infrared Temperature Gun
Zef Houssney — Mon, 21 Sep 2020 21:00:00 GMT
Modeling some pegboard mounts for a digital anemometer and infrared temperature gun.


How is FOIA doing in Italy?
Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 21 Sep 2020 21:00:00 GMT
Freedom of Information Act, you say? It could be more usable if…


Mobile homepage.html
voussoir.net/writing — Mon, 21 Sep 2020 07:27:16 GMT

Back to writing

[today_i_did_this]
Mobile homepage.html
I'm not satisfied with the new tab page on Android Chrome. So, I put together an html file that I can use as my own homepage. Here's a comparison:

I don't like the new tab page because:

You only get eight links.
You can't rename the links.
You can't rearrange the links.
I will never ever ever ever click to view the suggested content.
The web servers I run on my PC are on different ports, but they keep fighting for the same spot on the new tab page.

Unfortunately, new tabs still always go to the new tab page and I have to hit the home button. So while it's not as seamless as I'd like, I can live with the one extra tap.
Here's what I did:

Create the html file, which is as follows:
<!DOCTYPE html5>
<html>
<head>
    <title>Homepage</title>
    <meta charset="UTF-8"/>
    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<style>
body
{
    display: flex;
    flex-direction: column;
}
a
{
    display: flex;
    align-items: center;
    justify-content: center;
    margin: 8px 0;
    width: 100%;
    height: 50px;
    background-color: pink;
    border-radius: 8px;
    font-weight: bold;
    font-family: sans-serif;
}
div
{
    display: grid;
    grid-auto-flow: column;
    grid-gap: 8px;
}
</style>
</head>

<body>
    <a href="https://example.com">Example</a>
    <div>
        <a href="https://example.com">Example</a>
        <a href="https://example.com">Example</a>
    </div>
    <div>
        <a href="https://example.com">Example</a>
        <a href="https://example.com">Example</a>
        <a href="https://example.com">Example</a>
    </div>
</body>
</html>

Sync it between PC and phone with the amazing Syncthing.

Set it as the homepage using either file:///storage/emulated/0/<path>/homepage.html or file:///mnt/sdcard/<path>/homepage.html. Despite the name /sdcard/, this path should correspond to internal storage.
Note: With Android 10's new scoped storage permissions, you may get a permission denied error while opening the page. Your first option is to place the file inside the Android/data/com.android.chrome folder. E.g., an absolute path of file:///storage/emulated/0/Android/data/com.android.chrome/homepage.html.
Alternatively, I was able to bypass the scoped storage system altogether with this ADB command: adb shell sm set-isolated-storage off which rebooted the phone immediately and may have other consequences. I am not sure how it fares on Android 11.


Recently, I have been trying to use Android Firefox more, since it has extension support and thus uBlock. Astonishingly, Mozilla has removed the ability to set a homepage, which I believe occurred as part of the 2020-08-25 update. And their new tab page has an even bigger thing that I won't ever ever ever click on. I'm not sure what's going on over there.


View this document's history

2022-04-07 Archive mozilla link.
2021-06-07 Move S3 images to my domain.
2021-03-21 com.android.chrome folder is safe under scoped storage.
2021-03-13 Add scoped storage note and bypass.
2020-09-20 Add mobile_homepage_html.md.

Contact me: writing@voussoir.net
If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom



Using RDS Snapshots
Mon, 21 Sep 2020 03:00:00 GMT
Recently, I had a case where I needed to gain access to an RDS Instance that I had long since deleted. To add insult to injury, the Bastion host to gain access to the Database server had also been deleted and it's VPC, Security Groups and all the other architecture components! Yikes.
Fortunately, I had taken a final snapshot before it was deleted. Phew!
I thought it was as simple as downloading the snapshot, which I assumed was just an SQL Dump, import it into a local Database and then bobs your uncle. Not so fast... RDS Snapshots aren't just database dumps, after all that would just be too easy! Instead they are bespoke AWS format that is not able to be parsed and imported with a tool.
Despite there being documentation, it's fairly lacking so I thought I'd write a guide on how I gained access to the data within an RDS snapshot.
1. Create a new VPC
The first step is to create a new VPC. Name it what you like and give it any old CIDR Block. Leave everything else standard

  

2. Create a Security Group for the VPC
Next, you need to create a security group for the VPC you just created.
Simple add a name, then select your previously created VPC you created in step 1.
Next in the Inbound Rules pane, create an new inbound rule to allow traffic on Port 5432 (the PostgreSQL default port). The source should listed as "Your IP", which will prefill it on the current IP address you are accessing the AWS Console on.

  

3. Restore the RDS Instance
Now you've done that, go to RDS and then Snapshots.
Find the snapshot you're looking to get data from and click "Restore snapshot"



This will send you to a configuration page for the instance you are about to boot.
First, change the instance type to something cheaper because the default is something more powerful than Deep blue and costs about as much as a mission to the moon.
Afterwards, change the VPC that the instance is in, to the one created earlier and change the security group to the one created earlier. In my case "snapshot-access-vpc" and "rds-snapshot-sg". Additionally, select "Public Access".

  

Now hit Create and your instance will launch with the snapshot data.
Hey Presto!
You should not be able now connect to your RDS instance using the details on the Connectivity Pane when clicking on your RDS instance in the list. Hopefully you found this helpful!
Make sure to tear down this access after you finish using it as it is by no means secure. It should purely be used for grabbing data retrospectively if a client requests it.


Improving Koru's API Performance
Mon, 21 Sep 2020 03:00:00 GMT
If you don't know already - I love performance. It solves a genuine frustration for users and provides meaty problems to sink your teeth into. Koru's API performance was a little lacking since we had added a lot of features, rewritten large swaths of code and generally not thought about it, since our system is not dependant on returning results quickly. Never the less, we chose to do some performance improvement primarily aimed at reducing database load. This work was in preparation for moving to a multi-tenant architecture. When the move to a central database cluster is complete, reducing database load for each endpoint will be critical to ensure that we continue to have acceptable response times.
This work is still largely on going but here I'll discuss some of the foundation work we have done.
Gathering Data
Firstly, we started by collecting data on query performance in our system. We don't use an application performance monitoring, but this is simple enough by attaching onto the receive event on the PG Promise constructor. We then want to report queries that take over 500ms into Slack so that we can then focus our attention on them.
const pg = pgPromise({
  // {... other options }
  receive: async (data, result, e) => {
    winston.debug(
      `${e.query} received ${result.rowCount} row(s) in ${result.duration}ms`
    );

    if (result.duration && result.duration >= 500) {
      await Slack.reportSlowQuery(e.query, result.rowCount, result.duration);
    }
  },
});

This sends the query, the amount of rows it returned and the duration of the query to a Slack webhook which then produces a message like this:

	

Ok, so now we've got a steady stream of slow queries coming into our system! 🎉
Optimization
Quickly, we began to see a pattern. Most of the queries that were reported as slow contained two things:

A WHERE clause on the clients ID - added for prevention of accessing other clients resources
An ORDER BY of a property within a JSONB object

The first was a fairly easy change, simply add an index on the client ID column.
CREATE UNIQUE INDEX IF NOT EXISTS client_id_key ON clients USING BTREE (client_id);

I'm still unsure why we hadn't got an index here before, but I put it down to the fact that there was very little in the way of cross-client security. It was as if the system said "well you're authenticated so have what you want!". Thankfully, we have long since added these protections.
The second was a little more of a challenge. Initially, I began by looking up if it was possible to create an index on a certain property of a JSON structure or if you instead had to create just an index on the entire column.
Soon I discovered it was possible! But there was a lot of conflicting advise as to whether it actually improved query performance or not and if the building of the index netted a performance negative.
Anyway, I decided to create the index like this
CREATE INDEX candidate_response_score_indx ON responses((data->>'score'));

This took down the average query time from around 885ms to 7ms 🚀
Conclusion
There is still many more queries to improve the performance of but this was a great start. It taught me a lot about the internals of Postgres and experimentation.
Takeaways

Learn but don't be afraid to experiment - there becomes a point with learning that the best way to see if something will be a positive change or not, is to just try it out
"Reckons" don't always reflect reality - I believed there was lots of performance improvements to be made in a number of common operations but after trying the queries out and collecting data, they were already very performant. Therefore, be more data driven rather than "reckon" driven, even if those assumptions point you in the general direction.



[Computing] Serial Attached SCSI (SAS) is a circuit switched computer network
devever.net/~hl — Sun, 20 Sep 2020 22:31:00 GMT
(You may wish to read A brief introduction to SCSI before reading this article.)


A Bazel Persistent Worker for Rust
Nikhil's blog — Sun, 20 Sep 2020 22:09:41 GMT
Bazel persistent workers are a cool feature that allow Bazel to start up “compiler” instances that can accept multiple build requests. This brings benefits like saving startup time, saving the time to parse a standard library or share some cache across compiler invocations. This allows slight speedups in rebuilds, which can be valuable in speeding up the developer iteration cycle.
This is best exemplified in the existing persistent workers:
The Java and Scala rules benefit from paying the cost of process startup only once (warming up the JVM and so on.


Dune
Apurva Shukla's RSS Feed — Sun, 20 Sep 2020 16:51:31 GMT
⭐ ⭐ ⭐ ⭐ ⭐ This book is indescribable.

The harsh worlds and realities that Herbert presents intertwined with a inter-galactic feudalism, a…


Connecting R53 With External Domains
Prasham H Trivedi — Sun, 20 Sep 2020 09:47:31 GMT
In this article, we will see how we can use AWS’ Route 53 to connect with external domains and serve static S3 website.


2020-09-20
ho.dges.online — Sun, 20 Sep 2020 03:00:00 GMT



Our New Species (Part 1/2)
A Matter of Perspective — Sun, 20 Sep 2020 03:00:00 GMT

This is part one of this story. You can read part two here!

The Age of the Spiritual Machine
Boot Sequence
A softly toned, analog alarm pierced the quiet of Harrison’s city apartment. His forgotten dreams slipped through the cracks as his consciousness took over to start the day. Just another day.
He sighed, rolling out of bed, and switched on his phone. The morning ritual’s feed sprung to life with a firehose of ads, notifications, software updates, and Twitter notifications – all masquerading with self-importance.
Harrison scrolled through, filtering for anything important, or messages from loved ones. Amid the stupid tweet recommendations and ancient newsletters he really ought to unsubscribe from, a Calendar notification caught his eye.” Meeting reminder: Lunch with Jacob @ 12pm”. He was looking forward to this.

Stepping into the shower, he welcomed the hot water to wash away an average night’s sleep. Showers were his sanctuary, a private oasis to think, generate ideas, and relax without any distractions or Slack messages.
His mind booted up. He hadn’t seen anything in the mornings feed regarding his application for the Neuralink surgery. He grew anxious; it was just over a week since they said they would get back to him, and there was still radio-silence.
Feelings of jealousy arose with painful thoughts of his fellow programming colleague who’d recently received a Neuralink. Already, in the two weeks since he’d had the surgery, his co-worker had rebuilt the entire API service for his department, learned sales, marketing, and was now championing technical client relations. All with a new paycheck and high-praise from the CEO, of course.
Harrison was still stuck as a QA engineer, left to do the manual human testing of the AI’s complicated work. Despite the consistent late-night studying, algorithm practice, and competition submissions, he was only just managing to stay afloat at his job. 5 years ago, he had a high paying role as a senior engineer, but some things had changed in the past few years. It was hard to keep up.
A temporary blast of cold water shattered his stream of thought. He dried off in the kitchen and put on a brew of Golden Teacher mushroom tea. It had been 3 years since psychedelic psilocybin mushrooms had been approved for legal, personal use. Harrison brewed them daily. It was not enough to feel anything or create any hallucinations. The tea gave Harrison a necessary, though only slight, edge in the competition with the Connies. Others had also embraced the “natural augmentation” that psilocybin offered, eager for anything to expand the mind and stay relevant amongst the chaos. An increasingly common blend of techno-spirituality and psychonaut techniques permeated Harrison’s online social network.
Since the “Linked Revolution”, a “Connie” was a nick-name for those fortunate enough to afford and receive the Neuralink surgery. They were a different kind. The “Linked Revolution” took the world by storm. An increasing number of professionals, creatives, investors, business-people, and politicians itching to get the implant. They couldn’t build the installation machines fast enough.
The world underwent vast sweeping changes as the bandwidth between humans and machines exploded by several orders of magnitude. A bridge for humanity and AI to co-exist. Human 2.0.
Nothing was the same. Harrison sipped his tea.

Harrison’s best friend was his ryukin goldfish, Alice. The pair had a simple relationship, and he liked it that way. Harrison would talk, and she would listen. She was good at that.
He stood there in his towel, staring into her large bowl, watching her circle aimlessly.
“Are we still relevant, you and I, Alice?”
She bubbled in response.
Harrison wondered if normal humans and goldfish had grown more alike over the past few years.
“Are we trapped to the confines of a watery sphere? At the mercy of a greater force, one that can expand beyond the physical constraints of our world? Is our entire existence defined by the chance that they forget to feed us one day?”
He chewed his lip and concluded the tea was kicking in.
“Good chat Alice.”
Harrison cleaned his mug, slipped on his freshly washed jeans, and put on a WWDC 2030 shirt from earlier this year. The years alone at a computer all day and night had not done wonders for his health, and the jeans were a bit too tight. He opened Tesla Fetch on his phone and ordered a ride to his usual cafe spot in the city.
Running downstairs, he bumped into his neighbor, an old-lady chatting away on FaceTime to her grandchildren in Bali.
“Oh, Harrison, dear! Say hello to my oldest grandchild, Namibia. She’s in Bali, you know?” the old-lady enthused.
“Bali? That’s a dangerous place to be,” Harrison squinted at the tiny screen she was thrusting in his face, “How are things down there?”
“We’re here to enjoy the last of it before it goes under,” Namibia sighed, “Doing what we can to help the locals, you know?” She seemed sad.
Harrison worked for a startup, Rising Tide, that provided AI relocation assistance for struggling islanders. The thought of being in Bali right now made him queasy.
“Ah, ok. Well, look after yourself!” and he darted down the remaining stairs into the tiny backstreet of his city. Strangers weren’t his strong suit.

After a quick glance at Fetch, it told him the ride was still a few minutes away. Instead, he opened HackerNews and leaned against the wall as he consumed his most trusted source.

Microsoft VR Office overtakes Oculus sales with new Neuralink integration
New nuclear fission plants to replace California’s solar farms
SABER virus gains a second wave in South East Asia
Ask HN: Best Linux distros for Neuralink/AI development?
Successful Second Landing of SpaceX Mars Mission

The last one caught his eye. SpaceX had successfully established a Martian base in 2027, officially turning humans into an interplanetary species. Incredible efforts and resources were now being poured into the space industry. Transportation, energy innovation, various offshoots of spacecraft companies, asteroid mining expeditions, pirates fighting with the US Space Force; it was all very science-fiction and exciting.
Unfortunately for most, the experiences of this brave new world were limited to news articles and immersive video games. Space was reserved only for those who could afford it, or smart enough to work in it. The competition for a job in space was astronomical, and a non-Connie securing a position these days was almost unheard of. Harrison day-dreamed.
Not long after, the pleasant hum of the driverless, electric, Tesla Fetch zoomed into view, stopping with perfect deceleration right in front of Harrison. The doors swooshed open, and a gust of cool, conditioned air rushed to greet him as he stepped inside to sit on the comfy leather seat.
In the corner, a purple-haired tween was socializing loudly with her friends on some app.
“OMG, did you see her new TikTok video? I was like, ummmm, who are you trying to be, you know? Like, don’t pretend to do your own content when you’re just copying me. Be original, bitch. What? Oh yeah, I know. Yep. Mhm. She’s so fake. And it only got like 3 likes, so whatever. Mine got like, 300. Anyway, so what did you tell Ryan last night?”
There was another passenger, however, that intrigued Harrison much more. The strongly-built, middle-aged man was dressed only in black, sporting heavy boots, tinted sunglasses, and an expensive suit. His short brown hair was contrasted by his clenched, freshly-shaved jawline and tight lips. The air felt colder around him, and he seemed to exude a sense of danger and anger. He didn’t move a muscle as Harrison pulled out his phone, and the doors closed with a click.
“We’ll be ignoring him,” Harrison thought, uncomfortably resuming the perusal of today’s updates. The Fetch was speeding quickly through the city and took a slight turn, down into one of the city’s tunnel openings. The new underground system was AI-Designed, created by analyzing traffic patterns over the years. AI identified the most common routes and destinations people took, and humans drilled the tunnels. It was extremely efficient. Virtually nobody owned a car anymore, let alone drove – traffic was an issue of the past.
Harrison lamented on the hours he used to spend driving to the corporate office. All that wasted time, and for what? Since COVID, everything went remote, and the world had never looked back. SABER virus had only deepened the new workflow.
The Telsa slowed, stopping outside a mysterious, unmarked brick building. The agent (as Harrison supposed he must have been) promptly stepped out, without a sliver of expression, and entered the two reinforced steel doors of the building. Good riddance. Miss Tween was still happily gossiping away.
Harrion’s phone buzzed, notifying him he would arrive in a few minutes. 2 article headlines later, and they pulled up at The Electric Fig.

The Electric Fig
The tiny, hipster cafe was wedged in an alley between two gigantic skyscrapers, barely seeing the sun. It opened up to the street with a giant floor to ceiling window, inviting busy passerby’s in with scents of fresh, 4th wave coffee and down-tempo jungle beats. Harrison stepped through the open door, waving at some of the staff, and tiptoed his way past potted plants and laptop chargers. His regular seat was at the very back.
As usual, the energy was palpable, with the tiny room stuffed full of other remoters, designers, devs, and bloggers. He was grateful for his noise-canceling headphones. A few other regulars he knew smiled at him from their conference calls.
Harrison made his way to the counter and was met with an unfamiliar, pretty face. He felt the butterflies in his stomach as he stumbled over his words.
“Hey, ugh, you new here?” Great. What a line idiot.
“Yeah I am,” she smiled, she had a happy voice, “Just started today”
Harrison glanced at her name tag, “Ugh, Khaleesi? Nice name”
“Yeah,” she sighed, and seemed annoyed he’d said anything, “My parents are real fun. Most people just call me Kali”
“Oh, yeah, sorry. I’m a big fan, but yeah. Nice to meet you Kali” Why was talking to girls so hard? “Umm, can I get a cappuccino, please? No sugar”
“Sure”, she fumbled with the kiosk, “I’ll bring it out to you”
Harrison’s mind flooded with the million different ways he could have handled that interaction better. Returning to his table, he mumbled to himself. “Maybe I should renew my gym subscription” he thought.
He opened his laptop, Chrome, and Google meets to join his companies daily standup. He yawned; these meetings were always dull.
“Hi team!” His manager, Augustine, was annoyingly chipper, but today seemed especially intense, “How was everyone’s weekend?”
His team gave a half-hearted response, and those game enough turned on their mic to offer a “Good thanks”, or a “Yeah, alright”. Go, team.
Rising Tide was a relatively new startup, only founded in the last two years. The founder was a Jamaican refugee who was forced to leave his country with his family due to sea levels reducing the island’s capacity. He met the CTO in a holding camp near Toronto, and the pair decided to join forces to tackle the severe issue.
Harrison zoned out, opening a separate tab to check some of his favorite subreddits, and his mind drifted. He wondered about his lunch with Jacob today. He’d never met him in person, only on Twitch. What intrigued Harrison mostly was that Jacob claimed to be a Connie. What would he think of Harrison?
“Harrison? You still with us?” Augustine broke his focus.
“Oh, yeah, sorry, mic problems” he lied. “Uhh today I’m fixing that issue with the invalid tokens being sent to the client”
Augustine didn’t seem convinced, but Harrison wasn’t sure if it was an internet lag.
“Ok, thanks, Harrison. Have a great day everyone” The call ended with a patronizing chime.

Harrison couldn’t concentrate. Between the upcoming lunch, the boredom of his Jira tickets, and the frustrations of his interaction with Kali, he was distracted. The code in front of him blurred into colorful blobs, dancing in front of his eyes, taunting Harrison with its declarations, control flows, and loops. An artificially generated soup of complexity.
The mushrooms weren’t helping today, at least not for this.
Had coding shifted along with everything else over the last few years? With the Connies and their AI generating most of the code more often, how was a human supposed to keep up?
“It’s not like I’m dumb” Harrison complained to a small Telegram group of other programmer friends, “I spend so much time studying this shit. Are you guys struggling with this too?”
His message was seen by several, and gained a couple of thumbs-up emojis. Harrison was encouraged that he wasn’t the only one. Mindlessly, he opened up Gmail to check if there was any Neuralink update. Still nothing.
A Slack question from a more junior co-worker sucked him back into his work. The unique blend of caffeine and psilocybin was in full effect as he forced himself to zone in and be useful. He disabled his notifications so he could program in peace and spend the next few hours debugging.

By the time his Calendar reminded him it was time to leave, Harrison was well on his way, packed to go. He paid, waved awkwardly at Kali, and returned to the now empty, cold street. The sunlight valiantly sought for any small break between the skyscrapers, providing patches of illumination before returning to the darkness of steel, cement, and glass. Harrison reached the end of the street and turned left, the sky breaking out to a view of the lake. He enjoyed this walk, and today there was a spring in his step.
Jacob had suggested they meet at an upmarket lunch bistro that Harrison had never heard of. Google Maps told him it was a 22-minute walk, so he resumed an episode of the Joe Rogan Experience. Joe’s current guest was a Seargent of the Martian Space Core, discussing an ongoing referendum for Martian independence. Recent political tensions over trade agreements between Euro-Chinese leaders and the Martian settlers were a hot topic.
The Sergeant’s argument was that Earth was demanding too many critical minerals to be shipped on returning voyages – minerals that Mars needed. Mid 2025, a global state of emergency introduced heavy international rations on precious metals running out. There were insufficient physical resources to supplement the titanic consumption of energy produced by non-renewables with solar or wind. Despite the cultural perspective on nuclear fission opening to more research and deployments, we were not moving fast enough. Asteroids, though rich in assets, were an expensive and risky operation. Mars was ripe for the picking to sate Earth’s hunger.
The breezy bike-path along the lake gave way to chic, elegant architecture, stylish murals, and groomed hedge-trees. The people here dressed differently, and nobody said a word—Connie territory.
Harrison had arrived.

  To be continued…
  

  👉 You can read part two here! 👈 



The Mysterious Keka Slider
The David Brownman Blog — Sun, 20 Sep 2020 03:00:00 GMT
A quick dive comparing relative effectiveness different compression speeds and formats.

Read the whole thing.


2020–09–20: Downsizing the multi-boot image
megi's PinePhone Development Log — Sun, 20 Sep 2020 01:00:00 GMT



2020–09–20: Some ways to improve Pinephone safety
megi's PinePhone Development Log — Sun, 20 Sep 2020 01:00:00 GMT



About too much work being pointless
Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 19 Sep 2020 21:00:00 GMT
Is it still work, if it damages society?


Novelty Makes Novices
Pete's Points — Sat, 19 Sep 2020 19:07:15 GMT
My experience in the last decade have been that software developers are asked to learn new technologies with such frequency that novice level skills have become a semi-permanent norm that is expected and accepted.
Here's some technologies:

clojure
edn
kotlin
maven
VS Code
vim
elasticsearch
kafka
kafka connect
kafka streams
mongodb replication
event sourcing, change data capture architecture, CQRS
avro
graphql
graphql schema delegation
apollojs
nextjs
postgresql ltree plugin
yarn
advanced circleci configuration
advanced docker
mozilla sops
kustomize

Is that list everything I've used in my 20 years in the industry? No! That's just tech I had to learn and use in my 18 months at Reaction Commerce! That's more than one each month. Alomst every one of these represents "probably should read at least 1 whole book on this" depth.
Then of course we go through an acquisition and our tech stack changes pretty much completely. 2 new programming languages, 2 new databases, a new public cloud, a new RPC system, in-house frameworks and libraries, and so on.
In terms of my personal job satisfaction, I'm hoping that now that the acquisition transition is pretty much complete, I'll start to have at least a few stable bits of my tech stack: mostly the same DBs, programming languages, and cloud managed services. I enjoy doing work with intermediate or advanced skills a lot more than I enjoy learning new things and fumbling around as a novice. Of course others will crave and flourish with a series of new stuff to learn in a cursory way.
Thinking more broadly, I have some vague concerns about this might have an indirect causal relationship with the generally terrible state of technology at large, but I have no way to really test this, and the even second level more cynical side of me suspects maybe even if expert implementations were the norm, tech would still be mostly terrible because we don't really care about making it good.


Martin Schlachter and Justina Tschosick
David Schlachter — Sat, 19 Sep 2020 15:30:00 GMT
Everything I know about my great-great-great grandparents Martin Schlachter and Justina Tschosick.


Emacs: search/play Music (macOS)
xenodium.com @alvaro — Sat, 19 Sep 2020 03:00:00 GMT
While trying out macOS's Music app to manage offline media, I wondered if I could easily search and control playback from Emacs. Spoiler alert: yes it can be done and fuzzy searching music is rather gratifying.

Luckily, the hard work's already handled by pytunes, a command line interface to macOS's iTunes/Music app. We add ffprobe and some elisp glue to the mix, and we can generate an Emacs media index.
Indexing takes roughly a minute per 1000 files. Prolly suboptimal, but I don't intend to re-index frequently. For now, we can use a separate process to prevent Emacs from blocking, so we can get back to playing tetris from our beloved editor:
(defun musica-index ()
  "Indexes Music's tracks in two stages:
1. Generates \"Tracks.sqlite\" using pytunes (needs https://github.com/hile/pytunes installed).
2. Caches an index at ~/.emacs.d/.musica.el."
  (interactive)
  (message "Indexing music... started")
  (let* ((now (current-time))
         (name "Music indexing")
         (buffer (get-buffer-create (format "*%s*" name))))
    (with-current-buffer buffer
      (delete-region (point-min)
                     (point-max)))
    (set-process-sentinel
     (start-process name
                    buffer
                    (file-truename (expand-file-name invocation-name
                                                     invocation-directory))
                    "--quick" "--batch" "--eval"
                    (prin1-to-string
                     `(progn
                        (interactive)
                        (require 'cl-lib)
                        (require 'seq)
                        (require 'map)

                        (message "Generating Tracks.sqlite...")
                        (process-lines "pytunes" "update-index") ;; Generates Tracks.sqlite
                        (message "Generating Tracks.sqlite... done")

                        (defun parse-tags (path)
                          (with-temp-buffer
                            (if (eq 0 (call-process "ffprobe" nil t nil "-v" "quiet"
                                                    "-print_format" "json" "-show_format" path))
                                (map-elt (json-parse-string (buffer-string)
                                                            :object-type 'alist)
                                         'format)
                              (message "Warning: Couldn't read track metadata for %s" path)
                              (message "%s" (buffer-string))
                              (list (cons 'filename path)))))

                        (let* ((paths (process-lines "sqlite3"
                                                     (concat (expand-file-name "~/")
                                                             "Music/Music/Music Library.musiclibrary/Tracks.sqlite")
                                                     "select path from tracks"))
                               (total (length paths))
                               (n 0)
                               (records (seq-map (lambda (path)
                                                   (let ((tags (parse-tags path)))
                                                     (message "%d/%d %s" (setq n (1+ n))
                                                              total (or (map-elt (map-elt tags 'tags) 'title) "No title"))
                                                     tags))
                                                 paths)))
                          (with-temp-buffer
                            (prin1 records (current-buffer))
                            (write-file "~/.emacs.d/.musica.el" nil))))))
     (lambda (process state)
       (if (= (process-exit-status process) 0)
           (message "Indexing music... finished (%.3fs)"
                    (float-time (time-subtract (current-time) now)))
         (message "Indexing music... failed, see %s" buffer))))))

Once media is indexed, we can feed it to ivy for that narrowing-down fuzzy-searching goodness! It's worth mentioning the truncate-string-to-width function. Super handy for truncating strings to a fixed width and visually organizing search results in columns.
(defun musica-search ()
  (interactive)
  (cl-assert (executable-find "pytunes") nil "pytunes not installed")
  (let* ((c1-width (round (* (- (window-width) 9) 0.4)))
         (c2-width (round (* (- (window-width) 9) 0.3)))
         (c3-width (- (window-width) 9 c1-width c2-width)))
    (ivy-read "Play: " (mapcar
                        (lambda (track)
                          (let-alist track
                            (cons (format "%s   %s   %s"
                                          (truncate-string-to-width
                                           (or .tags.title
                                               (file-name-base .filename)
                                               "No title") c1-width nil ?\s "…")
                                          (truncate-string-to-width (propertize (or .tags.artist "")
                                                                                'face '(:foreground "yellow")) c2-width nil ?\s "…")
                                          (truncate-string-to-width
                                           (propertize (or .tags.album "")
                                                       'face '(:foreground "cyan1")) c3-width nil ?\s "…"))
                                  track)))
                        (musica--index))
              :action (lambda (selection)
                        (let-alist (cdr selection)
                          (process-lines "pytunes" "play" .filename)
                          (message "Playing: %s [%s] %s"
                                   (or .tags.title
                                       (file-name-base .filename)
                                       "No title")
                                   (or .tags.artist
                                       "No artist")
                                   (or .tags.album
                                       "No album")))))))

(defun musica--index ()
  (with-temp-buffer
    (insert-file-contents "~/.emacs.d/.musica.el")
    (read (current-buffer))))

The remaining bits are straigtforward. We add a few interactive functions to control playback:
(defun musica-info ()
  (interactive)
  (let ((raw (process-lines "pytunes" "info")))
    (message "%s [%s] %s"
             (string-trim (string-remove-prefix "Title" (nth 3 raw)))
             (string-trim (string-remove-prefix "Artist" (nth 1 raw)))
             (string-trim (string-remove-prefix "Album" (nth 2 raw))))))

(defun musica-play-pause ()
  (interactive)
  (cl-assert (executable-find "pytunes") nil "pytunes not installed")
  (process-lines "pytunes" "play")
  (musica-info))

(defun musica-play-next ()
  (interactive)
  (cl-assert (executable-find "pytunes") nil "pytunes not installed")
  (process-lines "pytunes" "next"))

(defun musica-play-next-random ()
  (interactive)
  (cl-assert (executable-find "pytunes") nil "pytunes not installed")
  (process-lines "pytunes" "shuffle" "enable")
  (let-alist (seq-random-elt (musica--index))
    (process-lines "pytunes" "play" .filename))
  (musica-info))

(defun musica-play-previous ()
  (interactive)
  (cl-assert (executable-find "pytunes") nil "pytunes not installed")
  (process-lines "pytunes" "previous"))

Finally, if we want some convenient keybindings, we can add something like:
(global-set-key (kbd "C-c m SPC") #'musica-play-pause)
(global-set-key (kbd "C-c m i") #'musica-info)
(global-set-key (kbd "C-c m n") #'musica-play-next)
(global-set-key (kbd "C-c m p") #'musica-play-previous)
(global-set-key (kbd "C-c m r") #'musica-play-next-random)
(global-set-key (kbd "C-c m s") #'musica-search)

Hooray! Controlling music is now an Emacs keybinding away: ø/
comments on twitter.
UPDATE1: Installing pytunes with pip3 install pytunes didn't just work for me. Instead, I cloned and installed as:
git clone https://github.com/hile/pytunes
pip3 install file:///path/to/pytunes
pip3 install pytz
brew install libmagic

UPDATE2: Checked in to dot files.


On school improvement
nutcroft — Sat, 19 Sep 2020 03:00:00 GMT
Few would disagree that the educational system of the —so-called— Western world could be improved. Eight partially concrete concepts on a potential improvement follow.


Less information. Slash the amount of information included in the courses. Aim to keep the essence, the fundamentals, the knowledge that a student is expected to either remember 10 years later or to have been influenced by it.


More humanities and art. There are many forms of art, and even the simplest change of dedicating a couple of hours per week on those would be beneficial (yet admittedly crude).


More bodily knowledge. The importance of physical activity both for physical health but also for psychological health cannot be overstated. More classes of exercising and sports is an easy first step.


Teacher quality assessment by students. Students are the essence and purpose of the whole operation. How could we not deeply value their impressions?


Teacher freedom. Teaching and learning are creative processes in their very core. If a teacher is micromanaged by laws on how to teach, then learning will probably not happen; only a knowledge auditory event will.


De-professionalise teaching. Accomplished educationalists talk about this idea for quite some time. If only, it's worth a serious attempt.


School as a more integrated part of society. Let's incorporate school as part of every society member's life. Non-parent professionals can teach. Others can learn. Is there an argument for not being inclusive?


Student freedom and abolishment of mandated studying. Every time a student is graded with a failure mark in an exam it is proof that people cannot force people to learn. Let's entrust students with the responsibility to decide what to learn, when, where, and how.


--
Also on this series: What’s wrong with school?


How to batch resize and compress images in linux
Shyam Jos — Sat, 19 Sep 2020 03:00:00 GMT







 
 



In this tutorial I will show you how to batch resize and compress images in Ubuntu or in any other Linux distros(debian,arch and fedora).
‘Convert’ command is a very powerful image manipulation utility which comes preinstalled in almost all Linux distributions (ubuntu,debian,arch and fedora) and it is a part of ImageMagick software suite.
Check if convert command is available on your Linux distro #
 convert -version
output:
Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
If the above command is not found on your Linux distro you can install it by typing the below command.
Install convert tool on Ubuntu / Debian #
 sudo apt-get install imagemagick
Install convert tool on fedora #
 sudo dnf install imagemagick
Install convert tool on arch #
 sudo pacman -S imagemagick
Note: It’s always good idea take backup of your images before running convert command.
Batch resize and compress images with convert command #
Below command will resize and compress all images ending with .jpg extension in a ‘for loop’ and save the compressed/resized image as ‘filename-Optimized.jpg’.
–quality : this option is used to set the image compression level in percentage
-resize : this option is used to resize the image to a given resolution , You can use -resize option with width ( -resize 1600x ) only or both width and height ( -resize 1600×900), In both cases convert command will automatically adjust the given resolution to get optimum aspect ratio so you don’t have to worry about stretched images.
Note: some jpeg image extensions may end with .JPG (in capital letters) ,Since Linux is case-sensitive you may need to adjust the bash script accordingly.
Open terminal and cd into the directory containing your photos and execute below command
 mkdir photos-Optimized;for photos in *.JPG;do convert -verbose "$photos" -quality 85% -resize 1600x900 ./photos-Optimized/"$photos-Optimized.jpg"; done
Voilà! Now you have successfully resized/compressed images, You can see the optimized images in the photos-Optimized directory.


Cybersecurity in the early 1990's
Jason Eckert's Website and Blog — Sat, 19 Sep 2020 03:00:00 GMT
Back in the late 1980’s and early 1990’s, I remember collecting MS-DOS computer viruses. I’d download them from an online Bulletin Board System (BBS) and store them on floppy disks in compressed format to prevent them from executing. For fun, some friends and I would occasionally release them on one of our systems to see what they did, and then reinstall MS-DOS  afterwards. Most viruses didn’t do very much, but some others (e.g. BBkiller) would remove or modify the contents of files.


Farewell Estes Park
Adriano Caloiaro's personal blog — Sat, 19 Sep 2020 03:00:00 GMT
This is less about saying farewell to Estes Park than it is about a ride I’ve been eyeing for over five years. Not only did I have the wrong bike when I was here in September, 2015, but I learned that this route existed the day after I finished an out-and-back run of Trail Ridge Road on my old Giant roady. TRR is the downhill paved segment of today’s route.
There are a multitude of reasons to visit this beautiful town nestled at the base of Rocky Mountain National Park. There are endless hiking trails both in and out of the park, mountain biking, tons of sport climbing routes (in fact, 18 routes literally in the downtown area), great trail running, and views for days. But at  the back of my mind, pulling into Estes Park five years after the last time, was the fall river road to trail ridge road loop, beginning in Estes. It’s a mixed gravel and paved route that’s known in the area as one of the greats. And indeed it is!
On September 9th winter came early in Colorado, following record-setting temperatures the previous three days over Labor Day weekend. About 8 inches of snow fell down in the valley, with up to 12-16 inches falling at the top of the of the Rockies. Initially the storm sent me into an immediate panic, because both Trail Ridge Road and Fall River Road were closed – possibly for the rest of the season. Neither of these are normal roads: they’re high, traverse the continental divide, and TRR holds the designation as the “highest continuously paved road in the United States”. So they’re kind of special.
Immediate panic transformed into eventual delight. TRR would probably reopen, but Fall River Road would probably not. That doesn’t bode well for a ride involving Fall River Road, but It all turned out to be great news. Why? First of all the entirety of RMNP is timed-entry restricted, meaning you have to have reservation in advance to enter with a car. Bikes are open game. This means the park is more empty and with fewer cars than it ever is. Secondly, gate security on a road like Fall River Road doesn’t exits, which meant I’d probably have the entire north section of the park to myself if I was willing to duck a gate and turn a blind eye to some pesky rules.



  


Indeed, I’ve never been more alone in RMNP. The ride was fantastic and without a soul in sight until I peaked out at the alpine visitor’s center. I have to admit that leading into this ride I was questioning whether I still had it because I’m far from peak personal fitness. I’ve been active recently, but not 1 mile of elevation gain active. So it was a bit of a surprise to finish this climb and decent in 3.5 hours. I was harboring fears of getting turned around by the elevation and steep grade, and no longer being in my 20s. I don’t know how long my first ride of TRR took back in 2015, but it was embarrassingly longer. Helluva way to end my 5 weeks in Estes Park!





Up next! I’m excited to be headed to Crested Butte bright and early tomorrow morning.
Cheers


An optimistic description of the Network Economy...
Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 18 Sep 2020 08:00:00 GMT
…with just a few weak spots.


Specify Bundler Version
Thomas Countz — Fri, 18 Sep 2020 03:00:00 GMT
If you’re like me, you may have versions of both Bundler 1 and Bundler 2 installed on your system. This can make it difficult to manage different codebases.

Say one requires an older version, like 1.13.6. You can run the following command to install it,

gem install bundler -v 1.13.6


but when you install gems,

bundle install


you might end up seeing something like

BUNDLED WITH
   2.1.4


in your Gemfile.lock file… which is not the same as bundling with 1.13.6.

You can tell Bundler that you’d like to use a specific version of Bundler by specifying the exact version number before the command you wish to run. Like so:

bundle _x.x.x_ install


To install gems for your project using Bundler 1.13.6, you can use this command to force Bundler to use the correct version:

bundle _1.13.6_ install


And you should see

BUNDLED WITH
   1.13.6


at the end of your Gemfile.lock file.

Hooray!


2020–09–18: Let's talk about safety of Pinephone
megi's PinePhone Development Log — Fri, 18 Sep 2020 01:00:00 GMT



And Another Thing Wrong with Social Media...
one mikro2nd — Thu, 17 Sep 2020 17:57:30 GMT
One of the sources of ill we experience with social media is this constant pressure to be exellent. To be sparkly and witty and clever. To be a Special Snowflake.
Among the myriad flora and fauna inhabiting social networks, there’s that Long Tail of us who have little to say beyond chatter to our friends and acquaintances. That’s fine. It’s what we came for in the first place. And then there’s the other 5%: Those who are there to promote their businesses, their books and music, their causes and politics.


Original ErgoDox is Better than Infinity
Pete's Points — Thu, 17 Sep 2020 16:37:26 GMT
I own both an original ErgoDox (full hand acrylic case, from I think the 2nd massdrop) and a newer ErgoDox Infitity. The original works better for me and I wanted to share my experience and reasoning. There's a few things that are just simpler/nicer:

flashing the original just requires a paperclip button press on the top of the keyboard and 1 command in the OS
Flashing the infinity is a dance of a paperclip press on the underside of one half, an OS command that requires sudo, changing a USB cable around, a second paperclip on the bottom of the other half, a second OS command, and finally putting the USB cable back.
2 of the infinity USB cables have the wiggly USB 3.0 micro shape which is a pain to connect
the cable connecting the 2 halves of the infinity is thicker than the ergodox's TRRS so its more of an intrusion into my desktop
The infinity full-hand case has outer bottom shape isn't a straight line. There's a tiny angle and curve that makes trying to tent it near impossible vs the original where you jam a piece of all-thread into one of the bolt holes and you're done.



Dear Computer, We Need to Talk
bastibe.de — Thu, 17 Sep 2020 13:25:00 GMT
After years of using Linux on my desktop, I decided to install Windows on my computer, to get access to a few commercial photo editing applications. I'll go into my grievances with Linux later, but for now:
I tried to install Windows, you won't believe what happened next
Like I have done many times with Linux, I download a Windows image from my university, and write it to a USB drive, then reboot into the USB drive. The USB drive can't be booted. A quick internet search leads me to a Microsoft Support page on how to Install Windows from a USB Flash Drive, which says that


Instead, one has to format the stick as FAT32, make it active, then copy the files from the image to it. So I follow the instructions and open the Disk Management program. It does not offer an option of FAT32, nor for making the partition active. I settle on (inactive) exFAT instead. It doesn't boot.
I switch over to Linux, where I can indeed make a FAT32 partition, and I can mark it as bootable, which I take as the equivalent of active. But Linux can not open the Windows image to copy the files onto the USB stick. So back to Windows, for copying the files. Except they can't be copied, because some of them are larger than 4Gb, which can't be written to a FAT32 partition. What now?
While researching how to download a different version of Windows 10, I stumble upon the Media Creation Tool, which automatically downloads Windows 10 and writes it to the USB stick correctly. Why was this not pointed out in the article above? Who knows. At any rate, it works. I can finally install Windows.
The installation process requires the usual dozen-or-so refusals of tracking, ads, privacy intrusions, and voice assistants. I wish I could simply reject them all at once. And then the install hangs, while "polishing up a few things". Pressing the helpful back button, and then immediately the forward button unhangs it, and the installation completes.
Next up are drivers. It feels anachronistic to have to install drivers manually in this day and age, but oh well. The new GPU driver to make screen tearing go away, a driver for my trackball to recognize the third mouse button, a wacom driver, ten or so Intel drivers of unknown utility. The trackball driver is not signed. I install it anyway. The GPU driver does not recognize my GPU and can't be installed. A quick Internet search reveals that my particular AMD/Intel GPU/CPU was discontinued from support by both AMD and Intel, and does not have a current driver. But fora suggest that up to version 20.2.1 of the AMD driver work fine. They don't, the driver crashes when I open images in my photo editor. An even older version published by Intel does work correctly. So now I am running an AMD GPU with an Intel driver from 2018.


Installing and setting up Firefox and my photo editors works without issue, thank goodness. Emacs has a Windows installer now, which is greatly appreciated. OpenCL and network shares just work. This is why I'm installing Windows next to my Linux.
But Windows is still not activated. I copy my university's product key in the appropriate text box, but hesitate: That's for Windows Enterprise, and I'd be just fine with Home. So I cancel the activation without activating. A helpful link in the activation systems sends me to the Microsoft Store to get my very own version of Windows Home for €145, which normally retails for around €95, so that's a no-go. Whatever, I'll go with my university's Enterprise edition. Except the activation box now says my product key is invalid. And the Store now literally says "We don't know how you got here but you shouldn't be here" instead of selling me Windows. After a restart it installs and activates Windows Enterprise, even though I never actually completed the activation.


I install Git, but in order to access my Github I need to copy over my SSH key from the Linux install. Which I can't boot at the moment, because installing Windows overwrites the boot loader. This is normal. So I download Ubuntu, write it to the USB stick, boot into it, recover the bootloader, boot into my old install, reformat the stick, copy the files to the stick, boot back into Windows, and the files aren't on the stick. Tough. Boot back into Linux, copy the files onto the stick, eject the stick, boot back into Windows, copy the files to the computer. Great user experience.
Now that I have my SSH key, I open a Git Bash to download a project. It says my credentials are incorrect. I execute the same commands in a regular CMD instead of Git Bash, and now my credentials are correct. Obviously.
There are several programs that claim to be able to read Linux file systems from Windows. They do not work. But Microsoft has just announced that you will be able to mount Linux file systems from WSL in a few weeks or months. So maybe that will work!
I set my lock screen to a slideshow of my pictures. Except my pictures do not show up, and I get to see Window's default pictures instead. An internet search reveals that this is a wide-spread problem. Many "solutions" are offered in the support fora. What works for me is to first set the lock screen to "Windows Spotlight", then to "Slideshow". Only in that order will my pictures be shown.
I will stop here. I could probably go on ad infinitum if I wanted to. This was my experience of using Windows for one day. I consider these problems relatively benign, in that all of them had solutions, if non-obvious ones.
Why install Windows in the first place?
Part of the reason for installing Windows was my growing frustration with Linux. I have been a happy user of KDE of various flavors for about seven years now. But ever since I got into photo editing, things began to become problematic:
My photo editor requires OpenCL, but the graphics driver situation on Linux is problematic, to say the least. I generally managed to get RocM running most of the time, but kernel updates frequently broke it, or required down- or upgrading RocM. It was a constant struggle.
I wanted to work with some of my data on a network share, but KDE's implementation of network shares does not simply mount them for applications to use, but instead requires each application to be able to open network locations on their own. Needless to say, this worked almost never, requiring many unnecessary file copies. Perhaps Gnome handles network shares better, but let's not open that can of worms.
Printing photos simply never worked right for me. The colors were off, photo papers were not supported, the networked printer was rarely recognized. Both for a Samsung printer and an Epson and a Canon. One time a commercial printer driver for Linux printed with so much ink it dripped off the paper afterwards. Neither Darktable nor Gimp nor Digikam have a robust printing mode. I generally resorted to Windows for printing.
I ran that Windows in a virtual machine. With Virtualbox, the virtual machine would be extremely slow, to the point where it had a delay of several seconds between typing and seeing letters on the screen. VMWare did better, but would suddenly freeze and hang for minutes at a time. Disabling hugepages helped sometimes, for a short while. The virtual machine network was extremely unreliable. Some of these issues were probably related to my using a 4K screen.
Speaking of screens, I have two screens, one HighDPI 4k and one normal 1440p. Using X, the system can be either in HighDPI mode, or in normal mode. But it can't drive the two displays in different modes. Thus the second monitor was almost useless and I generally worked only on the 4k screen. With Wayland I would have been able to use both screens in different modes, but not be able to color-calibrate them or record screen casts. Which is completely unacceptable. So I stuck with one screen and X. In Windows, I can use both screens and calibrate them.


Additionally, Linux hardware support is still a bit spotty. My SD card reader couldn't read some SD cards because of driver issues. It would sometimes corrupt the SD card's file systems. USB-connected cameras were generally not accessible. The web cam did not work reliably. The CPU fan ran too hot most of the time.
So there had been numerous grievances in Linux that had no solutions. Still I stuck with it because so many more smaller issues were actually fixable if I put in the work. In fact I had accumulated quite a number of small hacks and scripts for various issues. I feared that Windows would leave me without recourse in these situations. And it doesn't. But at least the bigger features generally work as advertised.
Where do we go from here?
Just for completion's sake, I should really find an Apple computer and run it through its paces. From my experience of occasionally using a Macbook for teaching over the last few years, I am confident that it fares no better than Linux or Windows.
Were things always this broken? How are normal people expected to deal with these things? No wonder every sane person now prefers a smartphone or tablet to their computers. Limited as they may be, at least they generally work.
There is no joy in technology any more.


The future of farming looks more and more like Star Wars
Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 17 Sep 2020 08:00:00 GMT
May the Force of the Right Hardware be with you.


True Heroism
ho.dges.online — Thu, 17 Sep 2020 03:00:00 GMT
True heroism is minutes, hours, weeks, year upon year of the quiet, precise, judicious exercise of probity and care - with no one there to see or cheer. This is the world".

David Foster Wallace, The Pale King



Rust 2021: Lower the barriers
Krishna's blog — Thu, 17 Sep 2020 03:00:00 GMT
Rust's success is guaranteed on a long enough timescale, but lowering the barrier to entry by making it easier to learn is crucial for its adoption rate.


2020–09–17: Video acceleration experiments with PinePhone
megi's PinePhone Development Log — Thu, 17 Sep 2020 01:00:00 GMT



Notes from Amir Shevat on Measuring & Managing Developer Relations
swyx's site RSS Feed — Wed, 16 Sep 2020 21:40:08 GMT
Notes from an a16z podcast about developer relations that I thought was valuable


Focused headings
Home on Pinecoder blog — Wed, 16 Sep 2020 14:00:00 GMT
I finished another feature on Chronicles: “Focused headings” (demo). Focusing a heading finds all content under that heading, filters everything else out, then pins the heading to the top of the journal.
A journal might typically have a range of topics across many days:
After “Focusing” a heading, the heading element is fixed to the top, and the view filters any content not under that heading:
The primary inspiration was wanting a way to take running notes on similar topics, then have a way to surface them all in one stream — as though the entire journal was about just that topic.


Building the USB audio passthrough board
Kaetemi — Wed, 16 Sep 2020 11:39:42 GMT
After trying out a USB isolator to fix the audio noise that was caused by a ground loop, I continued my search for a high quality plug-and-play solution to forward...


Why writing is important
Andrew's Stories — Wed, 16 Sep 2020 11:00:00 GMT
I have no idea how to run RSS feeds, so please read this story on my website here!


Verbal mathematics sequence
Junk Heap Homotopy — Wed, 16 Sep 2020 09:24:13 GMT
— a sequence introducing mathematical thinking to VerbalsThesis: (verbally loaded) mathematics is just philosophy, but where everything is precisely what we mean


Poisson on startups
Junk Heap Homotopy — Wed, 16 Sep 2020 09:23:04 GMT
Let’s start with some sage advice.A lot of curmudgeonly commenters on HN treat viable side-projects as a lottery. If we take that idea seriously, well, your chances of winning a PowerBall per ticket is 1 in 13 983 816. There are around 42620 programmers in Silicon Valley + the other US tech hubs (Seattle, Dallas/Fort Worth, DC, NYC)1. So if every single one of those Y programmers tries to found a startup, the expected number of startups we’ll get is:


Bicycle delivery, or SLOW delivery?
Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 16 Sep 2020 08:00:00 GMT
Where are the bigger gains?


What is Stealth Address technology and Why Does Monero Use It?
SerHack - Security Researcher — Wed, 16 Sep 2020 03:00:00 GMT
Monero is based on the CryptoNote protocol and utilizes the most powerful cryptographic techniques in an effort to protect the privacy of the sender, recipient, and obfuscate the amount transacted. -Continue to read the article on SerHack.me


Deprecating Marvin
Muffin Man — Wed, 16 Sep 2020 03:00:00 GMT
It is funny, I would never say I would be sad to deprecate a software project. Especially a project boilerplate. But I just merged the deprecation notice, and I do feel down a little.
I created Marvin about four years ago for my own needs. Since then it grew, became internal project at
Work&CoFun fact, our website is made on top of Marvin. and was used by thousands of people.
Why? #
When Marvin was created React landscape was looking a lot different than today. The ecosystem continues to relentlessly evolve and maintaining a single boilerplate to capture all these possibilities has become unproductive.
The repo will stay available, but we are stopping further improvements and maintenance.
Thank you! #
A big thank you to everyone who used and supported Marvin!


Analysing the Indian government cyberspace
Nemo's Home — Wed, 16 Sep 2020 03:00:00 GMT
I recently did some work on analysing the Indian government cyberspace, thought I should document them somewhere outside of my Twitter¹.

List of GoI websites

I’d made a list of Indian government websites in Jan 2019:

I ran @18F /pulse on Indian Government websites to see how many of them support HTTPS. A quick summary:

Total Websites: 14183
Total Live Websites: 11710 (82%)
Websites with Valid HTTPS: 4753 (40% of all live websites)

Raw Dataset for now: docs.google.com/spreadsheets
— Nemo (@captn3m0) January 15, 2019


The dataset was from 2 sources:


  GoI Directory
  crt.sh (All certificates ending in .gov.in were used)


I re-ran the scripts to get an updated list (12842 domains), then tabulated them against the public-suffix² for each. There is a long-tail, and I’ve published results here. Here are the top public suffixes for Indian government sites:


  
    
      Public Suffix
      Domains
    
  
  
    
      nic.in
      2454
    
    
      gov.in
      7259
    
    
      in
      528
    
    
      ac.in
      490
    
    
      com
      568
    
    
      co.in
      171
    
    
      org.in
      168
    
    
      edu.in
      117
    
    
      org
      844
    
    
      res.in
      134
    
    
      net.in
      12
    
    
      net
      38
    
  


Sanskari Proxy

This was a long standing idea on my ideas repo:


  A lot of Indian Government websites are inaccessible on the public internet, because
they geo-fence it to within Indian Boundaries. The idea
is to make a Indian Proxy service that specifically works only for the Geo-fenced Indian government
websites.

  For eg, if uidai.gov.in is inaccessible, hitting uidai.gov.sanskariproxy.in will get you
the same result, proxied via our servers.


Since I’d made an updated list of GoI websites, this seemed easy enough. I realized that setting up uidai.gov.sanskariproxy.in would likely count as impersonation under the Indian law,
so I did the next best thing: run an actual proxy. Here’s the announcement tweet:

Are you a security researcher outside India? Do you hate getting geoblocked to Indian government websites?

Well, I made a proxy for security researchers outside India to access Indian government websites without resorting to shady VPNs.
captn3m0/sanskari-proxy
— Nemo (@captn3m0) September 5, 2020


Project page is https://github.com/captn3m0/sanskari-proxy, and if you’d like to get access - please reach out.

Cyberspace Ownership

I’d planned to get a complete list of geoblocked websites next. While I’m progressing on this front, the results have been inconsistent/inaccurate so far. As an intermediate step, I’d made a list of IPs against every domain³, which looked like this:


  
    
      Domain
      IP Address
    
  
  
    
      aavin.tn.gov.in
      164.100.134.148
    
    
      abnhpm.gov.in
      14.143.233.34
    
    
      agnii.gov.in
      13.232.216.65
    
    
      ap.gov.in
      117.254.92.53
    
    
      aponline.gov.in
      125.16.9.130
    
    
      appolice.gov.in
      118.185.110.147
    
    
      attendance.gov.in
      164.100.166.114
    
    
      cgg.gov.in
      112.133.222.115
    
  


While running numerous nmap scans (and failing), I start checking the ASN⁴ for some of these IPs to see who was hosting each website - especially the ones I was finding were blocked.

I stumbled upon a bulk IP to ASN service by Cymru, ran all the IPs against it and published the results. Here’s the important graph:



As you can expect, NIC⁵ has the highest share, with NKN⁶, BSNL, and CtrlS following at roughly 5% each. There are a few other chart on the twitter thread, and the raw data is available here with interactive versions of each visualization.

What next?

I’m working on running and comparing connectivity scans to these IPs to get a better understanding of the geoblocking situation. There’s also some issues with the domain list, as it seems to be missing lots of domains - so more corrections are needed.


  
    
      Twitter decided to suspend 12 different accounts I had access to recently - I’m starting to get wary of using Twitter for archival now. ↩
    
    
      A “public suffix” is one under which Internet users can (or historically could) directly register names. For eg - nic.in or github.io. Mozilla manages the list at https://publicsuffix.org/. ↩
    
    
      There are issues with this approach, since domains do resolve to multiple IPs. But this is okay for the rudimentary analysis I’ve been doing so far. ↩
    
    
      Autonomous Systems (AS) is how the internet is sliced up and managed by different entities. Each AS (usually an ISP) is responsible for routing within its network, while announcing network routes on how it can be reached. ↩
    
    
      The primary government office (under MeitY) that provides infrastructure and support for government IT services. ↩
    
    
      National Knowledge Network is a multi-gigabit research and education network that provides a high speed network backbone for educational institutions in India. ↩
    
  



2020–09–16: PineBook Pro and Levinboot again
megi's PinePhone Development Log — Wed, 16 Sep 2020 01:00:00 GMT



Typesafe, Fullstack React & GraphQL with AWS Amplify
swyx's site RSS Feed — Wed, 16 Sep 2020 00:23:42 GMT
This is the blog version of a talk I am giving at React Summit 2020


Guidelines to improve your React folder structure
Max Rozen — Tue, 15 Sep 2020 12:48:48 GMT
People often say to 'move files around until it feels right', but what does 'right' even mean? Let's learn in this article.


Pointers are Easy, Optimization is Complicated
metablog — Tue, 15 Sep 2020 12:01:17 GMT
Just recently came across Ralf Jung's 2018 post titled Pointers are Complicated.  The central thesis is that the model that most C (and 
assembly language) programmers have that a pointer is just an integer that happens to be a machine
address is wrong, in fact the author flat out states:  "Pointers are definitely not integers."

That's a strong statement.  I like strong statements, because they make a discussion possible.  So 
let's respond in kind:  the claim that pointers are definitely not integers is wrong.


The example

The example the author uses to show that pointers are definitely not integers is the following:



int test() {
    auto x = new int[8];
    auto y = new int[8];
    y[0] = 42;
    int i = /* some side-effect-free computation */;
    auto x_ptr = &x[i];
    *x_ptr = 23;
    return y[0];
}



And this is the crux of the reasoning:


It would be beneficial to be able to optimize the final read of y[0] to just return 42. The justification for this optimization is that writing to x_ptr, which points into x, cannot change y.


So pointers are "hard" and "not integers" because they conflict with this optimization that "would be
beneficial".

I find this fascinating: a "nice to have" optimzation is so obviously more important than a simple
and obvious pointer model that it doesn't even need to be explained as a possible tradeoff, never
mind justified as to why the tradeoff is resolved in favor of the nice-to-have optimization.


I prefer the simple and obvious pointer model.  Vastly.


This way of placing the optimizer's concerns far ahead of the programmer's is not unique, if
you check out Chris Lattner's  What Every C Programmer Should Know About Undefined Behavior, you will note
the frequent occurrence of the phrase "enables ... optimizations".  It's pretty much the only
justification ever given.


I call this now industry-dominating style of programming Compiler Optimizer Creator Oriented 
Programming (COCOP).  It was thoroughly critiqued in What every compiler writer should know about programmers or “Optimization” based on undefined behaviour hurts performance (pdf).



Pointers as Integers

There are certainly machines where pointers are not integers, the most prominent being 8086/80286 16 bit
segmented mode, where a (far) pointer consists of a segment and an offset.  On 8086, the segment
is simply shifted left 4 bits and added to the offset, on 80286 the segment can be located anywhere
in memory or not be resident, implementing a segmented virtual memory.  AFAIK, these modes are
simplified variants of the iAPX 432 object memory model.

What's important to note in this context is that the iAPX 432 and its memory model failed horribly,
and industry actively and happily moved away from the x86 segmented model to what is called a
"flat address space", common on other architectures and finally also adopted by Intel with the
386.


The salient feature of a "flat address space" is that a pointer is an integer, and in fact this
eqivalence is also rather influential on CPU architecture, with address-space almost universally
tied to the CPU's integer size.  So although the 68K was billed as a 16 bit CPU (or 16/32), its
registers were actually 32 bits, and IIRC its address ALUs were fully 32 bit, so if you wanted
to do some kinds of 32 bit aritmetic, the LEA (Load Effective Address) instruction was your
friend.  The reason for the segmented architecturee on the 8086 was that it was a true 16 bit
machine, with 16 bit registers, but Intel wanted to have a 20 bit address space.
 

So not only was and is there an equivalence of pointers and integers, this state of affairs
was one that was actively sought and joyously received once we achieved it again.  Giving
it up for nice-to-have optimizations seems at best debatable, but at the very
least it is something that should be discussed/debated, rather than simply assumed away.



Building UI without AutoLayout
Whacky Labs — Tue, 15 Sep 2020 09:00:00 GMT
I love Auto Layout. It helps a lot when designing complex UI. But there are times when the UI is very simple and Auto Layout might feel a bit overkill, while other times the UI might be a bit too complex and Auto Layout actually starts affecting the app performance. Before auto layout there was another technique to creating UI, it’s called Springs and Struts (also known as Manual Layout to be in contrast with Auto Layout). I like Manual Layout a lot as well for its simplicity. Like with every other tool, there are trade-offs when selecting the best tool for the job, and it also applies when selecting Auto Layout vs Manual Layout.

The good thing is that, the Auto Layout has not been designed as an alternative to Manual Layout, rather more like an complement. Where instead of us having to calculate the frame, we start with a CGRectZero and let the Auto Layout fill in the frame value later. Most of the time it’s wonderful and doesn’t impact our flow. Other times we might have to wait for the layout pass run to read back the calculated frame values

  // let Auto layout calculate the frame values
  dispatch_async(dispatch_get_main_queue(), ^{
    // start using the frame values for something else.
  });


I often wish if the Auto Layout were not that tightly coupled with UIKit. In a sense, if I could just run Auto Layout without a layout pass. This inspired me to take another take of building UIs without Auto Layout with something I’d like to call as Simple Manual Layout.

Inspiration

The inspiration is from how UIBarButtonItem works with UIToolbar or UINavigationBar. If we wanted to build a UI like



We would create a UIToolBar and add a bunch of UIBarButtonItem

UIToolbar *toolbar = [[UIToolbar alloc] initWithFrame:toolbarFrame];
UIBarButtonItem *playButton = [[UIBarButtonItem alloc]
                               initWithBarButtonSystemItem:UIBarButtonSystemItemPlay
                               target:self
                               action:@selector(playVideo)];
UIBarButtonItem *pauseButton = [[UIBarButtonItem alloc]
                                initWithBarButtonSystemItem:UIBarButtonSystemItemPause
                                target:self
                                action:@selector(pauseVideo)];
UIBarButtonItem *rewindButton = [[UIBarButtonItem alloc]
                                 initWithBarButtonSystemItem:UIBarButtonSystemItemRewind
                                 target:self
                                 action:@selector(rewindVideo)];
UIBarButtonItem *forwardButton = [[UIBarButtonItem alloc]
                                  initWithBarButtonSystemItem:UIBarButtonSystemItemFastForward
                                  target:self
                                  action:@selector(forwardVideo)];
UIBarButtonItem *spaceButton = [[UIBarButtonItem alloc]
                                initWithBarButtonSystemItem:UIBarButtonSystemItemFlexibleSpace
                                target:nil
                                action:nil];
[toolbar setItems:[NSArray arrayWithObjects:
                   spaceButton,
                   rewindButton,
                   spaceButton,
                   playButton,
                   spaceButton,
                   pauseButton,
                   spaceButton,
                   forwardButton,
                   spaceButton,
                   nil]];


The interesting element here is UIBarButtonSystemItemFlexibleSpace. Which is documented as “Blank space to add between other items. The space is distributed equally between the other items.”. Similarly, there’s another one called UIBarButtonSystemItemFixedSpace which is documented as “Blank space to add between other items. Only the width property is used when this value is set.”.

I think this approach could be used to build a layout engine which is very simple in terms of mental model but can be used to build as sophisticated layouts as we’d want.

Simple Manual Layout

With that design in mind we can build out layout engine. If there is a class SLELayoutItem which is a placeholder for a UIView and another class SLELayout that takes in one or more of these SLELayoutItem and immediately calculates the frame of the every SLELayoutItem. Then we can use the calculated frame value when constructing our UIView objects.

So to create a full screen subview we should be able to create as:



SLELayout *layout = [SLELayout layoutWithParentBounds:_rootFrame
                                            direction:SLELayoutDirectionColumn];
SLELayoutItem *mainItem = [SLELayoutItem flexItem];
[layout addItem:mainItem];

UIView *redView = SLECreateView(mainItem.frame, [UIColor redColor]);
[_rootView addSubview:redView];


UIView *SLECreateView(CGRect frame, UIColor *color)
{
  UIView *view = [[UIView alloc] initWithFrame:frame];
  view.backgroundColor = color;
  return [view autorelease];
}


And a 2 subview layout, where the top is flexible and bottom is fixed



  SLELayout *layout = [SLELayout layoutWithParentBounds:_rootFrame
                                              direction:SLELayoutDirectionColumn];
  [layout addItem:[SLELayoutItem flexItem]];
  [layout addItem:[SLELayoutItem itemWithHeight:200]];

  CGRect topFrame = [layout frameAtIndex:0];
  CGRect bottomFrame = [layout frameAtIndex:1];

  [_rootView addSubview:SLECreateView(topFrame, [UIColor redColor])];
  [_rootView addSubview:SLECreateView(bottomFrame, [UIColor blueColor])];


A more interesting layout would be where we have a column that contains a row.



  CGFloat contentHeight = 200.f;
  SLELayout *mainLayout = [SLELayout layoutWithParentBounds:_rootFrame
                                                  direction:SLELayoutDirectionColumn];
  [mainLayout addItem:[SLELayoutItem flexItem]];
  [mainLayout addItem:[SLELayoutItem itemWithHeight:44]];
  [mainLayout addItem:[SLELayoutItem itemWithHeight:contentHeight]];

  CGRect headerFrame = [mainLayout frameAtIndex:0];
  CGRect toolbarFrame = [mainLayout frameAtIndex:1];
  CGRect contentFrame = [mainLayout frameAtIndex:2];

  SLELayout *contentLayout = [SLELayout layoutWithParentBounds:contentFrame
                                                   direction:SLELayoutDirectionRow];
  [contentLayout addItem:[SLELayoutItem flexItem]];
  [contentLayout addItem:[SLELayoutItem itemWithWidth:contentHeight]];
  [contentLayout addItem:[SLELayoutItem flexItem]];
  [contentLayout addItem:[SLELayoutItem itemWithWidth:contentHeight]];
  [contentLayout addItem:[SLELayoutItem flexItem]];

  CGRect content1Frame = [contentLayout frameAtIndex:1];
  CGRect content2Frame = [contentLayout frameAtIndex:3];

  [_rootView addSubview:SLECreateView(headerFrame, [UIColor redColor])];
  [_rootView addSubview:SLECreateView(toolbarFrame, [UIColor blueColor])];
  UIView *contentView = SLECreateView(contentFrame, [UIColor yellowColor]);
  [_rootView addSubview:contentView];

  [contentView addSubview:SLECreateView(content1Frame, [UIColor cyanColor])];
  [contentView addSubview:SLECreateView(content2Frame, [UIColor magentaColor])];


Implementation details

The implementation of this layout engine turns out to be not as sophisticated. If we provide a SLELayoutItem which can have some properties fixed and others flexible.

SLELayoutItem.h
@interface SLELayoutItem : NSObject

// no values fixed
+ (instancetype)flexItem;

// partially fixed
+ (instancetype)itemWithWidth:(CGFloat)width;
+ (instancetype)itemWithHeight:(CGFloat)height;

// total fixed
+ (instancetype)itemWithSize:(CGSize)size;

// would be filled by the layout engine
@property (nonatomic, readonly) CGRect frame;

@end


So we can mark any flexible value as kSLELayoutValueUndefined which is -1 in our case.

// SLELayoutItem.m
@implementation SLELayoutItem
- (instancetype)initWithSize:(CGSize)size
{
  self = [super init];
  if (self) {
    // requested frame
    _originalFrame = (CGRect) {
      .origin = { .x = kSLELayoutValueUndefined, .y = kSLELayoutValueUndefined },
      .size = { .width = size.width, size.height }
    };
    // will be updated later
    _finalFrame = CGRectZero;
  }
  return self;
}

// called by layout engine
- (void)setOrigin:(CGPoint)origin
{
  _finalFrame.origin = origin;
}

- (void)setSize:(CGSize)size
{
  _finalFrame.size = size;
}

@end


And we can have an internal setter interface only visible to SLELayout

// SLELayoutItem+Internal.h
@interface SLELayoutItem ()

- (void)setOrigin:(CGPoint)origin;
- (void)setSize:(CGSize)size;

@property (nonatomic, readonly) CGRect originalFrame;

@end


Next, within SLELayout we have an mutable array that contains SLELayoutItem. And whenever a new item is added we recalculate the frames per item.

// SLELayout.m
@implementation SLELayout

// ... 

- (void)addItem:(SLELayoutItem *)item
{
  [_items addObject:item];
  [self updateFrames];
}
@end


If we calculate only for one direction, say vertical. The updateFrames might look something like:

// SLELayout.m
@implementation SLELayout

// ... 

- (void)updateFrames
{
  // calculate total fixed height
  CGFloat fixHeight = 0;
  NSInteger flexItems = 0;
  for (SLELayoutItem *item in _items) {
    CGFloat itemHeight = item.originalFrame.size.height;
    if (itemHeight == kSLELayoutValueUndefined) {
      flexItems += 1;
    } else {
      fixHeight += itemHeight;
    }
  }

  // calculate height per flex item
  CGFloat flexHeight = _parentSize.height - fixHeight;
  CGFloat flexItemHeight = flexHeight / (CGFloat)flexItems;

  // update final frames per item
  CGFloat offsetY = 0.f;
  for (SLELayoutItem *item in _items) {
    CGSize itemSize = item.originalFrame.size;
    CGFloat itemHeight = (itemSize.height == kSLELayoutValueUndefined)
      ? flexItemHeight
      : itemSize.height;
    itemSize = (CGSize) { .width = _parentSize.width, .height = itemHeight };

    [item setOrigin:CGPointMake(0.f, offsetY)];
    [item setSize:itemSize];

    offsetY += itemSize.height;
  }
}
@end


And similar calculations for width.

And now it doesn’t seem hard to imagine to support alignment for sub views (currently they are all set 0.0f or all aligned to start) with something like:

typedef NS_ENUM(NSUInteger, SLELayoutAlignment) {
  SLELayoutAlignmentStart,
  SLELayoutAlignmentCenter,
  SLELayoutAlignmentEnd
};


If I actually start using this code in real life, I might start supporting it. The code for the Simple Layout Engine is available at github.com/chunkyguy/SimpleLayoutEngine


The unbearable US-ness of online discourse
Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 15 Sep 2020 08:00:00 GMT
My apologies in advance to all my US friends. But…


Hard Travel Case for Corne LP
trv.rs — Mon, 14 Sep 2020 18:17:00 GMT
When I am bored I get on Amazon and look for hard cases that might work for my favorite keyboards. This week… the new Corne LP. While there are people making soft sleeves for keyboards, there's not much in the way of a protective travel case for most keyboards, especially the weird hobbyist ones that I've been using recently.

This 5" hard case for a GPS is $11 on Amazon, and from the internal dimensions it looked like it'd fit the corne LP and some cables… and it does!
Included in the package is a very short usb-c -> usb-a cable, a short TRRS cable to connect the halves, and the Apple Camera Connection kit so that I can plug this into my phone.¹ Since soldering up the Corne, I've learned there are bluetooth chips I could put on it… but I'll just work with this for a while.




You have to reduce the power draw of your keyboard for this to work. You can see my commit here. ↩





Go-style coroutines in the FN language
Blog on blog.gnoack.org — Mon, 14 Sep 2020 13:15:00 GMT
My toy Lisp fn is an inherently
single-threaded language, but with its built-in stack-inspection
capabilities, it’s easy to build coroutines on top.
The
implementation
only has about 20 lines of code, including comments. This article
contains a simplified implementation in-line. In only rewrote some
comments to make more sense in context, and removed confusing
technicalities like fn’s custom method call syntax, the use of
dynamic-wind and remarks about tail calls.
Prerequisites
In most programming languages, the call stack is a chain of stack
frames which ends in the program’s entry point.






A conventional call stack


Stack frames are on the heap in fn.

In the case of C, this call stack is a contiguous memory area, but in
fn, the stack frames are actually allocated objects on the heap
which have pointers to each other. Each stack frame also stores the
information required to resume
it.

Functions can get a handle on their own stack frame

Additionally, fn has a natively implemented helper called
$get-frame.
When called, $get-frame returns the caller’s own stack frame as Lisp
object which can be manipulated. You can access it from an interactive
fn session as well:
fn> ($get-frame)
#<Frame () @1ff4437b5780>
fn> (frame-caller ($get-frame))
#<Frame () @1ff4437b97c4>

Implementation
A queue of suspended coroutines
For Go-style coroutine behaviour, we keep track of a queue of call
stacks of currently suspended coroutines. A coroutine which yields
execution queues itself last, and returns into the first call stack
from that queue.






A queue of suspended call stacks


Starting new coroutines

A new coroutine is started with thread-new!. thread-new! suspends
its caller and itself becomes the entry point for the new coroutine
which immediately executes. The helper function thread-die!
unschedules the current thread from execution by scheduling another
coroutine without adding itself back to the queue.
(defn thread-new! (thunk)
  ;; Suspend current call stack.
  (queue-push! *thread-queue* (frame-caller ($get-frame)))
  ;; Make sure we don't accidentally
  (set-frame-caller! ($get-frame) '*should-never-be-used*)
  ;; Call thunk, then call thread-die!.
  (thunk)
  (thread-die!))

(defn thread-die! ()
  (set-frame-caller! ($get-frame)
                     (queue-pop! *thread-queue*))
  nil)

Switching between coroutines

In some languages, next-thread! would be called yield

To switch between coroutines, all you need to do is to have multiple
call stacks and switch between them. In fn, this is done in the
next-thread! function. next-thread!, when called, will swizzle out
the calling stack frame stored in its own stack frame. That way, the
function can return into a different call stack than the one it was
invoked from.






next-thread! returns into a different call stack than it was called from

next-thread! is implemented as follows:
(defn next-thread! ()
  ;; Push our own frame's caller onto the queue.
  (queue-push! *thread-queue* (frame-caller ($get-frame)))
  ;; Remove a suspended stack frame from the queue,
  ;; and set it as our own caller, so we'll return to it.
  (set-frame-caller! ($get-frame)
                     (queue-pop! *thread-queue*))
  nil)

Caveats about tail calls
Manipulating your own stack frame leads to a simple and short
implementation, but the devil is in the details.
In particular, in languages with implicit and automatic tail calls, an
invocation such as (set-frame-caller! ($get-frame) ...) cannot be
put at the end of a function, of you would manipulate the caller of a
function that won’t ever return again. To work around this problem,
functions like next-thread! and thread-die! return nil
explicitly at the end, so that set-frame-caller! does not get
invoked with a tail call.
The full coroutine implementation is in the fn repository in
examples/threads.fn.


Coronavirus, Italian schizophrenia
Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 14 Sep 2020 11:00:00 GMT
Unsurprising, yes. But very weird to live in.


Real Deal Bridge
Apurva Shukla's RSS Feed — Mon, 14 Sep 2020 08:30:23 GMT
Over a period of four months our game studio team at No Moss Co, led the development of Real Deal Bridge - a realtime multiplayer, cross…


Book Review - Shoe Dog by Phil Knight
swyx's site RSS Feed — Mon, 14 Sep 2020 06:30:30 GMT
One of the greatest autobiographies by a business icon and great writer


Voting Roll Call
The David Brownman Blog — Mon, 14 Sep 2020 03:00:00 GMT
Voting won't solve everything, but it certainly won't make anything worse. These are instructions to help you make sure all your Facebook friends are registered to vote!

Read the whole thing.


Notes on Adversarial Interoperability
swyx's site RSS Feed — Mon, 14 Sep 2020 02:37:49 GMT
Summarizing thoughts from Seth Godin and Cory Doctorow on Interoperability


2020–09–14: Putting 13 PinePhone distributions on a 8GiB uSD card
megi's PinePhone Development Log — Mon, 14 Sep 2020 01:00:00 GMT



Did You Know Fonts Could Do All This?
Venam's Blog — Patrick Louis (Lebanon) — Mon, 14 Sep 2020 00:00:00 GMT
Freetype, included in the font stack on Unix, is quite complex. There are so many layers to get it to do what it does that it's easy to get lost. From finding the font, to actually rendering it, and everything in between. Like most of the world, I use a rather low screens definition (1366x768 with 96 dpi) and rather old-ish laptop, unlike some font designers that live in a filter bubble where everyone has the latest macbook. Thus, good and legible font rendering is important. Let's play with lesser known toggles available to us when it comes to font rendering and see what they do, let's have fun and explore possibilities.


Assassin 33 A.D.
🦉 olshansky 🦁 — Sun, 13 Sep 2020 16:04:38 GMT
Olshansky's review of Assassin 33 A.D.


Be careful with your Docker logs
Heitor's log — Sun, 13 Sep 2020 03:00:00 GMT
Logs are important. But they might fill your disk. Here I show how to configure Docker daemon to rotate logs.


Start With No
Dylan Paulus' Blog — Sun, 13 Sep 2020 03:00:00 GMT

Our default response to any idea that comes up should be: “Interesting. Maybe some day.” In other words, a very soft “no” that leaves all our options open. We don’t put it in a backlog. We give it space so we can learn whether it’s really important and what it might entail.


    _{"Shape Up" - Ryan Singer (Basecamp)}

Here's a story: At a previous employer, I had a boss who I thought was pretty tough. Not to me, but to our users. No matter the request, his response was always the same: "No." Eventually, we implemented around 40% of the requests, but the constant rejection of these requests stuck with me. Now that I lead a team, I notice myself using the same mindset. "No" isn't negative, and it's not dismissive. "No" is a way of protecting the team, the project, and the users.
No?
There is a natural progression to projects that looks something like this: first, the honeymoon phase. The pastures are green; the project has no technical debt, and every feature is a great idea. As we add functionality, our user base grows. If we're lucky enough to get traction, we move to phase two: the monolith. Our application now handles a bunch of use-cases. It does your taxes, picks up your dog's poop, and reminds you to call your mom. Only power users can fully grasp the purpose of the application. The technical debt is increasing with every new feature added. Finally, we get to the last phase. The application has become so bloated that it's hard to develop new features. New users aren't willing to learn the ins-and-outs of the product. A new startup comes around with a trimmed-down version of your application. It's so easy to use! Around this time, users are jumping ship.

Saying "no" is a buffer in our development process. It gives us, the developer, time to think of an optimal solution. Could a feature being asked for be solved in a way that satisfies multiple feature requests? Is there some domain logic we have yet to understand? Does the feature really make sense in this application? All these questions help reduce the lifecycle described above. It reduces feature churn by keeping the application simple and by solving issues the users actually have.
Be mindful of how features interact within your application. Nothing comes without a cost—especially in software development.
No.
"No" is a way of protecting people—the users and the development team. There is a gradient of experience when it comes to using your application. Users will be complete beginners, and others will be power users. As a product owner, we need to balance these two extremes. If the application is too power-user-centric, then it's harder for new users to grasp. When the application is too hand-holding, power users become frustrated. Saying "no" to one extreme is a way of keeping the scales balanced. But be mindful of what you say "no" to. Some features may be vital to the application's success. This is part of the art.
"No" gives time to find the optimal solution to a problem. Users ask for a solution to X and suggest implementing Y, but what they really need is K. Feature requests give us an opportunity to improve processes and workflows. Our users are domain experts but are prone to doing things the way they've "always done them." We can come in and find ways to simplify. Ask the user, "do we really need this step?" or "Could the system handle that instead of a person?" Once code is written and users get accustomed to the feature, it's really hard to change. Do this step upfront.
"No" keeps the development team doing meaningful work. Spending a sprint or two implementing a feature nobody uses or finds frustrating is demoralizing. Think about the workflow before writing any code. Question the feature and the users. Does the application need this? Could we implement it in a simple way? Reduce code churn; reduce developer churn.
No!
Work needs to get done. At some point, you need to say "yes," but instead of taking on every task, you get to be mindful of what to prioritize. You get to take on the tasks that add the most business value and delight users. Use "no" as a tool.


Don't Go Far Off by Pablo Neruda
ho.dges.online — Sun, 13 Sep 2020 03:00:00 GMT
Don’t go far off, not even for a day, because –

because – I don’t know how to say it: a day is long

and I will be waiting for you, as in an empty station

when the trains are parked off somewhere else, asleep.

Don’t leave me, even for an hour, because

then the little drops of anguish will all run together,

the smoke that roams looking for a home will drift

into me, choking my lost heart.

Oh, may your silhouette never dissolve on the beach;

may your eyelids never flutter into the empty distance.

Don’t leave me for a second, my dearest,

because in that moment you’ll have gone so far

I’ll wander mazily over all the earth, asking,

Will you come back? Will you leave me here, dying?


The Guilt Line - Red & Blue's Frontier
A Matter of Perspective — Sun, 13 Sep 2020 03:00:00 GMT
“STOP!” The police officer yelled, “This raucous is over!”
The music ceased immediately, and the end-of-year forest party came to an abrupt end. It was good while it lasted. Some of the teenagers ran for the darkness and safety of the trees, while others, horrified of what their parents would say, stood Blue–struck, red beer cups in hand.
What seemed like a fun, secluded, annual ritual – a chance to dance away the boredom of another school year of tests and assignments – was not looked upon so favorably by the strict, local law enforcement. After all, it was only last year that Johnny was seen handing out weed. The town was on edge.

Blue Vs. Red
Along the frontier of the Blue/Red boundary lies many a conflict:

Terrorists and trained armies.
“Surfer-dudes” and their frustrated bosses, from the cafes they sometimes show up to work for.
An eager youth and a strict school principal.
Criminals and the law.
A rowdy teenager and a concerned parent.
Etc

The hard line between expressive Red and a responsibility-driven Blue underpins many of societies’ interactions and much of our individual lives.
As we’ve learned from our previous forays into these complex perspectives, both are needed. Neither is “right” or “wrong”. And yet, when one is fueled by a righteous and ordered Blue drive or a carefree and in-the-moment Red fuel, we don’t always see the other side. They’re often the enemy: the misbehaving hooligans, or the stuck-up, prissy suits.
Sound familiar?
The Guilt Line

In the Spiral’s evolution, Blue clamps down on an overactive Red that was taking things too far. We learned that one of Blue’s powerful assets in controlling Red is guilt. A firm, disciplined, fear of doing something wrong in the eye of the law/god/ism/Truth/company policy.
In the diagram above, Red rises until, usually, something catastrophic happens, and society steps in. A concrete line is drawn, and anybody who crosse this line will be severely punished. You’ve been warned.
Here’s the thing, however. This line is dependent on how healthy your Blue is, AND your acceptance of Red.
A Low Guilt Line = No fun


As the guilt line drops, so too does healthy expression and self-acceptance of ones desires (culturally or individual)

For example, let’s take a party-going highschool-dropout, who after a dance with alcoholism and a brief intervention from the local church, get’s his shit together and becomes church’s accountant.

How do you suppose this “structure” (Red -> Blue) would respond to his children going out to party? I think it’s fair to say he would not be offering to pick them up at 1 am from the beach celebrations.
Likewise, the same applies to countries. After the bloodshed, chaos, and warring disputes, the “peace” that Blue brings is a breath of fresh air. Countries like Japan enforce incredibly strict social norms and rules to limit Red’s expression - often resulting in very unhealthy behavior.
“You can be Red, but only on a Friday or Saturday night, between the hours of 7 pm and 11:59 pm, and it but be contained to this small district we’ve set aside for you.”
This sentiment is exceptionally damaging (as is any over-suppression of the natural Spiral). It creates friction (internal and externally), crippling guilt, sexual frustration, “working one’s self to the bone”, stress, “uptight” ness, boredom, lack of creativity and joy, and many other issues.
In some cases, for those whose “guilt line” has been too low for too long, they’ll snap one day. They’ve sacrificed themselves for society/others, allowing nothing for themselves. Suddenly, the long-term, high-valued employee quits their job, gets a divorce, moves to Hawaii, and buys a Ferrari. A typical midlife crisis – Red violently bursts through Blue.
Too high a Guilt Line = Chaos


As the guilt line rises, order, discipline, and tempered rationality gives way to extreme behavior, ego, and power struggles.

Of course, the converse of everything above is also true. If Blue laxes too much, allowing Red to walk over it as it pleases, other problems arise.
For example, when crime becomes prevalent (sometimes organized by small amounts of Orange) and the police are too weak, corrupt, or under-funded to control it, Red can claim entire nations. Despite potential external meddling from the CIA, Pablo Escobar is an unfortunate illustration of Red dominating a still developing Blue Colombia.
At one point, an extreme Red was controlling Colombia’s entire country
from a “prison” that HE designed for himself. It crippled Colombia’s people (who thankfully have recovered) and much of South America’s reputation.
Red must be kept in check. Not so much that it limits the self to the point of a puristic chastise, but enough to protect and preserve the safety of all. Lowering the guilt line when Red has reigned for a time is a bloody business.
Finding the balance
I wonder if you can recognize your own guilt line. Indeed, it often fluctuates depending on the area of our lives. How far do you go before the sense that you’re doing something wrong kicks in? Do you end up hurting people (intentionally OR Unintentionally)? Or are you too strict on yourself and others, not giving space to “take the day off” or “dance the night away”.
If you’re fueled by Red, living in a dominant Blue (which is most of the world), is the line too low for you? Too high?
If Blue fuels you, how much Red can you tolerate? Do you put your foot down at the slightest sight of “unruly behavior” or “bad manners”? Or do you recognize that we all possess a bit of everything, and join in on the fun?
Indeed, different situations call for the raising and lowering of our Guilt Lines.
As always, there are no definitive answers to any of the above. There are only the tools we develop and discover – using and practicing them in the game of life. Like an archeologist, we brush away our own bias and dig a little deeper into the world around us – driven curiously to uncover the mysteries of others world-views.
And now we have another tool.


Modelling a Wealth Tax Correctly
Krishna's blog — Sun, 13 Sep 2020 03:00:00 GMT
This article examines the accuracy of modelling a wealth tax and critiques Paul Graham's argument against it.


You do not have a Right To KEEP Buying (*)
Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 12 Sep 2020 21:00:00 GMT
And that is BAD.


Cheese cake recipe (no crust)
xenodium.com @alvaro — Sat, 12 Sep 2020 03:00:00 GMT


Preheat oven
Preheat oven at 175°C.
Ingredients at room temperature
Ensure the cream cheese, sour cream, and eggs are at room temperature before starting.
Mix cream cheese

900g of cream cheese

Mix the cream cheese thoroughly.
Mix sugar

240g of sugar

Add half the sugar. Mix in thoroughly. Add second half and mix.
Mix sour cream, corn flour, and vanilla.

100g sour cream
40g corn flour
1tbsp vanilla bean paste

Add the three ingredients and mix well.
Mix eggs

3 eggs
1 egg yolk

Add the eggs and mix for 30 seconds.
Mix by hand
Finish mixing thoroughly by hand, using a wooden spoon.
Prepare pan

Springform pan
Parchment paper

A springform pan works best here. Wrap its plate with parchment paper and lock it in place.
Pour mix

Strainer

Pour the mix through a strainer and into the prepared pan.
Rest mix
Let the mix rest in the pan for 10 minutes to let air bubbles out.
Bake
Bake for an 1 hour 10 minutes. Maybe add another 10 minutes (or more) if surface is still pale. Turn the oven off, leave door half open, and let it sit for 20 minutes.
Cool off
Take out and let it cool off to room temperature.
Refrigerate
Refrigerate for 4 hours (or overnight) before removing the sides of the pan.
Eat!
Nom nom. Yum yum.
Bonus (topping)
I winged this one and it worked out well. Heated up frozen berries with some honey and used it as topping. The whole combo was pretty tasty.


I Want to Fix Goodreads
PREPEND — Sat, 12 Sep 2020 03:00:00 GMT
I use goodreads a lot because I like to read books and want to find more books to read. This morning I was reading a thread on hackernews on Sarah Manavis’s NewStatesman article and it reminds me of how much I want, I yearn, for useful book recommendations. I think about this pretty much every time I open the goodreads page or app and hope that maybe this is the day they finally fix recommendations, so I figure I would capture some ideas I have for fixing their recommendations.

I Love Goodreads

First, I love Goodreads. My profile says I joined May 2012. But I think that’s just whenever I started this profile and I have multiple, old profiles that I can’t find. My memory says I started using them soon after launch, but maybe I’m misremembering. But I’ve used them for a long time because I like to read books. I love browsing bookstores and libraries looking for new books. I experience joy finding a 50 or 100 year old book that I didn’t know about. One of my favorite things is reading a book so good that I can’t go to bed, or want to walk an extra mile. But that’s pretty rare for me, I’m lucky if I get one of those a year.

I have goodreads on my link stack with the tag “purdah” that I think means it is part of the stuff I do that uniquely identifies me.

Any time I get a recommendation from a friend or article or wherever, I add it to my “to read” shelf on goodreads. That helps me when I’m trying to buy a new book. So at least I can read through the 245 books that at least a past me wanted to read. This works well and is better than a vanilla list.

I also like seeing what my friends read and especially what they like or dislike. My friends have different tastes so it’s not as simple as just reading everything they read, but it helps a little. It’s one of the few places that I think facebook helps because at least I get to find out facebook friends who also are on goodreads. Every once in a while I find a new book through this feed, maybe once every 2 years. This might be because I don’t have many friends on goodreads (only 42), but I think it’s also because different people have different tastes in books.

I also like that goodreads will tell me when an author I’ve read releases a new book, or is having a book chat or something. Although I wish they would tell me when they come to a local book signing.

The Problem As I See It

This gets me to what I think is wrong and want to fix. I have read and rated 802 books with my average rating of 3.97. This makes sense as I’m probably only going to read books I think I’ll like, but I think this is the average goodreads rating, not the average of my ratings. Looking at what I rated, I’ve only given out 3 one stars, 42 two stars and tons of five stars. I keep scrolling on and on.

The user interface for goodreads is pretty bad- information is hard to find, layout is wasted, search doesn’t work well, hard to tell ads from user content- but that’s not what I’m talking about here. The main problem is in finding books that I will like. Maybe a UX overhaul from a great designer would help, but not that much. A craigslist interface would be great if they could just solve this one problem…


  What is the single book that I will love most in all the world?


I want to find the next amazing experience. Books are great because they are cheap and portable. It’s not like wine where the best is more than I’ll ever pay. Or travel where a first class flight can cost $20k. If I can find a book, I can read it. And maybe someone I know can read it too.

If I can mix in more wonderful books, my life will be better. My family’s life will be better. Etc etc. Maybe this is the answer to world enlightenment. Probably not, but it will be great, I think.

When I click on recommendations, goodreads shows me five books that I don’t want to read. And it’s weirdly bad. Some of it makes since why it’s bad. They recommend different versions of books I’ve read. They recommend two different versions of Lord of the Rings (one of my favorite books), but I guess they don’t know these are the same book. That’s just a waste of time and it’s weird that amazon isn’t smart enough to avoid duplicative books.



But then they recommend stuff like Stephen King’s The Long Walk. This is perplexing because I only have one book that goodreads knows I’ve read from King and I rated it two stars. I don’t have any of his books in my “to read” list. Why would they recommend this book? Two of my friends rated it four stars and one even wrote a recommendation. So maybe that’s why. I’m aware of this book and don’t want to read it. But other books like Shoe Dog by Phil Knight has no relationship to me. Why would they recommend this? I also know about this book and don’t want to read it. I wish there was some way to note books that I don’t want to read. Then at least they would stop showing me the same book over and over.

So it seems like goodreads is making pretty basic, rules-based recommendations. And they don’t help me pick new books.

My Perfect Wish

My idea for an 11-star experience ¹ in finding new books is that Goodreads knows me even better than I know myself and constantly recommends the perfect book. They recommend books that teach me, that spark enlightenment, that I hate but teach me to appreciate other books, books for when I want to be happy, and books for when I need to be sad. I want something like Jane from Ender’s Game who experiences everything I know but is fixated on finding me a good book. So the output of a book sommelier whose passion is seeing me sit there and love a book.

That’s what I would like.

A Little Story About Jeff Bezos

Once I was lucky enough to go to TED. It was great on so many levels, I highly recommend it and 2/3 of the cost was tax deductible. It was full of great stuff. I got to wait in the registration line in front of Neil Gaiman and Amanda Palmer, that was neat.

Anyway, one night Neil Gaiman was doing in a midnight ghost story reading in some neat four story townhouse type building ² that was totally empty and spooky and a great place for the event. The reading was on the top floor and there was a huge crowd waiting for the single elevator in this place. The lights were off and there weren’t any staff around so it was maybe 100 people waiting to go up three at a time. So I looked around the corner and took the stairs. About 2 or 3 other people had the same idea so we started racing up the stairs to get to the top and avoid the crowds.


photo by Joe Fox

This building was awesome. It seemed 100 years old, rare in Vancouver, and as we went up the stairs we saw each floor. The stairwell opened up to a big room that could probably seat a 50 person wedding dinner. The lights were off, and someone had opened a few windows and the white, wispy curtains were blowing with a little breeze. Even though it didn’t feel like a breeze. I said “cool, haunted house” and stopped climbing the stairs to look around. The person next to me said “yeah, cool” and we started quickly running around checking out the empty room. We did this on the third floor too and me and this random person did quick 60 second explorations of this spooky house.

It all happened quickly. Maybe 3 minutes to go up the stairs, run around each floor, and get to the top. About halfway through I realized this random dude was Jeff Bezos. I thought that was neat. I’ve bought books from Amazon since 1995 and I’m pretty sure I telnetted into it to order some books (although maybe that was cdnow before they bought them), so it was cool to think that Jeff Bezos likes haunted houses and still likes exploring empty buildings.

We got to the top and went into the venue and were maybe the first 10 people. Comically got there maybe after two or three elevators full of people. We went to the front where Gaiman was setting up at the podium. I got a comfy chair right in the front about five feet in front of Gaiman. Jeff Bezos sat in the chair to my left. The chair on my right had Neil’s wife, Amanda Palmer, who was carrying a ukelele case. I didn’t know it was a ukulele case until she told me. She also said she might play, but she didn’t.

Gaiman finished setting up and came up to talk to Bezos. Bezos asked if Gaiman got the books he sent. Gaiman said he did and they were really great and he enjoyed reading them. It seemed like this was the most recent in a series of book shipments.

Aside from nerd joy from getting to witness this small talk of people who interest me, I thought, at the time and many times since,


  How cool is it that Neil-freaking Gaiman is still finding new books that are great, and how cool would it be to get book recommendations from Jeff Bezos. I bet he has access to the best book recommendations in the world.


Also, TED has something called the “TED book club” where they mail you this box of books a month before the conference. This was the closest I’ve come to my dream. They maybe sent 10 and 5 are still on my bookshelf. But not the shelf behind me, so I don’t remember them specifically. I’ll look them up one day.

That’s probably better than Jane making recommendations. Amazon had just bought goodreads a year earlier and I was hopeful that any day, goodreads would roll out a wonderful recommendation engine any day now.

What I Want to Do to Fix

What I would like to do is to just get a data dump of mine and everyone on goodreads recommendations. I don’t need to know the identities, just to know what recommendations go with what individual. It would be neat to protect privacy and find individuals to potentially follow them on goodreads, but I think for the analysis I want to do, I don’t need to know the actual identities.

I think if I have this data, I could try to find people who rate a few hundred books like I have, then look at what books they’ve read the most, or books that they’ve rated highest. I think that would be a good signal that I might like it.

It would also be cool to see books they hated. Just to see uniqueness amongst the millions of readers.

It would also be cool to find my exact opposite. People who rate the absolute opposite from me, my antipode. ³

If I just had this data, I would do what goodreads isn’t doing for me. Maybe they’ve already tried and it’s impossible to achieve. Maybe it’s just one of those things where we have to increase opportunities and leverage serendipity.

They do have an api but it seems meant for integrators to do goodreads functions from an app or something. I can get my own recommendations, but there’s no way to get other people’s ratings, and unless I know the name of the book I can’t look up ratings. And that’s sort of the problem since I don’t know the name of books I don’t know. They do allow me to view a specific user’s info and even lets us compare shelves.

I wish they had a data dump somewhere. It’s strange to me that lots of users allow their ratings to be public (eg, this random friend user), so I can theoretically scrape all the reviews from just public users, but that’s a lot of work. I share my ratings because I want to help others. I assume that other users do so for the same reason so I wish Goodreads/Amazon would help facilitate this. Since userids are just 8 digits, I supposed I can use their API to read one user a second (their requested rate limit) and it would just take me and it would just take me 3.17 years, or maybe less since they may not have 99,999,999 users, but Statistica claims they had 90 million registered users on July 2019. ⁴

Housekeeping

Unanswered Questions


  Low quality of reading random people’s reviews and comments


Related Blog Posts


  How to identify yourself, when yourself changes? Neal Stephenson’s PURDAH.


Future Blog Ideas


  Describe the link stack idea.
  Emergent awesomeness. When you can’t explicitly do something, you just have to increase the rate of random awesome things and appreciate them when they happen.


Stuff I Read While Working on This Post (That You Might Want to Read Too)


  Why Goodreads is bad for books
  https://en.wikipedia.org/wiki/Amazon_(company)
  https://en.wikipedia.org/wiki/Goodreads
  Ender’s Game by Orson Scott Card
  Known knowns
  https://www.ted.com/attend/conferences
  https://www.neilgaiman.com/
  https://amandapalmer.net/
  https://www.goodreads.com/api
  https://en.wikipedia.org/wiki/Gentlemen%27s_club


Footnotes


  
    
      https://mastersofscale.com/brian-chesky-handcrafted/ Masters of Scale podcast with Brian Chesky where he describes AirBNB process of trying to design experiences where users rate something 11 stars (out of 5). I find it funny that AirBNB has never given me an 11 star experience, but lots of 5 stars. And I think that’s the message here. ↩
    
    
      https://vancouverclub.ca/ I think it was The Vancouver Club, so imagine this place but at midnight, empty and abandoned. Wikipedia says it was founded in 1889. ↩
    
    
      https://en.wikipedia.org/wiki/Antipodes I would also like to see this antipode functionality for twitter. Who is the person with the exact opposite of me on Twitter. Would be neat to read their ideas. ↩
    
    
      https://www.statista.com/statistics/252986/number-of-registered-members-on-goodreadscom ↩

Created:	October 2020
Size:	15x15cm and 13x13cm
Paper:	Fabriano Bristol 250gsm
Pens:	Molotow Blackliner, Koi Coloring Brush, Mechanical pencil

id	quality	codec	examples
571	4320p60	AV1	`youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ`
272	4320p30	VP9	`youtube-dl -F i6fWG4FxDZw`
337	2160p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
401	2160p60	AV1	`youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ`
305	2160p60	AVC	`youtube-dl -F -xNN-bJQ4vI`
315	2160p60	VP9	`youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI`
266	2160p30	AVC	`youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw`
313	2160p30	VP9	`youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
336	1440p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
400	1440p60	AV1	`youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ`
304	1440p60	AVC	`youtube-dl -F -xNN-bJQ4vI`
308	1440p60	VP9	`youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI`
264	1440p30	AVC	`youtube-dl -F -xNN-bJQ4vI i6fWG4FxDZw`
271	1440p30	VP9	`youtube-dl -F kFz9afj8lu0 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
335	1080p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
399	1080p60	AV1	`youtube-dl -F 1La4QzGeaaQ`
299	1080p60	AVC	`youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI`
303	1080p60	VP9	`youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI`
137	1080p30	AVC	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
248	1080p30	VP9	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
334	720p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
398	720p60	AV1	`youtube-dl -F 1La4QzGeaaQ`
298	720p60	AVC	`youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI`
302	720p60	VP9	`youtube-dl -F 1La4QzGeaaQ -xNN-bJQ4vI`
136	720p30	AVC	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
247	720p30	VP9	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
333	480p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
397	480p30	AV1	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ`
135	480p30	AVC	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
244	480p30	VP9	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
332	360p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
396	360p30	AV1	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ`
134	360p30	AVC	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
243	360p30	VP9	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
331	240p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
395	240p30	AV1	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ`
133	240p30	AVC	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
242	240p30	VP9	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
330	144p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
394	144p30	AV1	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ`
160	144p30	AVC	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
278	144p30	VP9	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`

id	quality	codec	examples
332	360p60 HDR	VP9	`youtube-dl -F 1La4QzGeaaQ`
18	360p30+96k	AVC+m4a	`youtube-dl -F S8Zt6cB_NPU`
134	360p30	AVC	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
243	360p30	VP9	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ -xNN-bJQ4vI i6fWG4FxDZw`
396	360p30	AV1	`youtube-dl -F S8Zt6cB_NPU 1La4QzGeaaQ`

id	quality	codec	examples
258	386k	m4a	`youtube-dl -F NMANRHz4UAY`
256	195k	m4a	`youtube-dl -F NMANRHz4UAY`
251	160k	Opus	`youtube-dl -F S8Zt6cB_NPU`
140	128k	m4a	`youtube-dl -F S8Zt6cB_NPU`
250	70k	Opus	`youtube-dl -F S8Zt6cB_NPU`
249	50k	Opus	`youtube-dl -F S8Zt6cB_NPU`

id	quality	codec	examples
22	720p30+128k	AVC+m4a	`youtube-dl -F S8Zt6cB_NPU`
18	360p30+96k	AVC+m4a	`youtube-dl -F S8Zt6cB_NPU`

Created:	September 2020
Size:	52x27cm
Paper:	200gsm
Pens:	Sakura Gelly Roll 0.8mm pen

类型	阻塞	超时
插入	putFirst(e)	offerFirst(e, time, unit)
移除	takeFirst()	pollFirst(time, unit)

类型	阻塞	超时
插入	putLast(e)	offerLast(e, time, unit)
移除	takeLast()	pollLast(time, unit)

队列方法	等效的双端队列方法
add(e)	addLast(e)
offer(e)	offerLast(e)
remove()	removeFirst()
poll()	pollFirst()
element()	getFirst()
peek()	peekFirst()

Public Suffix	Domains
nic.in	2454
gov.in	7259
in	528
ac.in	490
com	568
co.in	171
org.in	168
edu.in	117
org	844
res.in	134
net.in	12
net	38

Domain	IP Address
aavin.tn.gov.in	164.100.134.148
abnhpm.gov.in	14.143.233.34
agnii.gov.in	13.232.216.65
ap.gov.in	117.254.92.53
aponline.gov.in	125.16.9.130
appolice.gov.in	118.185.110.147
attendance.gov.in	164.100.166.114
cgg.gov.in	112.133.222.115

Hacker News Personal Blogs 2020 | All Blog Posts

My favourite books of 2020

A smart guide for your career as a software engineer

New Year's Resolution 2021

Soul

Object Detection at 2530 FPS with TensorRT and 8-Bit Quantization

Sex workers must go POSSE

Implementer's Guide to SOCKS

2020 Year in Review

My Favorite Privacy Apps and Services

2020 in 13 Seconds

Highlights

Lowlights

Underusing the NHS

Dynamic Programming (i): Multiplying Matrices

My Path to Developer Relations and Thoughts on the Future of DevRel

My Journey from No CS Degree to AWS from Age 30 - 34

Swyx Interview on Daily Dev Tips

Financialization of Big Tech is BIG

The Road Chose Me Volume 2: Three years and 54,000 miles around Africa

Python write_file utility

The ideal blog platform

Self hosted blogs

Addendum on hosting

BOINC dans un conteneur Docker

Installation de docker

The globalization of evil autocompletion

A Modernist Christmas

Single Sign-On and Resource Separation on AWS

Kendall Peak Lakes Snowshoe

Swift package code coverage (plus Emacs overlay)

Hiking bookmarks

LitElement Web Components in React App

Things I worked on in 2020

Accurate Estimations

Why Estimations are Hard

Two Types of Estimators

Playing Single-Pointed Darts

Playing Darts with Ranges

Simple Changes

Footnotes

Plan Your Year

node.example.com Is An IP Address

Working at Amazon

jlink's Missing Link: API Signature Validation

Speedrunning Computer Games History with a 6yr Old - Part 1

Why?

What do you have against Minecraft/Roblox?

A Modern “Home Computer”

Hardware

Software

Arcade

ZX Spectrum

Atari 2600

Apple II

Commodore 64

ZMachine

I’ll report back later…​

Horrors and errors of SOME independent book sellers

Wenatchee Crest Snowshoe

Patience

Futureproof

The Pragmatic Programmer - Review

❗️ Some criticisms of the book: #

Tip 17: Program close to the problem domain #

Chp 3 - The Basic Tools #

✅ Major (Useful) Points of the Book #

⚖️ Involve Your Users in the Trade-Off #

🍸 DRY #

📐 Orthogonality #

🎯 Tracer Bullets #

🤔 Estimates #

🐛 Debugging #

Various debugging tips #

📃 Design by Contract #

🎛 Design for Concurrency #

🧹 Refactoring #

When to refactor? #

How to refactor? #

📚 Requirements #

I’ll report back later…

# Final `test.dhall`