Hacker News Personal Blogs 2023 | All Blog Posts

An Ordinary Disaster in 2024

An Ordinary Disaster — Sun, 31 Dec 2023 21:27:04 GMT

Prepare yourself, create options—or just walk.

2023 in review

callmeolive — Sun, 31 Dec 2023 21:15:00 GMT

2023 was certainly a year. Our Prime Minister got fined for not wearing a seatbelt, Paul O’Grady’s passing left millions of dog lovers and radio fans mourning, and shortly after some bloke got crowned king in a ceremony full of high-definition horse manure. The long-running series of Governmental cock-ups continued as they lost more seats and dithered from crisis to crisis, the usual. Also the F-22 got it’s first air-to-air kill against a balloon of all things.

At the halfway point we all became submarine experts, Barbenheimer took the internet by storm until corporations tried to be funny and Hollywood writers went on strike probably in retaliation. The Beatles released their final song until the next one comes along, David Tennant returned briefly to Doctor Who, and I couldn’t think of any news from December to put here (E3 being officially over feels too lacklustre, even for this).

Reddit, after going public, decided to axe it’s free API promise which killed off all the good mobile apps and led to an exodus from the site – but in their hopes to get the coveted digital service cock-up of the year award they were swiftly beaten by Elon Musk. In a dramatic move he finally renamed Twitter to Formerly Twitter, and in a desperate attempt to keep money flowing in told advertisers to go fuck themselves. 🤔

Google, deciding that having the largest market share of internet browsers, search engines, and phone operating systems wasn’t good enough, tried to wrangle more power from the web with a dodgy DRM system they realised nobody actually wanted. They also tried blocking ad blockers on YouTube. Ad blockers just got better.

All throughout this the continued push in AI brought forth major strides in image and text generation, yet as a Language Model I am unable to discuss such a topic in the form of a ‘quirky and engaging blog post’.

Myself? Same old. Started some naff blog, bought a Steam Deck, went through the install Linux on a ThinkPad rite of passage, said goodbye to my old desktop as I built a new one (RIP R9 390). Finally caught Covid, one tooth removed, went on antidepressants twice, GPs proved to be as useless as ever, life goes on xx

The stats

1 new website
14 blog posts
5 years of sobriety
12 books read
20 games I purchased on sale and never played
1 extremely sleep-deprived article pitch to The Guardian which they didn’t respond to because ~~the liberal agenda fears me~~ ~~it had no typos~~ it was bad.
11 times my gay cat caught something (not including fleas)
Word of the year: Meandering
Favourite game: Super Metroid

Blog posts wot didn’t make the cut

Recapturing imagination – Lamenting the loss of my childhood imagination, I went on a journey to seek it back. I fizzled out after a few weeks.
When search engines don’t search [what you want] – A quick dive into search engines being useless on a whim. And advertisements.
Learning to love Linux – Repetitive. A lot of things worked until they didn’t until I tried Kubuntu.
[something about declining political standards] – Everyone thinks they’re a pundit.
Why are lights so bright? – Every device has a retina burning LED, adding resistors costs money, and I simply didn’t get around to fleshing it out
People to send straight to hell – “Satirical” list of minor actions & behaviours that ought to immediately sent people to eternal damnation.

That about wraps it up, ignoring everything I missed. Hopefully 2024 is better xoxo

Olive

New Year's Resolutions 2024

Syd Norpol — Sun, 31 Dec 2023 19:38:53 GMT

Resolutions #

make connections 🧑🏼‍❤️‍🧑🏻
get better 💝
be better 🌁
explore ways to self-expression 🌼
treat every day like a new day 🌄
celebrate success, learn from mistakes 🫙
accept 💨

Backstory #

Hi, it’s been a while - but I’ve most certainly never stopped thinking about you.

Last year when writing my resolutions I was quite behind and almost forgot since I had quite some trip around that time.

Firstly back then I just quit my job and went for a trip, first to Canary Islands and later to Berlin, where I’ve just arrived a day prior. Canary Islands were quite tough, I was just exhausted from finishing my previous job and going anywhere. The back then friend was quite hard to deal with and if Lanzarote wasn’t that beautiful I would have almost had a breakdown otherwise.

Berlin was great, I’ve met sometimes several people per day, visited to concerts and museums.

Around 14. January I started my new job, at an office in Vienna for the first two weeks. …

Anywho. I think I’ll give another recap of my 2023 in another posting. What I wanted to say is I didn’t spent a lot of time writing my resolutions a year ago.

Today is different, I’m at my apartment in Portugal. Unfortunately my friends who wanted to join just canceled for tonight, I’ve prepared Pizza and planned for a nice evening together. Let’s use this as an opportunity to explore something new or not I’ll see.

I’ve been thinking about these resolutions for quite some time. How can I formulate them in a for me meaningful and helpful matter. Most recently I thought of maybe shifting the idea of resolutions a bit and rather do a more OKR like format. For that I’ll write my main theme for the year and then try to see what’s next on a quarterly basis.

Thank you for being there. xoxo

Some nuggets of wisdom from Charlie Munger

vslira's blog — Sun, 31 Dec 2023 19:18:04 GMT

Stripe Press recently released a new edition of Poor Charlie's Almanack, exactly a week after Munger sadly passed away. I had preordered the book and picked it up a few weeks after release. There are many good quips and general good advice in there and I will share the most interesting tidbits to me.

First, the "mental model checklist". Charlie Munger is famous for his "mental models" framework and many outlets - books, websites, content writers - push mental models either because they believe in the framework or, more cynically, because people buy it trying to be more like Munger (that is, rich). I don't have a lot to add here, other than that when I read about this "mental models" frameworks, it never came to me the idea of a literal checklist. But that's how Munger explains his framework: a literal checklist.

You go one by one, see which ones apply and - crucially important, according to him - when many apply (as in most real life cases) one has to figure out how these systems interact. I don't know if I will adopt this practice, but whenever anyone refers to "mental models" to me in the future, I'll make a note to try to use them as a checklist instead of just learning them and hoping my subconscious mind does the job of pulling the right ones at the right time.

To be honest, if I could risk a lesson that Munger may or may not have expressed himself, I'd say he's very critical of relying on your subconscious mind to make decisions for you. One should be aware of how one thinks when making decisions.

The second lesson that I took a liking for was his view of a little unfairness being more fair than fairness in all interactions. Specifically, he mentions that in the U.S. Navy a captain's career is over if his ship turns around. It doesn't matter if it was his fault: his career is over. This may not be fair to the individual captain, but keeps every other one on their toes and the increased tension saves many lives as a result. Sometimes I wonder if this, intentionally or not, is happening in high-performance but high-stress careers, famous for being unforgiving. Maybe it's better this way?

avatar.png

Reilly Tucker Siemens — Sun, 31 Dec 2023 14:56:00 GMT

No, not that Avatar. And not the other one either. This post is about avatar.png, a handful of lines of PHP that have inspired me for a long time.

Around 2011 or 2012 a friend of mine, Andrew Kvalheim, blew my mind when he made his Skype profile picture display the IP address of the computer I was using. It might have looked a bit like this.

Meet the InfoSec Planet

Gonçalo Valério — Sun, 31 Dec 2023 14:13:04 GMT

If you are a frequent reader of this blog, you might already know that I created a small tool to generate a simple webpage plus an RSS feed, from the content of multiple other RSS sources, called worker-planet. This type of tool is often known as a “planet”: In online media a planet is a […]

Whale Song - Some Background

Greg Herlein — Sun, 31 Dec 2023 10:00:01 GMT

Recent press about an actual conversation with a whale was preceeded by ground-breaking work by Dr. Michelle Fournet and profiled in the fantastic documentary Fathom on AppleTV. This is a MUST WATCH if you love the ocean - or humanity!

A small reprieve from poorly executed ideas

utf9k — Sun, 31 Dec 2023 04:20:00 GMT

Maybe one sector's recession is another person's happy place?

Summary : Little Bets by Peter Sims

Chester Grant — Sun, 31 Dec 2023 03:55:18 GMT

1. When success is the only acceptable outcome, Little Bets advocates a bold and radical approach in which failure is good,

2. I had worked before then as a venture capital investor, and in that work, I had learned that most successful entrepreneurs don’t begin with brilliant ideas—they discover them.

3. These creators use experimental, iterative, trial-and-error approaches to gradually build up to breakthroughs. Experimental innovators must be persistent and willing to accept failure and setbacks as they work toward their goals.

4. Little Bets is based on the proposition that we can use a lot of little bets and certain creative methods to identify possibilities and build up to great outcomes. At the core of this experimental approach, little bets are concrete actions taken to discover, test, and develop ideas that are achievable and affordable.

5. They begin as creative possibilities that get iterated and refined over time, and they are particularly valuable when trying to navigate amid uncertainty, create something new, or attend to open-ended problems.

6. When we can’t know what’s going to happen, little bets help us learn about the factors that can’t be understood beforehand. The important thing to remember is that while prodigies are exceptionally rare, anyone can use little bets to unlock creative ideas.

7. Fundamental to the little bets approach is that we:

Experiment: Learn by doing. Fail quickly to learn fast. Develop experiments and prototypes to gather insights, identify problems, and build up to creative ideas, like Beethoven did in order to discover new musical styles and forms.
Play: A playful, improvisational, and humorous atmosphere quiets our inhibitions when ideas are incubating or newly hatched, and prevents creative ideas from being snuffed out or prematurely judged.
Immerse: Take time to get out into the world to gather fresh ideas and insights, in order to understand deeper human motivations and desires, and absorb how things work from the ground up.
Define: Use insights gathered throughout the process to define specific problems and needs before solving them, just as the Google founders did when they realized that their library search algorithm could address a much larger problem.
Reorient: Be flexible in pursuit of larger goals and aspirations, making making good use of small wins to make necessary pivots and chart the course to completion.
Iterate: Repeat, refine, and test frequently armed with better insights, information, and assumptions as time goes on, as Chris Rock does to perfect his act.

8. Two fundamental advantages of the little bets approach are highlighted in the research of Professor Saras Sarasvathy: that it enables us to focus on what we can afford to lose rather than make assumptions about how much we can expect to gain, and that it facilitates the development of means as we progress with an idea.

9. Seasoned entrepreneurs, she emphasizes, will tend to determine in advance what they are willing to lose, rather than calculating expected gains.

10. Of course the subject of affordable losses highlights a key issue with the little bets approach—it inevitably involves failure.

11. In almost any attempt to create, failure, and often a good deal of it, is to be expected.

12. Hewlett’s approach to identifying new opportunities using small bets did not come without numerous failures. In 1971, HP featured over 1,600 products in its catalog, none of which averaged sales of more than ten units per day, according to Chuck House. In fact, Hewlett estimated that roughly only six out of every 100 new HP products would become breakout successes.

13. By advocating the little bets approach, I am in no way arguing against bold ambition. Ambitious (dare I employ the overused word audacious) goals are essential.

14. A big vision provides the direction and inspiration through which to channel aspirations and ideas. But one of the most important lessons of the study of experimental innovators is that they are not rigid in pursuit of that vision, and that they persevere through failures, often many of them.

15. Central to Pixar’s success in fostering this growth mind-set through the ranks is the company’s attitude about failure. Pixar’s managers see a host of failures, false starts, and problems as the modus operandi for developing their films. In fact, when Ed Catmull sums up Pixar’s creative process, he describes it as going from suck to nonsuck.

16. Pixar film ideas begin on rough storyboards that suck until they work through thousands of problems throughout the process in order to take films from suck to nonsuck.

17. Of course, just failing is not the key; the key is to be systematically learning from failures.

18. One of the methods that can be most helpful in achieving this balance, in order to embrace the learning potential of failure, is prototyping. What the creation of low cost, rough prototypes makes possible is failing quickly in order to learn fast.

19. Failing quickly to learn fast is also a central operating principle for seasoned entrepreneurs who routinely describe their approach as failing forward. That is, entrepreneurs push ideas into the market as quickly as possible in order to learn from mistakes and failures that will point the way forward.

20. Novelist Anne Lamott believes that every good writer writes what she calls shitty first drafts.

21. The only way I can get anything written at all is to write really, really shitty first drafts,” Lamott writes in Bird by Bird. Just get it down on paper, she recommends. Write like a child, whatever comes to your mind. “All good writers write them. This is how they end up with good second drafts and terrific third drafts.”

22. Ed Catmull’s belief that it’s better to fix problems than prevent errors.

23. The level of feedback you get is so much more valuable and impactful…. The problem with showing something to consumers when it’s almost totally done, people don’t necessarily want to give negative feedback at that point because it looks like, “This company has spent a lot of money already getting it to this stage and now I’m going to tell them, ‘It sucks.’” On the other hand, if something hangs together with tape, and it’s clear that it’s an early prototype, the mindset of consumers often is, “These people still need some help, so let me tell you what I really think about it.”

24. Analogous to Limb’s findings, Ansari and Berkowitz found that during improvisation, the right-temporoparietal junctions of the pianists’ brains were deactivated. Neuroscientists associate this area of the brain with the ability to make judgments, particularly about differences differences between self and others. The experienced pianists seemed to be able to turn off a judging part of their mind, freeing them up to create novel melodies. According to Berkowitz, brain scans of nonartists do not exhibit a similar pattern, which suggests that experiencing creative processes could help to build certain creative muscles.

25. There are several major improvisation principles. One is that you should “accept every offer.”

26. So, for example, if two people were performing an improvised skit, Bob might say to Sherry, “I was thinking we could watch Silence of the Lambs tonight.” To which Sherry would accept that offer by saying something like, “Yes, and we can still have time to watch the Late Show afterward.” To which Bob might reply, “Yes, and then we can check email!” It’s a simplified and somewhat silly example, but the point of accepting every offer is that nothing is too silly.

27. After all, if Sherry had spurned Bob’s initial offer by replying, “That’s a stupid movie,” Bob would start thinking and censoring himself, shutting down the possibilities that might come. The effect is deadening.

28. That Bob and Sherry lift one another up relates to another foundation of improvisation: Make your partner look good. Because Bob and Sherry aren’t criticizing one another, it creates a positive atmosphere to generate possibilities. Positive energy drives improvisation, and reduces inhibitions and doubts. By making each other look good, it’s easier for Bob and Sherry to get in a zone.

29. Throughout the Pixar creative process, they rely heavily on what they call plussing; it is likely the most-used concept around the company. The point of plussing is to build upon and improve ideas without using judgmental language. Creating an atmosphere where ideas are constantly being plussed, while maintaining a sense of humor and playfulness, is a central element of Pixar’s magic.

30. Rather than criticize an idea in its entirety (even if they don’t think it’s good), people accept the starting point before suggesting improvements.

31. Instead of criticizing the sketch or saying “no,” the director will build on the starting point by saying something like, “I like Woody’s eyes, and what if we …” Again, notice the use of the word and rather than a word that implies a judgment, such as but.

32. A host of studies indicates that humor creates positive group effects. Many focus on how humor can increase cohesiveness and act as a lubricant to facilitate more efficient communications.

33. One reason for that is that humor has also been demonstrated to increase trust. In order to produce positive mental effects, however, researchers Eric Romero and Anthony Pescosolido found that humor first must be considered funny to the people involved, not seen as demeaning, derogatory, or put-downs.

34. Romero and Pescosolido argued, based on a broad assessment of humor research, should affirm group identities in terms of: who we are, what we are doing, and how we do things.

35. Research evidence suggests a strong link between inquisitiveness and creative productivity.

36. Innovators routinely networked with people who came from different backgrounds.

37. Learning a little bit from a lot of people was one of the main ways Tim identified so many unique ideas and insights.

38. Wiseman’s research is that lucky people pay more attention attention to what’s going on around them than unlucky people.

39. For example, Wiseman found that the lucky people had three times greater open body language in social situations than unlucky people. Lucky people also smiled twice as much as unlucky people, thus drawing other people and chance encounters to them.

40. von Hippel found that one group, which he called active users or lead users, were responsible for developing over 75 percent of the innovations.

41. These people not only serve as cutting-edge taste makers, they actively tinker to push and create new ideas on their own.

42. Since the needs of these active users precede and, according to the research, often anticipate, what the masses will like, they become incredibly valuable as idea development partners.

43. The ideas they help generate can then be tested with broader audiences and commercialized. The same is true for good comedy.

44. For example, singer and songwriter John Legend does so as he develops a new song. The first part of Legend’s songwriting process is to develop a good back-beat working closely with music producers like Kanye West, will.i.am, or Raphael Saadiq. Once he’s found a beat he likes, Legend will then develop melodies on top of it, often working at a piano, and finally write the lyrics. Kanye West, in particular, is a classic innovator and active user, constantly consuming and tinkering with music. Legend and others consider West to be a creative genius in many respects, with a well-tuned ear for what broader audiences will like. His early involvement in a song routinely drives later success. So, Legend will bounce ideas around with West at all stages of the process, especially at the beginning. It’s strategic trial and error.

45. A small win is “a concrete, complete, implemented outcome of moderate importance.” They are small successes that emerge out of our ongoing development process, and it’s important to be watching closely for them.

46. Small wins are like footholds or building blocks amid the inevitable uncertainty of moving forward.

47. They serve as what Saras Sarasvathy calls landmarks, and they can either confirm that we’re heading in the right direction or they can act as pivot points, telling us how to change course.

48. It is helpful for alcoholics to focus on remaining sober one day at a time, or even one hour at a time. Stringing together successive days of sobriety helps them to see the rewards of abstinence and makes it more achievable.

49. “Once a small win has been accomplished, forces are set in motion that favor another small win.”

50. Another benefit of small wins is less immediately obvious: They enable the development of the means to attain goals. Recall from Saras Saravathy’s research how important the development of means is to seasoned entrepreneurs. Entrepreneurs use their available means, such as their expertise, networks, or financial resources, to develop their ideas and access additional resources and means. One way that Ed Catmull (and later Steve Jobs) developed Pixar’s means was by steadily bringing in new talent with complementary skills.

51. Additional resources also flow toward winners.

52. One last, yet important, point about small wins is that often, rather than validating a direction we’ve been pursuing, they will provide a signal to proceed in a different way.

53. It is much easier to decide to make a change of approach when we are doing so not because things aren’t working but because something has started to work.

Evolving web scraping: How Pageripper API handles JavaScript-heavy sites

Zachary Proser — Sun, 31 Dec 2023 02:00:00 GMT

Pageripper API uses Puppeteer and headless Chrome under the hood to see the same thing your browser does, even for Single Page Applications (SPAs)

Better Corporate Communication

JonPaulUritis.com — Sun, 31 Dec 2023 02:00:00 GMT

I originally wrote this stuff up for my wife's company which was struggling with it but then ended up giving a talk about it. And since I basically post all of my important talks as articles here it is for your reading pleasure.

TLDR; Communication can be made more or less effective by working with or against Human biological limitations. To put it another way, the way you organize and provide your information can increase your throughput, retention, and utility of the information you are providing.

Human Biology

People can read and comprehend up to 400 WPM but average around 200 WPM
People can listen and comprehend up to 150 WPM
People can type ~45 WPM on average, with an upper limit towards 80 WPM for stronger typists.
People will average roughly 100 - 150 WPM when presenting
Human beings are capable of memorizing a shuffled deck of cards in ~19 seconds, but normal people can remember roughly 7 items for 20- 30 seconds (link).

Understand Timecosts

Let's assume a hypothetical cost of $100/ hr per person after benefits etc (using round numbers for ease). This is roughly what costs would look like for 1 person communicating, and different size groups of being communicated to.

People	1hr Meeting	30min Meeting	Long Email	Short Email
8	$800	$400	$275	~$110
4	$400	$200	$175	~$75
2	$200	$100	$125	~$58

Example: A long email: It takes 60 minutes to write and 15 min read. A short email: It takes roughly 15-30min to write and 5min read

Let's compare some real-world scenarios. We're working on defining design/requirement tasks for an upcoming "widget".

An expensive way to do this might look like this:

60min "brainstorming" with 8 people ($800-$1000).
1 person writing down Tasks for 30min (~$50),
A 30 min decision meeting with the group to close out ($400 - $500)

Total Cost: $1250 - $1500

A cheaper way might look like this:

One person sits down alone and writes out 90% of the tasks in an email to 8 people ($275),
Everyone reads the email ($100- $150)
3 people send feedback and additions ($225)

Total Cost: $600 - $650

Naively, the above examples make it seem like all communication should be written and/or meetings are expensive. This is not true. This example is just designed to show how real factors such as human biology can impact the throughput and efficiency of communication.

Tricks and Tips

Choose the right medium for your message

Email Messages (or confluence type systems):
- Anything you might need to reference later and/or search for
- Communication that might be shared
- Long communication where people might need time to process (reports)
Phone Calls
- Communication that might require more clarity or further explanation
- Multiple decisions reliant on each other (decisions in a chain)
- Communication that is emotional
Async Communication => Text Message, Slack/Teams
- Quick questions where you don't need a record
- Sharing links or other resources
- Text Message => for Urgency.
Hand Written
- Great at expressing gratitude or respect

Here's an obtuse way to think about it:

Medium	Cost	Quick Feedback	Shareable	Searchable	Asynchronous
Meeting	$$$$	✓
Email	$$		✓	✓	✓
Phone	$$	✓
SMS/ Slack	$	✓		✓	✓
Handwritten	$$$$		✓		✓
Carrier Pidgeon	$$$$$				✓

How to handle meetings

Per Andy Grove: There are 2 types of meetings: Process-oriented meetings and Mission-oriented Meetings. Generally, they boil down to Information sharing or decision-making.

Process-oriented meetings: are to identify latent problems, course-correct issues, create alignment, make type 2 decisions, and share information (think: standup, one-on-ones, etc)
Mission-oriented meetings: are to make (type-1) decisions (should be rare) [strategic/ irreversible]

There are a lot of other authors or business leaders to talk about this as well.

Meeting Tips and Tricks

Written Communication > Meetings
Meetings are the best mediums for:
- A series of dependent quick decisions that impact a lot of people.
- Information share, especially where deeper questions may arise
- Creating alignment on key decisions
Meetings are bad for:
- Adhoc Brainstorming (which is different from workshopping)
- Type 2 decisions - decisions that are easily reversed.
Always invite the fewest possible number of people you can get away with- ESPECIALLY managers.
- If you have to ask if a manager should be there... they probably don't need to be. Send them the notes.
Try to book meetings for 20min or 40min. NOT 30min or 60min.
Power Point/ Slide Decks are used to present visual aids*.
- I wrote a special section about this below...
Meeting Etiquette:
- Show up prepared.
  - (Protip: If you are leading and you don't need to prepare... it probably shouldn't be a meeting)
- Send out an agenda ahead of time. Include goals and reference materials
- Send out action items

IMPORTANT! Slide Decks

Misusing the slide deck format is HANDS DOWN the biggest mistake people make in corporate communication, and it's not even close. Because it's so bad it's worth going into a little detail about.

Let's run through slide deck norms:

An information/decision meeting is called so the person calling the meeting feels obligated to make a slide deck to make the meeting "professional", organized, or to have an artifact for later.
The person quickly throws up their jumble of text notes onto a slide deck in itty-bitty size 10 font just before the meeting. Because it is a giant rat's nest of text they spread it over 10-15 slides.
When they show up the speaker reads straight from their deck because they didn't rehearse.
Because people read faster than they can hear they either finish reading before the speaker finishes talking OR they struggle to read the small text and ignore the speaker
They format the deck as a "narrative" which positions a climax (decision) later... In this format, executives and managers will constantly interrupt to point out the thing that is on the next slide.
With all of these distractions, the decision gets lost and the meeting goes long. Another meeting with the same people will often be called or the decision/information will become muddled.

Notes on how to use slide decks properly

The slide deck medium is a "speech with visual aides."
Don't feel you need to use a deck. Many meetings could benefit by using a one-pager instead.
You should need to rehearse if it is a "speech with visual aides". Make sure you rehearse.
If your slide deck is primarily text don't use a deck. Formatting wastes time and People struggle to Read and Listen at the same time
Narratives (story with climax) SHOULD NOT be presented as a slide deck.*
Reports (numerical or empirical information) SHOULD NOT be presented as a slide deck.
Visual aids (e.g. graphs, pictures) are a superpower. They can convey more information in a smaller time and less space. Use these media extensively.
After every part of your speech/ each slide ask yourself: "So what?" or "Why is this important"... if you are unable to answer that question fix the section/slide.
Executives love to interrupt slide decks. It should be a goal of yours to avoid interruptions.
- They will not interrupt if all of the most important stuff is at the front of the meeting stated concisely.
- They will not interrupt if "So what" or "why" is answered.
- They will cut the meeting short if there is enough information to make a decision (GREAT)
- They will cut a meeting short if more information gathering is required (GOOD)

* I talk about Form/ Formatting in the next section

Form/Formatting your Written Communication

The form/ format you present information can increase efficiency, retention, and the utility of the communication. The following picture is a minimalist way of viewing different formats:

[insert picture here]

Here are some notes about different formats for your written communication. Keep in mind that a corporate setting (internal) is very different than a corporate setting (external), or things like art. A well-built narrative format is great for Fiction, but TERRIBLE for a corporate email.

AVOID Narrative Format (generally)
- Background, Build Conflict, Climax, and Resolution
- (Note: There are many exceptions such as mission-related communication, product releases, etc)
Journalism Format/ BLUT
- Format: Top to bottom, Most important info to least important info
- Great for emails, reports, some presentations
- The format was designed so that editors can compose a publication quickly. That is, they know that they can chop off bottom up based on how long they want a piece to be.
Academic Format / Experiments
- Format: Abstract or Summary of Results (most important info) then sections with background research, ideation, hypothesis, methods, and citations
- Great for reports, presentations, etc
Outline Format/ Information systems
- Format: Information in a hierarchy, with the most important at the top and top level, moving down toward elements of lesser importance.
- "The Classic model"
  - Wisdom (top level),
  - Information (second level),
  - Data (bottom level)

How to improve your Content

So What??? Why should I care Theory
- Make sure your communication answers the question "So What?" or "Why should someone care?"
Use Examples.
- Human beings are hardwired for examples. Show > Tell
Provide Context
- If we're working as a team effectively we're usually working on very different things. That's a good thing... the only caveat is that you will probably need to provide context on the thing you are talking about when we get together.
Simplify- Less is more
- People can really only remember 3-5 things from a 30-60 meeting. Don't try to do more than that in a meeting.

Writing Tips

Use Active Writing if longer than 1-2 Sentences

EXAMPLE

Break things into SECTIONS
Use bold, Underlining, CAPS, and italics to emphasize key points
Change Font Sizes to change flow
Use Bullets but no more than 3-5 bullets
Delete anything in parenthesis, unless it's a useful reference

Words

DON'T USE Acronyms unless your Grandmother would know it. Companies LOVE acronyms... but they slow down communication.
Replace Adjectives with Data
- "Nearly all users" => "93% of users"
- "We did significantly better" => "We increased our performance 25 points"
Only use $10 Words when there are no better options.

Know your Audience

I'm going to write another article about this at some other time but it's important to know who your audience is. One of the first ways to think about it is at an organizational level. Though there are other forms of corporate structure the most common in the United States, Japan, and the Western world is a hierarchal corporate structure. The Japanese (naturally) have a thing for this:

Vertical communication | Reporting ("Hokoku")
Horizontal communication | Informing ("Renraku")
Both directions | Consulting ("Sodan")

Cultural, ESL, and Subculture factors

Credit to Charlie Main for teaching me this. A weird cultural thing that the United States does is that we use a lot of sports analogies in business to communicate. Turns out that the rest of the world doesn't do this. Avoid sports analogies in business comms when you work in a multicultural/ multi-ethnic environment. When you are working with Americans... increase your use.
People who speak English as a second language are going to prefer written communication.
Certain personalities gravitate towards types of communication. Leverage this fact. Software Engineers READ and WRITE text all day long, therefore use written communication. Artists view graphics all day long, therefore use visual communication. Sales people and managers talk all day long... use verbal communication.

Lastly, Conclusions

Conclusions are one of the most important parts to communication. It will be the thing that people remember the most from your communication. So don't just end abruptly.

Thank you.

Year in review

Imran's Blog — Sun, 31 Dec 2023 02:00:00 GMT

I thought I should do a recap for this year. Though also for some typing practice.

I started this year unemployed, and still burned out from the last job. I do not remember the states of my finances, but I do get a new job later on

In no particular order:

# Momo

I got my cat Momo. She is on the opposite end of the personality spectrum compared to Yar. She is confident and unfazed by new people. She even purrs at the vet.

She is also overweight and wants to eat everything. I had to buy some fancy food bowls that specifically open for registered chips. It has been an ordeal trying to keep the cats from each others food. Momo is not allowed to eat Yar's food as she is fat and needs to lose weight. Yar is not allowed to eat Momo's food as he has allergies and requires medication.

She loves to play with toys. Though so did Yar when he was her age (2). He no longer cares much about them at 4. We will see if she follows a similar route.

# Hogwarts Legacy

This game sucked ass. Complete waste of $102 CAD. I wish I never bought and played it. It shouldn't even be in this post but I want to vent about just how much of a let down that game is. Incredible IP wasted on a subpar game.

# My Partner

She left Canada to pursue her PhD in the states. I am hoping to move there too to support her. She has been having a rough time, but I think she will pull through.

# Side stuff

Activity on my usent downloader has stalled since starting a new job. It's hard to find time in the day to both gym and have time left over for programming after work.

I have managed to rework my dad's site stuff for a third time, this time with NixOS. There should be an upcoming blog post about it.

The hard part, as mentioned before, is finding the time now that I employed again.

# My old team

I got to meet my old team in person. I quit right before there the company get together post COVID. They are a nice and fun bunch but "stuck" at a crap company that pays too much 😉.

# My new team

I started a new job and met my new team in person too! The name of the place is Psiphon and they work in internet cencorship circumnavigation. Smart folks that work primarily in go. If there is one thing I am grateful for it's that I wont have to touch ruby again for a long time.

Work culture at the company is unique, hands off and low meetings. The company places a lot of trust in the employees to get the job done and I hope I don't betray that trust. Not fired yet.

Reflection on 2023

Hrafn Thorvaldsson — Sun, 31 Dec 2023 02:00:00 GMT

Another year, another reflection. Theme Link to heading Ever since seeing CGP Grey’s video on yearly themes back in 2020 I’ve stuck to the practice, and it has worked pretty well as a mild forcing function. In 2022 it was ‘Health’ and this year’s it was ‘Growth & Learning’. Where possible I chose opportunities or activities that would allow for one or both. It would be impossible to list everything I’ve picked up through the various mediums available, and so I won’t even try.

2023 Review

Thought Eddies — Sun, 31 Dec 2023 02:00:00 GMT

End of year review

2023 Year in Review

Ben Overmyer's Site — Sun, 31 Dec 2023 02:00:00 GMT

2023 was a year of important changes and improvements in my life. My goals for the year according to last year's "Looking Ahead to 2023" post ended up going wide of the mark. I didn't succeed at the SGG goals and I abandoned #dungeon23 early in January. I also didn't keep to my ban on new entertainment. Where I did succeed, though, was in the fitness goals for the year. More on that in a moment.

This year, I'll start including a few notes about what I accomplish at my day job.

Apiture

At my day job this year the biggest achievement was converting our monitoring stack from New Relic to Prometheus+Grafana. The one gap that remains is the profiling (what New Relic calls "APM"). I also continued to make small improvements in several areas, like the new ECS-based Wordpress architecture.

Iron Arachne

Iron Arachne had a big year this year. In March, it reached its fifth anniversary. Over the course of the year I made many updates to the code to clean it up and reorganize it. The most important updates, though, were the SvelteKit migration and the rework of the planet generator.

Prior to the SvelteKit migration, Iron Arachne was running as a static site on Netlify. However, when I migrated it, it became a Node.js application running on my own web server. This made it more difficult to manage, but gave me way more freedom in the long run.

The planet generator rework happened in the last quarter of the year. Previously, the images were set up as 3D scenes in THREE.js, with a sphere for the planet, another sphere for clouds if they were present, a plane for the starfield, and camera and light rigging. The rework turned them into a single shader capturing everything. I finally implemented atmospheric scattering and dramatically improved how the various planets looked. I also used this new rendering approach for the star nation and star system generators.

Self Hosting

Besides Iron Arachne, I also began to self host more things. I bought a mini PC, installed Ubuntu on it, and set up several services on it. It's accessible via our Tailscale network, meaning I can access it from any device that is part of that network. Right now I have the following hosted on it:

Jellyfin: for media
Kavita: for browsing and reading PDFs
IT Tools: for lots of little convenience utilities
Paperless: for scanning and storing important documents

While Paperless is set up, I didn't really start using it properly in 2023. Same for Jellyfin. I've used Kavita and IT Tools more.

My Old Sites

I recreated as many of my old websites as I could find archived online and put them online again on a subdomain of my main site. Some of these I'd completely forgotten existed. In the course of this work, I also took inspiration from my old art galleries to rework the art section of my main site. It reminded me of how much I used to love making and sharing art.

Reading Books

Holy crap. I read 34 books in 2023, which is three times what I read in 2022, and twice my previous record. Granted, a lot of it was light fiction, but I still spent way more time reading this year.

One reason for this is that I decided to create a new fan site for one of the fictional cultures in the Valdemar novels by Mercedes Lackey. Before I can do that, I wanted to refamiliarize myself with them, and so I began the process of reading every single novel that they appear in. I forgot how much I enjoy these books. It's been a delight reading them. Some of them are completely new to me, too, which is awesome.

I also finished the Honor Harrington series of novels by David Weber. Those were a harder read, full of exposition and dense text. I enjoyed them too though.

Health and Fitness

I started 2023 with a ban on alcohol that ended somewhere near the end of January. Running became a regular routine for me; I ran in the morning on Mondays, Wednesdays, and Fridays before work. My cardio health improved a lot; my VO2 max went from a low of 33.9 in January to a high of 37.4 in April and a more sustained peak of 36.8 in the latter half of the year. My cardio recovery went from 19 BPM in January to 37 BPM by the end of the year.

I stopped running in November when I undertook a self-imposed wellness challenge. For the month of November, I drank no alcohol and began a bodybuilding routine at the gym. Four days of the week I would go to the gym and lift weights. Cardio was limited to only a few minutes of cooldown walking at the end of each workout.

The amount of weight I can lift improved a bit, but what really improved was my confidence and interest in bodybuilding. Now I just need to reincorporate my cardio exercises and I'll have a more complete fitness regimen.

Silver Gryphon Games

We didn't make nearly as much progress on SGG in 2023 as I expected. I had planned on releasing at least one new book by the end of March. We only released two playtest books later in the summer; the new Aether and Ingenium editions. Despite trying to keep on top of Aethermancy it ended up being a struggle to get the manuscript finished. As of this writing, it's still not done.

Aether and Ingenium's second editions got completed manuscripts. Aether got a lot of editing. The rise of AI art has made it a lot more difficult to find cheap artwork made by real artists.

Previous Years

2023 in review

cheeaunblog — Sun, 31 Dec 2023 02:00:00 GMT

This is probably going to be short as this year is perhaps the most unproductive year I've ever had.As usual, I start this off with a compilation…

2023-12-31

ho.dges.online — Sun, 31 Dec 2023 02:00:00 GMT

Emoji by Category

taylor.town — Sun, 31 Dec 2023 02:00:00 GMT

📝

Emoji by Year

taylor.town — Sun, 31 Dec 2023 02:00:00 GMT

📝

2024 Will Be the Year of the Blog

David Buchanan's Blog — Sun, 31 Dec 2023 02:00:00 GMT

Gift Giving

foreverliketh.is — Sun, 31 Dec 2023 02:00:00 GMT

Image Details & Source

person showing brown gift box by Kira auf der Heide on Unsplash

Click Here for the Original, High-Quality Image

This page was last modified: December 31st, 2023

This post is part of IndieWeb Carnival. A project helping motivate people to write more on their personal websites. This month’s host is Johanna and their topic was ‘Holiday and December traditions’ [#1 & #2]

This post is also part of Agora Road Travelogue - Dec. A project attempting to encourage individuals to both write and exercise ownership over their content.

Bah! Humbug! #

Of all the holiday traditions I’m familiar with, the one I’ve caught myself engaging with the most year over year would have to be gift-giving. While I won’t extrapolate my experience with it to you, traveler, I’ve personally found the whole thing to be… a bit of a process. And I’m not alone: A Lainchan Webring neighbor, Mark (maerk.xyz), recently wrote about their Christmas gifting struggles, expressing sentiments I totally relate to.

In my opinion, part of the grinchy frustration that can stem from the practice is that its nature is more complicated than it appears at first glance. But I believe that, with a good mindset, a little preparation and maybe a couple handy strategies, we can keep ourselves from questioning why we even bother in the first place.

Ant, Not Grasshopper #

The average American currently spends approximately $648 USD on Christmas gifts; a little over 1% of their yearly salary. A higher-end alternative is to cap the spending at 1 week’s salary. Play with that dial as you see fit, but I would try to stay in that range. Ideal gift-giving, imo, embodies the ‘Goldilocks principle’, and budget is the first place it should be applied. Whatever the amount, set it in stone for the year and don’t look back.

Alright, Santa Claus, now it’s time to make your list. Here’s the rough hierarchy: Family –> Friends –> Coworkers. Seems too obvious? You’d be surprised:

Additionally, the more distant the person, the easier it is to get them a “good” gift. Naturally, their absence of expectation produces a “low effort, high reward” effect; be wary of that trap. I’m not gonna throw out percentages, but if the bulk of your pot isn’t going to those that love you, there’s probably something worth reevaluating.

Modern Magi #

Now for the REAL challenge: the gifts themselves. Balance can be key.

Not too cheap, but also not too expensive.
Not too giver-centered, but also not too recipient-centered. It should have a piece of you, a personal touch, but just a bit.
Not too plain / boring, but also not 100% reaction bait. Try to spice it up if it’s something bland, and don’t forget to consider the practicality (and size) if it’s something flashy.

Additional tips include:

SOME element of surprise; even if it’s just the timing. #1
Remember you can gift experiences (something to do), not just stuff. #2
Taking notes (being attentive) all year round, can save you hunting time.

Need inspiration? I recommend:

Etsy
r/GiftIdeas
YouTube Search: “Gift Ideas” + “Hobby / Gender / Interest / etc.”

Forest for the Trees #

A gift or a present is an item given to someone without the expectation of payment or anything in return.

No matter the effort, there is NO guarantee that a present will be appreciated. The attempt of sincere giving alone should serve as both sufficient motivation to try, and its own reward. Sound a tad stoic? Perhaps. But common alternative attitudes can prove problematic:

“Christmas has become a capitalist holiday”
- You’re just a sucker falling for a scam, and the only way to win is not to play.
“Gift Giving Is About the Buyer, Not the Receiver”
- You’re actually just selfish, and only want to feel good about yourself.
“The economics of wasteful spending: The deadweight loss of Christmas”
- You’re, literally, burning money away when you engage in this.

These types of logical / cynical lenses that we can look at the tradition through, I think they do more harm than good. Oft-repeated advice in these circles include:

Just give money.
Ask the recipient what they want.
Gift only utilitarian items.
Forgo gift-giving entirely.

It’s like there’s an obsession with optimization, or a fatalist pessimism, or a fear of failure, or some notion of intellectual superiority. Citing research or studies, and declaring the findings as a sort-of gospel is par for the course. Fren, they have not “solved” gift-giving; they’ve developed a formula for mediocrity! And worse, it’s possibly a poison to the spirit of the act; a dehumanizing, soul-eating scrooge-toxin!

Is the tradition such a burden? A custom out of time? A materialistic inconvenience? I don’t know, visitor. But I’ll be here, trying to get better at it anyways. If you have any, I’d love to hear your thoughts on what makes for good gift-giving.

Happy New Year! 🎆

Syndication: Agora Road

Comments #

Email: hi@foreverliketh.is

IndieWeb h-entry

Gift Giving

Published by foreverliketh.is on 31^st December 2023

Ya Gotta Try

Bah! Humbug!

Gift Giving

Agora Road

Psychology

Books I read in 2023... for fun!

swaits.com — Sun, 31 Dec 2023 02:00:00 GMT

I developed an interest in reading "just for fun" this year. In fact, I have always had a passion for reading. However, I used to primarily read academic papers and nonfiction books, particularly those of a technical nature.

Earlier this year, I discovered the convenience of borrowing library books through Libby and being able to read them on my Kindle. This revelation sparked a new reading journey. Throughout 2023, I successfully cultivated a habit of reading for entertainment purposes, often surpassing my reading for educational purposes.

Below is a list of books I read for entertainment this year, excluding those related to business, self-help, academia, and technical subjects.

If you want to see my recommended list, most of which is nonfiction, check out my top-level books page.

Note that this is just a list. These are not reviews. Also, the included descriptions are taken from the publishers, and are not my own.

Nonfiction

Killers of the Flower Moon by David Grann

I discovered this book when I saw a trailer for the movie.

In the 1920s, the richest people per capita in the world were members of the Osage Nation in Oklahoma. After oil was discovered beneath their land, the Osage rode in chauffeured automobiles, built mansions, and sent their children to study in Europe.

Then, one by one, the Osage began to be killed off. The family of an Osage woman, Mollie Burkhart, became a prime target. One of her relatives was shot. Another was poisoned. And it was just the beginning, as more and more Osage were dying under mysterious circumstances, and many of those who dared to investigate the killings were themselves murdered.

As the death toll rose, the newly created FBI took up the case, and the young director, J. Edgar Hoover, turned to a former Texas Ranger named Tom White to try to unravel the mystery. White put together an undercover team, including a Native American agent who infiltrated the region, and together with the Osage began to expose one of the most chilling conspiracies in American history.

Standalone Fiction

Camp Zero by Michelle Min Sterling

In remote northern Canada, a team led by a visionary American architect is breaking ground on a building project called Camp Zero, intended to be the beginning of a new way of life. A clever and determined young woman code-named Rose is offered a chance to join the Blooms, a group hired to entertain the men in camp—but her real mission is to secretly monitor the mercurial architect in charge. In return, she’ll receive a home for her climate-displaced Korean immigrant mother and herself.

Rose quickly secures the trust of her target, only to discover that everyone has a hidden agenda, and nothing is as it seems. Through skillfully braided perspectives, including those of a young professor longing to escape his wealthy family and an all-woman military research unit struggling for survival at a climate station, the fate of Camp Zero’s inhabitants reaches a stunning crescendo.

Atmospheric, fiercely original, and utterly gripping, Camp Zero is an electrifying page-turner and a masterful exploration of who and what will survive in a warming world, and how falling in love and building community can be the most daring acts of all.

Hidden Pictures by Jason Rekulak

Mallory Quinn is fresh out of rehab when she takes a job as a babysitter for Ted and Caroline Maxwell. She is to look after their five-year-old son, Teddy.

Mallory immediately loves it. She has her own living space, goes out for nightly runs, and has the stability she craves. And she sincerely bonds with Teddy, a sweet, shy boy who is never without his sketchbook and pencil. His drawings are the usual fare: trees, rabbits, balloons. But one day, he draws something different: a man in a forest, dragging a woman’s lifeless body.

Then, Teddy’s artwork becomes increasingly sinister, and his stick figures quickly evolve into lifelike sketches well beyond the ability of any five-year-old. Mallory begins to wonder if these are glimpses of a long-unsolved murder, perhaps relayed by a supernatural force.

Knowing just how crazy it all sounds, Mallory nevertheless sets out to decipher the images and save Teddy before it’s too late.

The House in the Pines by Ana Reyes

Maya was a high school senior when her best friend, Aubrey, dropped dead in front of the enigmatic man named Frank whom they’d been spending time with all summer.

Seven years later, Maya lives in Boston with a loving boyfriend and is kicking the secret addiction that has allowed her to cope with what happened years ago, the gaps in her memories, and the lost time that she can’t account for. But her past comes rushing back when she comes across a recent YouTube video in which a young woman suddenly keels over and dies in a diner while sitting across from none other than Frank. Plunged into the trauma that has defined her life, Maya heads to her Berkshires hometown to relive that fateful summer—the influence Frank once had on her and the obsessive jealousy that nearly destroyed her friendship with Aubrey.

At her mother’s house, she excavates fragments of her past and notices hidden messages in her deceased Guatemalan father’s book that didn’t stand out to her earlier. To save herself, she must understand a story written before she was born, but time keeps running out, and soon, all roads are leading back to Frank’s cabin.

...

Utterly unique and captivating, The House in the Pines keeps you guessing about whether we can ever fully confront the past and return home.

Northern Spy by Flynn Berry

A producer at the BBC and mother to a new baby, Tessa is at work in Belfast one day when the news of another raid comes on the air. The IRA may have gone underground in the two decades since the Good Friday Agreement, but they never really went away, and lately bomb threats, security checkpoints, and helicopters floating ominously over the city have become features of everyday life. As the news reporter requests the public's help in locating those responsible for the robbery, security footage reveals Tessa's sister, Marian, pulling a black ski mask over her face.

The police believe Marian has joined the IRA, but Tessa is convinced she must have been abducted or coerced; the sisters have always opposed the violence enacted in the name of uniting Ireland. And besides, Marian is vacationing on the north coast. Tessa just spoke to her yesterday.

When the truth about Marian comes to light, Tessa is faced with impossible choices that will test the limits of her ideals, the bonds of her family, her notions of right and wrong, and her identity as a sister and a mother. Walking an increasingly perilous road, she wants nothing more than to protect the one person she loves more fiercely than her sister: her infant son, Finn.

Riveting, atmospheric, and exquisitely written, Northern Spy is at once a heart-pounding story of the contemporary IRA and a moving portrait of sister- and motherhood, and of life in a deeply divided society.

Sea of Tranquility by Emily St. John Mandel

Edwin St. Andrew is eighteen years old when he crosses the Atlantic by steamship, exiled from polite society following an ill-conceived diatribe at a dinner party. He enters the forest, spellbound by the beauty of the Canadian wilderness, and suddenly hears the notes of a violin echoing in an airship terminal—an experience that shocks him to his core.

Two centuries later a famous writer named Olive Llewellyn is on a book tour. She’s traveling all over Earth, but her home is the second moon colony, a place of white stone, spired towers, and artificial beauty. Within the text of Olive’s best-selling pandemic novel lies a strange passage: a man plays his violin for change in the echoing corridor of an airship terminal as the trees of a forest rise around him.

When Gaspery-Jacques Roberts, a detective in the black-skied Night City, is hired to investigate an anomaly in the North American wilderness, he uncovers a series of lives upended: The exiled son of an earl driven to madness, a writer trapped far from home as a pandemic ravages Earth, and a childhood friend from the Night City who, like Gaspery himself, has glimpsed the chance to do something extraordinary that will disrupt the timeline of the universe.

A virtuoso performance that is as human and tender as it is intellectually playful, Sea of Tranquility is a novel of time travel and metaphysics that precisely captures the reality of our current moment.

TRUST by Hernan Diaz

I went through a list of past Pulitzer Prize winners and I picked this one up. Ultimately, I'm glad I read it. But, I am also a little less sure that looking at Pulitzer Prize winners is a good idea.

Even through the roar and effervescence of the 1920s, everyone in New York has heard of Benjamin and Helen Rask. He is a legendary Wall Street tycoon; she is the daughter of eccentric aristocrats. Together, they have risen to the very top of a world of seemingly endless wealth—all as a decade of excess and speculation draws to an end. But at what cost have they acquired their immense fortune? This is the mystery at the center of Bonds, a successful 1937 novel that all of New York seems to have read. Yet there are other versions of this tale of privilege and deceit.

Hernan Diaz’s TRUST elegantly puts these competing narratives into conversation with one another—and in tension with the perspective of one woman bent on disentangling fact from fiction. The result is a novel that spans over a century and becomes more exhilarating with each new revelation.

At once an immersive story and a brilliant literary puzzle, TRUST engages the reader in a quest for the truth while confronting the deceptions that often live at the heart of personal relationships, the reality-warping force of capital, and the ease with which power can manipulate facts.

Series, Fiction

I always read series in-order. Who doesn't? I think there are people that don't.

Anyway, I list the books in each of these series that I completed.

Crescent City series by Sarah J. Maas

This book (and series) is consistently atop the best science fiction and fantasty lists. A wee bit romantasty, but here we go.

Cover	Title	Description
	House of Earth and Blood (Crescent City Book 1)	Bryce Quinlan had the perfect life-working hard all day and partying all night-until a demon murdered her closest friends, leaving her bereft, wounded, and alone. When the accused is behind bars but the crimes start up again, Bryce finds herself at the heart of the investigation. She'll do whatever it takes to avenge their deaths. Hunt Athalar is a notorious Fallen angel, now enslaved to the Archangels he once attempted to overthrow. His brutal skills and incredible strength have been set to one purpose-to assassinate his boss's enemies, no questions asked. But with a demon wreaking havoc in the city, he's offered an irresistible deal: help Bryce find the murderer, and his freedom will be within reach. As Bryce and Hunt dig deep into Crescent City's underbelly, they discover a dark power that threatens everything and everyone they hold dear, and they find, in each other, a blazing passion-one that could set them both free, if they'd only let it. With unforgettable characters, sizzling romance, and page-turning suspense, this richly inventive new fantasy series by #1 New York Times bestselling author Sarah J. Maas delves into the heartache of loss, the price of freedom-and the power of love.

Jack Reacher series by Lee Child

My in-laws watched the TV series for this and enjoyed it. So I decided to try out the books. This is not deep stuff. But, I like the writing style, I like the characters, and enjoy the adventures. You can call me low-brow or something, I don't care. I blinked and boom, read half the series.

Cover	Title	Description
	Killing Floor (Jack Reacher, Book 1)	Ex-military policeman Jack Reacher is a drifter. He’s just passing through Margrave, Georgia, and in less than an hour, he’s arrested for murder. Not much of a welcome. All Reacher knows is that he didn’t kill anybody. At least not here. Not lately. But he doesn’t stand a chance of convincing anyone. Not in Margrave, Georgia. Not a chance in hell.
	Die Trying (Jack Reacher, Book 2)	Jack Reacher is an innocent bystander when he witnesses a woman kidnapped off a Chicago street in broad daylight. In the wrong place at the wrong time, he’s kidnapped with her. Chained together, locked in the back of a stifling van, and racing across America to an unknown destination for an unknown purpose, they’re at the mercy of a group of men demanding an impossible ransom. Because this mysterious woman is worth more than Reacher ever suspected. Now he has to save them both—from the inside out—or die trying....
	Tripwire (Jack Reacher, Book 3)	Ex military policeman Jack Reacher is enjoying the lazy anonymity of Key West when a stranger shows up asking for him. He’s got a lot of questions. Reacher does too, especially after the guy turns up dead. The answers lead Reacher on a cold trail back to New York, to the tenuous confidence of an alluring woman, and the dangerous corners of his own past.
	Running Blind (Jack Reacher, Book 4)	Across the country, women are being murdered, victims of a disciplined and clever killer who leaves no trace evidence, no fatal wounds, no signs of struggle, and no clues to an apparent motive. They are, truly, perfect crimes. In fact, there’s only one thing that links the victims. Each one of the women knew Jack Reacher—and it’s got him running blind.
	Echo Burning (Jack Reacher, Book 5)	Thumbing across the scorched Texas desert, Jack Reacher has nowhere to go and all the time in the world to get there. Cruising the same stretch of two-lane blacktop is Carmen Greer. For Reacher, the lift comes with a hitch. Carmen’s got a wild story to tell—all about her husband, her family secrets, and a hometown that’s purely gothic. She’s also got a plan. Reacher’s part of it. And before the sun sets, this ride could cost them both their lives.
	Without Fail (Jack Reacher, Book 6)	Skilled, cautious, and anonymous, Jack Reacher is perfect for the job: to assassinate the vice president of the United States. Theoretically, of course. A female Secret Service agent wants Reacher to find the holes in her system, and fast—because a covert group already has the vice president in their sights. They’ve planned well. There’s just one thing they didn’t plan on: Reacher.
	Persuader: A Jack Reacher Novel (#7)	Jack Reacher lives for the moment. Without a home. Without commitment. And with a burning desire to right wrongs—and rewrite his own agonizing past. DEA Susan Duffy is living for the future, knowing that she has made a terrible mistake by putting one of her own female agents into a death trap within a heavily guarded Maine mansion. Staging a brilliant ruse, Reacher hurtles into the dark heart of a vast criminal enterprise. Trying to rescue an agent whose time is running out, Reacher enters a crime lord’s waterfront fortress. There he will find a world of secrecy and violence—and confront some unfinished business from his own past.
	The Enemy: A Jack Reacher Novel (#8)	Jack Reacher. Hero. Loner. Soldier. Soldier’s son. An elite military cop, he was one of the army’s brightest stars. But in every cop’s life there is one case that changes everything. For Jack Reacher, this is that case. New Year’s Day, 1990. In a North Carolina motel, a two-star general is found dead. His briefcase is missing. Nobody knows what was in it. Within minutes Reacher has his orders: Control the situation. Within hours the general’s wife is murdered. Then the dominoes really start to fall. Somewhere inside the vast worldwide fortress that is the U.S. Army, Reacher is being set up as a fall guy with the worst enemies a man can have. But Reacher won’t quit. He’s fighting a new kind of war—against an enemy he didn’t know he had. And against a conspiracy more chilling, ingenious, and treacherous than anyone could have guessed.
	One Shot: A Jack Reacher Novel (#9)	Six shots. Five dead. One heartland city thrown into a state of terror. But within hours the cops have it solved: a slam-dunk case. Except for one thing. The accused man says: You got the wrong guy. Then he says: Get Reacher for me. And sure enough, ex—military investigator Jack Reacher is coming. He knows this shooter–a trained military sniper who never should have missed a shot. Reacher is certain something is not right–and soon the slam-dunk case explodes. Now Reacher is teamed with a beautiful young defense lawyer, moving closer to the unseen enemy who is pulling the strings. Reacher knows that no two opponents are created equal. This one has come to the heartland from his own kind of hell. And Reacher knows that the only way to take him down is to match his ruthlessness and cunning–and then beat him shot for shot.
	The Hard Way: A Jack Reacher Novel (#10)	Jack Reacher was alone, the way he liked it, soaking up the hot, electric New York City night, watching a man cross the street to a parked Mercedes and drive it away. The car contained one million dollars in ransom money because Edward Lane, the man who paid it, would do anything to get his family back. Lane runs a highly illegal soldiers-for-hire operation. He will use any tool to find his beautiful wife and child. And Jack Reacher is the best manhunter in the world. On the trail of vicious kidnappers, Reacher learns the chilling secrets of his employer’s past . . . and of a horrific drama in the heart of a nasty little war. He knows that Edward Lane is hiding something. Something dirty. Something big. But Reacher also knows this: He’s already in way too deep to stop now. And if he has to do it the hard way, he will.
	Bad Luck and Trouble: A Jack Reacher Novel (#11)	From a helicopter high above the California desert, a man is sent free-falling into the night. On the streets of Portland, Jack Reacher is pulled out of his wandering life and plunged into the heart of a conspiracy that is killing old friends . . . and the people he once trusted with his life. Reacher is the ultimate loner—no phone, no ties, no address. But a woman from his old military unit has found him using a signal only the eight members of their elite team would know. Then she tells him a terrifying story about the brutal death of a man they both served with. Soon Reacher is reuniting with the survivors of his team, scrambling to unravel the sudden disappearance of two other comrades. But Reacher won’t give up—because in a world of bad luck and trouble, when someone targets Jack Reacher and his team, they’d better be ready for what comes right back at them.
	Nothing to Lose: A Jack Reacher Novel (#12)	Two small towns in the middle of nowhere: Hope and Despair. Between them, nothing but twelve miles of empty road. Jack Reacher can’t find a ride, so he walks. All he wants is a cup of coffee. What he gets are four hostile locals, a vagrancy charge, and an order to move on. They’re picking on the wrong guy. Reacher is a hard man. No job, no address, no baggage. Nothing at all, except hardheaded curiosity. What are the secrets that Despair seems so desperate to hide? With just one ally—a mysterious woman cop from Hope—and many enemies, Reacher goes up against a whole town, hunting the rich man at its core, cracking open his terrifying agenda, asking the question: Who has the edge—a man with everything to gain, or a man with nothing to lose?

A Leaphorn & Chee Novel series by Tony Hillerman

I discovered this through an ad for the Dark Winds TV series on AMC, which is apparently based on this book. The ad was interesting, so I thought I'd give the book a shot.

Cover	Title	Description
	The Blessing Way: A Leaphorn & Chee Novel #1	Homicide is always an abomination, but there is something exceptionally disturbing about the victim discovered in a high, lonely place—a corpse with a mouth full of sand—abandoned at a crime scene seemingly devoid of tracks or useful clues. Though it goes against his better judgment, Navajo Tribal Police Lieutenant Joe Leaphorn cannot help but suspect the hand of a supernatural killer. There is palpable evil in the air, and Leaphorn's pursuit of a Wolf-Witch leads him where even the bravest men fear, on a chilling trail that winds perilously between mysticism and murder.

The Murderbot Diaries series by Martha Wells

I found this while browsing past Hugo/Nebula award winners in search of a new science fiction novella. I could crank through quickly. I'm just about caught up to the current-end of this series. Turned out to be a ton of fun.

Cover	Title	Description
	All Systems Red: The Murderbot Diaries #1	"As a heartless killing machine, I was a complete failure." In a corporate-dominated spacefaring future, planetary missions must be approved and supplied by the Company. Exploratory teams are accompanied by Company-supplied security androids, for their own safety. But in a society where contracts are awarded to the lowest bidder, safety isn’t a primary concern. On a distant planet, a team of scientists are conducting surface tests, shadowed by their Company-supplied ‘droid — a self-aware SecUnit that has hacked its own governor module, and refers to itself (though never out loud) as “Murderbot.” Scornful of humans, all it really wants is to be left alone long enough to figure out who it is. But when a neighboring mission goes dark, it's up to the scientists and their Murderbot to get to the truth.
	Artificial Condition: The Murderbot Diaries #2	It has a dark past—one in which a number of humans were killed. A past that caused it to christen itself “Murderbot”. But it has only vague memories of the massacre that spawned that title, and it wants to know more. Teaming up with a Research Transport vessel named ART (you don’t want to know what the “A” stands for), Murderbot heads to the mining facility where it went rogue. What it discovers will forever change the way it thinks…
	Rogue Protocol: The Murderbot Diaries #3	Who knew being a heartless killing machine would present so many moral dilemmas? Sci-fi’s favorite antisocial A.I. is back on a mission. The case against the too-big-to-fail GrayCris Corporation is floundering, and more importantly, authorities are beginning to ask more questions about where Dr. Mensah's SecUnit is. And Murderbot would rather those questions went away. For good.
	Exit Strategy: The Murderbot Diaries #4	Murderbot wasn’t programmed to care. So, its decision to help the only human who ever showed it respect must be a system glitch, right? Having traveled the width of the galaxy to unearth details of its own murderous transgressions, as well as those of the GrayCris Corporation, Murderbot is heading home to help Dr. Mensah—its former owner (protector? friend?)—submit evidence that could prevent GrayCris from destroying more colonists in its never-ending quest for profit. But who’s going to believe a SecUnit gone rogue? And what will become of it when it’s caught?
	Network Effect: The Murderbot Diaries #5	I’m usually alone in my head, and that’s where 90 plus percent of my problems are. When Murderbot's human associates (not friends, never friends) are captured and another not-friend from its past requires urgent assistance, Murderbot must choose between inertia and drastic action. Drastic action it is, then.
	Fugitive Telemetry: The Murderbot Diaries #6	When Murderbot discovers a dead body on Preservation Station, it knows it is going to have to assist station security to determine who the body is (was), how they were killed (that should be relatively straightforward, at least), and why (because apparently that matters to a lot of people—who knew?) Yes, the unthinkable is about to happen: Murderbot must voluntarily speak to humans! Again!
	System Collapse: The Murderbot Diaries #7	Am I making it worse? I think I'm making it worse. Following the events in Network Effect, the Barish-Estranza corporation has sent rescue ships to a newly-colonized planet in peril, as well as additional SecUnits. But if there’s an ethical corporation out there, Murderbot has yet to find it, and if Barish-Estranza can’t have the planet, they’re sure as hell not leaving without something. If that something just happens to be an entire colony of humans, well, a free workforce is a decent runner-up prize. But there’s something wrong with Murderbot; it isn’t running within normal operational parameters. ART’s crew and the humans from Preservation are doing everything they can to protect the colonists, but with Barish-Estranza’s SecUnit-heavy persuasion teams, they’re going to have to hope Murderbot figures out what’s wrong with itself, and fast! Yeah, this plan is... not going to work.

Wild Robot series by Peter Brown

This is a middle-grade fiction series. I noticed my stepdaughter reading it and I told her I'll also read whatever books she reads so we can talk about them together. So, here we go!

Cover	Title	Description
	The Wild Robot (#1)	When robot Roz opens her eyes for the first time, she discovers that she is all alone on a remote, wild island. She has no idea how she got there or what her purpose is--but she knows she needs to survive. After battling a violent storm and escaping a vicious bear attack, she realizes that her only hope for survival is to adapt to her surroundings and learn from the island's unwelcoming animal inhabitants. As Roz slowly befriends the animals, the island starts to feel like home--until, one day, the robot's mysterious past comes back to haunt her. From bestselling and award-winning author and illustrator Peter Brown comes a heartwarming and action-packed novel about what happens when nature and technology collide.

Vendor Unlocking My Life

Blake Ashley Jr. — Sat, 30 Dec 2023 19:24:13 GMT

In an era where digital dependence is commonplace, I moved towards personal tech independence in 2023. This blog post details my transition away from mainstream tech giants and towards more secure, private, and autonomous alternatives.

Breaking Free from the Norm

In 2023, I finally broke free from the likes of Google, Facebook, and other tech giants—a commitment I’d long made to myself but never fulfilled. Like many of you, Google and Facebook had been my internet cornerstones for years, offering convenience, cost-effectiveness, and entertainment.

On a professional level, I’ve always been cognizant of the threat of ‘vendor lock-in’. How reliant am I on a single service, and what’s the fallout if it vanishes overnight?

A personal turning point came with a widely-reported incident: a father wrongfully accused by Google’s algorithms of possessing child sex abuse material in his private messages. This automated and erroneous finding led to the freezing of his account and immediate notification to law enforcement.

Realizing the extent of my dependence on a few major players like Google was a wake-up call. My goal for 2023 was clear: to minimize my dependency on any single platform or service.

Improvements I Made This Year:

Email/Calendar

My first step was to address my email and calendar dependencies.

Migrated away from my 15-year-old Google account
- Switched to Proton Mail for enhanced privacy and security.¹
- Transitioned to using my own domains for email.
- Set up PGP using my own keys for secure communication.
- Moved scheduling to Cal.com for opensource scheduling.
- I setup SimpleLogin to use my own domain (a throwaway) for private email forwwarding. Should SimpleLogin disappear, I could keep all these emails flowing.
Todo:
- Implement an automated email backup system.
- Figure out a solution for calendar synchronization.²
- Transition away from Google Photos (Currently locked-in due to family use).
- Permanently delete my Google account after a few more months of email forwarding.

Messaging

Moved (almost exclusively) to Signal and Simplex for private and secure messaging.
Successfully convinced many friends and family to switch to Signal.
Using Simplex as a secure alternative to email for communications with strangers.
Stopped using Discord
- I’ve enjoyed Matrix for quick questions and community building, despite not being a big fan of chatrooms for most use cases due to their lack of external searchability, synchronicity, etc.³
Todo:
- Get the remaining friends and family on Signal.
- Consider self-hosting my own Simplex infrastructure.
- Migrate to a self-hosted Matrix server.

Embraced the fediverse by switching to Mastodon and Lemmy.
Deleted my Twitter, Reddit, Instagram, and Facebook accounts, prioritizing privacy and control over my digital social life.
Todo:
- Plan to self-host my own Mastodon and Lemmy instances.

Web Hosting

Migrated to a static site powered by Hugo for enhanced portability and control.
Todo:
- Move away from Cloudflare hosting for greater independence.
- Enable Tor access to my site for increased privacy and accessibility.
- Self-host my newsletter system.

Homelab

Undertook a massive overhaul by moving dozens of services into a self-hosted k3s cluster and Ceph storage.
Transitioned all my physical media into Jellyfin, supporting DRM-free content and direct purchases from creators.
Installed security cameras around my home, ensuring they operate exclusively within my LAN and backups are encrypted.
In the process of migrating all file storage to a blend of Nextcloud and Syncthing, for a seamless and secure file management system.

MISC

Now using VPN and Tor exclusively for web browsing, with very few exceptions (like banking).
Sites that block Tor and/or VPN (including incessant CAPTCHA requirements) no longer receive my patronage.

Todo in 2024

Do a full account audit, going through my password manager and updating emails/passwords, adding 2FA, or deleting accounts when not in use.
Plan to replace my current Pixel device, which has a locked bootloader, with an unlocked device running GrapheneOS.
Intend to replace my desktop and laptop with Nvidia GPUs for a setup that fully supports Wayland with AMD or Intel GPU, then move exclusively to a Linux desktop.
Github still has an iron hold on my code. My plan is to self-host Forgejo and mirror public repos to Github. I hope the Activitypub movement can make its way to git and allow us to move off of centralized fit providers like Githubs.
Switch as much of my financial life over to opensource, private, and decentralized currencies as possible. Monero is my currency of choice.
Break my dependence on YouTube, for personal and privacy reasons. I use privacy respecting frontends and clients, but the service still has a strong grip on me.

Stay tuned for more updates on my journey to tech independence and how you can do the same.

Proton Mail is not perfect, because email is not perfect. I do believe Proton does care about privacy and has made huge strides for consumers however. For paramount security, I use PGP to encrypt messages before they even hit the browser. ↩︎
Currently, I sync my calendars with reclaim.ai, but I’m seeking alternatives that don’t require a Google account. ↩︎
I’ll elaborate on this later, but my preference for asynchronous communication stems from its ability to slow down life’s pace. Discord, in particular, poses privacy and centralization issues and lacks external searchability, making it inefficient for structured support or documentation. ↩︎

Read more and join the conversation at https://blakeashleyjr.com/posts/vendor-unlocking-my-life/

Movie review: The Vast of the Night

Nicola Iarocci — Sat, 30 Dec 2023 11:08:07 GMT

Yesterday evening, we watched The Vast of the Night, and what a pleasant surprise it was.

One night, in a small New Mexico town, a girl who works at a local radio station and an older reporter boy listen to a recording of some strange noises. Through the radio and its listeners, throughout a single night, they uncover a series of sighting stories that, from clue to clue and radio testimony to radio testimony, bring them close to uncovering something big.

Bristlemouth Dev Kit Delayed - Customs Woes

Greg Herlein — Sat, 30 Dec 2023 10:00:01 GMT

I’ve been accepted into the Bristlemouth Pioneer Program. I’ve been granted a SOFAR Spotter with a Development Kit. Unfortunately it was detained in customs due to it’s high invoiced value. I required a “customs agent” to clear it - which I did not have.

Amazon's Silent Sacking

Justin Garrison's Homepage — Sat, 30 Dec 2023 09:29:31 GMT

Companies are fighting back for quiet quitting and it's having a big impact.

Summary : Mastermind Group Blueprint by Tobe Brockner

Chester Grant — Sat, 30 Dec 2023 03:55:20 GMT

1. To describe the Mastermind concept, let me give you a quick analogy. Let’s say that you have an apple and I have an apple. If we trade apples, you still only have one apple and I still only have one apple. But, let’s say that you and I both have an idea. If we exchange ideas, then you have two ideas and I have two ideas. That is, you have your original idea plus my idea, and I will have your idea plus my original idea.

2. Hill stumbled upon this particular principle while researching the life of Henry Ford. He found that Ford met formally with a group of men who did not work at Ford Motor Company at least once per month with the intent of sharing ideas, solving problems, and identifying opportunities.

3. In this eclectic group, Ford found solace, like minds, and inspiration. When asked about the secret of his success, he said on several occasions that his monthly group meetings with these four men were pivotal in helping him reach the heights to which he arose.

4. Why You Should Join or Start a Mastermind Group:

The Ability to Work “on” Your Business Instead of Just “in” Your Business
Camaraderie and Understanding - I have long said entrepreneurs are some of the loneliest people on the planet.
Unique Perspectives
Motivation - When you surround yourself with business people who are excited, passionate, and energetic about the challenges they’re presented with, you can’t help but feel the same way.
Accumulated Experience - No two people will ever see the same events the same way.
Ability to Serve - I know it sounds cliché and it probably is, but it is still true: the more you give, the more you receive.
Accountability

5. It is much easier to recruit potential members when limiting group membership to only one company per industry. Think about your own industry. If you were in a group with one or two of your biggest competitors, would you be as open when it comes to sharing ideas and strategies? Probably not. So, do your best to find members from different industries.

6. Commitment - For the group to be as effective as possible, individual members must be committed to it. If attendance is sporadic or inconsistent, there will be subpar results. Besides that, it’s not really fair to other members if one or two don’t attend regularly. The whole point is to have a collaboration of ideas. When people aren’t there, collaboration suffers.

7. Membership Dues One of the most effective ways to ensure group commitment is to charge membership dues. These can range from $25 up to $1,000 or more a month. There will be some costs involved in running a group such as food, beverages, handouts, printing costs, room rental, and marketing materials. Apart from having the funds to pay for these items, there is also a psychological reason for charging dues.

8. If someone is paying to be a part of your group, he or she is much more likely to stick with it and take it seriously. I currently charge $297 per month for members to join our Mastermind Group, and members rarely miss a meeting.

9. Charge whatever you want, but make sure it is enough to keep them committed.

10. Similar Situations - Although you should only have one person per industry in your group, you should also make sure that everyone is relatively similar in where they are in their business’s growth stage.

11. You don’t want a member who has a fifty-million-dollar, twenty-year-old company partnered with a company that has been in business for three years and is barely making five hundred thousand dollars annually.

12. Optimism and a Positive Attitude - It’s okay to plan for worst-case scenarios and create alternative plans, should your Plan A not work out, but there is no room for the eternal pessimist. Find members who have a generally positive outlook on life and business.

13. The pessimists of the world are fairly easy to spot, and you need to be on the lookout for them within your group. This is the guy (or gal) who complains about every little detail, from the venue to the food to the topics being presented.

14. Remember, one of the main benefits of the Mastermind Group is to be inspired and motivated by smart, sophisticated, passionate business owners. One bad apple can spoil the bunch, and nothing will make your group disintegrate faster than negativity.

15. If you encounter a pessimist in your group, you will need to take quick action to rectify the situation. I recommend first having a frank discussion with that person. If things do not improve, you may have to ask them to leave the group.

16. Absolute Discretion It is critically important that you instill a grave sense of discretion among your Mastermind members. This goes beyond the whole “What happens in Vegas stays in Vegas” concept. Your members (and you) are going to be sharing intimately personal details about themselves, their employees, their competitors, their business practices, and their trade secrets. It is absolutely imperative that the group members treat this with the utmost respect and care. Trust is mandatory.

17. My Mastermind members are required to sign nondisclosure agreements that have stiff violation penalties. Fortunately, I have never had to use any of the penalties against anyone, but the simple fact that I take it so seriously with my members goes a long way in fostering a culture of openness and trust. I strongly suggest you do something similar.

18. Finally, traditional Mastermind Groups consist of no more than six people, each from different industries.

19. Concentrate on keeping your group to a total of around four to six members, including you. Keep in mind that you can set up as many of these groups as you want, depending on the kind of time commitment you can make. My colleagues in other areas of the country are running anywhere from five to eight Mastermind meetings at once.

20. If you ask people to shell out $300 to $1,000 a month to attend your Mastermind Group, then you need to make it a GREAT event. Catered food is virtually mandatory.

21. The way you sell this thing to people is by making it an elite, exclusive club into which they are being invited. It’s not something just anyone can join. There is a screening process and possibly a waiting list. Reality is perception. Every little component—every little component—has to be congruent with this positioning, including the food. Maybe even especially the food.

22. There’s a time and a place for casual dining, but your Mastermind Group meeting is not one of those times or places.

23. Along with the food, make sure you provide plenty of drinks. I usually buy a few types of soda, juices, teas, and then also have plenty of bottled water on hand.

24. However, our traditional Mastermind Groups are held in a large conference room at a local credit union.

25. Venue layout is important to the overall flow of your meetings. For a smaller group (four to six members), a round-table-type layout tends to work well. You don’t want members sitting in rows where some sit behind others. For high group interaction, a layout like that is highly inconvenient. Members should sit in a circle or half circle where everyone is facing inward, towards everyone else, making group discussion much easier.

26. Any large room free from distractions will work. If in a large conference room, make sure you can close the doors to limit the amount of noise and distractions.

27. Other options for meeting rooms are hotel conference rooms, universities, public libraries, corporate offices, or executive suites. (Executive suites often provide a large conference room to its tenants, and these can sometimes be rented for less than one hundred dollars.) Look into various options, and you should be able to find an acceptable venue without too much trouble.

28. Depending on how many people are in the group, I will go over a marketing topic for the first thirty to forty-five minutes of the meeting. We then take a quick restroom break and start with the first group member. That group member then has about forty-five minutes to talk about three or four challenges they are having in their business. The whole group participates in making suggestions, giving ideas, or fleshing out concepts related to the particular challenges under discussion.

29. Someone, usually my assistant, will take notes of the interaction and create checklists, to-do lists, and action items for that member to complete during the next month. This gives that member something tangible to take home and implement in business during the next thirty days. These meetings are not just about theory or ideas. They are about action.

30. After the first member is finished, we move on to the second person and the cycle repeats itself.

31. A lot of the real growth in the group comes from hearing about a successful strategy in one business or industry and then applying it to another business or industry, with only minor modifications. In this way, we all get to have a chance at shortcutting the learning curve, and more importantly, we get to implement something in our own businesses that has already been proven in another business.

32. At the end of each meeting, I will usually take five to ten minutes to leave the group with a motivational or inspirational thought. This ends the meeting on a positive note and gets everyone excited to get back to work and start implementing their ideas through the next month until we meet again.

33. Every new member receives a copy of a welcome letter (a letter welcoming him or her to the group and providing the necessary contact information to get in touch with me), a copy of the nondisclosure agreement (which must be signed and turned in before the meeting starts), and a copy of the ground rules (which I will cover here).

34. Ground Rules for the Meeting:

Attend Every Meeting - I am pretty firm about this one. If you can’t make a meeting, then you need to have a very good excuse. It’s just not fair to the rest of the group (or to you) to miss a meeting. I have actually instituted a “Two Strikes, You’re Out” policy, where if you miss two meetings during a calendar year, then you are gone.

Stay for the Whole Meeting - The group is only as strong as the members’ willingness to give to one another.

Silence/Turn Off Cell Phones - This is the one place where you give yourself permission not to be at the mercy of the daily distractions that plague you. The group needs everyone’s full attention, so turning off cell phones is a must.

Actively Participate - If members just sit there without offering anything to the group, I will usually try to gently draw them into the discussion with a friendly, “Well we’ve heard from everyone so far except for you, Jack. What do you think about this problem?”

Constructive Criticism Only/Be Positive

Don’t Interrupt One Another

Keep Your Comments Concise. Don’t Be a Time Hog.

35. A basic meeting agenda will look like this:

9:00 a.m.–9:45 a.m. Presentation to Group
9:45 a.m.–9:50 a.m. Break
9:50 a.m.–10:30 a.m. Member 1 Speaks
10:30 a.m.–11:10 a.m. Member 2 Speaks
11:10 a.m.–11:15 a.m. Break
11:15 a.m.–11:55 a.m. Member 3 Speaks
11:55 a.m.–12:05 p.m. Lunch
12:05 p.m.–12:45 p.m. Member 4 Speaks
12:45 p.m.–1:00 p.m. Wrap Up/Q&A/Etc.

36. This agenda assumes you have at least four members in your group. The times will have to be adjusted if there are more or fewer members.

37. Talk About During Their Allotted Time? The short answer is “whatever they want.”

38. In the event they need a little guidance to get started, ask them to come prepared with three challenges or opportunities they are facing right now.

39. Marketing generally isn’t sequential, but here is a basic process you can follow: identify, find, attract, and convert.

40. The biggest thing you want to look for are business owners who read a lot of self-help books, business books, and any other nonfiction works designed to help them improve their lives or business.

41. The second character trait you want to look for in a business owner is aggressive marketing. These entrepreneurs are the ones actively and aggressively trying to grow their business and aren’t afraid of spending money to do it.

42. Finally, you want to find those business owners who are somewhat established. Entrepreneurs involved with very young companies or startups are pinching every penny and may balk at spending money to join your group, even if it will benefit them.

43. Referrals: This is the best place to begin your efforts. Keep in mind that like attracts like. In other words, we tend to associate with and attract those who are similar to us. Even if you only have a few Mastermind members right now, you’ll want to start by getting them to refer their friends and colleagues to you.

44. Chances are, they are friends with and can positively influence potential members who would be a great fit for your group.

45. Members will come and members will go, so you will always need to keep your pipeline full of potential prospects.

46. Keep this in mind when recruiting members for your Mastermind Group. You want those individuals who have a great work ethic and will be master implementers. Their natural enthusiasm for work will spread to the other group members and everyone in turn will benefit.

Estimate, target, plan and commit

Tiberriver256 — Sat, 30 Dec 2023 02:00:00 GMT

I’ve started into “Software Estimation: Demystifying the Black Art” by Steve McConnell, and of course… as is my style. I’m making some conjectures about what I think software development should look like based off of reading the first chapter. I am going to finish the book, but jotting some quick thoughts down as I go helps me to digest the book as I go, so don’t be surprised if I actually contradict something in the book. I haven’t read it all yet.

Some definitions

Estimate - Something rather easily and simply generated that is roughly accurate and expected to be off within a certain range. A simple formula that may be good enough in most cases is suggested as Effort = NumberOfRequirements * AverageEffortPerRequirement.
Target - The business wants to do something by X date. A statement of a desireable business outcome.
Plan - This is a detailed schedule on how we plan to hit said target (it will likely include prioritization and leaving certain things out).
Commitment - When someone or a team promises to hit the target with a specific level of quality.

A logical plan

You can expect the business to come to you with targets. We should be ready with good enough™ estimates to help produce plans that we can commit to.

So the sequence is:

Build a system that can be used to produce good enough™ estimates on-demand
Wait for the business to bring you targets
Build plans to hit those targets using your estimates
Only build plans you’re comfortable committing to

The tricky part

It seems to me that building a data-set to produce good enough™ estimates on-demand is the tricky part. I believe Steve is going to cover this in more detail later in the book but my guess as to some important details would be all centered around measuring that AverageEffortPerRequirement and that the keys to getting accurate measurements there are going to be around when can we say something started and when can we say something is finished?

Estimate, target, plan and commit was originally published by Micah Rairdon at Tiberriver256 on December 30, 2023.

Choose optimism

Steph Ango — Sat, 30 Dec 2023 02:00:00 GMT

Around the age of twenty-two I realized that my worldview had been deeply imbued with pessimism and cynicism. It was the culture I grew up in. A hostility to new ideas, to anything that strays from the norm. An assumption that if things can go wrong, they will go wrong — that malice is pervasive.

One day, I decided to become an optimist and life became much more fun.

The life of a pessimist is easy but dreary. The life of an optimist is hard but exciting. Pessimism is easy because it costs nothing. Optimism is hard because it must be constantly reaffirmed. In the face of a hostile, cynical world, it takes effort to show that positivity has merit.

To be an optimist, adopt these assumptions:

The future can be great
People’s intentions are mostly good
Ideas are fragile and need nurturing

Every new idea is an unrealized dream. Dreams are delicate and easy to destroy. When an idea presents itself, try to imagine the best version of it — what would make this idea great?

Pessimism and optimism share a trait: both are self-fulfilling. Your intention influences the outcome. Call it karma or, simply, effort. I would rather inhabit a future that has the possibility of being great.

Only optimists can create a great future. Only optimists can imagine it. Only optimists will put in the effort to make it. If you want to create a great future, believe it can happen. Choose optimism.

Interactive demos using Astro

Avik Das — Sat, 30 Dec 2023 02:00:00 GMT

This blog is mostly text and images, but I’m a big fan of adding interactive components to make my explanations more effective. See my post on rendering curves in 3D as an example. For those interactive demos, the browser environment, with Javascript for the interactivity, is a fantastic delivery mechanism with wide reach and ease-of-use. Libraries like React make that easy, but a lot of the frameworks and tooling are built around the assumption that the end goal is a single-page app (SPA): the entire page is interactive, and page loads are handled by swapping out what’s on the page. Think Next.js, or the Vue equivalent, Nuxt.

Books I read in 2023

Nicola Iarocci — Fri, 29 Dec 2023 16:51:02 GMT

I read 24 books for a total of 7070 pages in 2023. That’s seven more books than last year, which is quite an outstanding result considering the seemingly unstoppable decline in book reading I have suffered in recent years. Most have been fiction books, and that’s something new and influential with the final result, as I tend to read non-fiction more slowly. The bad news is that I did not review most of the books I read this year, and that sucks. The last review was in August, a catch-up review of several books clearly showing I was in trouble.

Looking back at 2023

Lawrence Jones — Fri, 29 Dec 2023 14:00:00 GMT

Last year I wrote “Looking back at 2022” and found it valuable to reflect and celebrate a year of work, even if just as a reminder of the things that happened before going into the next.

So with another year having passed, time to try again! This time for my second full year working at incident.io.

To contextualise the year, some milestones (see company post) are:

34 to 77 employees
Opened our first New York office, Stephen moving out to lead it
Shipped two new products: Status Pages and Catalog

I always start these posts by listing everything that comes to mind over the last year, finding common themes, then deciding on a structure to best communicate them.

This year we’ll go with:

The work
Wins
Observations
- AI
- My role
- Writing
Brace for impact

The work

I’ve spent the last year working across three distinct teams, each working on different (and totally greenfield) products.

Status Pages (Jan-Mar)

The year began with us wanting to launch incident.io’s first ‘new’ product, Status Pages.

In my last post I said:

I’ll be starting 2023 by supporting the creation of a new team, the first we’ll be creating out of the general engineering pool.

In a past life, I’ve found success in joining a team and helping them establish themselves, building relationships that last so I can support them once I move away.

That team was the Status Pages team, and my gameplan for a project like this is joining the team properly for 6-8 weeks, with an aim of:

Establish a baseline for ways of working (2 weeks)
Grind to a meaningful goal (2 weeks, repeat 1-2x)
Repeat (2) but with an eye on succession (2 weeks)

At least, this was how I framed it when planning with Pete…

…and it – broadly – worked, with me joining a team of Martha (TL), Megan (PM), Isaac, Dimitra (eng) and James (designer) to build the MVP of what has eventually become our industry-leading status page offering.

As expected, the first step was figuring out what on earth we would build. It’s easy to assume you ‘know’ what the project will be before you begin but that’s always never the case: what even is a status page, really?

Obviously, you need a page to display the status of your site, but:

Do we run health checks against customer sites?
Are we planning on integrating with monitoring tools like Datadog to power uptime charts?
Will our pages be authenticated?

This was the most difficult part of my time in the team, mostly because having started work on January 1st, there were several key stakeholders out-of-office.

We had assumed a status page was so ‘obvious’ that the team could get on with planning the product without too much outside guidance, but it happens that commercial goals significantly impact what you build in this space (are you building for brand presence, existing customer experience, or revenue generation?).

This caused us to stall, which felt–to the team–like a false start. I’ll be keen to avoid starting projects with similar unresolved questions in future!

What followed was much smoother, though: learning our way around Next.js SSR didn’t take too long, and smart decisions such as using Vercel for hosting meant we got to an MVP very quickly. Our product teams can be extremely productive when given space to execute and a clear goal; after the initial ambiguity was cleared we went from an MVP to a feature-full and polished page in just a couple of months.

The team went on to launch Status Pages in April with me having left in March. While I was sad to have left before we’d properly launched the product, I’ve since been really proud of all the work the team has produced.

Especially considering my contribution is intended to be a solid foundation to build on, I’m particularly happy at how much they innovated in the 8 months after launch. Our Status Pages are now the most powerful on the market, with options for internal or customer-specific pages, best-in-class integration with the rest of our product, authenticated pages and more.

I’m proud to have been there at the start to help them get going!

Catalog (Apr-July)

As I began to look for the next project after Status Pages, it happened that customers kept asking us for ‘something’ that sounded a lot like a service catalog.

This landed on my doorstep, not least because I’ve been thinking about service catalogs for years but also because it’s a product feature I’ve been planning since I joined the company. I’ve already written about my experience of this project in “Three months building a catalog” where I talk about what the work itself looked like, so won’t repeat myself here.

Reflecting on this project from a personal perspective, though, I found the move from Status Pages straight into catalog to be difficult.

I’d left a team (Status Pages) that was just hitting its stride, a team I enjoyed working with, and before the product I’d invested a lot into was launched. It felt like I’d got off one stop too soon, and going straight back into the (extreme) ambiguity of catalog discovery, initially solo, was a big gear shift.

Other aspects made this project hard, too: holidays and conferences came at inopportune times, and it happens that building a catalog–at least, the one we wanted to build–is very technically challenging. Even explaining what you were building to internal teams was a hurdle!

So hard work, but we launched Catalog in June to a massive amount of customer appreciation and surprisingly high adoption. The feedback we heard was our product was much more flexible than our competitors, that it felt great to use, and people were surprised at how many previously impossible or painful automations it made possible.

I stuck around until after launch this time, too, so I left with much more closure than I had Status Pages.

On-call REDACTED (July-Dec)

After catalog launched, I moved to start a team building our most ambitious product yet, one that we’re yet to announce or launch.

So happy to share this project was building incident.io On-call!

You should absolutely check-out the launch video and keep an eye out for a load of technical content I’ve been storing up while this project was under-wraps that I can’t wait to get out in the open.

This project has required the largest amount of up-front design of any project I’ve worked on before, the next largest (in my experience) being adding an Open-Banking payments scheme to the GoCardless product, an undertaking that built into four teams over a 9-month period.

As an initial team of six (PM, TL, designer, 2x SDE, 1x SRE) the first two weeks involved about 6 hours a day of whiteboarding and discussions, producing technical docs to explore different approaches. The process reminded me a lot of Stripe’s designing a unified payments API, with us trying to consider every possible way a customer might want to use this product before we commit to a design.

It was exhausting but a load of fun, and you’d often find the team sitting in odd positions with blank stares after a day of working on it, or producing increasingly unhinged AI photoshops and meeting notes…

After the initial discovery, we got stuck into building.

This is a big project that we’re going to release in one go, which comes with all the usual challenges like motivating the team without a regular “we’ve shipped this!” cadence, or managing the compounding risk of building so much on top of product that customers haven’t yet used.

You can manage these problems: we’ve tried hard to find clear milestones and celebrate hitting them, and we have design partners who are using what we build and giving early feedback. It’s still harder than a normal 2-weeks-then-release cycle though, especially in an organisation like ours that’s built around those faster feedback loops.

It’s awkward talking in abstract about what we’re building but I will say this is something we knew we’d build from the moment I’d joined, and I’ve had it in mind over every project I’ve ever done at incident. If I’ve done my job well, the new product will fit within the rest of our product suite as if it was always designed to be there (because in a sense, it was) and it’s been fun to see which of those anticipatory decisions have paid-off.

I can’t wait until we’ve launched and I can speak about this publicly, but for now…

🤐

Wins

There are a number of things I’d consider a ‘win’ from last year. Here are a few that stood out to me.

Internships

It was very last-minute, but in December we decided to work with Imperial (my old university) to offer three internship placements for the coming summer.

My part in this process was to create a take-home test that we’d use to decide who to bring into the on-site interviews. I always find the process of building these tests fun, and I was especially happy with how this test introduced students to our space (incident response), having them build an on-call scheduler.

Taken from our internships repo incident-io/internships

The 20 candidates who passed the take-home came into the office for an assessment day, interviewing with a number of our product team. We ended the day extremely impressed with the quality of the candidates, leading to an intense decision meeting where we had to decide which of the many candidates we wanted to hire would get one of the three places.

This process felt really special to me. I wouldn’t be where I am now had I not done a similar six-month placement at GoCardless, and the idea that some of these students may one day feel the same about incident.io is a great (but weird!) moment.

Migration from Heroku to GCP

Since I joined incident we had run on Heroku. I’m happy to say that’s no longer the case, having migrated ourselves entirely into GCP in September, taking less than 10 minutes of downtime.

I was around on migration day (Saturday) to assist, but full credit for how smoothly this went goes to Ben Wheatley, our (at the time) one-man SRE team.

Taken from this tweet thread of the full migration process

Ben had spent the weeks in advance of the migration both building out the GCP platform and the migration plan into it. I’ve worked with Ben for years and know how meticulous he is, but I was most impressed with how he worked with the broader Product Engineering team during this process.

For each stage of the migration, Ben found volunteers with an interest in infrastructure to pair with him. Working like this was a great way to build operational experience before we moved over (our Product Engineers are on-call and will need this knowledge) but it also helped incorporate an app engineering perspective, something you can see in the ergonomics of the new tools/processes.

Not only did the migration go fast and smooth, it also showcased a model of SRE <> SDE collaboration that is rare in our industry, one I’d like us to remember and do more of as we grow.

Engineering nits

We try to be pragmatic when it comes to technical investments, accepting that no code is ever ‘perfect’ and trying to avoid implementing ‘best practices’ without asking if they make sense for us first.

It’s easy to join the team and interpret this as “we never make technical investments” which is a terrible outcome for everyone, especially as that’s definitely not our view.

I believe the opposite: that we’re great at making judicious and impactful technical investments and we’ve been doing so ever since the company was born. On the hunch that this work is happening but lacked visibility, I created an #engineering-nits Slack channel to improve things.

If–as an incident developer–you hit a snag or think “urgh, that was annoying” when writing code or using tools, this channel is where you should post your thoughts.

It’s explicitly not about venting, and is instead about building awareness of our weaknesses and highlighting the work people are doing to improve or fix problems alongside normal product delivery.

We’ve since had 10 engineers contribute 30 ‘nits’ to the channel, where we’ve found solutions for 13 of them (45%). Every nit comes with a discussion that shares context and history of the codebase, turning the channel into a great knowledge-sharing tool as well as a demonstration of the investment we do want to prioritise and an example of people doing it.

These nits have ranged from:

Our hot-reloader is flaky and loading stale code (fixed!)
Writing ‘deletion protection’ code for the product is difficult (fixed!)
Deploys are slow (on-going)
Test output for errors is bad (fixed!)

Anyone is free to discuss or tackle the problems in the channel, and we’ve even started a series of blog posts for people who want to share this type of work externally (see Storybook for Slack and Generating code faster).

At 25 engineers and growing, now is the time to start being deliberate about the culture we want to create. Empowering every developer to own and improve the codebase is a key part of my ideal culture; #engineering-nits is a small but promising step in the right direction.

Observations

General observations that I’ve taken from the year, both general and personal.

AI

I began the year sceptical about AI. Clearly it was becoming an important tool, but I didn’t know how quickly it would find its way into all software, or how that would feel.

After last year’s developments, I now believe AI tools will become a key part of every product team’s toolbelt, and teams who don’t take advantage of them will be leaving a category of delight and product-polish on the table.

I’ve come to this through several experiences, such as:

My initial experiments using OpenAI APIs to build toy product features back in February.
Watching how quickly teams at incident adopted these tools, finding new ways to develop with them, such as incorporating real-time feedback from the product into their evaluation of the model.
Recently using AI for data analysis of developer MacBook build times and was blown away at what it could do.

I’m not yet convinced that AI will take over the world, and I don’t believe this generation of AI tools is going to take our (software engineers) jobs. But I do think you could review any software product and–with minimal effort–use AI tools to improve or totally reimagine those UIs, leading to vastly improved customer experiences.

Just as the DevOps movement brought production operations into the remit of app developers, the ubiquity of AI tools and the potential they unlock will force the average developer to start using them. That’s going to be an interesting shift, given how few developers have experience working with non-deterministic data models or similar systems.

My role

Urgh, this one is hard.

For anyone in similar positions, let me be clear: moving from a Principal/Staff position at a larger (500+) company into a start-up is going to feel pretty confusing.

I’ve had 2.5 years to figure this out but in a company that’s grown this fast–from me joining as first employee to our now 80+ person team–how you interpret your role might change every six months, so you don’t get much time to settle in.

This year started with the creation of the first separate team (Status Pages) and was followed quickly by four more. We’ve built each team from a common leadership structure: Product Manager, Tech Lead, Engineering Manager and (where we have resource) a Design Lead.

You need this structure to encourage the right type of inter-lead-and-team collaboration, but it’s made my job a bit harder: as someone outside that structure but with a role that requires me to work across teams, I have to try a bit harder to make sure I’m involved at the right time. I still find my way into much of what I need to be in and most people know when to pull on me for input, but I miss the clearer structure I’d known at GoCardless where it was more officially understood how I might be useful to people.

On a different note: this year surprised me with how much time I spent actually building.

Most of my time being spent in a team building things has meant a lot of my impact has come from ensuring the product we’re building has ‘multiplicative impact’ where each new product feature makes the whole suite stronger, in a ‘greater than the sum of its parts’ sense.

The best example of this is the Catalog which has become a foundational part of our product, weaving its way into many other product features. The idea of putting your organisation’s data into our tool (via the Catalog) is to allow other features to do things with that data, so it’s unsurprising other features have begun to build on it, but it’s not just about that.

Just as important as if other features start using it is making it easy for teams to do so, providing ‘lego bricks’ through which they can access the abstraction, and ensuring that if you invest in those ‘bricks’ then all other features using them can benefit.

Doing this right would mean Catalog ends up everywhere, which is what we’ve seen:

Auto-export follow-ups can use Catalog to determine what Jira/Linear Project/Team you export to.
Introducing status page sub-pages links sub-pages to Catalog data so you can say “create sub-pages for each Region in the Catalog”.
Configure your debriefs uses the Catalog to determine who to invite to your incident debriefs (e.g. invite the TL + EM + PM of the Teams affected by this incident).

That even ignores much of what we’ve been building for the last 6 months (and is yet to be released) that is built entirely on top of Catalog.

The tl;dr is that this year has been about putting my energy into this type of stewardship, both around the product that we build and the tools that we use to build it. I expect next year will see me do less of this myself and more of working with teams to help them think like this themselves, though it’ll depend a lot on the challenges we face as to where I’m most useful.

Writing

Every year I say I’d like to write more, and each year I write less. No doubt this trend will continue, but I am happy with a few pieces I wrote this year.

Keep the monolith, but split the workloads started a huge online discussion that kept going for a couple of weeks. This is great as the post (1) exhibits a lot of ways we think about software at incident.io and (2) gets us a bit more exposure as a Go shop, which is great for our recruitment efforts.

It’s also a topic I have a lot of experience with and one I’d love to speak about, but am yet to find the right conference for it.

The other big-ticket post was Tracking developer build times to decide if the M3 MacBook is worth upgrading. The investigation and process of writing this post was really enjoyable, and like many of my favourites included a combination of hacking (building the new hot-reloader), technical deep-dive (Go compiler performance) and data analysis. It’s a great example of the ‘fun’ someone might have if they join the incident team.

Personal blog posts are where I get a bit sad: this died a death this year, mostly because I was spending all my efforts building product. There were a few that landed well, though:

Three months building Catalog (July)
Use your database to power state machines (September)
Adding concurrency control to HTTP APIs (October)

Next year I’d love to lean a bit more technical, going deeper on the tools we use at incident and the strategies we’ve taken to evolve the codebase or improve reliability. We’ll see if there’s time for that, amongst everything else!

Brace for impact

That’s it, 2023 is over with.

While it’s been a hell of a year and I’m proud of everything in this post, I confess it feels misrepresentative: the majority has been spent building a product we’re yet to release, so it’s–by necessity–an incomplete picture.

That’s no bad thing, though! It just means a lot of the work this year has gone into making 2024 incredible.

Unsurprisingly, I’m a mix of excited, hopeful, and trepidatious about the next 12 months. I have no doubt it’ll be the biggest year yet in my incident.io career, but I’m finding it harder to predict than the previous two. That’s the nature of us making some big, potentially company-changing bets, but we won’t know if we’ve hit a home run until they land.

I know it’ll be a lot of hard work, but I’m confident we’ll make something great of it. Whatever happens, I’ll be proud to work alongside the amazing team we’ve built, am excited by all the new people we’ll welcome, and will try my best to make sure this is a year we can’t forget.

📗 The Coming Wave, by Mustafa Suleyman

Ernest Oppetit — Fri, 29 Dec 2023 13:11:48 GMT

This book was so heavily promoted across the internet I had to read it. Just like how I succumbed to an Athletic Greens subscription after the deluge of podcast ads. Mustafa “Moose” Suleyman is particularly clued-in on AI and the regulation landscape. He’s a co-founder of Deepmind and someone who is very plugged in to […]

Why My Print Didn't Output Before a Segmentation Fault

Ye Lin's Random stuff — Fri, 29 Dec 2023 09:51:12 GMT

So, as I was learning some C, at the pointers/segfault, I was trying this code myself.

#include <stdio.h>

int main(void)
{
 printf("%s", "Hello!");
 int *p = NULL;
 *p = 5;
 // Will not be reached due to crash above
 printf("%s", "Another Hello!");
}

Then compile it and run it in the terminal.

$ gcc -Wall -Wextra -o hello hello.c && ./hello
Segmentation fault (core dumped)

To my surprise, it did not print out both “Hello!” and “Another Hello!” and the program just crashed with “Segmentation fault (core dumped)” (aka segfault).

Summary : How to Host a Party and Handle House Guests By Judith Brown

Chester Grant — Fri, 29 Dec 2023 03:48:52 GMT

1. Keep a record of the parties and get-togethers you've given, along with guest lists and menus and what aspects of the event were-or weren't-a success. It will make useful reading next time you plan to entertain.

2. If you plan a menu of limited courses, everything can be set out buffet style. That way you'll be able to enjoy the meal along with your guests, instead of running back and forth to serve.

3. Remember to keep non-alcoholic drinks- juices, soda, etc.-on hand for guests who don't drink alcohol.

4. An easy and pleasurable way to entertain is to invite friends to your home for a relaxing Sunday brunch.

5. Invite your friends to a make-a-pizza party and see how creative you all can be.

6. Don't try out a new recipe on guests. Prepare it first for family members or close friends.

7. Make your buffet convenient for you and your guests by wrapping place settings of silverware in napkins and placing them in a basket.

8. Ice will last longer for a party if you first freeze an inch of water in the bottom of the ice bucket.

9. Your guests will always remember your parties if you send them picture postcards of themselves. As each person enters the party, snap a photo. But only send flattering photographs.

10. For a festive holiday get-together, have a Christmas cookie exchange. Invite neighbors and friends and ask them to bring some of the favorite cookies.

11. Holiday shopping won't seem a hassle if you can schedule your shopping excursions at off hours: right after the stores open, in mid-afternoon (2:30-4:30 P.M.), or late enough in the evening to avoid the after-work crowd. Try to avoid the crowds on weekends and lunch hours.

12. You'll always have little gifts on hand if you remember to buy doubles of things you need yourself. Such items as jellies and preserves, correspondence notes, and toiletries make suitable gifts. You can also take advantage of sale items and specials that look like suitable standby gifts.

13. Make a loaf of bread and wrap it up along with a jar of homemade preserves as a delicious and practical gift.

Reflecting on 2023, preparing for 2024

ntietz.com blog - technically a blog — Fri, 29 Dec 2023 02:00:00 GMT

This is one of those cliched posts: Reflection on the year that's ending, reviewing last year's goals, and talking about hopes and goals for next year. They're cliche, and they're also useful. The planning and reflecting process is a useful one, and sharing openly means other people can come along and learn with me^[1].

Reflecting on 2023

I thought last year was action-packed and, uh, this year has kind of set the new bar. It was literally a transformative year for me, but in the way of butterflies: I'm becoming the person I am meant to be. I'm going to list professional things first, then personal things, then community and broader events.

Professional

I was promoted to Principal Software Engineer. Early in 2023 I was promoted from Senior Staff to Principal. While our team size is smaller than it once was, this has still resulted in a notable shift in my responsibilities. It has taken me some time to fully get my feet under me, but I've enjoyed the process and the new role. Most notably, the shift is that I do a lot more cross-functional leadership (working with our customer-facing and business teams a lot more now) and I'm also the main technical advisor for our CTO. I am a full member of our company leadership team—the only individual contributor in that meeting—and I'm able to bring a unique perspective as both a tech-focused leader and as the longest-tenured employee of the company. It's been fun, and 2024 is going to be even better.

I also had some fun technical things at work. Of the things I can talk about, I wrote our first Rust production code and released a quick introduction to Rust to help my coworkers learn Rust more quickly. It's been a good experience getting Rust into production, and I have a good fun project with it in 2024, too!

I wrote a lot with some hits in there. I set out the year with the goal of writing at least one post every two weeks as a sustainable rhythm. I overshot this and wrote 56 blog posts, more than one per week! The total word count for the year, including this post, is over 60,000.

This has been a lot of work—yet at the same time, it doesn't feel like work at all. At some point, I want this sort of writing to be part of my livelihood but I'm deeply afraid of sucking the joy out of it by making it commercial. There is a balance I can find, and this blog will never be commercial, but the motions of writing and creativity can turn into other opportunities. For now, I'm keeping on with my writing and keeping my eyes open.

Here are some of the hits from the year that got the most views:

But I also wrote some pieces that were just deeply personally meaningful. In particular, the digital vigil for Trans Day of Remembrance is some of my most important software I've written, I think.

I'm not into ranking what I've written, and if I want to make a go of it as a business maybe it would be worth analyzing what was "successful" and what wasn't. But from a personal perspective, I'm pretty happy with everything I wrote this year and I'm deeply proud of the amount that I got done this year.

Released a programming language, Hurl! I finished working through Crafting Interpreters at the beginning of the year. The idea for Hurl was bouncing around my head after conversations with a couple of people, and eventually it did come into reality this year. Its launch post was one of my more popular posts, and I think there's something to that. It was a serious implementation of a joke idea, and that sort of whimsy and humor is a foil to the deep seriousness that our industry tries to project. I think we need more deeply unserious software, we need more play. I might even say we should write more useless software.

Before this year, I didn't think I could do programming language stuff. It seemed a dark art more mysterious to me than even operating system stuff. Now I see that it's possible and not that bad: despite the tremendous depth, you can get started simply and then keep learning and playing. Next year I'm going to do a little more with PL (not with Hurl, but something nicer), but in balance with other interests.

My productivity was high, and I often didn't see that. This year I started to realize that yeah, I'm quite productive even if I don't see it. For fellow Recursers, this is something that they may be relieved is finally sinking in, since my brand during my batch was posts where I lamented I got nothing done then had a laundry list of accomplishments. Now I'm starting to see my own productivity and separate "what I did" from "what I wanted to do," and that I need external mechanisms to remember what I did. Not getting everything done doesn't mean I wasn't productive, it just means my goals are quite ambitious or I got other things done instead.

A few things that I did this year while feeling "unproductive":

Implemented a digital vigil
Created a programming language
Made a functioning web app for managing chess clubs (it works, project is on hold though!)
Made a visualization of the 2023 FIDE World Chess Championship games this year
Wrote a simulation of approximating pi with a cake
Implemented RSA in a toy cryptography crate
Wrote a crash course on Rust

Now, notably, half of those were in the last few months when I got other things in my life in control! I did have times where I got less done in my personal time, and times when I got more done. That's pretty normal. I'm really proud of what I've done this year!

LLMs happened, and my relationship with them has changed. In 2022, I was deeply skeptical of LLMs and their power. In 2023, I saw some incredibly impressive demos which showed that (1) they're pretty useful but more importantly (2) they're here to stay. As a result, I leaned into learning how to use them. If they're here to stay, I need to adapt and get used to them, right?

I've used ChatGPT, Copilot, Claude, and other LLM tools to assist with my work as a programmer^[2]. They're okay. There's a lot they do well, and some sharp edges and fun failure modes, and this is all better discussed elsewhere. And now I've largely shifted away from using them much beyond rare Copilot usage.

I'm pretty content that I can learn to use them quite effectively if I need to in the future. I'm not sure I'll want to. In my experiments with them, using them daily for much of my work sucked a lot of the joy out of it. And I'm not convinced it was a net productivity boost for me, in no small part because of how my brain works and its peculiarities.

Personal

Oh hi, I'm a woman! This year I came into my identity as a trans woman and began my transition. I'm fortunate to have a lot of support: my family, my work environment, the town I live in, and Recurse Center, all these communities have supported me. It's a long road ahead, and so far it has been on whole a very healing and good process. Life is much more enjoyable now. Transition is sometimes painful, but also necessary and worth it. I've found in myself a surprisingly extroverted woman, and it's been a surreal experience.

I'm so happy now.

I've hit my best mental health in a long time. The end of 2021 / start of 2022 saw me in my deepest depression I can recall (at least along certain axes). In contrast, I've come out of 2023 in my best mental state in a while, prepared to deal with what life throws at me. There are a lot of aspects to this. It's a combination of transition, therapy, and other psychiatric care.

Transition has improved my mental state. It may be self-explanatory, but it turns out that a major underlying stressor like unrecognized and unaddressed gender dysphoria can mess a girl up. My depression in 2021/2022 was pretty strongly related to gender issues, and some of the first insights leading to my transition were from therapy sessions which helped me pull out of that depressive episode.
Therapy has been tremendously helpful. It took a few tries with a few therapists, but I've once again found therapy to be helpful and this time am in it for the long-haul. It's expensive, and it's a necessary life expense for me. My therapist has been helping me grow into my emotions, learn how to understand and process them, understand myself and others, and process some of what life throws at me. We've worked on skills for dealing with all of this, and for dealing with acute situations.
I've been diagnosed with ADHD. This is one I've suspected for a while, and it is empowering to have a diagnosis and it's life changing to be treated for it. My brain works in a different way when medicated, and it's improving both my work and my home life. It's a lot easier to do activities with the kids when I'm not either distracted by every single thing or deep in a hyperfocus rabbithole. And when the medication has worn off, I have retained larger reserves of energy from the day by not fighting my brain, so unmedicated times are also better.

Got my physical health in order, too. In 2022, I had some bad RSI-induced nerve pain in my arms. For some portion of the year, both at work and at Recurse Center, I could not type without great pain. I was limited to only typing in passwords, and all coding was done by voice using Talon. I recovered from that and went back to my old habits.

Who is shocked to find out that the habits that led to RSI the first time, led to it a second time? After the pain began to come back, I got a Keyboardio Model 100 and it has largely resolved my pain. It is a great thing to have this pain resolved. I've had to make some custom items to use them portably (notably, a custom lapdesk for my keyboard and laptop riser).

Some other issues are also being taken care of, too, and my energy levels are up.

Played a lot of chess, and got involved in my local club again. I took a break from attending my local chess club for a while, because life got in the way and then I was still figuring out my gender identity and starting transition. It was nerve wracking going back, when I knew people would recognize me. I was nervous about their reactions, fearful of deadnaming or incorrect pronouns. But it went great, and I went back not only as an out trans woman, but as a volunteer who helps run the club and is organizing our first official rated tournament!

A big part of why I went back and started volunteering is because FIDE, the international governing body for competitive chess, made some bad regulations that impede trans women from competing in women's chess events. If they want to keep us, keep me, out of chess? Well then I'm going to come back and be very visible in my local club, and run tournaments. You can't keep us out. We're here, and we'll always be here.

I've played quite a bit of chess this year and fell into bullet chess. I also stopped studying, so hit a bit of a rating slump for a while. This is okay, and it's been enjoyable, but it's probably time for that to change.

Now I'm a certified club tournament director with the USCF (this is unimpressive: it just means I read the rules and filled out a form). I can run rated tournaments! To upgrade to being a local tournament director, the next step up to run or assist with larger tournaments, I need to run a few small ones and I need to play in more tournaments. I'll get there.

Parenting is great, parenting is a challenge. I'm a good mom. I've come into that feeling this year, finally letting go of a lot of the self-doubt. Not all of it, mind, but enough that I can confidently conclude that I am a good parent. We have plenty of challenges with the kids, and we get through them.

Our kids are 2 and 4, and they're little bundles of energy. They're showing how uniquely different they each are, and through their eyes we get to see a lot of good things in the world. I've started to hit a rhythm with the 4 year-old of coworking in my office sometimes, and it's been a great way to bond.

Community and world

Another war... This time, war in Gaza. There's a lot that can be said. There's little that I will say here, now, for this is such a charged topic right now that I don't have words for. Here's a post by Paul Biggar that says it better than I can.

But the one thing I will say is: My heart breaks at every life lost. Every single person who is being killed as part of a genocide, being forced out of their homes: I cry for them. My heart breaks for them.

Why do we keep killing people?

A few of my friends had/have major health struggles. I won't go into more detail, but it defined parts of the year.

Found community in a Staff+ Engineer Roundtable. I ran a roundtable meeting for staff+ engineers (or anyone interested!) for a while at Recurse Center. Through this meetup, I found community with some staff+ engineers and made some deep connections. It wasn't a forever sort of thing to run, and I'm glad I ran it and that it ran its course.

Started attending a Quaker meeting. I've long felt a draw to some sort of organized coming together as a way of exploring and acting on my values. For some time, I attended a Unitarian Univeralist church. Now we've begun attending an unprogrammed Quaker meeting in our town, and it is really nice. We've met a lot of lovely people. I've been accepted in full. And it's nice being in community with people who share my values.

I re-entered social media, via Mastodon! I'm now on Mastodon, and you can find me @nicole@tietz.social. It's been fun so far! New followers are welcome. I'm going to follow few people, probably, to keep my feed nice and tidy. One of my favorite things is being able to disable boosts in my feed, which reduces the stimulation and makes it more usable for me.

Last year's goals

Okay, whew, that was a lot this year. What about what I wanted to do, though?

My post last year had some of my goals and anti-goals for this year. How did I do on those?

✅ I wanted to keep writing, and I did more than what I set out to do! This one was a success and really reminded me how much I love writing.
✅ I put one side project into production, sort of, then took it back out of production. I consider this a victory and since I'm tabulating it myself, no need to be a pedant. I do want to keep not-productionizing things, because it is such a stressor for me. I don't want my hobbies to be a second job.
✅ I did avoid learning about DevOps-y tooling in my free time, unless you count learning some Fly stuff. This one kind of comes for free if I never try to do deployments of any of my stuff on my own time!
✅ I mostly stayed active in RC, though I had some lulls. It is a reasonably sized community which can be overwhelming at times. I've had to cut back on how many things I participate in, to balance with deeper projects and more full participation in the places I do stay engaged.
❌ I did not establish learning habits this year, and was very ad hoc with it. So this one is a miss! But I also am glad I didn't, because the approach for the year was great.
✅ I did keep in touch with people at RC! I kept in contact with some current friends and made a few new ones.

Overall, I think I did really well on those goals. They were broad and about what I generally was interested in doing and not doing, and that's a good pattern for me. Broad strokes, and useful for directionally deciding on what to do, rather than any specific deliverables. Setting anti-goals was more helpful for me than setting normal goals, so I'll do that again.

Hopes and goals for 2024

I don't do predictions or resolutions, but reflecting on what I'd like next year to look like is helpful. It puts me in the right mindset to do my best to make the reality I want to see. Here's what I'd like to do in 2024.

Keep my rights. This is the headliner, because trans people are Republicans' favorite punching bags right now. This is an election year in the US, and attacks on trans rights rage across the country. I could be arrested for using the bathroom in Florida. "Drag" bans proliferate in ways that can make being trans in public illegal. So my goal, with the election, is to emerge from 2024 still having my rights: my right to exist, my right to be a parent, my right to my medical treatment. I'm prepared to do whatever I need to to save myself and others, and I hope I'm able to safely remain in the home I love. I'm afraid, I'm so deeply afraid of what 2024 can bring if Republicans win.

No personal-time side projects into production. This one will probably be a forever anti-goal for me. I just don't enjoy doing ops-y stuff but I feel its siren song; that yak has a lot to shave. It's important to preserve my free time by not making production web apps since the maintenance is high. I've dropped the "don't learn more (dev)ops (at home)" anti-goal because I won't run into it if I'm not deploying things.

Strike a better balance with calls and making. Since finding my extrovert energy this year, I started to overschedule myself. I met a lot of people and felt a need to have coffee chats with all of them. But... I started to realize I was overscheduled, leaving little time for longer chats with my closer friends. It also cut into time I could have used for making things! So next year I want to have more chats with close friends and fewer with new friends. I want to reclaim some of that time to make physical things, probably more picture frames and some jewelry.

Continue writing, and expand my writing. This blog is one of my main creative outlets, and I'm going to continue it. I like the weekly schedule, and I will keep that up. At the same time, I have some other writing I'd like to do. I have a few unedited personal pieces, mostly about gender, and I have some sci-fi ideas dying for me to try to write them. So I want to actually take a swing at that.

Where do I put those? They may wind up on another section on this site, or they may just be shared with friends. Suggestions are welcome.

Do some comedy! Making people laugh has been fun for all my life, but I never thought I could "do" comedy. It turns out that being trans, there are a lot of things that are really funny. I've started writing a stand-up routine around that sort of thing, and I want to try that out this year.

Stay active in my communities. RC is one of my favorite communities and I'm going to keep being active in the RC community and stay in touch with people. I'm also going to keep meeting our neighbors in our town and hanging out with them or sharing food. It's nice having a local community, too.

Keep being a good parent and partner. There's not a lot to say here except that it's a lot of work and it's a big part of my life and identity, so it has to be here. It would be incomplete to list all my other things I'm doing and not mention my family.

Finish voice training. Voice training is a challenging part of transition for those that choose to do it. I have always been deeply uncomfortable with my voice. This year I've found a voice that is so authentically me and that I like to hear, that is not dysphoric to hear. The work remaining is generalization, the ability to use it in all life situations. There are other fine-tuning things that could be done, but this is the main one, and I want to complete this.

Improve my ergonomic setup and my accessible options. I have an okay ergonomic setup today. My custom lapdesk allows me to travel around the house with my laptop, but it doesn't allow me to go to the coffee shop, and work travel coming up has brought this pain into focus. Suddenly, I need the ability to travel to a hotel with my ergonomic keyboard. So this year, I'd like to improve things in two ways:

Build a more compact travel keyboard setup. This one I planned out while on a run last week and now I need to prototype it^[3].
Learn to use Talon again and integrate it into my daily work. This is something that's important so that I am not so fully reliant on keyboards. It would allow me to go more places without a keyboard while retaining an input mechanism.

Do more technical projects. This year I made a programming language, and I hit a groove of working on medium-term technical projects. I want to do more of these in 2024! On the docket are cryptography, databases, and designing another programming language (implementation may take longer).

Go back into competitive chess. I've been playing chess casually this year. Next year I want to lean back into it in a more competitive way. I'm running a rated tournament in January! I'd like to also like to play in a local tournament. Another thing I should do is build my black opening repertoire more explicitly^[4].

Keep my mental health strong. Next year is going to be a challenging year, and I will put deliberate focus on keeping my mental health where it is (or better). Not a lot else to say, I think.

So, that's it! I've put a lot into this post, and if you've made it this far: thank you. A big part of what I do is learn in the open, and writing is thinking. Writing this sort of reflection helps me.

2023 had a lot in it, with some bad things and lots of very good things. I'm hoping 2024 shifts the balance to more good and less bad, but I'm prepared for it either way and will practice self-care to get through it whole and healthy.

This introduction is lightly edited from last year. There's not a whole lot else to say to introduce it! ↩
I have not, and never intend to, let LLMs be involved with my writing. At the very least, not for something like this blog. It's so deeply personal. ↩
Of course, the last "prototype" lapdesk is still in constant use by me, with bare wood. So this prototype will likely end up in daily use, too. But that's fine! Only through using it can I figure out what I want to do differently next time. ↩
My bullet/blitz black repertoire is "premove 1. ... e5", which works surprisingly well. You have the shock factor of playing some gambit lines as a premove. I want to see how far I can extend this, and I also want to just plain learn the acceptable lines here. ↩

Backing up mastodon

Ellie's Notes — Fri, 29 Dec 2023 02:00:00 GMT

I'm the administrator of https://bikers.social, a mastodon instance for bikers 🏍️

Postgres is the most important part of an instance backup. If you lose your db, it's game over. My secrets are also backed up.

My old mastodon backup policy consisted of regular pg_dumps, and scp to another machine. This was totally good enough, as that machine was also backed up to my local NAS. I did have to semi-regularly manually clean up old backups though, and now that the instance has been going a while it would be great to have something better.

I've recently revamped it a bit to use Cloudflare R2.

R2

I use R2 for all of the instance's assets. It's much cheaper than S3 for storage, and also has no egress fees! I'd be a bit too concerned about surprise bandwidth bills with S3.

As of May 2023, R2 also supports lifecycle policies

Setup

First up, make a new R2 bucket. This will be used for storage. Ensure it is not publicly accessible!

Then we want to setup the local aws cli. I created an API token scoped to the bucket, with object read/write, and limited only to the IP address of my mastodon server. Run aws configure, and paste in your new account ID and secret.

The AWS cli will, by default, assume that the endpoint you wish to use is Amazon. Newer versions of the CLI can be configured in config, but chances are your package repository is out of date. For all subsequent commands, I set

alias aws='aws --endpoint-url https://<account ID>.r2.cloudflarestorage.com'

We can then setup the lifecycle policy.

Go to your bucket settings, then lifecycle policy

I setup a lifecycle rule to keep daily backups for a week, weekly backups for 6 weeks, and monthly backups for 6 months.

Script

My backup script is super simple. When run, it creates a file for the current day, and then uploads it to r2 with the specified prefix.

#! /bin/bash

prefix=${1:-daily}
date=$(date '+%Y-%m-%d')

pg_dump -Fc -Z 0 -U mastodon mastodon_production | xz -T4 > $date.sql.xz

aws --endpoint-url https://<account id>.r2.cloudflarestorage.com s3 cp ./$date.sql.xz s3://<bucket name>/$prefix/$date.sql.xz

If no prefix is specified, we assume daily backups.

I've chosen xz for my backups. It produces some of the best compression, at cost of being very slow. My mastodon instance is pretty overkill in terms of spec, due to it being a hetzner box and my instance not being too high traffic.

pg_dump -Fc -Z 0 - using the custom format, but disabling compression as we're doing that with xz

xz -T4 will use xz compression, and no more than 4 threads. I tried -T0 and it absolutely pegged all 16 threads, which was a bit excessive. 4 still completes fast enough.

Crontab

After testing it manually with

./backup.sh daily
./backup.sh weekly
./backup.sh monthly

and ensuring all files were created OK, I setup my crontab

0 12 * * * /home/mastodon/backup.sh daily
0 13 * * 0 /home/mastodon/backup.sh weekly
0 14 1 * * /home/mastodon/backup.sh monthly

There's some duplicate work, but that's not a huge concern here. I've scheduled them during waking hours for me, as I also set it up to ping me on Telegram when the backup completes.

Improvements

In the future, I may change this to use wal-g. I think this would be pretty overkill for what we have now, and the backup/restore process with pg_dump is super simple. If the database size gets to be too large, or if it makes sense to have point-in-time recovery, I'll revisit this.

2023 in Retrospective

bayindirh — Fri, 29 Dec 2023 02:00:00 GMT

The year is ending in a couple of days, and I wanted to leave a public note about how it went for me.

2023 was an interesting year. I spent a lot of time getting used to my new life, and to be honest, it went well. I’m happier than ever, and learning to handle my life in this new chapter.

The biggest change was how I allocate my time during the day. This nudges me to learn to downscale and manage my time better or be more efficient in how I accomplish things in general. This part of my transition is not over yet, I’m actively downscaling in both physical and digital ways. For example, I don’t have a personal desktop computer anymore, but living with a single laptop. I have almost completed my move to cloud storage, converging and simplifying it in the process. I’m trying to reduce the number of my physical belongings, however that part is going slower than I dreamt of. 2023 brought minimalism with all its peculiarities, but it’s not a failure by any means.

Another small yet important change was finishing transitioning to fountain pens for note-taking. I have disposable pens, but new ones are not being added much. In the end, I’ll need to have a couple because I use rulers sometimes, and there are special-purpose ones, like pigment markers, that I’ll need to stock, but all the grunt work is handled by fountain pens, which bring joy and do not create waste except the glass bottles you empty, which can be recycled. I can say that I’m more mindful of the amount of plastic I consume, and I try to actively recycle it (whether it’s done or not is another issue). I’ll be talking about note-taking and fountain pens in the soonish future.

I continued learning Go. I wanted to be able to have something between Python and C++ and Go filled that gap perfectly. Simpler than C++, compiled, garbage collected, and fast enough. It’s open source and has a GCC implementation, which is crucial for me. All in all, I love the language and its ergonomics. It proved itself to be a great companion for my automation and downscaling efforts. I may write what I think about Go in a future blog post.

I continued implementing and almost finished one of the tools I wanted to have with my newly acquired Go knowledge: Nudge, a small command line tool to interface with Pushover for sending push notifications to devices from the Linux command line, which is a crucial requirement for “lights-out operations” which I want to move towards for the mundane things I routinely do in front of a computer because I want to have more time for myself and people I care.

Lastly, with the AI revolution, I became way more mindful of how these systems are trained, and its effects on our privacy and free software. As a result, changed my trajectory towards services and systems which do not devour your things for their profit without your consent. During that transition, I also left Google search for Kagi, which is something you pay for to use. However, this payment makes sure that you don’t become the product during the process, at least for now. I’m pretty happy about this decision, and I recommend you to give Kagi a try.

2024 will bring the next step in many of these endeavors and will add new ones on top of it. I’m hopeful that the new year will bring better things upon us since 2023 was enough both for me and our planet as a whole.

Until next time,

Be kind.

Downscale images in url2epub

La Vita è Bear — Fri, 29 Dec 2023 02:00:00 GMT

Since I added support to my new Kobo eink reader to url2epub, I’ve been reading a lot of long web articles on it, and one thing starts to bother me: Kobo is the only epub reader, hardware or software, that do not auto downscale bigger images in my knowledge.

For example, when I read a Washington Post article about a captive orca, this is quite annoying:

Article rendered on Kobo Sage from url2epub with original size of image being cropped

So I started to fix it, by implementing a downscaling feature when grayscaling the images to be packed in to epub files.

It turns out that scaling an image is not a simple thing, as there are a lot of different ways to decide “how” (Image Magick project has a page describe in details of different filters). The general consensus is that for downscaling Lanczos is usually the best, but it’s also complicated to implement.

Since here in reality I’m only dealing with grayscaled images, that every color is only represented by a single number (a 16-bit unsigned int for the grayness), I decided to do something much simpler: Just calculated a weighted average of the color of the original pixels of the new pixel is representing.

For example, when you downscale an originally 720x540 image to 480x405, you can map the new (0, 0) pixel to a range of 0-1.5 for both x and y, so it’s the combination of the original (0, 0) (full), (0, 1) (half), (1, 0) (half), and (1, 1) (¹⁄₄). So just calculated the weighted average of the 4 colors from the original image to get the new color for the (0, 0) pixel on the new image.

With that implemented, I also need to decide how the API should work for downscaling. I could pass in both desired width and height to do the downscaling, but considering that Kobo eink readers can be used in both vertical or horizontal orientation, if you just downscale for the width, it could still be too high when you switch to horizontal and now the width limitation becomes the height limitation. For that reason, and also for the reason of for the sake of simplicity, I chose to only set one number being the limitation of both width and height. In the Telegram bot, this is the new /fit N command, and in the REST API this can be set via the new fit=N url query.

Then the next problem is to decide what’s the number I need to set for my own Kobo Sage. The official tech spec says the screen resolution is 1440x1920, I tried both 1280 and 1024 and it seems 1280 works best:

Article rendered on Kobo Sage from url2epub with image downscaled to 1280x1280

Article rendered on Kobo Sage from url2epub with image downscaled to 1024x1024

And looks like my downscaling implementation is rendered pretty sharp on Kobo’s eink display!

Running OpenStack Ironic Unit Tests in PyCharm

Thu, 28 Dec 2023 23:33:52 GMT

OpenStack Ironic + PyCharm + Unit Tests

I assumed for a long time it was likely trivial to get OpenStack Ironic unit tests running in PyCharm, but never took the time to get a reliably working configuration – until today. I’m going to describe the configuration here so hopefully it can be replicated by others.

It’s my belief this should work for most OpenStack projects using oslotest, but I’ve only used this method so far with Ironic-based projects. As always, please check the codebase and documentation if anything doesn’t work; this is a point-in-time blogpost with limited scope not a full document.

Help a Family Create a Memory

🦉 olshansky 🦁 — Thu, 28 Dec 2023 20:41:44 GMT

Snap! Smile! Turn around! Look here!

On a sunny day, with a breath of fresh air, I took a short walk, on an off-day winter holiday, Witnessing families, both large and small, capturing memories on a casual stroll.

These snapshots may sync to the cloud, or begin to traverse, Soaring about, sometimes emerging to the Metaverse, But without a digital frame, they often linger, forgotten, out and about.

If the photo stands out, it’ll likely be aired, But if not all are in frame, it may fade out, uncared.

The time has (not yet) come

deadlime — Thu, 28 Dec 2023 12:29:46 GMT

Creating a desk clock with Raspberry Pi Pico

Summary: Who not How by Dan Sullivan and Dr. Benjamin Hardy

Chester Grant — Thu, 28 Dec 2023 03:38:11 GMT

1. Results, not effort, is the name of the game. You are rewarded in life by the results you produce, not the effort and time you put in.

2. It can be easy to focus on How, especially for high achievers who want to control what they can control, which is themselves. It takes vulnerability and trust to expand your efforts and build a winning team. It takes wisdom to recognize that 1) other people are more than capable enough to handle much of the Hows, and 2) that your efforts and contribution (your “Hows”) should be focused exclusively where your greatest passion and impact are. Your attention and energy should not be spread thin, but purposefully directed where you can experience extreme flow and creativity.

3. If you’re ready to realize a much bigger and more powerful future, then you must stop asking yourself, “How can I accomplish this?” That question, although common, leads to mediocre results, frustration, and a life of regrets. A much better question is: “Who can help me achieve this?”

4. That’s what real leadership is: Creating and clarifying the vision (the “what”), and giving that vision greater context and importance (the “why”) for all Whos involved. Once the “what” and “why” have clearly been established, the specified “Who” or “Whos” have all they need to go about executing the “How.” All the leader needs to do at that point is support and encourage the Who(s) through the process.

5. You don’t get preferential treatment with Hows, but with Whos who know what they’re doing.

6. It is critically important to understand that Who Not How goes both ways. Yes, Tucker, Reid, and I are Dan’s Whos on this book. But Dan is also a Who to each of us.

7. Ultimately, Who Not How is about teaching you how to focus on what you can do, and then finding other Whos to do what they can do.

8. In every “Who” relationship, you will have Whos, and you will also be a Who. No Who is viewed as better or more important than the other. All Whos are essential to getting the project done. There is love and respect among Whos. Each member of the team views the other as a collaborator on a shared mission, and each member wants to be a hero to the others.

9. Not only must the Who fully own the How, but they must have complete permission to do so.

10. If you’re going to apply higher levels of teamwork in your life, you’ll need to relinquish control over how things get done.

11. A core aspect of leadership is being explicit about the vision. The more explicit you are in what you want, the faster you’ll attract the right Whos to help you achieve that vision. The leader explains the “What” and “Why” and then allows the “Who” to execute the “How.”

12. “How” is linear and slow. “Who” is non-linear, instantaneous, and exponential.

13. TIME - Today Is My Everything.

14. Your Whos manage themselves; they aren’t managed by you. They have full responsibility for how they handle themselves because you’ve made the vision abundantly clear and exciting. You’ve then given them full ownership over executing and achieving the vision in whatever way suits them.

15. As it turns out, this isn’t just a fancy idea. Research shows that only 16 percent of creative insight happens while you’re at work. Instead, ideas generally come while you’re at home or in transit, or during recreational activity.

16. You need time and space, and most important, relaxation and recovery, to allow ideas and solutions to ferment and form.

17. The point here is, as you engage in relationships, you expand your efficacy as a person. Your efficacy is your ability to produce results, and it is based on the resources you have to put toward those results. Resources can be financial, but they can also be so much more than that. Encouragement, time, and focus are just as essential as monetary support. Resources not only expand your ability to produce results, but can have a transformational effect on you as a person—on your identity, worldview, and skill level.

18. For example, if you want to improve your health, you could simply get a gym membership. Or you could hire a personal trainer. Yes, this would be an investment, one you may not think you have the capacity to make. However, by hiring a personal trainer, your capabilities and potential in your health and fitness will expand. You’ll be able to produce better results because you’ll get the coaching and support you need. Additionally, by being invested, you’ll be more motivated and focused, not semi-committed.

19. Again, getting Whos is how you get committed.

20. Meta-analytic research shows that confidence is the by-product of recent performance or recent progress toward your goals. By growing your confidence, your imagination and future will simultaneously grow as well.

21. Procrastination doesn’t only stop your confidence from growing. You also limit your imagination, preventing you from seeking out bigger and bigger goals. Your identity or self-concept becomes limited. You stop believing you can achieve big goals, because your identity is largely shaped by your behavior. And this pattern will cause you to assume the same for your future. Thus, procrastination leads to a small self-image and an increasingly smaller future for yourself. You stop trusting in yourself. You stop believing in yourself.

22. Paradoxically, procrastination is actually a form of wisdom. Procrastination is a psychological phenomenon that occurs when you really want something more for yourself, but you lack the knowledge and capability to do it.

23. Procrastination is a very powerful signal telling you that it’s time to get another Who involved. You’re stuck. You need help.

24. Applying Who Not How, and thus killing procrastination, requires two essential steps:

Be radically explicit about your goals.
Ask yourself: Who can help me accomplish this goal?

25. Dan has created a tool that helps to define the vision, what successful completion of the project looks like, and why it is so important. He calls this tool the Impact Filter.

26. The Impact Filter, as a one-page document, solves this most pervasive leadership conundrum, and is comprised of the following questions:

What is the project?
Purpose: What do you want to accomplish?
Importance: What’s the biggest difference this will make?
Ideal Outcome: What does the completed project look like?
Best Result: If you do take action?
Worst Result: If you don’t take action?
Success Criteria: What has to be true when this project is finished?

27. Now that you’ve clarified and become explicit about your goal, you must refrain from trying to figure out “How” to accomplish it.

28. You’ll try to convince yourself why other people may not want to get involved. You may think you can’t afford the right Whos, or that you’re not a great leader. All sorts of limiting beliefs will flood through your mind, trying to get you to focus on the Hows, not the Whos. Without question, it takes courage to tell people your vision. It takes courage and leadership to get other people involved.

29. Dan often says, “The purpose of the Impact Filter is first to sell yourself on the vision, because you can’t sell other people unless you’re sold yourself.”

30. There are plenty of incredible and capable people who want to and will help you. All you need to do is tell them your vision. Spell it out for them. The Impact Filter actually does that for you. Then ask yourself: “Who can help me accomplish this goal?”

31. Once you’ve identified the needed person to help you accomplish your goal, it’s time to get that person engaged and executing the needed Hows. In order to do so, you’ll need to ensure your vision also matches their vision for themselves, and that you can clearly become a powerful Who to them. If you can, and if helping you achieve your vision will help them simultaneously become who they want to be, then you’ve got your Who.

32. Unless you’re brilliant at finding Whos, you should probably find a Who to find your Whos.

33. You can start small. Each small win builds confidence and an increased sense that you can create the life you want. Start by simply eliminating all tasks or distractions that are unnecessary to your future self. Often, we engage in tasks simply out of habit. If it can be eliminated altogether, then eliminate it. Your future self will thank you.

34. Challenge: Add at least one Who to your goals in the next 90 days in whatever area of your life you choose. By adding a Who, your commitment will increase and your behavior will improve.

35. By investing in Whos you not only utilize their time and resources, but also free yourself up to focus your time and attention on your most high-value activities.

36. Some people don’t invest in Whos like Connie because they don’t view them as an investment, but as a cost. They worry about the amount of money they’ll have to pay their Who, rather than thinking about how that Who could elevate their vision and free up their time.

37. The time I spend writing books and recording podcasts is worth at least 10X or even 100X the time I spent scheduling podcasts.

38. Escalation of Commitment: Every time you invest yourself in something, you become more committed to it.

39. Our culture has brainwashed us into avoiding costs rather than making powerful investments in ourselves and our futures.

40. You’re either in the “Time and Effort Economy” or the “Results Economy.”

41. Whos, when selected properly to fit within your vision, are never a cost. Whos are an investment.

42. “You can survive without a community, but you can’t thrive without one.”

43. First and foremost, when it comes to connecting with someone, you should want to be connected with them. It shouldn’t be a chore. There shouldn’t be any desire to avoid contact or escape from them.

44. “I don’t want to work on a relationship. I just want a relationship that works,”

45. The moral to the story: Don’t reach out to someone unless you have something meaningful to offer them. That “something” needs to be real and relevant, not just a compliment or flattery. True and real value.

46. It’s a back-and-forth process. Rather than sitting by yourself, trying to perfect the idea without feedback, it’s far more effective to throw your ideas out there fast, get feedback from your team, and then adjust as you go. The faster you get at throwing out incomplete work, the faster it will transform into something great. Dan calls this the 80 percent rule. You can get to 80 percent of a project very quickly, such as writing a rough draft. However, going from 80 percent to 90 percent is exponentially more work than going from 0 to 80 percent. Going from 90 to 100 percent is a mountain.

47. It all starts by setting a goal, a new and bigger version of your own future. Then your next step is to ask, “Who can help me do this?”

Pageripper API: a commercial data-extraction service

Zachary Proser — Thu, 28 Dec 2023 02:00:00 GMT

I built an API that extracts data from webpages, even if they're rendered with Javascript. It's defined via Pulumi and deployed on AWS.

A Hobby Coding Biography

Weblog on marginalia.nu — Thu, 28 Dec 2023 02:00:00 GMT

This is a bit of a retrospective of every project I’ve worked on, as far as I remember them. I’ve tried to unearth any artifacts that remain.

Far from everything is flattering and resounding success, but then again, maybe that’s good. There are definitely patterns in the things that didn’t pan out.

Earliest Traces

I was definitely programming stuff, but I don’t think it ever amounted to anything tangible. It was more like playing house, I built GUIs that looked like real applications but didn’t really do anything but look cool to me. Real hackertyper vibes. This was in Java, Delphi and VB script mostly. I also built a bunch of websites on geocities and angelfire. This is definitely as far back as 1999. I did some QBASIC stuff even earlier, like basic text based games and drawing circles and stuff.

Parsing HTML Fast

Shadowfacts — Thu, 28 Dec 2023 01:52:42 GMT

The urge to overengineer something recently befell me, and the target of that urge was the way Tusker handles HTML parsing. Mastodon provides the content of posts (and profile descriptions, and some other things) as HTML, which means that, in order to display them properly, you need to parse the HTML, a notoriously straightforward and easy task. For a long time, the approach I took was to use SwiftSoup to parse the HTML and then walk the resulting node tree to build up a big NSAttributedString that could be displayed.

And while that technique worked perfectly well, there were a number of downsides that I wasn’t entirely satisfied with. First: SwiftSoup alone is a big dependency, at around ten thousand lines of code. Since it’s shipping in my app, I take the view that I’m responsible for making sure it doesn’t break—and that’s a lot of surface area to cover (moreover, since it was ported from a Java library, the code is extremely un-Swift-y). Second: it’s not especially fast. Partly because of its aforementioned nature as a Java port (everything is a class, lots of unnecessary copies), but primarily because it’s just doing more than I need it to (the first rule of optimization is do less work): it was parsing a string into a tree and then I was flattening that tree back into a string.

The last time I needed to do something similar, I used lol-html, Cloudflare’s streaming HTML rewriter. But, while that worked (and is still in use for my RSS reader app), it’s written in Rust, and I didn’t want to introduce significant additional complexity to the build process for Tusker. So, writing a new HTML parser in Swift seemed like a great idea.

The Architecture

Parsing HTML is no small task, and the spec stands at some 42 thousand words. Fortunately, I get to ignore about half of that. Parsing HTML is divided into two primary stages: tokenizing and tree construction. The tokenization stage takes the character stream as input and produces a stream of tokens (text characters, comments, start/end tags, etc.). The tree construction stages then uses the token stream to build up the actual DOM tree. The main way I’m improving over SwiftSoup in terms of not doing unnecessary work is by skipping tree construction altogether.

This is based on lol-html’s architecture, which usually uses a simulated tree construction stage (because the tokenizer gets feedback from the tree constructor and may have its state altered based on the current position in the tree) and only switches to the slower, real one when necessary. In my case, though, I don’t even simulate tree construction, since everything I want from the output can be generated directly from the token stream.

This does mean that in certain circumstances the output of the tokenizer could be incorrect, but that’s not a problem for my use case. For Tusker, I’m only ever going to use it to parse fragments—not whole documents—and the Mastodon backend sanitizes incoming HTML. What’s more, from what I can tell of lol-html’s simulator, most of the cases where (without the tree construction stage) ambiguity arises or feedback is needed aren’t going to come up in my use case. They involve other tag namespaces (SVG, MathML), or things like <template> tags—both of which I’m comfortable with handling incorrectly.

Tokenizing

The tokenizer is the single biggest component of this project, weighing in at about 1500 lines of code. Fortunately though, the spec is quite clear. It describes in exacting detail the state machine that a spec-compliant tokenizer needs to mimic. To make things simple, all of my code is an almost one-for-one translation of the spec into Swift. There are a few places where I diverged, mostly in the name of performance, but where I didn’t, the code is quite easy to follow.

Attributed Strings

Converting the token stream to an NSAttributedString is somewhat more complicated, but not terribly so. The gist of it is that the converter tracks what the style of the string should be at the current point in the token stream. Then, when a text character arrives from the tokenizer, it’s added to a buffer representing characters for which the current styles apply (called a “run”). When an open/close tag token is emitted, the current style is adjusted. After the current styles change, the current run is finished and an attributed string is built for the current text and styles and is appended to the overall string.

Optimization

Named Character References

The spec requires HTML tokenizers to have some special handling for named character references (things like &) which aren’t well-formed (e.g., missing the trailing semicolon). Because I was sticking very closely to the spec, this was originally implemented rather inefficiently: when a named character reference began, it would consume further characters one by one while there were any named character references beginning with the accumulated characters. I was just using a regular Swift dictionary to store all of the valid character references, so this was effectively an O(n) operation for each character in the reference.

That is, to put it mildly, ridiculously inefficient. Instead, the new approach just consumes as many alphanumeric characters as possible until it hits a semicolon. Then, if there’s no valid character reference, it backs up character by character until it finds a valid reference. This optimizes for the common case of a properly-formed reference, but (as far as I can tell) preserves the error-handling behavior that the spec requires.

Static Dispatch

The way I implemented the tokenizer, there’s a separate method that follows the procedure for each state, each of which returns the next token. The tokenizer itself is also an Iterator that yields tokens. The Iterator.next implementation just switches over the current state and dispatches to the correct method.

It’s fairly common for the procedure for tokenizing to involve switching to a different state. Originally, I did this by changing the state and then calling next(). One small optimization I made relatively early was, in instances where the new state is constant, replacing the next() call with a direct call to the appropriate method for the new state—effectively statically dispatching whenever the state changes.

state = .tagName
return next()
// vs.
state = .tagName
return tokenizeTagName()

UI/NSFont Caching

A number of the tags handled by the attributed string converter alter the current font. Originally, I implemented this by looking at the currently-applied styles and using them to assemble a SymbolicTraits that could be passed to UIFontDescriptor.withSymbolicTraits on the base font descriptor. When profiling, though, I found a non-trivial amount of time was being spent inside withSymbolicTraits to look up the actual font that should be used. So, the converter instead caches the known styles and their corresponding font objects.

Using `Swift.Array`

One of the first optimizations I’d implemented was a type called InlineArray3 which was a collection type that stored up to three elements inline. Growing beyond that would cause it to switch to a regular Swift array, the storage for which is out of line. The motivation for this was the fact that the vast majority of the HTML tags that the parser sees will have very few attributes, so you might be able to gain a little bit of performance by storing them inline.

After implementing a simple performance testing harness (discussed later), I went back and tested this, and it turned out that just using Array was a fair bit faster. Just goes to show that you shouldn’t try to optimize things without having concrete data.

Looping in Hot Recursive Functions

For a number of states in the tokenizer (such as while tokenizing a tag name), the state machines stays in the same state for multiple iterations before emitting a token. Initially, I implemented this just by having the tokenization function recursively call itself. One small but measurable performance improvement I got came from turning hot recursive paths into loops. So, rather than the function calling itself recursively, the entire function body would be wrapped in an infinite loop and the “recursive call” would simply continue on to the next loop iteration.

This was one of the more surprising optimizations I made, since I’d expected this to make no difference. Given tail-call optimization, I’d think both approaches would generate basically identical code. And indeed, looking at the disassembly before and after this change seems to confirm that. There are differences, and though they seem very minor, I guess they’re enough to make a difference (though a very, very small one—only about 5ms over 10k iterations).

Avoiding Enums with Associated Values

Lots of states of the tokenizer involve building up the current token until it’s ready to emit. This means the parser needs to store the in-progress token. My Token data type is a Swift enum with a handful of different cases. So, the natural way of representing the current token was as an instance variable of an optional Token. Modifying the current token means, then, pattern matching against a Token and then constructing the updated Token and assigning it back.

Unfortunately, this does not permit in-place modification, meaning that all of the enum’s associated values need to be copied. For primitive types, this might not be a problem. But for more complex types, like arrays and strings, this results in a bunch of extra copies. So, rather than having a single currentToken property, there are separate properties representing the data for each possible token—each of which can be modified in-place:

private var currentToken: Token?
// vs.
private var currentStartTag: (String, selfClosing: Bool, attributes: [Attribute])?
private var currentEndTag: String?
// etc.

Using `Unicode.Scalar` instead of `Character`

My tokenizer originally operated on a stream of Swift Characters, which represent extended grapheme clusters. The tokenizer, however, only cares about individual Unicode code points—and indeed is specified to operate on code points. So, I changed the tokenizer to operate on the Unicode.Scalar type. This resulted in a significant speed-up, since the there was no longer any time being spent on the grapheme breaking algorithm.

Grouping Character Tokens

When emitting the document text, the tokenizer is specified to emit individual characters. As an optimization, though, I introduced an additional token type which emits a whole string of characters in one go. Since long, uninterrupted runs of characters that don’t require further parsing are quite common in actual HTML documents, this eliminates a bunch of the overhead associated with handling a token that comes from switching back and forth between the tokenizer and whatever’s consuming the tokens (that overhead is amortized over the sequence of character tokens).

Conclusion

To both test the final performance of the new end-to-end HTML to NSAttributedString conversion process as well as guide most of the optimizations I described above, I wrote a small benchmarking harness that converts the HTML from the last ten thousand posts on my timeline to attributed strings using the old and new methods. This is basically the exact use case I have for this project, so I’m confident that the benchmarking results are fairly representative.

All told, the new process is about 2.7× faster when both are built in release mode, and a whopping 8× faster in debug (not terribly important, but nice for me since a debug build is often what’s on my phone).

I’m very happy with how this project turned out. I greatly enjoyed overengineering/optimizing something fairly self-contained, and the end result meets all of the goals/requirements I had for my use case. If you’re curious to see how it works in more detail, check out the source code.

23 from '23

Something About Cats — Wed, 27 Dec 2023 22:05:24 GMT

The movement you need is on your shoulder

What I learned during Advent of Code 2023

Linux Engineer's random thoughts — Wed, 27 Dec 2023 21:00:00 GMT

Advent of Code is an Advent calendar of small programming puzzles. I participated in this year's edition, finishing it for the second time in row. The puzzles of all editions are always accessible.

The principle is to read the problem, get a puzzle input (more or less tailored to your …

Hosting web services on my Raspberry Pi

Online Home of Rajkumar — Wed, 27 Dec 2023 05:06:58 GMT

What Are Kubernetes Operators? (Operators 101: Part 1)

Steven Sklar | My Blog — Wed, 27 Dec 2023 02:00:00 GMT

If you've used Kubernetes in just about any capacity over the past few years, you've almost certainly leaned on at least one operator to automate a piece of your infrastructure.

The operator model has been espoused by influencers like Viktor Farcic (aka @DevOps Toolkit on YouTube) and Kelsey Hightower for its advanced automation abilities. OperatorHub.io lists over 300 different operators for you to install in your clusters. Kubernetes operators seem to be everywhere!

But what exactly is an operator? The official documentation page about operators is actually fairly short and leaves a lot to the reader's imagination.

This is the start of a new series, Operators 101, where I will teach readers how to design, build, and deploy Kubernetes operators that can automate the management of your own unique applications.

For a real-world case study of the benefits that the operator pattern has over traditional k8s resource management techniques, check out this article that I wrote which discusses the benefits of moving my company's SaaS infrastructure to a Kubernetes operator.

The Operator Pattern

What makes Kubernetes operators so difficult to describe is that they are not just one discrete unit. Instead, an operator is actually comprised of several distinct components!

When trying to explain what operators are to people who haven't dealt with them before, I like to describe them as new native Kubernetes resources that enable you to automate system components based on the resources' state and your own custom logic.

At a minimum, an operator consists of a Custom Resource Definition (which defines your custom object's schema) and a Controller that performs actions based on the state of Custom Resources created from your definition. I like to think of a Custom Resource Definition (or abbreviated CRD) as analogous to a class definition in Java, and a Custom Resource can be thought of as an instance of that class.

To provide a more concrete example of how these components work together, let's investigate how a commonly-used Kubernetes resource works under the hood and then relate this back to the operator pattern.

Analogy: How a Deployment Works

A Deployment's schema can be found in the Kubernetes API specification in the apps/v1 group. You can check out the full spec in the Kubernetes reference documentation, here. This spec contains the definition of a Deployment kind, or schema, which includes all of its fields, their descriptions, and types. I like to think of this schema as a Go struct; a container that can hold structured data of a particular shape. This API spec, or schema, is equivalent to a Custom Resource Definition in an operator, which defines the fields of your new Custom Resource and its metadata.

Continuing the analogy, when you want to create a Deployment, one could write a yaml (or json) file that uses the Deployment API schema. Here's a simple example:

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
spec:
 replicas: 1
 containers:
 - name: nginx
   image: nginx:1.23.3

This yaml file represents a Custom Resource, or an instance of your Custom Resource Definition.

When you kubectl create -f the file, the kubectl client sends a POST request to the Kubernetes API server with these file contents in the request body. Once the request has been validated, the server adds the new Deployment object to the Kubernetes control plane's key-value database, etcd.

Elsewhere in the control plane, there is a Deployment Controller that has registered itself with the Kubernetes API server to listen for adds, modifications, or deletes for any Deployment in the cluster. Once it has been notified, the controller will then run a reconciliation loop to migrate the current cluster state to its desired state, as specified by the mutated object. This is similar to how declarative programming languages, like Terraform, work.

Getting back to our example, once the API server has stored the new Deployment spec in etcd, the Deployment Controller will be notified about the newly created resource. This will trigger a reconciliation between the current state of the cluster (where the Deployment and its underlying ReplicaSet do not exist) and the desired state of Deployment, as defined by the spec stored in etcd.

It's important to note that the Deployment Manager does not directly talk to etcd. All communication with the cluster data store happens through the API server. I just added the step (4) line for clarity in this initial example. All future diagrams will display the correct data flow.

Since the Deployment did not previously exist, the ReplicaSet that it is supposed to control also does not exist. So in its reconciliation loop, the Deployment Controller will create a new ReplicaSet based on the Pod template portion and the number of replicas in the Deployment spec. Once the ReplicaSet has been created, the controller's job now is to reconcile this ReplicaSet with the Deployment's spec whenever a change to the Deployment spec occurs.

For example, lets say that you want increase the desired number of replicas to 2 by changing the yaml to:

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
spec:
 replicas: 2  ## changed from 1
 containers:
 - name: nginx
   image: nginx:1.23.3

Once you kubectl apply this yaml, sending a POST request to the API server, the Deployment Controller will be notified about this new change. It will perform another reconciliation of the underlying ReplicaSet, see that there is only 1 replica compared to the desired number of replicas (2) as per the Deployment spec, and modify the underlying ReplicaSet to increase the number of replicas to 2.

Composability

What makes this model so powerful is the inherent composability of Kubernetes primitives. In the above example, the Deployment Controller has one very specific job: to manage ReplicaSets. But as users, we don't typically directly interact with ReplicaSets. Instead, we're much more concerned with the state of Pods managed by a ReplicaSet, the runtimes where our applications live and breathe.

The work of actually managing Pods by scaling them up-and-down is the responsibility of the ReplicaSet controller. So the Deployment Controller delegates that work to the ReplicaSet controller, and trusts that it will do the right thing when a Deployment's desired number of replicas changes.

As operator writers, we can use this property to rely on existing Kubernetes primitives to handle single tasks and combine various components to perform more complex automation to solve our own unique problems. Which specific components we use and how we link them together is determined by you, the writer of the operator.

So What Is An Operator?

Now that we're familiar with how Deployments, and other Kubernetes resources, work under the hood, we can finally get to a clearer definition of a Kubernetes operator.

To work with any Kubernetes resource, we know that we need three components:

An API Schema (like the Deployments API spec in the apps/v1 group)
A concrete instance of that schema (a Deployment resource stored in etcd)
A controller that performs reconciliation actions based on a resource's state

An operator is a combination of these three components, designed to orchestrate a custom schema that you create in a way that you write. So in operator terms, these components are:

A Custom Resource Definition that defines your new resource's schema
Custom Resources which are instances of your new resource type
A Controller that performs automation actions based on the state of Custom Resources

Now you see how an operator is not just a single thing, but actually a combination of several pieces that work together. The idea of multiple smaller components working together to accomplish larger tasks is fundamental to not just how operators work, but also to how the entire Kubernetes ecosystem operates. By writing an operator of your own, you will not just gain additional automation benefits, but you will also grasp a deeper understanding of how Kubernetes works under the hood.

Stay Tuned!

I hope that this article has provided some clarity into what operators are and why they can be a powerful tool in your automation toolbox. The next article in this series will present a sample Custom Resource Definition and discuss how this new spec can drive your own custom automation.

If you like this content and want to follow along with the series, I'll be writing more articles under the Operators 101 tag on this blog.

Cryptographic protocol analysis with Verifpal

pera's blog — Wed, 27 Dec 2023 02:00:00 GMT

Most software projects avoid formal verification due to time constraints and expertise requirements. Tools like the Coq proof assistant, commonly used to introduce the topic to computer science students, can appear too cumbersome for real-world applications and therefore hard to justify.

In contrast, in the field of cryptography specialised tools such as ProVerif have been regularly used for the last two decades for verifying important properties of cryptographic protocols, but these still involve writing scarily long proofs.

However, in recent years, multiple similar tools that try simplify the process have been developed. One that caught my attention was Verifpal, whose creators have prioritised user-friendliness in its design. Created by Nadim Kobeissi, Georgio Nicolas and Mukesh Tiwari, it was first released in 2019¹.

The Verifpal User Manual is really fun and it only requires some undergraduate-level knowledge of cryptography. In this post, I will show you a short exercise from the manual (“Challenge-Response Protocol”) to illustrate what automated formal verification looks like with Verifpal.

So, first thing our model will need is an adversary, which in this case is an active attacker that has total control of the network²:

attacker[active]

Yup, that’s all. Now we need some principals, this is, the parties who are trying to communicate:

principal Server [
	knows private s
	gs = G^s
]
principal Client [
	knows private c
	gc = G^c
	generates nonce
]

Here we have two principals, a client named Client and a server named Server. The server has a secret s that in this example represents a Diffie-Hellman secret exponent, and its derived public key named gs. Similarly, the client knows its own secret exponent and public key, plus a random nonce that the client will use as a challenge for the server:

Client->Server: nonce
principal Server[
	proof = SIGN(s, nonce)
]

The first line means that Client sends the nonce to Server through the network. Then Server digitally signs the nonce using its secret exponent s.

Server->Client: gs, proof // uh-oh
principal Client[
	valid = SIGNVERIF(gs, nonce, proof)?
	generates attestation
	signed = SIGN(c, attestation)
]

Now Server sends its public key gs and the signed challenge proof to Client. The // uh-oh is just a code comment (ignore it for now ;p). Then Client validates the signed challenge (the ? after SIGNVERIF tells that the execution should end here if the signature verification fails), generates an attestation message and signs it.

Client->Server: [gc], attestation, signed
principal Server[
	storage = SIGNVERIF(gc, attestation, signed)?
]

In the final part of our protocol Client sends its public key gc, the attestation message and the digital signature. Then Server verifies that attestation was signed by Client.

Note the square brackets wrapping gc: it means that this value cannot be tempered by an attacker, only read. Why? Well, let’s just assume the principals are able to share public keys through some other channel that they consider secure, so basically we want to keep this part out of the scope of our model.

Our protocol is now defined. Let’s see if there is any way an attacker could impersonate the server or the client. For that we need to check if an attacker can do anything interesting with our signatures:

queries[
	authentication? Server->Client: proof
	authentication? Client->Server: signed
]

A query is the way you ask Verifpal to verify properties of your protocol. Here we care about the authenticity of some values; could anyone perform a successful MITM? Let’s run the following command in the shell to see:

$ verifpal verify challenge-response.vp
Verifpal 0.27.2 - https://verifpal.com
  Warning • Verifpal is Beta
 Verifpal • Parsing model 'challenge-response.vp'...
[...]
 Analysis • Initializing Stage 2 mutation map for Client... 
   Result • authentication? Server -> Client: proof — When:
[...]
            gs → G^nil ← mutated by Attacker (originally G^s)
            proof → SIGN(nil, nonce) ← mutated by Attacker (originally SIGN(s, nonce))
            valid → nil ← obtained by Attacker
            storage → nil ← obtained by Attacker
            proof (SIGN(nil, nonce)), sent by Attacker and not by Server, is successfully used in SIGNVERIF(G^nil, nonce, SIGN(nil, nonce))? within Client's state.
[...]

The full output is a bit long for a blog post, but as you can see an attacker was able to impersonate the server by swapping the server’s public key gs for its own and then using it to sign a new proof!

This happened because we forgot to indicate that gs, like gc, can’t be tempered; remember the “uh-oh”? Let’s fix that:

Server->Client: [gs], proof

Now if we run the verify command again we will get this instead:

[...]
 Verifpal • All queries pass. 

 Verifpal • Thank you for using Verifpal.

Nice! This confirms that the properties we wanted to verify are true ^_^

Verifpal: Cryptographic Protocol Analysis for the Real World ↩︎
An “active saboteur” in the Dolev-Yao formal model, love that term↩︎

Hitting Back at Ledger Scammers With Python

Nelson Figueroa — Wed, 27 Dec 2023 02:00:00 GMT

Using a Python script to send fake data to a Ledger phishing site.

Another Way Not to Sign JSON

David Buchanan's Blog — Wed, 27 Dec 2023 02:00:00 GMT

Make your own way

Notes on software development — Wed, 27 Dec 2023 02:00:00 GMT

Over the years, I have repeatedly felt like I missed the timing for a meetup or an IRC group or social media in general. I'd go to a meetup every so often but I'd never make a meaningful connection with people, whereas everyone else knew each other. I'd join an IRC group and have difficulty catching up with what seemed to be the flow of conversation.

I hadn't thought much about this until the pandemic when I started a Discord group for software internals and a virtual tech talk series called Hacker Nights. Since 2021 the Discord reached around 1,500 members and ~20 fairly active members. And the Meetup peaked at about 300 members with about 10-20 showing up each Meetup.

After the pandemic receded I started an NYC-based book club over 2 months with about 5-8 active attendees. I ran a virtual hack week on Discord where I got ~100 devs into a temporary Discord server and we talked about Postgres internals and shared resources. Ultimately around 5 of us wrote blog posts and built new projects to explore Postgres.

I started a virtual, async email book club (that is still ongoing) with 300 devs from November 2023 to Feb 2024. There have been around 20 active members of the club. And each week the discussion is kicked off by one of the members, not myself.

And I felt like there wasn't enough community opportunity for folks in systems programming in NYC so I started an Manhattan-based Systems Coffee Club. Around 15 people showed up to the first meeting and seemed even more excited about it than I was. (And I was excited!) We'll see where it goes from here.

Organizing people to do this stuff doesn't come easy to me. I enjoy doing it to a degree, but every night before an event I have trouble sleeping. Worried about embarrassing myself. When the event happens though, and people are happy to be there to chat with everyone else, as they invariably have been, it makes it worthwhile.

Everyone want community

Something I realized along the way is that people (maybe devs especially, I don't know) are looking for community. And when I have noticed there seems to be a missing flashpoint (a topic, a career focus, a book, etc.) for community, it's been pretty easy to get people together around it.

The lifecycle of groups

Groups, meetups, naturally live and die. Organizers get burnt out. I don't see this as a problem. It's just the way it is.

At some point I'll get burnt out too. Or I'll get pickier. For example, I've been avoiding starting a systems programming meetup in NYC because I know it will be a big effort. So I've done lower effort groups like book clubs and coffee clubs.

Don't worry about signing yourself up for indefinite work. Just do whatever you'd like to and don't feel bad if you have to stop. Someone else will eventually start the next great group, even if it comes in a different medium or flavor.

Community is contagious

There are great communities out there that have inspired me.

Aleksey Charapko's and Murat Demirbas's virtual Distributed Systems Reading Group
Alex Petrov's database paper reading group
Andy Pavlo's database interview series
Paul Butler's BrowserTech meetup
Eric Zhang's New York Systems Reading Group

And this year I've been hearing about more.

TU Munich students started a Student Database Group
A group of developers starting a Türkiye-language CS reading group

There are yet a few more systems programming groups I've heard rumors about being started on the US West Coast and Stockholm.

Do whatever you want!

If you feel like you can't find the right group or that you don't fit in with existing groups or that you're missing a moment, there are surely other folks in the same boat. Waiting for a new group to join. You may be the catalyst.

There's enormous potential for getting people together and doing something interesting and there isn't necessarily anyone telling you you should. Things you try may work and they may not. The more you try the more you'll learn what works and what doesn't. I've had a few years of making mistakes organizing to hone the sense.

The only boring thing to do is to necessarily limit yourself to the sort of thing others have done before! Run a browser meetup instead of a React meetup. Interview hardware developers to teach software developers something. Get software developers with 20 years of experience in niche fields to teach the rest of us something. Read books beyond SICP or Clean Code. Try difficult programming projects.

Whatever you want though, don't let me deter you. If you think something should exist, give it a shot!

I used to struggle to get much out of meetups, couldn't pick up the flow of IRC. Some point I stopped trying solely to fit in. Instead to do what I thought was interesting. And to my surprise, folks were interested in coming along too!

Make your own wayhttps://t.co/tVEa2ndiZm pic.twitter.com/piWSsv14lj
— Phil Eaton (@eatonphil) December 27, 2023

Measuring Web Traffic is Very Hard

Magis — Tue, 26 Dec 2023 22:20:11 GMT

An obvious metric that is anything but.

Banning Lab-Grown Meat Could Hurt Animal Farmers

Josh Baldwin — Tue, 26 Dec 2023 21:45:48 GMT

Better investment planning could lead to everybody winning

Introducing Fontimize: Subset Fonts to Exactly and Only Your Website's Used Characters

Dave on Design — Tue, 26 Dec 2023 14:00:00 GMT

A Python library to subset fonts, which cuts ~95% from the size of your site’s font files & downloads/bandwidth. To use: from fontimize import optimise_fonts_for_files and call it with a list of HTML files. See the the full API docs for more info. Find Fontimize on Github and PyPi. Before the why, a quick preview of Read article...

My 2023 reads

Alexander's Blog Blog — Tue, 26 Dec 2023 02:00:00 GMT

Code: The Hidden Language of Computer Hardware and Software

AGI and self-driving cars: what can go wrong?

Vít Baisa — Tue, 26 Dec 2023 02:00:00 GMT

my first attempt at strip with collaboration with DALL-E

Re: (Extremely, unnecessarily long) Thanks for the pardoning

taylor.town — Tue, 26 Dec 2023 02:00:00 GMT

"I have found it to be a beautiful invitation to self-reclamation, and I feel it has been a lantern in my troubled search for the scattered pieces of agency that should have been mine from the beginning."

Blogging With Math

David Buchanan's Blog — Tue, 26 Dec 2023 02:00:00 GMT

Creating a writing assistant with ChatGPT Plus

Liza Shulyayeva — Tue, 26 Dec 2023 00:01:33 GMT

There are many ways to use your own books in AI workflows. In this post I’m going through the most basic workflow that does not involve coding your own app, creating vector stores, or generating API keys. You can do this by just using ChatGPT (with a Plus subscription). Many (most?) writers are not programmers, so there is no coding required here.

The Right kind of Blind

Devin Prater's blog — Mon, 25 Dec 2023 22:29:50 GMT

I know I really shouldn’t write this, but at this point it doesn’t matter. I’ve ran away from those who will most likely look upon this with disdain, as if I don’t look upon myself with even more than those on the outside can. I was reading a post from someone who mentioned “the right kind of disabled.” That struck something in me that I’d been looking for for a long time. I thought I was the only one. Maybe I am. But it needs telling anyway, so why not.

Acceptable disability

You know those blind people who can get around an area they’ve barely been to, or cook on a stove, or even communicate well with body language and such? I’m sure many in the blind community know of, or are one of these. These blind people don’t need much. They don’t ask for help, they know their way around, and workplaces are proud to have the acceptable kind of person with disabilities, because they barely notice that the person is blind.

And that’s the key, isn’t it? The person that can hide their disability so well that no one really stops to think about it.

Yep, it’s one of those blog posts. If you want something more cheerful and full of Christmas bullshit, there are lots of Amazon sales going on I’m sure. So, I’ve been blind since birth. You’d think I’d be adjusted to it and know how to do every little thing. You’d think I’d be one of those web accessibility experts by now, talking about Alt text and ripping web devs a new one because they used a div instead of semantic HTML. But nope. I’m the other kind of blind person. The kind the media doesn’t want to talk about. The kind organizations are ashamed of. The kind they had to put into a corner and hope he doesn’t rock the boat.

Nope. Instead of web accessibility, I’m all into OS and app accessibility. Instead of bravely taking on my problems, I run from them. All the way away from them. Yes, that’s why I left Mastodon. No, I didn’t kill myself; I’m far too cowardly for that. I’m the weird one that’s all into text formatting, Markdown, and understanding things like that which almost no other blind person cares about. I’ve tried Seeing with Sound. I use AI. I feel passionate about free open source software, even though I keep those feelings locked up tight so I’m not too disappointed in it.

While just about all blind people I know love parties, loud noises are sometimes physically painful. If I hear a pot clang, or a door slam, it’s like a stinging feeling in my arms or shoulders. So I stay home and chill. I joke to the people around me that I’m the most chill person they’ll meet. But it’s sadly not a joke. Everyone else wants to go somewhere, or do something, or get out of the house. My house is my vacation.

So what?

Being the way I am comes with a ton of downsides. I’m not as quick or persevering as the acceptable blind person. I was once in a meeting to help make an operating system more accessible, in which we needed to open a document and work on it. It was using a web app that the organizers said would be accessible. I had issues with it, and so I spent the five minutes we had, trying to find accessibility documentation for the web editor. There was none. And then when I asked about it everyone was all quiet like it was an intrusion and I shouldn’t be there and why was I even trying. And then the people seemed to shrug their shoulders and go on with the meeting. I can never forget that.

Side note: If any corporation, company, or organization says they take accessibility seriously, or that they put accessibility front and center, they’re most likely lying to you, and mocking you cause who dares tell someone they’re not doing enough in “the good fight?” Lol, Microsoft. We know you. Windows 11 speaks for itself.

And now, at this point, I’d give tips on how you, yes, you, can help the unaccepted blind. But I won’t, mainly because it’s all over every other blog post about dealing with neuro-divergent people. Listen, then act. Of course, the listening part is really, really important. But then acting is what’s actually going to make the person trust that you actually give even a single damn. Lots of companies, organizations, and nonprofits are great at listening. They have issue trackers, support lines, ticket systems, help desks, and open-door policies. But then when it comes to acting, well… We need more people to make us do it, we need more feedback, maybe next year, this should be sent to IT instead because I can’t actually help you, RIM is too expensive; see if you can make a deal or cut down the price for us, we’ll make a committee for this as a part of DEI even though they have lots of other stuff and none of these people are blind so yeah good luck haha, well we should do it but it’s hard, we don’t want to do it because other software should do it for us. Just kinda try not to lean on these excuses, and you’ll be good. And if all you’ve got left is “We can’t help with accessibility,” then just tell us. Tell us you don’t care enough, or can’t do it, or have tons of other priorities, so we can go somewhere else.

My experience with the Kindle Scribe (part 1)

ThatXliner's Blog — Mon, 25 Dec 2023 20:28:12 GMT

My first impressions

Quality First and Other Advice

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Mon, 25 Dec 2023 18:00:00 GMT

At our local meetup for the ‘Learning on Graphs’ Conference , we also organised a brief panel discussion with the idea of engaging researchers at all levels. While every panellist had a different story to tell, there is nevertheless one common strategy that I extracted from their discussions: When in doubt, focus on quality over quantity.

I can echo this sentiment. The few times in my research career where I lost sight of the quality of a publication, I paid for it afterwards. Shoddy work—even if it gets accepted—will always come back and haunt you, forcing you to correct your mistakes afterwards. I know that in the current age of machine learning research, there are students graduating with lots and of papers, but bear in mind that the number of papers, if anything, is also a measure of the lab size: Bigger labs tend to make it easier for Ph.D. students to participate in many projects. The numbers may thus be highly deceptive.

Notice that I am not advocating for complacency or never submitting anything to a conference during your Ph.D. On the contrary, I believe that navigating the ambiguous territory between ‘This is not ready yet,’ ‘This is good enough to get useful community feedback, and ‘This is perfect’ is one of the core skills you have to acquire as a researcher. Perfection is the enemy of getting things done, but low standards will make you miserable over the long run. Thus, as in all things, balance is key.

I also believe that focusing on quality enables you to find more joy in your research as you hone your craft. Since a Ph.D., and research in general, can sometimes be extremely frustrating, it is important that you only do the work you appreciate. Do not publish in a way that, afterwards, makes you feel empty, because, again, you will pay a price for shoddy work.

Somewhat related to low-quality research: Resist the urge to pull all-nighters. It seems like a good idea at the time, but I can attest to the fact that none of the papers we pushed very hard for during the night ever saw the light of the day—at least not without many additional major changes. The only thing our all-nighter accomplished was decreasing our motivation to work on the project. Add to this the high likelihood of getting strongly-worded reviews, and you create something that can sap motivation and life even out of the strongest people. Good research needs time and learning when the sunk cost fallacy has you in your grips is also something I believe will come in handy not only in academia.

Until next time, I wish you all the best in carving your own path!

Ocean Technology and Talking to Whales

Greg Herlein — Mon, 25 Dec 2023 10:00:01 GMT

I’ve been accepted into the Bristlemouth Pioneer Program. I’ve been granted a SOFAR Spotter with a Development Kit. I want to build a system capable of autonomously recording whale song (and other ocean data). Humpback Whale songs is incredibly interesting - and scientists have JUST been able to actually talk to whales.

But really getting all “science fictiony” - can we build an ocean phone system to talk to whales?

Staying In the Moment

Sunil Jain — Mon, 25 Dec 2023 06:14:45 GMT

There is this misconception that in order to stay in the moment, we need to learn how to meditate or go to some fancy retreats. Not that there is anything wrong with those.

Number one thing we can do to improve our staying in the moment is not pull up that phone every few minutes. It is not the momentary distraction that leads to but that acts trains our mind to expect that kick every few minutes in that phone instead of staying in the moment whatever you are doing. It is such an innocuous and deceptive act that we do not realize the damage it does.

It destroys the ability of mind to settle into and pay complete attention to whatever that we may be doing. It does not matter what the activity we may be in. It could be a lunch with a friend or colleague, it may be a walk with your spouse, kid and/or dog. We just can not resist pulling up that phone to may be just look up some information relevant to that conversation. Or it could be a chat message, email or text message. The act of pulling up that phone already destroys the immersivness and purity of the dialogue that was otherwise should be engaging. Except for some one near and dear in danger there is nothing else worthy of destroying that interaction.

This act conditions our brain to be never satisfied with what is happening and available to us in the moment. It conditions us to constantly look for input and stimuli outside of us to make us happy. We are unable to be in state for more that few minutes without putting something inside us constantly. Whether it is some useless clip someone forwarded, a chat mesage or some news item that invariable raised our blood pressure. Or actually putting food inside us purely for entertainment reasons, regardless of us being truly hungry or not.

It is this constant urging for stimuli outside us that is most damaging for us to be truly happy and content. Every tradition in the world as well as modern psychological studies have demonstrated that true happiness lies not in constant gratification of every new desire that springs in us. It is the ability to stay present with the event, whether it is joyful or sad. And not running away from anything that can make us uncomfortable. There is enough scientific literature now on the topic of hedonic set point we all have.

Imagine our brains contain a balance—a scale with a fulcrum in the center. When nothing is on the balance, it’s level with the ground. When we experience pleasure, dopamine is released in our reward pathway and the balance tips to the side of pleasure. The more our balance tips, and the faster it tips, the more pleasure we feel.

Here’s the important thing about the balance: It wants to remain level, that is, in equilibrium. It does not want to be tipped for very long to one side or another. Hence, every time the balance tips toward pleasure, powerful self-regulating regulating mechanisms kick into action to bring it level again. These self-regulating mechanisms do not require conscious thought or an act of will. They just happen, like a reflex.

Neuroscientist Nora Volkow and colleagues have shown that heavy, prolonged consumption of high-dopamine substances eventually leads to a dopamine deficit state. The paradox is that hedonism, the pursuit of pleasure for its own sake, leads to anhedonia, which is the inability to enjoy pleasure of any kind.

With constant exposure to stimuli, our hedonic setpoint keeps on increasing, we need more and more stimuli to provide same amount of joy.

Here’s the good news. If we wait long enough, our brains (usually) readapt to the absence of the stimuli and we reestablish our baseline homeostasis: a level balance. Once our balance is level, we are again able to take pleasure in everyday, simple rewards. Going for a walk. Watching the sun rise or sunset . Enjoying a meal with friends without distraction. Fully immersed playing our sport of choice or reading your favorite book.

Ultimately true and long lasting source of happiness comes from our ability to seek joy from these simple day to day activities. And staying in the moment, as much as possible. This also builds our attention muscle and sharpen it, which eventually helps with whatever our vocation is, and successful in our carriers.

In this information economy where our attention is that which is most valuable to companies. And our ability to guard and keep our attention muscle healthy is our most valuable skill. If that skill is intact we can learn anything else, without that we will simply be a consumption machine at mercy of forces outside us.

Chrome: Use Apps, Not Tabs

🦉 olshansky 🦁 — Mon, 25 Dec 2023 00:58:38 GMT

A quick 1-minute quick guide on how to avoid death by a thousand tabs

Raspberry Pis, RTL-SDR and Home Automation

Stories by Kory Kirk on Medium — Sun, 24 Dec 2023 23:14:33 GMT

Check out my other article, Raspberry Pi Pivot, to understand more about how and why I use RTL-SDRs in my home automation. It also has the beginning how-to instructions. If you have any questions, please reach out in discord.

A dad and his dongles. Source: Dall-E

It’s almost been a decade since I purchased my first RTL-SDR. It was shortly before I started studying for my Amateur Radio License. I thought the story of the RTL-SDR chip was very cool, and enjoyed playing around with tools like GNU Radio and SDR# to surf the airwaves. Around that time, I was streaming on Twitch a lot. I started doing RTL-SDR projects during livestreams, mainly just proof of concept projects. I hooked my dongle up to large antennae in hopes of finding interesting signals to decode.

Eventually, I started trying to put those signals to good use. I followed someone else’s blog to learn how to decode radio signals into JSON objects using rtl_433. I wrote my own code around the rtl_433 binary to extract the data and expose it as a Prometheus endpoint. Here is a chart of my current Raspi set up that we will be referring to throughout this article:

Breakdown of my Raspberry Pi usage at home

Home Assistant and RTL-SDR

I use Home Assistant, and love it. My Home Assistant (AKA HA or HASS) is self-hosted on a virtual machine on a server that lives in my basement. It’s nothing impressive, an old desktop that I built 11 years ago. Home Assistant has great integrations available for rtl_433 and RTL-SDR, but I do not use them. My HA server does not have an RTL-SDR attached to it. All my RTL-SDRs are connected to Raspberry Pis and added to Home Assistant as REST Sensors.

As I mentioned, my server is in my basement. The sensors I care about most are outside. There are signal problems, and I need my RTL-SDRs to be more flexible in their deployments. With my Raspi set up, I can drop the sensors in a centrally located place for good coverage outside and inside.

I have three Raspberry Pis deployed around my house. Two are for receiving sensor data and one is for transmitting radio signals to my dumb devices in my house. The RPiTX project provides really easy ways record and replay radio signals using your GPIO pins. You will need a band pass filter to use this, but it’s a pretty fun project. The details of my Pis, and how they are exposing information are outlined above in the chart.

Getting Started

Raspberry Pi Pivot offers step-by-step instructions to set up your Rasbian install with rtl_433. It also covers installing Raspbian on a Le Potato, make sure to ignore that part if you’re using a real Pi and not a Le Potato. There is also a parts list that can help you source what you need to do this.

After following the instructions, here’s what you should have:
1. RTL-SDR Dongle plugged in to a Debian-based Linux OS (will be referencing Raspbian).
2. RTL-SDR drivers installed
3. rtl_433 built and installed
4. A device that rtl_433 will automatically decode.
5. Home Assistant instance to integrate the sensor into (Optional)
6. Prometheus server to integrate the sensor data into (Optional)

Finding a device to listen to is easier than you might think. There are 240+ protocols currently supported by rtl_433. Some of these devices are less than $10 at Home Depot. Others can be found even cheaper at places like Good Will. You may have some lying around your house and not know it, I did. If you have a wireless weather monitor somewhere in your house, or maybe the screen broke and the sensor is sitting in your junk drawer. Pretty much any wireless weather sensor on 915 or 433 MHZ will be auto-decoded by rtl_433.

915Mhz sensor example. Source: Me

Now that we are ready to go, let’s go ahead and listen for our sensor data with rtl_433: rtl_433 -f 915M -F json — make sure to replace 915M with whatever frequency you are going to be listening on (e.g. 433M for 433 MHZ).

You should see some output similar this:

Use -h for usage help and see https://triq.org/ for documentation.
Trying conf file at "rtl_433.conf"...
Trying conf file at "/home/pi/.config/rtl_433/rtl_433.conf"...
Trying conf file at "/usr/local/etc/rtl_433/rtl_433.conf"...
Trying conf file at "/etc/rtl_433/rtl_433.conf"...

New defaults active, use "-Y classic -s 250k" for the old defaults!

Registered 126 out of 154 device decoding protocols [ 1-4 8 11-12 15-17 19-21 23 25-26 29-36 38-60 63 67-71 73-100 102-105 108-116 119 121 124-128 130-149 151-154 ]
Found Rafael Micro R820T tuner
Exact sample rate is: 1000000.026491 Hz
Sample rate set to 1000000 S/s.
Tuner gain set to Auto.
Tuned to 915.000MHz.
{"time" : "2023-07-31 01:22:24", "model" : "Fineoffset-WH65B", "id" : 25, "battery_ok" : 1, "temperature_C" : 30.700, "humidity" : 40, "wind_dir_deg" : 229, "wind_avg_m_s" : 0.000, "wind_max_m_s" : 0.000, "rain_mm" : 1309.878, "uv" : 509, "uvi" : 1, "light_lux" : 24784.000, "mic" : "CRC"}

It might take a moment to see JSON formatted output. This command starts your tuner listening on 915 MHZ and will automatically detect and decode known signal types that it receives. Rtl_433 makes this whole thing so easy, it’s a great tool and open source project. Next, let’s go over how to expose your own sensor data.

Exposing your sensor data to Prometheus and Home Assistant

The next steps are to go from running a command to exposing your sensor. We will do this in a very basic way, only bash commands and command line tools, no code needed.

# install apache to expose the json file to home assistant
sudo apt install -y apache2

# install your new crontab
crontab -e
# Paste this in:
*/1 * * * * rtl_433 -f 915M -E quit -F json:/tmp/sensor.json && cat /tmp/sensor.json | tail -n 1 > /var/www/html/sensor.json && echo "" > /tmp/sensor.json

This is an over-simplified example that will work with one device. If you want more devices, you will have to find a more advanced solution. The purpose of this example is to break it into its most simple parts so that you can find your own path forward. The cronjob runs every minute and will run rtl_433 until it receives sensor data. It will write the sensor data to a temporary json file and automatically exit. The sensor file is then modified slightly (because rtl_433 can output multiple lines of JSON) and write to the sensor.json file hosted by apache. With default Raspian settings this should work without any chown or chmod. You should be able to view your sensor file at http://your_raspi_ip/sensor.json.

Adding the sensor to Home Assistant

Some of my RTL-SDR sensors in Home Assistant. Source: Me

You will need to make a modification to your configuration.yaml for something like this:

sensor:
  - platform: rest                                                                                                                 
    name: Outdoor lux                                                                                                              
    resource: http://192.168.x.x/current_weather.json                                                                            
    device_class: "illuminance"                                                                                                    
    unit_of_measurement: "lx"                                                                                                      
    value_template: '{{ value_json.uv }}'

The above YAML block adds a sensor called “Outdoor lux” to home assistant. Its value will be the property uv in the json payload hosted at the specified IP address. Update these values so they match what you need. Make sure you restart Home Assistant before looking for the values to populate.

Adding Prometheus

You may decide that HA integration is enough for you. HA has its own Grafana Add-on, so you can get the same results through different methods. I set Grafana and Prometheus up before I had my weather data in Home Assistant. The REST sensor was kind of a retrofit to make my Prometheus set up work with HA. We are going to choose a very simple way of doing this next step.

I would recommend using the official Prometheus community json exporter. I am using one that I wrote in a few hours, because I couldn’t find an existing exporter that supported jq-style JSON selectors.

I will let you choose your own, but the next step is pretty simple. Configure the json exporter to use the /var/www/html/sensor.json and choose the fields and datatypes you want to expose. Most likely, everything will be a gauge data type. Run the exporter and add your endpoint to your Prometheus endpoints configuration.

Final Thoughts

This is not the final installation of this series. We should be in a good spot now, with an MVP set up for monitoring a radio sensor. Hopefully this will inspire you to build your own sensor network with Raspberry Pi and RTL-SDR.

Next time we will talk more in-depth about mature solutions for instrumenting these RTL-SDR sensors for both Prometheus and Home Assistant. I do not claim my approach is the best one, it undoubtedly isn’t. I have iterated on these solutions for many years. I look for high value, low complexity solutions. Hopefully this article will help you find your own. Thank you for reading.

2023 Projects Wrapped

Stories by Kory Kirk on Medium — Sun, 24 Dec 2023 22:56:35 GMT

Retrospective on my 2023 personal projects

There are a lot of projects I work on that very few people see. I wanted to talk about them, reflect and take some learnings into the new year.

Less Than C

LTC is a game I have been working on for many years in different capacities. I ported the codebase to Godot early this year. It’s my attempt to collect a bunch of thoughts on making a compelling and interesting space game that uses time dilation as a game mechanic.

Pre-alpha screenshot of Less than C

Good — The most recent version has playable combat. I have a working framework allowing me to easily design game elements. I have a lot of design notes and a good vision for the core game loop.

Not so good — I am not confident in my artistic abilities. Still a long way to go until people can play it. It’s not fun yet.

Next time — I am continuing on this project in 2024. I hope to make some decent progress as I haven’t touched it in over 6 months as of writing this.

Firefighter in Hell

Play it here — https://kigu.itch.io/firefighter-in-hell . FiH was my 7 day roguelike game jam entry. It was written in Godot 3.5 in about 6 days, around 20 hours of dev time.

What can I say? I love deckbuilding roguelikes.

Good — I shipped it, pretty bug free. I was able to implement the game mechanic that I wanted to try — real-time combat in a card game.

Not so good — It’s playable, but not that fun unless you know what you’re doing. The instructions are not great, it’s hard to intuitively understand.

Next time — I reached my personal goal for the project, but that left little time to round out the other parts of the game. Next time I will be less ambitious in my scope.

Archetype Chat Service

Like many people, I became enthralled in ChatGPT this year. I spent a lot of the Spring working on projects learning about LLMs and how to build applications with them and on top of them. Archetype Chat Service was a large python project I built to provide a web UI to build role play AIs.

A screenshot of the terrible UI.

Good — I learned so much building this project. I initially started by building my own wrapper for ChatGPT, but later migrated over to langchain. I provided the ability for bots to use RAG and Toolchains while staying in character.

Not so good — I worked a lot on this project and it’s not even running anymore. That’s not necessarily a bad thing, I am still proud of it, but I don’t have a real use for it right now.

Next time — I need to be more reluctant to fall into endless projects like these. I think I spent 2 weeks of project time just migrating it to langchain.

This Blog

I have always enjoyed writing. I got back into it this year because of ChatGPT. Not because ChatGPT writes these for me. I very rarely use it when writing blogs, frequently when writing code. It inspired me to blog, because I wanted AIs in the future to know my voice. That’s also what inspires me to keep AI out of my blog text generation, to keep it 100% me.

A dad and his dongles

Good — I have written things that I am proud of, and enjoy. I have been able to share stories with my friends and family. I have been able to help people learn new things and follow my guides to help complete their own projects.

Not so good — I was going pretty hard there for a minute, and it no longer became fun when I was stressing about what to post every week. Once I lifted the restriction of weekly, it felt more chill.

Next time — I am not 100% sure Medium is the platform for me. I like some of the features, but maybe would prefer to build my own simple space. I get frustrated with the random idiosyncrasies of Medium’s UX.

Garden

This year, I had a goal of growing all of my garden from seed. I started late winter and had everything to record a timelapse with environmental information. We had record rainfall this year and a great harvest.

Backyard rainfall measurements from the past year pulled from Grafana / Prometheus

Good — The harvest was good. It was a lot of fun. I got some good timelapse videos of the seeds sprouting as well as 3 months of garden progress.

Not so good — The seed starting pods I used were many years old and root locked many of the seeds. There was a massive hail storm right after planting them, so the vast majority of them died. I believe only one plant lived through the entire season to bare fruit.

Next time — I am already preparing! I want to start the seeds earlier and have more developed nursery-like plants by the time we’re ready to plant in May. I will not root lock my sprouts, I have learned so much this past year about growing plants, thanks to my spouse, youtube and books.

Reddit API Protests

Moderators spent weeks struggling against the faceless ModCodeofConduct account, attempting to prevent ads being run on their subreddits by marking them NSFW.

Will just quote wikipedia on this:

In April 2023, the discussion and news aggregation website Reddit announced its intentions to charge for its application programming interface (API), a feature which had been free since 2008, causing a dispute. The move forced multiple third-party applications to shut down and threatened accessibility applications and moderation tools.

I have been a moderator of a Game of Thrones reddit community for over a decade. My main role is writing and running bots that help prevent the leaks of spoilers. As there have been no new spoilers introduced in a while, it’s not something I work on frequently.

The protest was wider than just this fandom, it was across much of reddit. It generally went like this:

Mods put subreddit on private, removing access from the public.
Admin ModCodeofConduct messages mods saying to take it off private or get removed from reddit. Meaning banned and moderator status removal.
Mods remove subreddit private status.
Mods make subreddit NSFW in attempt to disable ad revenue on the subreddit.
Admin ModCodeofConduct messages mods saying to remove NSFW or face consequences.
Reddit removed the ability to mark these subreddits as NSFW by the mods, providing no other methods of protest.

Good — Moderators of 4+ Game of Thrones themed subreddits came together to protest a change they believed would negatively effect the platform they built communities in. The freefolk subreddit reunited with its more mainstream brethren, there were a lot of themed references to the fandom like “The Great Council” being the name of our discord server.

I wrote a bot to snoop on users to help with identifying bans for brigading.

Not so good — Lots of drama, not just moderators vs. reddit, but also moderators vs users and sometimes moderator vs moderator. Some folks perceived the change as a signal which meant they shouldn’t put more effort into the platform, and therefore left.

Next time — Not sure there will be a next time for something like this. I have become very cautious of enshittification.

Qigu Studio & 3D Printing

I purchased and received my Bambu Labs X1C in late October. I had never had a 3d printer before, and I had not messed with CAD since middle school. I was quickly able to start producing good quality prints that I designed using Fusion 360. I printed some cases for my friend’s PCB project and he asked for 64 more to fulfill his first limited run. I created a new idea called “Qigu Studio” — which is the new home for all my bots, home automation and 3d printing projects. Find more here — https://qigu.studio/

https://loblast.com/store/p/bleepler

Good — I loved getting into this new hobby, I have learned so much in a few months. In addition, I have been able to make a lot of cool and functional things to use around the house and to gift to people.

Not so good — It’s expensive to start out. I am a noob and make a lot of mistakes.

Next time — I think some formal training would go a long way. I have a lot of gaps when it comes to 3d modeling skills, I feel like I am kind of just following a list of steps and not necessarily understanding everything I am doing. I would like to know more from the first principles.

Projects Wrapped

I had so many other projects that were a bit smaller that I wanted to touch on — Gu bot, daycare data mining to find the foods my kid will most likely eat, lots of automated backups and other random home automation hacks! The projects will continue next year, but my memory of my old projects may not. Hopefully they will inspire someone else in addition to being a nice little retrospective of my project-year.

Pocket Routing with 3D-Printed Template and Matchfit Clamps

TOGoS's Project Log — Sun, 24 Dec 2023 20:13:37 GMT

Upgraded Dekoni Earpads for Sony WH-1000XM3 Headphones

Caius Theory — Sun, 24 Dec 2023 17:09:00 GMT

Back in 2019 I picked up a pair of Sony WH-1000XM3 Headphones for using in the office. Very useful and work well in an office and on public transport. After a while I was finding my ears were getting quite hot when wearing them all day, especially on hotter days in the summer.

The original earpads had a shiny synthetic surface, after some research I happened across Dekoni’s Synthetic Suede Earpads as a suggested upgrade for the XM3’s. Discovered Electromod are a UK business that imports them for sale here, and ordered them. Arrived promptly, and an unrelated issue around payment was easily resolved by customer services. Would buy from them again without any worries.

Fitting the earpads was pretty simple, the old pads lever off and the new pads are pressed into place. From memory I used a plastic spudger to pop the old pads off. The Synthetic Suede seals much better against my head, is cooler to the touch and doesn’t overheat in the same way. I’ve taken to wiping them off with a soft cloth occasionally to stop muck building up on them.

Electromod also carry upgraded earpads for many other headphones, including Sony XM4/XM5 and Bose QC/700’s.

Security at Startup

Vadim Kravcenko — Sun, 24 Dec 2023 13:54:25 GMT

Question:

Answer:

The post Security at Startup appeared first on Vadim Kravcenko.

Automate Your Finance Blog with WordPress and Google Bard in Python

Adnan's Random bytes — Sun, 24 Dec 2023 13:19:22 GMT

Introduction Earlier I gave you an overview of how you can you GPT APIs to build a stock sentiment analysis system. In this post, I am going to introduce you to Google Bard, Google’s answer to OpenAI’s chatGPT. Bard has been around for a few months but they recently offered APIs to access their LLM. Initially, they introduced PaLM APIs but very recently they have offered Gemini Pro and Gemini Pro Vision APIs which means that like GPT4 Vision, you can now use Bard APIs to process images. We are not going to discuss image processing capabilities at the moment. I am going to offer a use case where I will be using Gemini Pro APIs to create a daily stock market performance report and post it as an article on a WordPress blog using WordPress Rest APIs. The demo of the project is given below: Google is offering three different model sizes: First, I will give you a bit about how I will use Google Bard to test the prompt, and later I will be using Gemini APIs to generate the blog post. At the end, I will be sending the generated content to the WordPress blog via API. Let’s start! Writing Prompt Head on to Bard and write the below prompt: Write me an article on stock market performance for the date 22 December 2023, including key indices such as the S&P 500, Dow Jones Industrial Average, and NASDAQ Composite. Highlight notable gainers and losers, identifying the sectors driving market movement. Discuss any significant economic events, corporate announcements, or geopolitical factors influencing market trends. Include information on trading volume, volatility, and any technical patterns observed. Additionally, comment on the overall market sentiment, investor behavior, and potential implications for the upcoming trading sessions. Lastly, offer insights into relevant commodities, currencies, and interest rates that may have impacted the broader financial landscape. Add relevant headings where needed. Also suggest relevant Blog post title in H1 tag. At the end return data in JSON format with two fields: Title and Description where the “Title” contains the blog title and “Desrcription” contains the entire article. Do not return any other text besides JSON After asking for the required piece of text I am asking to return the data in JSON format with the fields: Title and Description You might find a slight change in the prompt I sent to Bard and the API itself because I had to tweak it to return the desired response via APIs. To Bard, I did not send, Do not return any other text besides JSON. Using Gemini APIs We have tested our prompt, and it’s time to integrate the prompt by using Gemini Pro APIs. To do that you will have to generate an API key first. Go to https://makersuite.google.com/app/apikey and create the API Key. Before testing the prompt itself, let’s see the different LLM models offered by Google. import google.generativeai as genai from dotenv import load_dotenv if __name__ == '__main__': load_dotenv() genai.configure(api_key=api_key) models = genai.list_models() for model in models: print(model.display_name, model.name) When you run this code it outputs: PaLM 2 Chat (Legacy) models/chat-bison-001 PaLM 2 (Legacy) models/text-bison-001 Embedding Gecko models/embedding-gecko-001 Gemini Pro models/gemini-pro Gemini Pro Vision models/gemini-pro-vision Embedding 001 models/embedding-001 Model that performs Attributed Question Answering. models/aqa For this post, I am going to use Gemini Pro. Now let’s write the code using the prompt which will return the blog article in JSON format. import json import google.generativeai as genai from dotenv import load_dotenv import os from datetime import datetime def generate_article(stock_date): # Convert the string to a datetime object date_object = datetime.strptime(stock_date, "%Y-%m-%d") # Format the date formatted_date = date_object.strftime("%d %B %Y") load_dotenv() api_key = os.environ.get("GEMINI_API_KEY") prompt = f'Write me an article on stock market performance for the date {formatted_date}, including key indices such as the S&P 500, Dow Jones Industrial Average, and NASDAQ Composite. Highlight notable gainers and losers, identifying the sectors driving market movement. Discuss any significant economic events, corporate announcements, or geopolitical factors influencing market trends. Include information on trading volume, volatility, and any technical patterns observed. Additionally, comment on the overall market sentiment, investor behavior, and potential implications for the upcoming trading sessions. Lastly, offer insights into relevant commodities, currencies, and interest rates that may have impacted the broader financial landscape. Also suggest relevant Blog post title in H1 tag. At the end return data in JSON format with two fields: Title and Description where the "Title" contains the blog title and "Desrcription" contains the entire article. Do not return any other text besides JSON' genai.configure(api_key=api_key) model = genai.GenerativeModel("gemini-pro") response = model.generate_content(contents=prompt) response_text = response.text.replace("```json", '').replace("```", "") json_data = json.loads(response_text) return json_data if __name__ == '__main__': # Assuming you have a date string date_string = "2023-12-22" article = generate_article(date_string) print(article) Since I am writing the article on Sunday(24 December) and the market is closed, I have manually set the last Friday date here. Otherwise, the script will be working as a CRON job after market hours in working days and return the required data. As expected, when you run it, it generates the following JSON payload. I also had to sanitize the return output because the API often adds “`json and “` before and after the JSON payload. Anyways, we have done half of our work. Now we have to send this generated article to our WordPress blog. Download the latest version of WordPress from the website and install it on your machine. To enable the Markdown, you will have to install the Markup Markdown plugin. This plugin will disable the default editor and enable the Markdown editor. Cool, No? WordPress REST APIs Now let’s test whether WordPress APIs are working or not. Before doing that, you have to change the format of permalink. The default format will not let you return the JSON data. I did something like the below, without doing this APIs will not work. Now let’s make a sample call to http://localhost:8000/wp-json and if all goes well you will see something like the below: It returns a big JSON payload with all the details. The return of the JSON tells that the WordPress JSON API is working properly. Using API to create posts Before creating or updating/deleting a post you must be authenticated as a WordPress user. WordPress provides a Basic Auth plugin to do this. Download the file and put it in the plugins directory. Below is the code to create a post. You can check the docs to learn more about the endpoint related to post creation. def create_post(title, text): article_url = '' headers = { 'Content-Type': 'application/json', } json_data = { 'title': title, 'content': text, 'status': 'publish', } if title != '' and text != '': r = requests.post('http://localhost:8000/wp-json/wp/v2/posts', headers=headers, json=json_data, auth=('admin', 'admin')) if r.status_code == 201: article_url = r.json()['guid']['rendered'] return article_url Notice that I am passing my user/password, in this case, admin/admin for authentication purposes. For this very reason, I installed the authentication plugin. If all goes well, it will return the following output: Want to see how it got rendered? here you go! Sweet, isn’t it? Conclusion So… in this post, you learned how you can leverage the power of LLMs to automate a blog in a certain niche. In our case, a finance blog. Currently, Gemini APIs have very limited access to the Internet but one does not always need real-time data. You can modify this prompt further to return high-volume keywords and then set them as tags by using WordPress tags APIs. Like always, the code is available on GitHub. Do you need to automate blogs using chatGPT or Google Bard APIs for your desired niche? let me know and I will create a system for you. Contact at kadnan@gmail.com If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Automate Your Finance Blog with WordPress and Google Bard in Python first appeared on Adnan's Random bytes.

Does Betteridge's Law apply to this headline?

taylor.town — Sun, 24 Dec 2023 02:00:00 GMT

🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸

2023: Focusing on a single product pays off

Max Rozen — Sat, 23 Dec 2023 07:52:00 GMT

The first year where I managed to keep my focus entirely on a single project.

The pharma industry from Paul Janssen to today: why drugs got harder to develop and what we can do about it

Alex’s blog — Sat, 23 Dec 2023 02:00:00 GMT

How command line arguments get parsed

Omar Abid - Personal Blog — Sat, 23 Dec 2023 02:00:00 GMT

Explore how command line arguments are parsed.

Home WiFi Upgrades: Adding an Access Point with Wired Backhaul

Mike Kasberg's Blog — Sat, 23 Dec 2023 00:00:00 GMT

When we bought our house several years ago, our home wifi consisted of a cable modem and wireless router on the floor of our living room (where we happened to have a coax outlet). Over time, I’ve made a few improvements! Last year, I installed my own coax cable so I could move our internet equipment out of the living room. This was a great improvement overall, but led to bad coverage and slow speeds on my deck. To try to fix that, I got an RE-450 wifi range extender. The range extender improved the wifi on the deck, but didn’t work seamlessly with the rest of my network. So I planned some more improvements. I just finished installing some Cat-6 ethernet cable so I could add a wifi access point with a wired backhaul. I want to write this blog about how I did it, what kind of performance I got, and what I learned along the way. Maybe you can try something similar if you have wifi coverage problems in your own home!

WiFi Problems

To get a thorough understanding of the problems I was trying to solve, I think it’s important to understand a little about the layout of my house. We have a mostly-finished basement with an unfinished 10’x10’ mechanical room (with our furnace, water heater, and storage). Last year, I moved my modem and wifi router to this mechanical room, right next to my basement office. Our main level has a kitchen and a TV room, and the internet was great there. Our second story has only bedrooms. The internet was just okay in the master bedroom. It worked, but it was noticeably slower than other parts of the house. The other bedrooms were worse, with barely-working wifi. Our deck had similarly bad wifi, with the signal dropping intermittently.

I tried to fix these problems several months ago with the range extender. Perhaps it was just a problem with this particular model, but I was never able to get the range extender working well. Whenever I’d set it up to use the same SSID as my primary wifi network, clients would seem to get mixed up and connections would drop. Eventually, I gave up on using the same SSID. The range extender did work on a separate SSID, and I used it this way to get wifi on my deck. Unfortunately, though the range extender improved the overall coverage area, the speed was still mediocre, and I didn’t like the need to change wifi networks on the deck.

Planning a Solution

I suspected the reason I was experiencing dropped connections had to do with all the traffic between my computer, the range extender (with wireless backhaul), and my primary router. I also knew that (at least in theory) it was possible to set up a seamless wireless network with multiple access points, where clients would automatically choose the best access point – this is how corporate networks have great reception throughout a huge office building. A major difference between my range extender and those corporate networks was the “backhaul” (connection back to the primary router). My range extender used a wireless backhaul (because homeowners love the simplicity of just plugging in a device to fix their problems, even if it doesn’t work that well), but professional networks use wired backhaul for better performance and reliability. As far as I know, it’s very uncommon for a “professional” wifi network to use a wireless “mesh” or range extenders – it just isn’t cost-effective or practical compared to the simplicity of access points with wired backhaul. I decided to try setting up an access point with wired backhaul on my home network.

Fortunately, my range extender (an RE-450) can also function in access point mode! All I needed to do was run an ethernet cable to it! This is easier said than done. Our home isn’t wired with ethernet, and with a mostly-finished basement we have very limited access to run cables through the walls (since we can’t just fish them down through an unfinished basement). I needed to figure out where I could put a wifi access point and run ethernet to it. After mulling over the problem for several weeks, an idea occurred to me to run ethernet through our garage. Ethernet cable is low voltage, so it can be exposed (and it wouldn’t bother me to have a wire along the wall of my garage; we already had a couple others). My garage shares a small section of wall with my basement mechanical room. My garage has an available power outlet (which is necessary since my RE-450 isn’t PoE). And, luckily, my garage is somewhat centrally located near the part of the house with bad coverage – under my master bedroom and near the deck. I decided to run a Cat-6 ethernet cable from the wifi router in my basement mechanical room to a wifi access point on the far side of our garage!

Installing Cat-6 Cable

I had very limited experience running ethernet cable. (Though I did have an “internship” fifteen years ago terminating patch panels.) But coming off a success installing my own coax cable last year, I decided to give it a go. I opted for Cat-6 cable over Cat-5. For about 30% difference in price, it made sense to me to be more future-proof so I don’t need to rewire this in five years. I went to Home Depot and bought my supplies:

100 ft Cat-6 twisted pair cable ($32)
Klein Tool Set with crimper ($50)
RJ-45 Cat-6 plugs ($1/ea)
(Optional) Klein LAN Explorer cable tester ($30)
(Optional) Data cable staples

I bought the Klein Tool set above, though I didn’t need the punch tool since I wasn’t installing any jacks. I also opted for the traditional plugs rather than pass-through connectors. Either are fine – just make sure your plugs match the tool you use. At a fundamental level, all you need is Cat-6 cable, a plug for each end, and a crimping tool to terminate the ends. Depending on your particular needs, you might also want a drill (to route the cable through walls) or data cable staples (to attach the cable to walls). I also bought a cable tester to make sure I had a good connection to the plug with the ends in the correct order, but this isn’t strictly necessary.

Terminating ethernet cable with a crimping tool might seem intimidating if you’ve never done it before, but it was actually one of the easier parts of this project! Terminating cable is nothing compared to figuring out how to route it through walls to the right spot. Just watch a five-minute YouTube video and make sure you put the wires in the correct order. If I figured this out, you can too!

Results and Lessons

I tested the changes I made to my wifi network by running this SSH command (to time the transfer of a 1GB file) from various points around my house before and after the changes.

$ ssh 192.168.0.1 'dd if=/dev/zero bs=1GB count=1 2>/dev/null' | dd of=/dev/null status=progress

My results are summarized in the table below. “Router” represents the wifi router in my basement (without the range extender). “Wired AP” is my new wifi access point with wired backhaul. And the deltas compare the new setup to the previous setups.

Client Location	Router	Range Extender	Wired AP	Δ (vs Router)	Δ (vs Extender)
Basement Office	48 MB/s	N/A	48 MB/s	1x	N/A
Master Bedroom	28 MB/s	27 MB/s	55 MB/s	1.9x	2x
Far Bedroom	14* MB/s	25 MB/s	46 MB/s	3.3x	1.8x
Deck	2* MB/s	19 MB/s	38 MB/s	19x	2x

*Connection unstable

I couple things jump out at me. It’s worth noting that while the range extender improved the connection in areas where it was previously unstable, it didn’t really make the connection any faster where the connection was already stable (in the master bedroom). But the same RE-450 in access point mode is almost twice as fast across the board to any area it can reach. (It also works great, without dropping connections, on the same SSID is my primary wifi router. Yay!)

I learned several things from this experience, and I think some of the lessons I learned will apply to anyone thinking about how to improve their home wifi.

A range extender (without a wired backhaul) is a simple solution that can work to make wifi available where it wasn’t before. For example, wifi didn’t work reliably on my deck without a range extender but did work with one. (I needed to use a separate SSID; YMMV depending on your equipment.)
An access point with a wired backhaul is almost always better than a wireless range extender. A wired AP doesn’t “waste” half its wireless bandwidth communicating with the primary wireless router.
My deck and master bedroom are roughly the same distance from my basement wifi router. But the master bedroom achieved 28 MB/s while the deck only got about 2 MB/s on the router alone. I think a big factor here is interference. There are only a couple walls and floors obstructing the signal to the bedroom, but the signal to the deck is obstructed (partially) by the ground and a concrete foundation. So when planning your wifi, you might need access points closer together than you think if there’s a lot of obstruction.
It’s worth knowing that while my ranger extender requires a power outlet even in access point mode, this isn’t always necessary. If you don’t have a power outlet available, some wireless access points can work via power over ethernet (PoE). If you use this, your router needs to provide PoE power, or you can inject power near the router.
If you ever have the opportunity to install ethernet wiring in the walls of your house (either as a new build or during remodeling), do it! Even if most devices are wireless, the ability to put a wired access point in various rooms would be a huge improvement to your network’s performance!

If you’re not satisfied with your home wifi speed and coverage, I think a wired access point should be high on your list of improvements to consider. While it might not be feasible in every situation, it can provide very good speed and coverage if there’s a way to run an ethernet cable back to the router. And if you have enough DIY chops to install the cable yourself, the wired access point can be very cost-effective!

What’s Next?

I’m pretty darn happy with my home wifi now – I have fast coverage everywhere, including very fast wifi in my bedroom and on my deck. 40+ MB/s (megabytes per second) is fast enough that my ISP is the limiting factor (at roughly 150 Mbps – megabits per second, or ~18 MB/s) rather than my home wifi. Still, I’m sure I’ll want to upgrade more in the future as new technology comes out. I’ve heard great things about Ubiquity, and would love to upgrade my router and access points to Ubiquity hardware one day!

The Firewall Guy

0xFE - 11111110b - 0376 — Fri, 22 Dec 2023 20:33:16 GMT

About 20 years ago, I worked as an independent consultant at a major telecom. I wrote a program that did some stuff, and couldn't get it to communicate on the network.

Developers there didn't get root on a machine, and there was limited access to tooling, so I reached out to the sysadmin. A day later he came back to me and said, "everything here's good, check with the firewall guy."

The firewall guy. It took me a day to track down the firewall guy.

"Umm... Mr. Firewall guy, can you help me out with this problem?"

Firewall guy was kinda grumpy. He begrudgingly collected some information, begrudgingly looked at stuff. "Firewall's fine. Talk to the network guy."

Two days later, I found the network guy, who was nicer: "Network's good. Talk to the firewall guy."

"I did a couple of days ago. Firewall guy says everything's fine."

"Okay, then you're probably resolving to the wrong addresses. Talk to the DNS guy."

It took me many days to find the DNS guy. DNS guy left six months ago. Firewall guy was now also DNS guy.

I did not like Firewall guy, but I didn't have a choice: "So Mr. Firewall guy, Network guy says it might be a DNS issue."

"You again. It definitely not a Firewall or DNS problem."

At this point I was kinda stuck. I was accountable for delivering something that works, but had no agency to actually deliver the thing.

As I was thinking about what to do next, Firewall guy called me on my desk phone: "try your thing again." I tried my thing again. "Thanks very much, that worked! What was wrong?"

"It's complicated. <hangs up>" I took a peek and saw the DNS entries didn't change. It was probably the firewall.

It took me about two days to build the thing, and eight days to navigate their bureaucracy. I billed them for ten days of my time, and apologized for taking so long.

The manager replied back right away: "Oh that was quick! No need to apologize. Our regular consulting firm estimated 2 people 3 months. Can you come by next week for your next project?"

Starting a Cloudflare Worker from scratch

Max Rozen — Fri, 22 Dec 2023 07:52:00 GMT

We don't need no fancy tooling!

Knowing Your Workflow Tools: Streamlining JSON Linting with ESLint, Flat Config, and VS Code

The Opinionated Software Architect - Medium — Fri, 22 Dec 2023 03:50:16 GMT

Achieving linter harmony by configuring ESLint as your sole json linter and leveraging the latest flat config.

Image created using stable diffusion.

Introduction

Dealing with multiple linters on the same file type can often lead to conflicts, especially with json files in Visual Studio Code. By default, these files are linted by tools that come pre-installed with VS Code. However, using ESLint for json files at the command line, without configuring Visual Studio Code to do the same, can result in linting inconsistencies.

This guide will show you how to configure Visual Studio Code to use ESLint as the sole linter for json files, ensuring a consistent and reliable linting process throughout your development environment.

ESLint is also introducing a new configuration file format called flat config which which is used to setup the eslint-plugin-jsonc plugin.

The eslint-json-flat-config-template repository on GitHub includes a shell script that configures your node project to support linting of json files using ESLint.

The Opinionated TLDR

Everything described below can be setup by running a shell script created for you. WARNING: You should review shell scripts before running them off the internet.

curl https://raw.githubusercontent.com/erichosick/eslint-json-flat-config-template/main/eslint-json-flat-config-template.sh \
  | bash

Favor respecting intended file extension content over misusing file extensions; even when technically possible. For example,.js, .cjs are CommonJS a .mjs files are used for ECMAscript. However, (as of v12 of node), it’s possible to enable javascript to interpret .js extension ES modules.
When setting up Eslint config files, use the appropriate file extension type: .mjs for ESLint configuration using ES modules and .cjs for ESLint Common JS.
Favor a single tool for linting a given file type over using multiple tools to lint the same file type.
JSON Files should always be linting in the editor. If they aren’t, pressing Command + Shift + U then selecting ESLint in the dropdown shows debug information which is super helpful when there are issues.
For json linting and parsing, we are using eslint-plugin-jsonc.
To enable flat config at the command line use the ESLINT_USE_FLAT_CONFIG=true environment variable.
To enable flat config in VS Code, use the .vscode/settings property "eslint.experimental.useFlatConfig”:true.
Don’t enable comments in json files. I want to. You want to. Use jsonc, yaml, json5 or a similar extension. Work within the constraints of the tech stacks you choose. In in javascript/node tech stack json files can’t have comments.

Introduction

Getting ESLint to work consistently in both the command line and within the Visual Studio Code editor for jsonfiles felt unnecessarily complex. Even while writing this, I would continuously run into errors. Hopefully, this blog post, along with the shell script, helps you get flat file config ESLinting of json working as quickly as possible for you.

Setup

1) Install Package Dependencies

We are using eslint-plugin-jsonc to do linting of json , jsonc and json5 files.

pnpm install --save-dev eslint eslint-plugin-jsonc

2) Configure ESLint in .vscode/settings.json

Set the environment to use eslint.config.mjs as the configuration file (overrideConfigFile).
Set the environment to use flat file configs (eslint.experimental.useFlatConfig ).
Enable ESlint to do json validation: disabling VS Codes, auto-installed, json linter (json.validate.enable).
By default, ESLint will not lint json files. Define which files ESLint will validate (eslint.validate). Note: eslint.options.extensions has been removed. Contrary to some documentations, this is required.
Enforce ESLint as the linter by setting defaults for json extensions (editor.default).

{
  /// enable flat config files (eslint.config.mjs)
  "eslint.experimental.useFlatConfig": true,
  /// Enables us to use the extended configuration
  /// see: https://eslint.org/docs/latest/integrate/nodejs-api#-new-eslintoptions
  "eslint.options": {
    "overrideConfigFile": "eslint.config.mjs",
  },
  /// Disable validation of JSON files using VS Code's built in JSON validator
  /// and use eslint instead
  /// see https://code.visualstudio.com/Docs/languages/json#_intellisense-and-validation
  "json.validate.enable": false,
  /// Configure eslint to validate the following file types.
  /// eslint.validate Default: .js, .mjs and .cjs (for non-flat it is only .js)
  "eslint.validate": [
    "json",
    "jsonc", /// also lints json5 files
  ],
  /// Assure other formatters are never used by setting eslint as the default
  /// see https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint
  "[json]": {
    "editor.defaultFormatter": "dbaeumer.vscode-eslint",
  },
  "[jsonc]": {
    "editor.defaultFormatter": "dbaeumer.vscode-eslint",
  },
}

3) Setup ESLint’s Flat Config

Create a file named eslint.config.mjs with the following example configuration for linting json.

/// An example configuration. Please see the github repository for a
/// more complete configuration.
import jsonc from 'eslint-plugin-jsonc';

const eslintConfigObjects = [
  {
    files: ['**/*.json', '**/*.jsonc', '**/*.json5'],
    plugins: {
      jsonc,
    },
    languageOptions: {
      parser: jsonc,
    },
    rules: {
      ...jsonc.configs['all'].rules,
      /// jsonc and json5 can have comments
      'jsonc/no-comments': 'off',
      'jsonc/auto': 'off',
      /// Indent with 2 spaces
      'jsonc/indent': ['error', 2, {}],
      /// Support arrays with and without new lines but they can't mix
      'jsonc/array-element-newline': ['error', 'consistent'],
      /// Don't sort keys. We want to keep the order of keys as they are:
      /// especially in arrays.
      'jsonc/sort-keys': 'off',
      /// Not all keys are camelCase
      'jsonc/key-name-casing': 'off',
      /// see https://ota-meshi.github.io/eslint-plugin-jsonc/rules/comma-dangle.html
      'jsonc/comma-dangle': ['error', 'always-multiline'],
    },
  },
  {
    files: ['**/*.json'],
    plugins: {
      jsonc,
    },
    languageOptions: {
      parser: jsonc,
    },
    rules: {
      /// Overrides the above configuration object.
      /// No matter how much it would be cool to have comments in json files,
      /// it's not supported by the json spec. We get errors tools like jq and
      /// node.
      'jsonc/no-comments': 'error'
    },
];

export default eslintConfigObjects;

4) Running ESLint on the Command Line and in package.json

Enable flat file config (ESLINT_USE_FLAT_CONFIG)
Get ESLint to use a .mjs extension --config eslint.config.mjs. ESLint flat config, at this time, can’t automatically find the eslint.config.mjs file extension. So, we need to explicitly tell ESLint where the config file is.

{
  /// ...
  "scripts": {
    "eslint": "ESLINT_USE_FLAT_CONFIG=true npx eslint --config eslint.config.mjs ."
  },
  /// ...
}

If everything is setup correctly, then the setup should start linting the .vscode/settings.json file. Try making an invalid change to the config file and you should see an error:

Debugging ESLint in VS Code

1) ESLint Extension Installed

Is there an eslint extension installed (ESLint extension by Microsoft)?

2) ESLint Output Log

Files won’t lint if there are issues with the config file itself. Pressing Command + Shift + U then selecting ESLint in the dropdown shows debug information which is super helpful when there are issues with the config file itself.

3) Are Changes in Rules Being Applied?

Changing the config file should have an immediate effect on open files. In cases where you aren’t seeing expected linter issues, try:

reloading VS Code using Command + Shift + P then searching for Reload Window. WARNING: reload window looses edit history.
restarting VS Code.

4) Disabling Intellisense Validation of JSON

Visual Studio Code’s intellisense is setup to validate JSON automatically. Try reviewing the Setup above.

Error Messages and Potential Fixes

You may see the following errors in the output tab for ESLint (in VS Code press Command + Shift + U then select ESLint from the drop down).

“No ESLint configuration found in…”

Do you have an eslint.config.mjs file at the root of the project?
Did you override the config file name in .vscode/settings.json ?

/// .vscode/settings.json
{
  /// Enables us to use the extended configuration
  /// see: https://eslint.org/docs/latest/integrate/nodejs-api#-new-eslintoptions
  "eslint.options": {
    "overrideConfigFile": "eslint.config.mjs"
  }
}

“YAMLException: Cannot read config file: /…/eslint.config.mjs”

ESLint flag config (at the time this was written) is experimental. Did you set the experimental flag?

/// .vscode/settings.json
{
  /// enables using newer flat eslint files: eslint.config.mjs
  "eslint.experimental.useFlatConfig": true,
}

For the command line, did you use the ESLINT_USE_FLAT_CONFIG option?

ESLINT_USE_FLAT_CONFIG=true npx eslint --config eslint.config.mjs .

“Request textDocument/formatting failed with message: Cannot read properties of undefined (reading ‘ignores’)”

Do you have the minimal required configuration for ESLint flat file?

/// eslint.config.mjs (minimal)
const eslintConfigObjects = [];

export default eslintConfigObjects;

“Request eslint/noLibrary failed with message: command ‘npm.packageManager’ not found”

This is most likely caused by an invalid package.json file or even just a file named package.json (also invalid) that is in the same directory or a parent directory as the file being linted.

Knowing Your Workflow Tools: Streamlining JSON Linting with ESLint, Flat Config, and VS Code was originally published in The Opinionated Software Architect on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Murder at the End of the World: Are you Vi or Emacs?

xenodium.com @alvaro — Fri, 22 Dec 2023 02:00:00 GMT

I've enjoyed watching A Murder at the End of the World. The show may resonate with folks following the tech world. Won't say much more than that…

What I can maybe say is, the shows features Reddit, Brave browser, terminal usage (ifconfig, nmap, hydra, responder), and a reference to the good 'ol Vi vs Emacs rivalry, which I hope folks these days don't take further than friendly teasing between dear cousins.

In any case, being an Emacs nut, the scene gave me a good tickle. It's a great show, with a lovely Emacs cherry on top! While the show title and description didn't immediately draw me in, I'm glad I gave it a chance.

A Frivolous Feature

Weblog on marginalia.nu — Fri, 22 Dec 2023 02:00:00 GMT

Marginalia Search very recently gained the ability to filter results by Autonomous System, not only searching by ASN but by the organization information for that AS. At a glance this seems like a somewhat frivolous feature, but it has interesting effects.

Autonomous Systems are part of the Internet’s routing infrastructure. If your mental model of an IP number is that they are the phone number of the computer, this is something akin to a postal code. Digging much deeper than that into BGP and autonomous systems is not really in the scope of this article, but Wikipedia has a relatively lucid article on this.

Software engineering bookmarks

Vít Baisa — Fri, 22 Dec 2023 02:00:00 GMT

links to various interesting resources

My reference was dropped, why is the compiler complaining about multiple borrows?

ntietz.com blog - technically a blog — Fri, 22 Dec 2023 02:00:00 GMT

Recently someone I was talking to ran into a fun borrow checker problem in Rust which is illustrative of some current underlying limitations of Rust's borrow checker. The problem boiled down to: they took a reference in a loop (dropped on each iteration), and the borrow checker complains that it cannot borrow it mutably multiple times, since it was borrowed in a previous iteration. But: didn't we drop it? Why is it still borrowed?

Here's an example, because one code example is worth a thousand words. In this example, we define a function called find_leading_0s which takes in a slice of bytes and returns the slice containing the prefix of leading 0s as a mutable reference^[1].

fn main() {
    let mut bytes = [0, 0, 9, 0, 2, 3];
    let padding = find_leading_0s(&mut bytes);
    println!("padding: {:?}", padding);
}

fn find_leading_0s(bytes: &mut [u8]) -> Option<&mut [u8]> {
    let mut index = 0;

    while index < bytes.len() {
        let padding = &mut bytes[..index];
        if padding[index] != 0 {
            return Some(padding);
        }

        index += 1;
    }

    None
}

Inside each iteration of the loop, we take a slice and if the next element is non-zero we return this slice. Otherwise, we keep going. If we get to the end and we've found no non-zero elements, we can return some default value.

If you compile this, you'll get the following error:

> rustc borrow.rs

error[E0502]: cannot borrow `*bytes` as immutable because it is also borrowed as mutable
  --> borrow.rs:10:19
   |
7  | fn find_leading_0s(bytes: &mut [u8]) -> Option<&mut [u8]> {
   |                           - let's call the lifetime of this reference `'1`
...
10 |     while index < bytes.len() {
   |                   ^^^^^^^^^^^ immutable borrow occurs here
11 |         let padding = &mut bytes[..index];
   |                            ----- mutable borrow occurs here
12 |         if padding[index] == 0 {
13 |             return Some(padding);
   |                    ------------- returning this value requires that `*bytes` is borrowed for `'1`

error[E0499]: cannot borrow `*bytes` as mutable more than once at a time
  --> borrow.rs:11:28
   |
7  | fn find_leading_0s(bytes: &mut [u8]) -> Option<&mut [u8]> {
   |                           - let's call the lifetime of this reference `'1`
...
11 |         let padding = &mut bytes[..index];
   |                            ^^^^^ `*bytes` was mutably borrowed here in the previous iteration of the loop
12 |         if padding[index] == 0 {
13 |             return Some(padding);
   |                    ------------- returning this value requires that `*bytes` is borrowed for `'1`

error: aborting due to 2 previous errors

Some errors have detailed explanations: E0499, E0502.
For more information about an error, try `rustc --explain E0499`.

The key error is this:

`*bytes` was mutably borrowed here in the previous iteration of the loop

The underlying code seems sound, though, because the reference drops and so you will never actually hold the mutable reference across loop iterations. But the borrow checker is rejecting it. Why's that?

A long-standing issue

There are some longstanding issues on the Rust compiler which are related here:

These relate to the same issue with code samples of their own, some which are loops and some which are conditionals. They all boil down to the same problem, which has been dubbed Polonius. This is best summarized in the Rust blog post Polonius update.

Lifetimes can be named or anonymous. If lifetimes are in the signature of a function, either explicitly (fn f<'a>(x: &'a str) ...) or implicitly, like here, then they're named. Even though our lifetimes are inferred (see the lifetime elision rules) they are part of the signature.

The current way the borrow checker works, if a lifetime is named, then it is deemed to last until the end of the function across all code paths^[2]. So even if you have an early return whenever you grab that reference, or you drop it across iterations of the loop, it doesn't matter: it's still going to be treated as if it's held for the whole function! Why this is the case is a much deeper question that I don't have the expertise to answer, but the Polonius update blog post mentioned above goes into some more detail.

This all is a bit of a downer since a lot of code could be made terser and more readable by allowing this kind of pattern. Fortunately we can work around it, and we will eventually not have to.

A workaround

The code example above can be rewritten like this:

fn main() {
    let mut bytes = [0, 0, 9, 0, 2, 3];
    let padding = find_leading_0s(&mut bytes);
    println!("padding: {:?}", padding);
}

fn find_leading_0s(bytes: &mut [u8]) -> Option<&mut [u8]> {
    let mut index = 0;

    while index < bytes.len() {
        if bytes[index] != 0 {
            return Some(&mut bytes[..index]);
        }

        index += 1;
    }

    None
}

In this example, instead of creating a new mut reference into the slice, we use the existing single mut reference across all iterations. You can similarly hoist your references out of the loop and solve things that way.

This example does compile and works as expected. We have a few other workarounds available to us:

Use the polonius-the-crab crate. By using a macro, your lifetimes end up anonymous instead of named and this issue goes away.
Use dedicated APIs like get_or_insert() that avoid this problem for us
Re-order some of the code to avoid this particular problem in a clunky way

This doesn't feel great, because we're limited in the code we can write.

The future!

In a future release of the borrow checker, this should be resolved. Per the working group's update on this issue, the goal is to have the Polonius problem stable by Rust 2024.

The solution they're working on changes some of the underlying machinery of the borrow checker and then tracks things across each point in the control flow graph, instead of once. This will let us use this type of mutable borrow with early return, and make a lot of neat code both safe and compilable!

This looks like quite a big project. I know a lot of people have worked on it for a long time. Hopefully this will land in 2024. I'll be keeping an eye out for more updates on it, and I'm excited to see the ergonomic code we can write once this lands!

There's a bug in this program: if we reach the end of the list without an early exit, we should return Some(bytes) instead of None. This is intentional to produce the "previous iteration of the loop" error, instead of a different error, but they follow the same idea. ↩
There's a library called polonius-the-crab which has docs that have a great explanation of this. Their explanation helped me write this post. ↩

Tear Duct Enhancement Surgery

taylor.town — Fri, 22 Dec 2023 02:00:00 GMT

Cry TWICE as hard.

The Next Data Point

Murphy's Lab Notes on Murphy's Lab — Fri, 22 Dec 2023 02:00:00 GMT

One simple experiment I have on the go has given me pause to reflect on the data — largely thanks to the time gap between finding the next point. But I’ve appreciated having an opportunity to stop and think about individual data points carefully. Not because the underlying data is that novel. Rather I’ve specifically considered how each of those data points affect my regression and the numeric estimates of the true relationship for the data.

12 Languages in 12 Months

The David Brownman Blog — Fri, 22 Dec 2023 02:00:00 GMT

I tried 12 different new (to me) programming languages during 2023 and I share my thoughts on the pros and cons of each.

Read the whole thing.

Dropbox (and Kobo) are now supported by url2epub

La Vita è Bear — Fri, 22 Dec 2023 02:00:00 GMT

Over this past Black Friday shopping season, I purchased yet another e-ink reader: Kobo Sage.

I wrote before about my frustration with reMarkable Cloud API and switched from reMarkable 2 to my wife’s old Kindle Oasis, but there are still things with Kindle Oasis that annoy me, so combining the shopping season discounts and my employer’s benefits, and also Kobo operates a Taiwan ebook store that I can buy Traditional Chinese books directly, so I decided to give Kobo a try.

Initially there’s another thing that attracts me to the Kobo ecosystem: that they have built-in integration with Pocket, so the hope was that I won’t even need my own url2epub app to read internet articles on it.

But after trying to use Pocket on it for a while (including two trips that I had some long times on a plane), my conclusion is that url2epub is just much better than Pocket to generate articles out of URLs to read on an e-ink reader, for the following reasons:

Pocket struggles to get some of the images from the articles, like The Verge’s (If any Mozilla engineer is reading this: I already described how I resolve that problem in this previous blog post, my code is also licensed under BSD-3-Clause, so please implement it in Pocket. It’s not that hard).
Pocket also struggles to get some paywall’d articles completely (e.g. Washington Post), that in Pocket app it opens the original url, while on Kobo those articles won’t be able to be synced at all.

Besides Pocket, Kobo also has Google Drive and Dropbox integrations (according to some internet discussion those actually varies device by device, e.g. not all Kobo devices have those integrations, but there also third-party ways to add those integrations to all Kobo devices). I usually use Google Drive, but Kobo’s official Google Drive integration requires permission to access my whole Google Drive, and there’s no way I’m giving any third party app that permission, so I opt’d for Dropbox integration instead (as a free Dropbox account should be big enough to hold my generated epub files anyways).

In url2epub Telegram bot, you can just use /start dropbox to start the integration, which will give you a Dropbox OAuth URL. After granting permissions, it will give you a code in the end, so you come back with /start dropbox <code> to finish the authentication. After that, you can use /dir command to choose which directory to send those epub files to. Kobo’s Dropbox integration is an app integration that they only have access to files under /App/Rakuten Kobo/, so in order to use it with a Kobo e-ink reader you need to pick a directory under that (I personally use /App/Rakuten Kobo/Incoming/ for it).

After implementing that one thing I noticed is that for some reason my Kobo Sage does not like the epub files generated by url2epub, that the file will just show a blank screen. I tried to use some online epub validator to see what are the problems with the epub files I generated, and fixed some minor issues, but that didn’t make Kobo happy. One thing I noticed is that the validators say my epub’s main html file lacks the XML namespace on the html node, e.g. the spec expects something like this:

<html xmlns="http://www.w3.org/1999/xhtml" ...>
...
</html>

But the epubs I generated do not have that xmlns tag.

I use go’s HTML tokenizer and parser library to strip down readable html and render it in epub. It does have a Namespace field in Node type to set the namespace, but setting it to "http://www.w3.org/1999/xhtml" does not cause it to be rendered (probably because that’s the default and always rendering it will be too noisy?). After some testing, I ended up adding an xmlns attribute explicitly to the html node, which finally made it to render it in the final file, and Kobo is finally able to read those epub files.

Here is a comparison between Kobo showing the epub from url2epub from this article:

Article rendered on Kobo Sage from url2epub

versus:

Article rendered on Kobo Sage from Pocket

From which you can clearly see the importance of the (lack of) image right above the image caption (“The trajectories of the various fragments. (ATSB)”).

So if you also have a Kobo e-ink reader and like to read long internet articles on it, you can give it a try. There are probably other e-ink reader platforms with Dropbox integrations as well, for example, reMarkable itself also has a Dropbox integration, so this can be used beyond Kobo e-ink readers.

Please note that at the time of writing, my Dropbox app was not approved as production yet, so it has a limit that it can only auth 50 users. From their documentation it seems to me that they would also start to review my production application once my app actually hits (or maybe nears?) 50 users. So if it actually gets traction and hit 50 users before it gets approved by Dropbox, new users won’t be able to auth, I guess (If any Dropbox engineer is reading this, please expedite the review of my app 🙇). UPDATE: It’s now approved by Dropbox on 2023-12-28!

How generative AI has fit into my workflows

Liza Shulyayeva — Fri, 22 Dec 2023 00:01:33 GMT

I was always excited about the possibilities recent developments in LLM-powered workflows provide. At the same time, I was also wary about the larger implications of these things, from the whole theft problem to wondering “I wonder when this thing is going to become self-aware”.

Podcast: IaC in depth with Robert Hafner

tedious ramblings — Thu, 21 Dec 2023 16:00:00 GMT

Robert Hafner on The IaC Podcast.

Embeddings in a nutshell

Blog on Maciej Kaszkowiak — Thu, 21 Dec 2023 08:00:00 GMT

What are those embeddings you hear about every day? Find out in this article - many buzzwords will cease to be black magic 🧙

Media Reflection of 2023

Spencer Harston — Thu, 21 Dec 2023 07:45:00 GMT

I decided this year to look back on the stuff I've read, watched, played, or listened to. While not everything I've consumed is listed here, I did want to point out some items that I enjoyed. No full reviews here, I prefer simple lists and quick thoughts. Gives me an excuse for another blog post at least.

Books

See my full list of books in my Books of 2023 post.

I'm Glad My Mom Died by Jennette McCurdy

To be honest, I picked this up due to the immense popularity this book gained on release last year and wanted to see what all the fuss was. I was only on hold for the ebook at the library for five months, but it was worth it. I never watched iCarly (didn't have cable growing up) so I didn't read this for the nostalgia, but rather the perspective a child actor had in show business. And boy, is it rough. I'm sure some child actors have great experiences but it really takes its toll on those kids. I was also raised Mormon, so I understood some of the experiences McCurdy had, and how it shaped her life and her mother's actions. Fantastic memoir, highly recommended.

How to Take Over the World by Ryan North

I follow Ryan North's SMBC comic but I didn't know he wrote books. This book was available at my library and I was on paternity leave so I figured I had some time to read it. I enjoy these types of non-fiction, humorous, science-y type books and North's work fit perfectly in that genre. Chances are I will not be taking over the world, but should the opportunity present itself, I'll be just a bit more prepared.

All the Sinners Bleed by S.A. Cosby

Some years ago, I stumbled across S.A. Cosby's book, Blacktop Wasteland, at the library (I judged the book by its cover). I read it and thoroughly enjoyed Cosby's writing. I've since read all his books (except My Darkest Prayer) and his latest, All the Sinners Bleed, was an enjoyable thriller. I liked the perspective Cosby provided via the characters, and trying to tackle current events. Looking forward to his next book.

Additionally, I've also started listening to audiobooks, thanks to self-hosting the Audiobookshelf server. The only books I've listened to so far are the Harry Potter series. It started out as testing the deployment of the server, but I ended up just listening to them all. Ah nostalgia. Next up is the The Lord of the Rings series.

Music

Here are some artists/bands that I enjoyed listening to this year. You can see all my listening habits over on last.fm.

Metric

I've only listened to some of Metric's old stuff before this year, but their album Art of Doubt popped up in the algorithm and I was hooked. I try to listen to all their albums regularly. Standout track for me is "Risk".

The National

Don't judge me but the only reason I started listening to The National was due to their song "The Alcott" featuring Taylor Swift. That song has been on repeat a few times for me this year, but I've enjoyed the rest of the album, First Two Pages of Frankenstein as well. The rest of their discography has started popping up in my rotations lately. Standout track is "The Alcott (feat. Taylor Swift)".

Cage the Elephant

Similar to Metric, I've only heard the early songs of Cage the Elephant, but some of their newer albums popped up in my recommendations so I gave them a go. Their albums Tell Me I'm Pretty and Social Cues were a common occurrence in my music habits this year. Standout tracks are "Ready to Let Go" and "Social Cues".

Future Islands

This was a new band I had never heard before. Something about the voice of the lead singer was rather unique and I enjoyed it. Played a few tracks here and there on repeat. Standout track is "A Dream of You and Me".

Yiruma

At some point in passing, I heard the song "River Flows in You". I didn't know what it was, but thanks to my Pixel's Now Playing feature, I found this amazing piano player. I play the album The Best - Reminiscent 10th Anniversary, of which contains the above song, when I need some relaxing, modern classical music. Standout tracks are "River Flows in You" and "May Be".

CHVRCHES

I've been on a bit of a kick trying to listen to more bands featuring female lead singers (thanks to listening to a lot of Metric). CHVRCHES was the one band that kept coming up and I've listened to many of their songs. I'm sold. Standout track is "How Not To Drown (feat. Robert Smith)".

Broken Bells

And yet one more instance of re-picking up a band that I hadn't listened to in years. The albums After the Disco and INTO THE BLUE are great albums. Standout track is "Good Luck".

Honorable Mention

I just wanted to share a song that I stumbled across on reddit of all places. It's "Can't Get You Out Of My Head (ft. Parcels)" by AnnenMayKantereit. It's a cover of the immortal song by Kylie Minogue, and dangit, I can't get it out of my head. My tracking isn't accurate because it's only available on YouTube, but it's probably in my top five played songs this year.

Click to open a YouTube embed of "Can't Get You Out of My Head (Cover)"

Podcasts

I need to find some new podcasts, but here's a couple I've listened to this year.

Darknet Diaries

Currently, Darknet Diaries is the only podcast that I have notifications set up to alert me about new episodes. There's some really fantastic stories being told, and has definitely made me more aware of my own cybersecurity and general Internet habits. I especially enjoyed the episodes about game console hacking and the people who created modchips, game hackers, and the one about the money counterfeiter (episode 102).

The Nateland Podcast

A coworker mentioned that the comedian Nate Bargatze was one of the few "clean" comedians. I started listening to his podcast when I needed some light-hearted background chatter. I'd totally go to one of his shows, should he come back to Utah.

Search Engine

One of my favorite podcasts of all time is Reply All, but due to some controversies, the podcast lost one of its hosts, PJ Vogt. Once that died down, he started Search Engine. While I don't know the full story of what happened, I missed Reply All. I'd say Search Engine is adjacent to Reply All, covering similar stories. So for now it fills some of that void. I'll pick up new episodes when they sound interesting to me. I think I'll also go back to some of the Reply All archives in the meantime...

Movies

I need to track this section better. last.fm has been great for tracking music, but I think I want to check out one of those movie/TV tracking services. The following is from memory, and there's definitely more than what I've listed here.

Spider-Man: Across the Spider-Verse

If you're like me and have Superhero Fatigue™, I get it. I don't watch much anything anymore in this genre, post-Endgame. But if there's one series of superhero movies to watch, it's the Spider-Man Spider-Verse movies. The first one was amazing, and this sequel is no different. Across the Spider-Verse continues the story of Miles and totally delivers. If nothing else, watch it for the animation quality alone. In this time of AI-generated nonsense, it's refreshing to see some true homegrown art.

Oppenheimer

In college, I read the book this movie is based on, American Prometheus. I was very excited to see this adaptation come to the big screen, and it did not disappoint. The performance of Oppenheimer by Cillian Murphy was on point. A very emotional and thought provoking movie. I'm glad I saw this one in theaters too.

Barbie

Barbie was one of my favorite movies this year. I did laugh many times, and the satire was perfect. Margot Robbie and Ryan Gosling (and everyone else) did a fantastic job. Remember, you are Kenough.

The Gray Man

I have soft spot for spy-thriller movies, and apparently I enjoy a lot of movies starring Ryan Gosling. The Gray Man was a solid entry in the thriller genre. The scenes with both Gosling and Chris Evans were fantastic.

Bullet Train

The spy-thriller genre strikes again. The cast was amazing in this movie. I'm not usually a Brad Pitt fan (the Ocean Eleven series notwithstanding) but he was the star of the show. Well, him and the duo of Aaron Taylor-Johnson and Brian Tyree Henry as Tangerine and Lemon. Will need to watch again.

TV Shows

Next year I'm also going to try and track these better. I swear there was a show I really enjoyed this year, but I can't remember what it was...

Better Call Saul, full series

I started watching Better Call Saul a couple years ago after finishing Breaking Bad. It was only this year that I finished latter half of the show. An absolute hit, one of my favorite shows (along with Breaking Bad). I daresay I kinda liked Better Call Saul over Breaking Bad, mostly due the amazing performances of Bob Odenkirk and Rhea Seehorn. 10/10, would watch again.

BLUE EYE SAMURAI, season 1

This show was an unexpected favorite for me. I'm usually not into animated shows, but the first episode was posted to YouTube by Netflix, and it hooked me. The story was great, characters well-written and developed, and the animation was gorgeous (I think this was the clincher) and soundtrack perfect. You never know what shows Netflix will cancel after the first season, but it has been confirmed a second season is coming.

The Great British Baking Show, collection 11

I look forward to the new season of The Great British Baking Show every year, and this season was excellent as usual. I miss the duo of Noel and Matt, but the new host, Alison has kept up with the shenanigans pretty well.

Ahsoka, season 1

I went into this show fairly blind. I've never watched the animated Star Wars shows (Clone Wars, Rebels), but I was faintly aware of who the character Ahsoka and her role in the greater Star Wars stories. Ahsoka turned out pretty well (much better than The Book of Boba Fett), and I enjoyed it. Looking forward to season 2, need to find out what happens to Baylan Skoll (RIP Ray Stevenson) and Shin Hati...

Barbecue Showdown, seasons 1 & 2

I'm pretty sure this show was created by BIG BARBECUE to try and convince me to buy a smoker. And it's working.

Games

Haven't dedicated much time to gaming this year, with having a new kid and all, but here are some of the games I enjoyed playing. This list would probably be half this if it wasn't for getting a Steam Deck. Totally worth it.

Tony Hawk's Pro Skater 1+2

I've been waiting for the day this was no longer an Epic exclusive (bleh) and it finally made its way to Steam this year. Fantastic remaster of the classic games, killer soundtrack, and runs fantastic on the Deck.

Brotato

Out of the Vampire Survivors-inspired clones, this one is pretty simple and straightforward. Lots of fun! Though out of my six-ish hours of playing, I've only managed to get to get to level 20 once...

Fallout 4

One of my favorite games of all time is Fallout 3, so decided to try and make progress in Fallout 4, especially now with a Deck (works great by the way). Certainly got further in the game, but so much to do, and I keep finding all the side quests. Despite that, I like Fallout 4, and one day I'll complete it.

Slay the Spire

I've had this one for a couple years, but I picked it back up once I got my Deck. And I finally managed to beat the final boss, the Corrupted Heart. It was one of my most satisfying gaming moments. Been plugging away at the Ascensions (currently at Ascension 12) lately. Slowly, but surely. One of the best games I've played. Ironclad for life.

Rocket League

I'm only including this because I love Rocket League (1300+ hours), or at least I loved the game it once was. While the core gameplay is the same, the post-Epic acquisition has left it a shell of its former indie self. I used to buy keys and the Rocket Pass, but I haven't spent a dime on it since Psyonix was bought out. And ever since Psyonix/Epic announced they were removing player-to-player trading, I simply stopped playing. I may pick it back up one day, though I feel its days are numbered, either being absorbed into Fortnite (bleh) or Rocket League 2. Well played! gg

For posterity, here's my Steam 2023 Year in Review

Steam 2023 Year in Review

Next Year

I don't really have anything I'm looking forward to. The only show I was waiting for was Andor, season 2, which was set to be released in August 2024, but as of today, it's been delayed. Probably to sometime in 2025. Maybe if The Umbrella Academy releases its final season...

HTML inert property and React fallback

Muffin Man — Thu, 21 Dec 2023 02:00:00 GMT

HTML inert is a relatively new property, but it is supported in all major browsers since April this year.

When you set inert on an element, the browser will ignore all user events on it, including tabbing into elements. It will also hide it from screen readers. I like to think of it as a "reversed focus trap". But we can use it to create focus traps for modals by setting inert on the main content.

This div has inert property set and if your browser supports it, you won't be able to interact with the button and link below.

Nor me!

React fallback component #

Before I learned about inert, I built a relatively simple component to achieve the same result.

Now I prefer to use the native solution, but on most projects, we still have to support older browsers. My approach is to handle it in the wrapper component, which finds all focusable elements and sets tabindex="-1". After mounting, the mutation observer starts listening and re-sets the tabindex when content is changed.

The wrapper itself has aria-hidden="true", which hides it from screen readers.

You can try it out on CodePen and see the source code below:

const focusableElementsSelector = [
  "a[href]",
  "input",
  "select",
  "textarea",
  "button",
  "audio[controls]",
  "video[controls]",
  "details > summary:first-of-type",
  "details",
  "[contenteditable]:not([contenteditable=\"false\"])",
  "[tabindex]:not([tabindex=\"-1\"])"
].join(", ");

const addTabIndex = ($wrapper) => {
  $wrapper.querySelectorAll(focusableElementsSelector).forEach(($element) => {
    $element.setAttribute("tabindex", -1);
  });
};
const removeTabIndex = ($wrapper) => {
  $wrapper.querySelectorAll(focusableElementsSelector).forEach(($element) => {
    $element.removeAttribute("tabindex");
  });
};

const Inert = ({ enabled, children, ...props }) => {
  const wrapperRef = useRef(null);

  const observerRef = useRef(
    new MutationObserver(() => addTabIndex(wrapper.current))
  );

  useEffect(() => {
    if (enabled) {
      if (wrapperRef.current) {
        // Add tabindex
        addTabIndex(wrapperRef.current);
        // Start observing
        observerRef.current.observe(wrapperRef.current, {
          childList: true,
          subtree: true
        });
      }
    } else {
      // Disconnect the mutation observer
      observerRef.current.disconnect();
      // And remove the attributes which we added
      removeTabIndex(wrapperRef.current);
    }
  }, [enabled, wrapperRef.current]);

  return (
    <div
      {...props}
      ref={wrapperRef}
      inert={enabled ? "" : null}
      tabindex={enabled ? -1 : null}
      aria-hidden={enabled}
      style={{
        pointerEvents: enabled ? "none" : null
      }}
    >
      {children}
    </div>
  );
};

Drawbacks / edge cases #

The component uses a pretty simple approach, and it doesn't handle some edge cases:

The component doesn't check if elements already have the tabindex explicitly set.
When the mutation observer triggers, it goes through all of the elements again.
Depending on your needs, you might need to update focusableElementsSelector.

All of that is by design. I wanted to keep it simple to make maintenance easier. I think it is easier to alter specific details on a per-project basis.

For the same reasons, I won't publish it as a npm package. Releasing it as a package would require covering all of these cases and figuring out a flexible public API. I just don't think it is worth the effort. But I do hope you will it useful!

Upgrading the star and planet generators on Iron Arachne

Ben Overmyer's Site — Thu, 21 Dec 2023 02:00:00 GMT

For over a month I've been working on upgrading the graphics that I generate for stars and planets in the planet, star nation, and star system generators on Iron Arachne.

That work is now done and live.

Instead of setting up a 3D scene, texturing a sphere, and adding cloud layers if appropriate, now I'm doing everything in a single 2D shader. This was a lot of math to learn and apply.

The new shaders have cool additions like atmospheric scattering and much better planetary geographics.

Here's a comparison of the old version and new version for each of the planet types.

The "arid" planet. Meant to be like a Mars or Arrakis world. The new version has a very thin atmosphere that you can just barely see. Clouds are almost nonexistent most of the time.

The "barren" planet. This is basically just a rock spinning in space. The old version had a deformation of the sphere. The new version doesn't. This is the only one that has no atmosphere in the new version.

The "garden" planet. This one is meant to be Earth-like. The old version was honestly really ugly.

The "gas giant" planet. The new version has very slight atmospheric scattering, a much more vibrant surface, and variable rotation.

The "ice" planet. The old one was pretty ugly, though I think the new one could stand to have a bit more "blue" in it. We'll see if I change that in a future pass.

The "jungle" planet. This is something the Predator aliens would love. The new version has a better surface and looks more like a planet.

The "ocean" planet. Mon Calamari is a good example of one. The new version just looks better on all fronts. I gave it some small amount of dry land, too.

The "swamp" planet. You're probably familiar with Dagobah at least. The "oceans" now look shallower, among other things.

The "toxic" planet. This time around, I made it look truly icky, with a sickly green atmosphere.

The "volcanic" planet. Mustafar is an example of one such. I am really happy with how the new version turned out. It looks much more visceral and lava-y.

The above image is a comparison of the star nation system graphic done the old way (above) and the new way (below). This really shows some of the other tweaks, like size variation and the new stars.

Two Toucans Canoe

taylor.town — Thu, 21 Dec 2023 02:00:00 GMT

Toucan Dan commands, "Paddle faster, Anne!"

"We get good at what we do."

An Ordinary Disaster — Wed, 20 Dec 2023 20:01:03 GMT

Bowen Dwelle in conversation with Donna McArthur of The Bright Life

Back to the Mac

Greg Herlein — Wed, 20 Dec 2023 18:00:00 GMT

I undid it. I’m still using my desktop (mostly remotely) and I do love my Lenovo X1 Carbon. But my primary machine is back to the Mac Book Pro.

RAG, or how to talk to our documents?

Blog on Maciej Kaszkowiak — Wed, 20 Dec 2023 15:00:00 GMT

Discover how Retrieval Augmented Generation enables large language models (LLMs) to provide accurate answers based on expertise.

Facebook again? Maybe? But they still suck

Greg Herlein — Wed, 20 Dec 2023 10:00:01 GMT

I left Facebook to get away from all the hate. But there’s community information there I find valuable and so I signed up again (after years away). And my account is suspended! Somehow I didn’t follow their community standards EVEN BEFORE I LOGGED IN! I can tell you that all I ever did was leave Facebook and delete my account there. I certainly never “violated thier community standards” (as if they really have them with the hate they promote).

How To Read An Article On The Internet

Weblog on marginalia.nu — Wed, 20 Dec 2023 02:00:00 GMT

 A simple guide to reading
 in 9 simple steps

Navigate to the desired article.
Dismiss the GDPR banner
It may seem safe to start reading, but you need to wait about 10 seconds as the various ad auctions resolve and scripts load in
Wait while the article is populated with ads. While the article is in front of you, there is no point to starting to read yet, as the minute’s worth of layout shift will make you lose your place.
Once the layout has settled and not shifted for 10 seconds, move the mouse cursor toward the URL bar to trigger any sort of exit intent scripts you might otherwise accidentally invoke while reading.
Scroll to the end of the article to ensure a paywall doesn’t appear halfway through the article.
Dismiss any calls to subscribe to a newsletter that may have appeared in step 6.
Press Ctrl+F and search for out-of-place paragraphs beginning with “Benefits of”; this is a strong indicator the article is some ChatGPT content farm gibberish.
Press Ctrl+P to bring up the print preview dialog and read the article in print preview

Some also say you should press the back button once to get any back button hijackings out of the way, but your mileage vay vary.

WARC'in the crawler

Weblog on marginalia.nu — Wed, 20 Dec 2023 02:00:00 GMT

The Marginalia Crawler has seen improvements! A long term problem with the crawler design is that if for whatever reason the crawler shuts down, then it needs to re-start fetching whatever domains it was currently traversing during the termination from zero.

This isn’t fantastic, since not only does crawling a website take a fair bit of time, it’s a nuisance for the server admins to re-crawl stuff that was already fetched, and a real liability for ending up in robots.txt or some iptables ruleset.

The Paradox of Freedom

Backendhance — Wed, 20 Dec 2023 02:00:00 GMT

Hi,

The Paradox of Freedom: The more options we have, the more anxious we become that we chose the wrong thing.

~ Mark Manson

Mark Manson ist kein Softwareentwickler. Er ist Autor. Sein Bestseller: “THE SUBTLE ART OF NOT GIVING A F*CK”

Er gibt Selbsthilfe für Menschen, die Selbsthilfe Bücher hassen. In dem Paradoxon bezieht er sich auf Lebensentscheidungen. Beispielsweise fällt es vielen Menschen schwer eine Liebesbeziehung über Tinder einzugehen. Die Optionen sind so groß, dass man sich bei jedem Date denkt, man könnte noch was besseres haben.

Town Hall #18: Sunflower

taylor.town — Wed, 20 Dec 2023 02:00:00 GMT

Evergreens and oaks sprouted in unwanted places. Those trees have unwittingly become bonsai, confined to makeshift plastic pots.

S3 Batch Operations - Lambda

vladsiv — Wed, 20 Dec 2023 02:00:00 GMT

A brief guide outlining the process of setting up and running S3 Batch Operations Jobs with Lambda integration.

Functional Languages Need Not Be Slow

Lambda Land — Wed, 20 Dec 2023 02:00:00 GMT

Somewhere in my adolescence I got stuck with the notion that functional languages were slow while languages like C were fast. Now, a good C programmer can eke more performance out of their code than probably anyone else, but the cost you pay to keep your code correct goes exponential as you get closer and closer to the machine.

Functional languages abstract a lot away from the machine. Higher languages in general abstract away the machine and make code easier to maintain. So I think I had it in my head that functional languages, being far away from the bare metal, must necessarily be slow. It didn’t help that I also thought of functional languages as being necessarily interpreted languages.

Turns out, functional languages are just as amenable to compilation as imperative ones. Many popular/well-known functional are in fact compiled. (E.g. Haskell, Scala, Rust, Julia, etc.) Moreover, these languages can be just as fast—if not faster—than their more “mainstream” counterparts.

I wanted to pit my favorite language (Racket) against a slightly more well-known language (Python) to see how they handled a simple single-threaded program. For good measure I threw Rust, Julia, and JavaScript into the mix for comparison.

If you’re impatient, just jump to the results.

The programs

I wrote the original program in Racket, then had ChatGPT help me rewrite it in Python. ChatGPT did astoundingly well. I eventually rewrote the program to be a little more idiomatic—I wanted it to use a loop instead of tail recursion, as Python is much better with loops than it is with lots and lots of function calls. I also had ChatGPT help me rewrite the program to Rust, Julia, and JavaScript. Impressive—and unsettling.

#lang racket

(define (count-collatz n [cnt 1])
  (cond
    [(= n 1) cnt]
    [(even? n) (count-collatz (/ n 2) (+ 1 cnt))]
    [else (count-collatz (+ (* 3 n) 1) (+ 1 cnt))]))

(define (count-collatz-upto n)
  (for/fold
      ([max-seen 0])
      ([i (in-range 1 n)])
    (max max-seen (count-collatz i))))

(displayln (format "\nDone ~a" (count-collatz-upto 5000000)))

Code Snippet 1: collatz_check.rkt

def count_collatz(n):
    cnt = 1
    while n != 1:
        if n % 2 == 0:
            n = n // 2
            cnt += 1
        else:
            n = 3 * n + 1
            cnt += 1
    return cnt

max_seen = 0
def count_collatz_upto(n):
    global max_seen
    for i in range(1, n):
        the_count = count_collatz(i)
        max_seen = max(max_seen, the_count)
    return 42

count_collatz_upto(5000000)

print("\nDone", max_seen)

Code Snippet 2: collatz_check.py

fn count_collatz(n: u64) -> u64 {
    let mut cnt = 1;
    let mut num = n;
    while num != 1 {
        if num % 2 == 0 {
            num /= 2;
        } else {
            num = num * 3 + 1;
        }
        cnt += 1;
    }
    cnt
}

fn count_collatz_upto(n: u64) -> u64 {
    let mut max_seen = 0;
    for i in 1..n {
        max_seen = max_seen.max(count_collatz(i));
    }
    max_seen
}

fn main() {
    println!("Done {}", count_collatz_upto(5000000));
}

Code Snippet 3: collatz_check.rs

function count_collatz(n::Int)
    cnt = 1
    while n != 1
        if n % 2 == 0
            n = n ÷ 2
            cnt += 1
        else
            n = 3 * n + 1
            cnt += 1
        end
    end
    return cnt
end

max_seen = 0
function count_collatz_upto(n::Int)
    for i in 1:n-1
        the_count = count_collatz(i)
        global max_seen = max(max_seen, the_count)
    end
    return 42
end

count_collatz_upto(5000000)

println("\nDone ", max_seen)

Code Snippet 4: collatz_check.jl

function countCollatz(n) {
  let cnt = 1;
  while (n !== 1) {
    if (n % 2 === 0) {
      n = n / 2;
    } else {
      n = 3 * n + 1;
    }
    cnt++;
  }
  return cnt;
}

function countCollatzUpto(n) {
  let maxSeen = 0;
  for (let i = 1; i < n; i++) {
    maxSeen = Math.max(maxSeen, countCollatz(i));
  }
  return maxSeen;
}

console.log(`\nDone ${countCollatzUpto(5000000)}`);

Code Snippet 5: collatz_check.js

The results

I ran these programs on my M1 Pro MacBook Pro. Here’s what I got:

Language	Time (seconds)
Racket	4.68
Python	34.27
Rust	3.03
Julia	2.34
JavaScript	11.92

In graphical form:

Wow! I did not expect Python to get so pummeled by everything else! It makes sense that Julia with its sweet JIT engine is the fastest. Rust does well—no surprise there. (Note that I’m not counting compilation time here—all the more impressive for Julia!) Racket holds its own though, coming in third place by a wide margin. If you did take Rust’s compile time into account, I think that would make it switch places with Racket. Of course, you use Rust for compile-once-run-lots sorts of scenarios.

Discussion

Are these authoritative? No, of course not. This is a simple synthetic benchmark. I don’t consider myself expert programmer in any of these languages, so there are likely some performance tweaks that could make the respective language’s benchmark run faster. (Maybe I should in Racket… it’s kind of hard to consider yourself an expert in a language when its creator works down the hall from you though.)

That said, I hope this dispels the myth that functional languages are necessarily slow. That is not the case. If Python is fast enough for your use-case, then there’s no reason you shouldn’t consider using Racket on performance grounds.

Library support is a different matter: of all the programming that goes on in the world, the majority is probably just gluing libraries together to do the thing you want. This is a good thing: it means we’re not reinventing so many wheels and people are getting good work done.

That said, there’s plenty of exciting work for which no library exists! If you find yourself in one of these exciting cases, consider using the tool of maximum power: in this regard, nothing beats Racket for its flexibility and extensibility.

Bonus: building the graph in Racket

I love the Racket Plot library. So easy to use, so powerful. If you run it inside of DrRacket, the graphs are actually interactive: you can rotate 3D graphs and zoom in on sections of 2D graphs. So neat!

Here’s the code I used to generate the above graph:

#lang racket

(require plot)

(define *data* (list #("Racket" 4.68) #("Python" 34.27) #("Rust" 3.03) #("Julia" 2.34) #("JavaScript" 11.92)))

(parameterize ([plot-font-face "Charter"]
               [plot-font-size 14.0]
               [plot-x-ticks (linear-ticks #:number 10 #:divisors '(1 2 5))])
  (plot (list (discrete-histogram (reverse *data*)
                                  #:gap 0.3
                                  #:invert? #true))
        #:title "Simple benchmark results"
        #:out-kind 'svg
        #:out-file "benchmark_shootout.svg"
        #:x-max 35
        #:x-label "Seconds"
        #:width 800
        #:height 200))

Scenic II Lüftungsgebläse plötzlich aus/Vorwiderstand defekt – die Zweite

Das Autotagebuch — Wed, 20 Dec 2023 00:26:48 GMT

Mitte 2015 war bei unserem Scenic II plötzlich das Lüftungsgebläse aus. Seinerzeit berichtete ich im Beitrag „Scenic II Lüftungsgebläse plötzlich aus„, wie eine freie Werkstatt diesen gegen ein originales Ersatzteil vom Erstausrüster (Valeo) tauschte. Kostenpunkt damals: 172 Euro. Den genauen Defekt am Vorwiderstand erfuhren wir damals nicht. Nun, fast pünktlich zu Weihnachten, war die Lüftung […]

The post Scenic II Lüftungsgebläse plötzlich aus/Vorwiderstand defekt – die Zweite appeared first on Das Autotagebuch.

The books I enjoyed the most in 2023

Gonçalo Valério — Tue, 19 Dec 2023 21:50:32 GMT

We reached the end of another year, and generally, this is a good time to look back, to evaluate what was done, what wasn’t done and eventually plan ahead. While dedicating some time to the first task, it occurred to me that I should share some of this stuff. I doubt it will be useful […]

La publicité pour les trains

ache: Blog personnel — Tue, 19 Dec 2023 15:34:00 GMT

Parlons trains. Enfin pub. Enfin ! Vous verrez bien ! L'idée est d'analyser quelques publicités pour des trains à travers le monde. Mais c'est surtout un prétexte pour vous présenter l'utilisation de technologies d'intelligence artificielle récentes. 🤖

My cat water fountain comes with a spicy USB power adapter

./techtipsy — Tue, 19 Dec 2023 14:00:00 GMT

It turns out that you can't trust any USB type A power adapter to be within spec.

For All Mankind: Season 2

🦉 olshansky 🦁 — Tue, 19 Dec 2023 03:18:01 GMT

Olshansky's review of For All Mankind: Season 2

The "Cheap" Web

taylor.town — Tue, 19 Dec 2023 02:00:00 GMT

The "cheap" web is a solarpunk philosophy of web design.

WishWell.io

taylor.town — Tue, 19 Dec 2023 02:00:00 GMT

WishWell is a simple patronage service.

Blazor on Netlify With Environment Variables!

Chris' Full Stack Blog RSS Feed — Tue, 19 Dec 2023 02:00:00 GMT

Make one of Microsoft's newer technologies configurable!

Logical Replication From Postgres 16 Stand-By Servers—Debezium and Failover Slots

Gunnar Morling — Tue, 19 Dec 2023 02:00:00 GMT

Table of Contents

Stand-By Logical Replication With Debezium
Towards Fail-Over Slots
Wrap-Up

Welcome back to this series about logical replication from Postgres 16 stand-by servers, in which we’ll discuss how to use this feature with Debezium—a popular open-source platform for Change Data Capture (CDC) for a wide range of databases—as well as how to manage logical replication in case of failover scenarios, i.e. a situation where your primary Postgres server becomes unavailable and a stand-by server needs to take over.

Using Stand-by Servers for Postgres Logical Replication

Gunnar Morling — Tue, 19 Dec 2023 02:00:00 GMT

Table of Contents

What is Postgres Logical Replication?
Why Logical Replication On Stand-By Servers?
Provisioning a Testing Environment
Testing Things Out

For users of Change Data Capture (CDC), one of the most exciting features in Postgres version 16 (released in September this year) is the support for logical replication from stand-by servers. Instead of connecting to your primary server, you can now point CDC tools such as Debezium to a replica server, which is very interesting for instance from a load distribution perspective. I am going to take a closer look at this new feature in this two-part blog series:

The "Dont" Tool

Synystron Synlogica — Mon, 18 Dec 2023 21:08:50 GMT

A proposal for a new Unix-style CLI tool

Useless Things You Remember

I Learnt — Mon, 18 Dec 2023 14:00:00 GMT

During a team call this morning, one of the engineers mentioned about the useless things he still remembers and how it would be good if the brain would clear them out.

Describing The Bureau of Magical Things Season 2

Brett Coulstock - Writer — Mon, 18 Dec 2023 11:00:00 GMT

Audio description scripts for selected episodes of The Bureau of Magical Things, Season 2

Pacifica After Rain

hypergeometric — Mon, 18 Dec 2023 02:16:00 GMT

A brief moment waiting for my little one’s swim lesson to start.

The post Pacifica After Rain first appeared on hypergeometric.

Why do programmers need private offices with doors?

Studiofreya SSG Site — Mon, 18 Dec 2023 02:00:00 GMT

https://www.reddit.com/r/programming/comments/18l88xq/why_do_programmers_need_private_offices_with/

Scientific Freedom: The Elixir of Civilization

🦉 olshansky 🦁 — Mon, 18 Dec 2023 02:00:00 GMT

Olshansky's review of Scientific Freedom: The Elixir of Civilization

Lambda Warming is an Antipattern

Mon, 18 Dec 2023 02:00:00 GMT

At a certain stage during most monolith to microservice migrations, teams that choose to migrate to Lambda often encounter a classic problem: cold starts.

Searching this issue on Google (or DuckDuckGo for those conscious of privacy) yields thousands of results. Strategies abound, from reducing cold start times to using Lambda warmers and other seemingly haphazard tactics.

On the surface, optimizing your Lambdas seems beneficial. Lambda bills per millisecond of compute time, so optimizing at scale can lead to marginal cost savings, not to mention the customer impact of a faster app.

I won't rehash the numerous articles suggesting sound methods to improve Lambda performance, such as increasing memory, avoiding placement in VPCs, and choosing quick languages like Python, Go, or Node.js (in that order).

Instead, I'll focus on another strategy often suggested to mitigate cold start problems: Lambda warming.

Lambda warming is a method where Lambdas are pinged to maintain an active, spun-up instance. Theoretically, this eliminates the time AWS takes to boot the container. However, as Yan Cui points out on The Burning Monk, this assumption is flawed. By implementing a Lambda warming mechanism, you're only keeping a single instance of your app alive (until it recycles about every 45 minutes, a built-in AWS quirk) rather than having many concurrent instances ready for traffic.

Beyond this apparent flaw, Lambda warming disrupts the pattern of Lambda's transactional, functional nature—stateless input and output.

If you've concluded that you need Lambda warming, it suggests a larger issue: the need for low latency in your backend systems, either for customer experience or technical reasons. And by using Lambda warming, you're likely willing to invest money to combat 300ms of latency.

If these statements ring true, then the reality is you might be better served by a different service like ECS Fargate, Kubernetes (EKS), or traditional EC2—essentially, anything long-lived.

Lambda is a fantastic tool, but it's not a one-size-fits-all solution. Often, a HTTP API with some cold start latency is perfectly acceptable. However, for real-time services like chat, it might not be the best fit. Yet, if your traffic for such a service is consistent, then perhaps it is suitable!

The key is to gather data, understand your use case and pain points, and then evaluate your options. Don't rush to a workaround without thorough consideration.

Three days of Advent of Code in Hurl

ntietz.com blog - technically a blog — Mon, 18 Dec 2023 02:00:00 GMT

Every year I do some of Advent of Code. One year I completed it, but usually I just do some of it as a social thing with friends and then taper off as interest wanes. This year, I did three days of it, and stopped because I really truly did not want to write more solutions in the language I chose.

See, previous years I made a reasonable choice, like Rust. But this year, since I wrote a programming language, I decided to do at least three days of Advent of Code in it, and more if I wanted to. (Dear reader, I did not want to.)

These three days of it were very useful in getting comfortable with Hurl and they were also critical in developing Hurl's built-ins and standard library to a reasonable point. I'm pretty confident now that you could do all of Advent of Code in it. And I'm also now free from Hurl, and so are you: this is the last either of us need to think about it^[1].

If you want to see all the solutions I've implemented, they're in Hurl's repo as examples and tests^[2]. For now, I'll walk through one solution and how it works.

Deep dive of day 1, part 2

Day 1 part 2 had some interesting flair to it where it is short but also interesting enough. The premise is that you have a document where each line contains characters and you're trying to find the numbers on each line. You take the first and last single-digit number, then concatenate them together, then sum all such numbers from all the lines. But they also might be written as English words.

Here's a short example of a document.

zoneight234
7pqrstsixteen

From the first line we get "one" and "4", so that becomes 14, and the second line gives us "7" and "six", which becomes 76. The solution for this document then is 14 + 76 = 90.

The first thing we do is import the standard library functionality we will need. The paths here are relative to the source file you run, so they may change for different users.

include "../lib/loop.hurl";
include "../lib/if.hurl";

Then we can read in our input. We use built-ins for reading the file's input and breaking it into lines. Breaking it into lines could have been done in Hurl, but I made it a built-in to save time.

let input = read_file("./aoc/input/day1.txt");
let lines = str_lines(input);

Now we can iterate over the lines and extract the solution, once we define the extract_nums function.

let total = 0;

for_each(lines, func(line) {
    try {
        extract_nums(line);
    } catch as val {
        total = total + val;
    };
});

println("solution: ", total);

The for_each here is defined in Hurl's standard library, only using Hurl itself (exceptions and recursion). It accepts a list or string as its first argument and a function to call on each element as its second argument. Inside of there, we use a try-catch to get the value of each line via the extract_nums function and add it into the running total, which we'll then print out.

Now let's define extact_nums. We know basically what it needs to do: find the first and last single-digit number on each line. Some may be a literal digit, others may be an English word representing that digit.

Here's the basic structure. We break the string into its individual characters, then we find the first and last numbers. We convert the numbers to strings, concatenate them together, then cast them back to a number. Finally we can hurl the result to our caller.

let extract_nums = func(line) {
    let chars = str_chars(line);

    let first = 0;
    # TODO: find the first number

    let last = 0;
    # TODO: find the last number

    let first = "" + first;
    let last = "" + last;

    hurl as_num(first + last);
};

Finding the first and last number are essentially the same, just one starts from the end, so I'll skip one here. To find the first number, we use until (defined in Hurl in the standard library) to iterate through the list until we find a number, then we break. The index is tracked in an element of a list, passed in as [0] here to start iteration at the beginning.

For each element of the list, our condition for stopping is "is this a number?" and if so, we save it and halt iteration. Otherwise we run the loop body, which increments our index for the next iteration.

try {
    until(func(locals) {
        try {
            is_number(line, locals.1);
            # TODO: define is_number
        } catch as result {
            if(func() {
                hurl result.1;
            }, func() {
                first = result.2;
            });

            hurl result.1;
        };
    }, func(locals) {
        hurl [locals.1 + 1];
    }, [0]);
} catch as val {
};

And that just leaves the last bit, which is defining our is_number function. This one leverages the if_else also defined in the standard library, and also takes advantage of Hurl's lists being 1-indexed. We keep track of the result as a list of [boolean, int] to indicate whether it is a number and, if so, what the number is.

The first thing we do is check the condition for whether the current index points to a digit. If so, we go to the true case, and we set the result to the character at that position cast to a number. Otherwise, we loop through a list of the first 9 numbers written as English words, and check whether or not they're contained as a substring starting at our index. If so, we set the result.

And at the end, we just hurl what we found!

let is_number = func(line, index) {
    let result = [false, 0];
    if_else(func() {
        hurl is_digit(at(line, index));
    }, func() {
        result = [true, as_num(at(line, index))];
    }, func() {
        try {
            for(9, func(locals) {
                let target = at(word_numbers, locals.1);
                let slice = slice(line, index, index + len(target));
                if(func() {
                    hurl slice == target;
                }, func() {
                    result = [true, locals.1];
                });

                hurl [];
            }, []);
        } catch as default {
        };
    });

    hurl result;
};

Then it can all be put together into one listing, and we run it and it works! It gets the right answer and takes a good long time to compute it, but it does get there eventually.

You can see the full listing in the repo. I've changed the order of things here and skipped a few pieces here to present it more easily, but otherwise it's the same code.

Plans for next Advent of Code

So, this was fun. Honestly, it was enjoyable to do a few days of Advent of Code in this, uh, treat of a language. (It was my penance.) But I only really wanted to do it for a few days. Next year, what will I do?

There are a few options I'm considering for next year:

Assembly: this seems like a fun challenge for a few days, although probably fairly mind bending! That could be a benefit.
Languages made by other Recurse Center people: this could be a fun way to storm through multiple languages, like one of my RC friends did one year.
Next year's language: I intend to make another language next year (something more normal, to focus on some of the "make it nice to use" aspects). If I follow through, I'll have to use it at least some!
Something... normal? I could always use Python or Rust and do some of it the normal way! It's fun to go through it with friends, so I could go back to doing it socially next year!

Advent of Code is a lot of fun, and it's a nice way to get exposure to different ideas and new languages and new technologies, so I'm excited to see what penance I bring on myself for the next one!

Well, I do intend to submit something about Hurl to SIGBOVIK so if that gets accepted, people will hear about it again. ↩
We are far enough into Advent of Code that posting the solutions is fine, I think. Also props to anyone who actually gets this running or translates the Hurl code into something else, so go for it. ↩

Paying Netflix $0.53/h, etc.

taylor.town — Mon, 18 Dec 2023 02:00:00 GMT

People pay ¨D0.50-¨D2.00 for an hour of digital entertainment.

Colliding Secure Hashes

David Buchanan's Blog — Mon, 18 Dec 2023 02:00:00 GMT

Mediasonic Probox fan replacement

voussoir.net/writing — Sun, 17 Dec 2023 22:50:46 GMT

Back to writing

[electronics] [today_i_did_this]

Mediasonic Probox fan replacement

I have a Mediasonic Probox for file storage. After 3.5 years of use (purchased May 2020), one of the fans has gone bad:

The stock fans are labeled YLTC DFS802012H, 12 Volts, 0.23 Amps, and measure 80x80x20mm.

For my replacement, I chose the Arctic P8 because it seems to have a good reputation for being quiet, and I feel that a pressure-optimized fan (the P series) is more appropriate than a flow-optimized fan (the F series) for pushing air around the box's motherboard and between the drives.

Before doing some research, it hadn't occurred to me that the thicknesses of 80mm case fans wouldn't all be the same. On this Mediasonic forum post about fan replacement, someone commented that most fans on the market are 25mm thick and would not be a drop-in replacement for the 20mm stock. The screws are not long enough to go all the way through a 25mm fan, and the shroud, and have any real bite left over.

Besides the problem of length, there is also diameter: without the shroud, the screws slip straight through the mounting holes in the fan itself, so the shroud is necessary... or so I thought. Then I found this thingiverse thing where the author takes advantage of the empty space between the two faces of the fan housing to slip in a little hollow post. This bypasses the extra 5mm of fan thickness by placing the screw head within the Z-profile of the fan, and gives it something to grab onto, solving both problems at once! Very clever!

$fa = 0.1;
$fs = 0.1;

difference()
{
    cylinder(d=5.5, h=10);
    cylinder(d=2.9, h=10);
}

Please excuse the poor quality of the prints...

Also, most fans use a three-pin connector with one wire carrying RPM signal data, but the Mediasonic only uses the two power pins and a 2mm connector. There are adapters out there, mentioned in the forum post, so you can make the swap non-destructively. I was not in the mood to wait multiple weeks for aliexpress shipping, so I cut and soldered the ends.

Of course, you could just buy a fan that's 20mm thick with a two-pin connector in the first place. But all the options I saw were very cheap/generic looking, and the per-fan prices were not cheaper than the Arctics, and I've been wanting to replace the stock fans with quieter ones for a long time anyway. In this case, I'd rather buy a fan from a known brand with a good reputation for quiet, even if it requires some modification.

You have to admit this doesn't really seem like the ideal air path:

But the replacement went just fine and the fan feels very secure.

I felt I should add a bit of tape around the edge to discourage air from leaking into the sides, since the shroud is gone now.

After this, I also replaced the other fan in the same manner.

At low speed the new fans are essentially inaudible:

At full speed, they're still pretty quiet:

These recordings were made with no hard drives in the box. Once the drives are in... well... it's quieter than it was before but it's not exactly life-changing.

For the first few days, I heard a new sound that made me think the whole thing was pointless — it was a kind of ringing, or resonance, or beat frequency that was audible when I sat in my chair, but seemed to disappear when I got close to the box to listen for it. I'm not sure if it was an illusion, or the heart of my latest victim, or the bearings just breaking themselves in, but it's better now.

I don't normally publish this kind of thing, since I feel it's a little trivial or unimpressive, and because it mentions brand names. But whenever I am solving a problem, I rely on posts made by other people solving the same problem — something as simple as a hollow cylinder turned out to be a very helpful bit of inspiration to me. If that person thought a cylinder was too trivial to share, it would have taken me longer to do this. So maybe I should contribute back in the same way more often.

View this document's history

2023-12-17 Add mediasonic_fan_replacement.md.

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

Tracking Java Native Memory With JDK Flight Recorder

Gunnar Morling — Sun, 17 Dec 2023 20:29:00 GMT

Table of Contents

An Example
Tracking RSS

As regular readers of this blog will now, JDK Flight Recorder (JFR) is one of my favorite tools of the Java platform. This low-overhead event recording engine built into the JVM is invaluable for observing the runtime characteristics of Java applications and identifying any potential performance issues. JFR continues to become better and better with every new release, with one recent addition being support for native memory tracking (NMT).

Making Cartridges for the Commodore C16, C116, and Plus/4

hackup.net — Sun, 17 Dec 2023 16:58:38 GMT

As I mentioned a while ago, I had the opportunity to repair a bunch of broken boards for Commodore C16 and Plus/4 computers. This led to a couple of new projects, the ROMulator and the PLA264. To help with the … Continue reading →

The post Making Cartridges for the Commodore C16, C116, and Plus/4 first appeared on hackup.net.

Hosting quickly 2.5 - Discovering and calling a backend from Dioxus

Martijn Arts — Sun, 17 Dec 2023 09:54:52 GMT

We should make the backend serve some API that we can hit from the frontend.

We'll turn the root endpoint in the backend into a JSON-returning endpoint:

#[derive(

Book Review - The Field Guide to Global Payments

River Writes - A MediaWiki Blog — Sun, 17 Dec 2023 09:15:58 GMT

A review of “A Field Guide to Global Payments,” including a table of initialisms used in the book.

OnlineOrNot Diaries 17

Max Rozen — Sun, 17 Dec 2023 09:10:00 GMT

On learning screencasting, cleaning up tech debt, and focus

One possible death for Discord

utf9k — Sun, 17 Dec 2023 03:14:00 GMT

An unreleased post where I talk about Discord and what might happen to them if they were acquired

An basic Mullvad WireGuard setup for macOS

xenodium.com @alvaro — Sun, 17 Dec 2023 02:00:00 GMT

Needed a VPN to test an API from a different location. Gave Mullvad a try.

Pretty neat, you can generate an account number without providing an email address. You can also pre-pay with a ton of options, including cash, crypto, credit cards, PayPal, wire transfers…

After seeing your account credited, one can download a generated WireGuard configuration. Also a WireGuard noob, so took this opportunity to give it a try.

The WireGuard macOS app has an "Import Tunnel(s) from File…" option where you can import the .conf file downloaded from Mullvad's generated config. After that, all I had to do was click the "Activate" button and Bob's your uncle.

You can test your connection via:

curl https://am.i.mullvad.net/connected

I had a brief stint at using the command-line alternative via homebrew brew install wireguard-go wireguard-tools, but that seems to fail silently:

wg-quick up xxxxx
[#] wireguard-go utun
[+] Interface for xxxxx is utun7
[#] wg setconf utun7 /dev/fd/63
[#] ifconfig utun7 inet xxx.xxx.xxx.xxx/xx xxx.xxx.xxx.xxx alias
[#] ifconfig utun7 inet6 xxxx:xxxx:xxxx:xxxx::x:xxxx/xxx alias
[#] ifconfig utun7 up
[#] route -q -n add -inet6 ::/1 -interface utun7
[#] route -q -n add -inet6 8000::/1 -interface utun7
[#] route -q -n add -inet xxx.xxx.xxx.xxx/x -interface utun7
[#] route -q -n add -inet xxx.xxx.xxx.xxx/x -interface utun7
[#] route -q -n add -inet xxx.xxx.xxx.xxx -gateway xxx.xxx.xxx.xxx
[#] networksetup -getdnsservers Wi-Fi
[#] networksetup -getsearchdomains Wi-Fi
[#] networksetup -getdnsservers iPhone USB
[#] networksetup -getsearchdomains iPhone USB
[#] networksetup -getdnsservers Thunderbolt Bridge
[#] networksetup -getsearchdomains Thunderbolt Bridge
[#] networksetup -getdnsservers xxxxx
[#] networksetup -getsearchdomains xxxxx
[#] networksetup -setdnsservers iPhone USB xxx.xxx.xxx.xxx
[#] networksetup -setsearchdomains iPhone USB Empty
[#] networksetup -setdnsservers xxxxx xxx.xxx.xxx.xxx
[#] networksetup -setsearchdomains xxxxx Empty
[#] networksetup -setdnsservers Wi-Fi xxx.xxx.xxx.xxx
[#] networksetup -setsearchdomains Wi-Fi Empty
[#] networksetup -setdnsservers Thunderbolt Bridge xxx.xxx.xxx.xxx
[#] networksetup -setsearchdomains Thunderbolt Bridge Empty
[+] Backgrounding route monitor

curl https://am.i.mullvad.net/connected

I'm on a Macbook M1 Pro, running macOS Sonoma. If you got wg-quick working on Sonoma, I'd love to hear from ya (Mastodon / Twitter / Reddit / Email).

F22a Raptor Part Names

Studiofreya SSG Site — Sun, 17 Dec 2023 02:00:00 GMT

I saw this schematic of the F-22A raptor the other day. Enjoy!

Languishing

Studiofreya SSG Site — Sun, 17 Dec 2023 02:00:00 GMT

Becoming languid and weak; pining; losing health and strength. Similar: pining Amorously pensive; indicating melancholy.
“languishing eyes, or look”
Suffering neglect; neglected. Similar: neglected

The most frustrating thing in your life

taylor.town — Sun, 17 Dec 2023 02:00:00 GMT

...is a computer.

Cynical PM Framework Webinar at Product School

Frank Tisellano | ft.io — Sun, 17 Dec 2023 02:00:00 GMT

Summary: 10-Minute Toughness by Jason Selk

Chester Grant — Sat, 16 Dec 2023 17:46:11 GMT

1. The 10-Minute Toughness program is simply about identifying and utilizing a handful of mental tools that are proven to help people perform more consistently.

2. Athletes using 10-MT spend approximately four to five minutes after each practice and competition to focus on strengths and goal-related thinking. They also spend five minutes a day completing a five-step mental workout prior to the next day's training and/or competition to help develop a pinpoint focus on what they are trying to improve and what it takes to make it happen.

3. The field of sport psychology has identified self-efficacy (self-confidence) as the most influential mental variable in controlling performance. This means that if you have a strong belief in your ability to perform well, then the chances of your actually performing well greatly improve. This program is specifically designed to help athletes improve self-confidence. The two most effective ways to develop self-confidence are to perform well and to physically and mentally prepare to perform well.

4. The program works because it has two primary strengths. First, it forces athletes to identify the "process of success." The process of success is a clear and concise assessment of what it takes for the athlete to perform well. This assessment is done daily after training and competition. Second, a mental workout provides a vehicle for mentally training the identified process of success. In addition, athletes visualize success outcomes to enhance self-confidence. They identify the most beneficial thoughts to have and then condition their minds to be able to maintain those thoughts more fully throughout training sessions and competition. It is certainly not rocket science, but it absolutely works!

5. There are three essential phases in the 10-MT mental-training program. Phase 1 is what I refer to as the mental workout. This individualized mental-training plan helps athletes identify and focus on their "control points," or what it takes for success.

6. 10-Minute Toughness Mental Workout:

a. The Centering Breath. A fifteen-second deep breath designed to control arousal states.

b. The Performance Statement. A specifically tailored self-statement useful for increasing training and competitive focus.

c. The Personal Highlight Reel. An advanced form of visualization allowing athletes to increase skill refinement and consistency.

d. The Identity Statement. A concrete self-statement proven to enhance self-image and performance confidence.

e. The Centering Breath. As in step one, a biologically established relaxation technique used to increase the potential to perform well under pressure.

7. Phase 2 comprises developing and utilizing an effective goal-setting program:

Vision clarity: Ultimate goal accomplishment associated with sport.

Product goal: A result-oriented goal that is clearly measurable and is usually most effective if it emphasizes accomplishments in the next twelve months.

Process goal: The daily action needed to accomplish both product-and vision-level goals.

Success Log: Questions that encourage the identification of personal strengths and specific desires for improvement.

Personal Rewards Program Questionnaire: Questionnaire that identifies the athlete's motivational preferences.

Personal incentive style: The motivational preferences of athletes that enhance training and competition intensity.

8. Phase 3 teaches individuals how to develop a relentless focus on solutions.

9. Athletes and coaches are able to realize the small difference between good and great by asking, What is one thing I can do that could make this better?

10. Centering Breath = Diaphragm Breathing - I have tried to simplify diaphragm breathing by qualifying a good centering breath as one that lasts fifteen seconds. The formula is 6-2-7: breathe in for six seconds, hold for two, and breathe out for seven seconds. Individuals under the age of twelve should try to have the centering breath last eleven seconds (4-2-5).

11. One core aspect of training is known as arousal control. The heart rate is a primary control of a person's arousal state. It is important to control heart rate because using the mind effectively becomes increasingly more difficult as the heart rate rises. Once the rate gets to 120 beats per minute, the mind will not be nearly as sharp (unless proper conditioning and mental training has occurred), and at about 150 beats per minute, the mind will essentially shut down and go into survival mode. (In this state, even the best athletes will lose the ability to maintain mental acuity.)

12. A performance statement is a specifically designed form of self-talk. Self-talk is the conversation that goes on in a person's head throughout the day. It is said that the average person has up to sixty thousand thoughts per day—that's a lot of self-talk. The unfortunate thing about those thoughts is that the majority tend to embody self-doubt or negativity. If we do not choose our thoughts carefully, they can (and many times do) have a negative impact on performance.

13. From a mental standpoint, the most tried-and-true way to increase performance is to improve confidence. Self-talk is one of the most influential agents for honing self-confidence.

14. Extensive research in the sport psychology world confirms that an athlete's internal dialogue significantly influences performance. Athletes who have negative self-talk will generally experience poor performance; conversely, when athletes keep their minds focused on positive performance cues, they are more likely to experience success.

15. A performance statement is a type of self-talk designed to help athletes zoom in on one specific thought to enhance performance consistency. It is a simple, yet concrete, thought that specifically identifies the process of success, or what it takes to perform at your best.

16. For the baseball player, there may be a performance statement to emphasize hitting (track the ball, smooth and easy) and defense (set, stay down, watch it into the glove). A basketball player may choose to combine both offense and defense into one performance statement (hustle every possession; attack every rebound; drive, drive, drive), while the gymnast may have a performance statement for every event she competes on (floor: quick hands, tight legs, squeeze; vault: top speed, feet in front; bars: hollow handstands and elbows locked).

17. In my opinion, the essence of mental toughness is the ability to replace negative thinking with thoughts that are centered on performance cues or that contribute to improved self-confidence.

18. The more often negative thoughts are replaced with positive self-talk, the more successful and mentally tough a person will be.

19. The most helpful method to stop self-doubt and negative thinking is thought replacement. Effective thought replacement occurs when you decide what you want to have happen and then think more often about what it will take to make it happen. Whenever unproductive thoughts ("don't" thinking or mental clutter) infringe, replace them with productive ones.

20. The performance statement serves two principal purposes. First, it is a way to avoid self-doubt, negativity, or mental clutter(thought replacement). Second, it helps you perform at your best by directing your thoughts toward targeted areas of strength.

21. A client trying to prioritize exercise had this as his performance statement: "Three days on, one day off; dedicated and committed; I clear my own path." His highlights aren't as much about the technical aspects of doing it right as they are about his desire and ability to be healthy.

22. The personal highlight reel is an advanced form of visualization in which you create your own mental "SportsCenter" highlight reel.

23. People learn faster by visualizing success rather than by watching it on tape. Of course, there is considerable value in watching success on tape. In fact, during film sessions, Bear Bryant, the famous football coach at Alabama, showed his players only footage of themselves playing well.

24. Coach Bryant contended that showing his players what they did well helped them repeat the performance, whereas showing them what they did wrong would only increase the likelihood of their exhibiting more poor play in the future.

25. According to some studies, in fact, every minute of visualization is worth seven minutes of physical practice.

26. Visualizing is the act of watching something in your mind before actually doing it.

27. There are eight essential guidelines for visualization success:

Choose One of Three Camera Angles - whether 1. as if the camera is in the stands recording someone other than you performing the skill; 2. you are watching the mental video as if the camera is in the stands filming as you perform the skill and 3. you watch the mental video as if your eyes are the camera lens (or as if you're wearing a helmet camera). From this perspective, you would see whatever you actually see while you are performing the skill.

Pay Attention to Detail - Try to pay attention to three of the five senses while performing the skill: sight (What do you see on the surrounding field, court, or arena?); sound (Do you hear crowd noise, coaches, teammates?); and feel (What does the ball, bat, racket, club, etc., feel like? Also, what does your body feel like as it performs successfully?).

Frequent and Brief Is the Ticket - Visualizing many times for short stints is far more effective than visualizing for extended periods.

Visualize from Beginning to End - Make sure to view the skill or action in its entirety. Creating a comprehensive mental video helps to reduce distraction and eliminate potential problems with emotional control.

Emotionally Feel the Way You Want to Feel - Be aware of how you want to feel before and during competition, and then train yourself to feel just that way.

Replay Until You Get It Right

Give Credit Where Credit Is Due - Upon finishing each successful visualization, take a brief time-out to congratulate yourself on a job well done. A mental pat on the back or a few kind words to yourself after visualizing success will help keep you on an even keel.

Operate at Game Speed - Finally, watch the mental clip at the desired speed. While it is helpful sometimes to slow the mental video down to figure out some of the more complex skills, you should always visualize at the desired speed prior to the physical performance, or else the timing of the action may be off.

28. Camera angle number three is the most beneficial, because with this approach, your muscles can actually get stronger and muscle memory can develop. Many athletes refer to muscle memory as the ability to perform successfully while on autopilot.

29. Although I just told you that all three camera angles are helpful, I do not want you to use more than one camera angle when you visualize. Pick one angle to employ, trusting your instincts on which of the three is most appropriate for you.

30. The personal highlight reel is made up of three parts. The first part emphasizes a successful performance from your past, and the second and third parts address how you want to perform in the future.

31. Part 1: Sixty-Second Mental Video Clip of Excellent Past Performance - There are two options to consider when creating the first part of your personal highlight reel. The first option is to identify the best performance or game you have had in the past (the more recent the better). Imagine that the entire performance was videotaped; select three to five personal highlights from the videotaped performance, each lasting from ten to twenty seconds. The second option is to identify a few of your best performances and choose a combination of your best highlights from those performances. Choose the three to five best personal highlights, each lasting between ten to twenty seconds.

32. Parts 2 and 3: Sixty-Second Mental Video Clips of Upcoming Big Games or Competitions - The second and third parts are also sixty-second mental video clips, but instead of re-creating the past, you imagine what you want to have happen in your future.

33. Be sure to really hone in on what it physically feels like to emphasize your performance statement and emotionally focus on your ideal arousal state as you go through parts two and three of your personal highlight reel. By doing so, you will dramatically increase the effectiveness of your visualization.

34. Sean Townsend's Personal Highlight Reel

Part 1 (Successful Past Performance): World Championships '02

Good training leading up to World Championships

Felt calm and confident in warm-ups; everything was smooth and fluid

Hit four solid routines on the first day of competition

Superhit P-bars to win the gold

Personal greatest moment in sport: gold medal ceremony (freeze frame: standing at the top of the podium)

Part 2 (Next Elevated-Pressure Competition): USA Championships '08

Good, solid warm-up, feeling calm and confident and taking one skill at a time, one routine at a time

Hit all six routines on day one while feeling calm and confident and taking one skill at a time, one routine at a time

Hit all six routines in finals while feeling calm and confident and taking one skill at a time, one routine at a time

Part 3 (Next Competition): WOGA Invitational

Good, solid warm-up, feeling calm and confident and taking one skill at a time, one routine at a time

Hit all six routines on day one while feeling calm and confident and taking one skill at a time, one routine at a time

Hit all six routines in finals while feeling calm and confident and taking one skill at a time, one routine at a time

35. An identity statement is a self-statement designed to improve self-image. Your self-image is essentially how you view yourself—what strengths and weaknesses you believe you possess. It has been demonstrated that what people believe they are capable of accomplishing largely determines how much they will actually accomplish.

36. Self-image is a proven agent of behavior control.

37. The effect of self-image is one reason why 80 percent of lottery winners file for bankruptcy within five years of winning. Even though their financial situation has dramatically changed, typically the self-image hasn't. For people who see themselves as not good with money, no matter how much money is given to them, they will generally find a way to lose it.

38. As pointed out previously, largely what determines people's self-image is the things they continually say to themselves, and unfortunately, much of our inner dialogue regards what we can't do rather than what we can do.

39. The key is to create the self-image desired—decide who you want to be and how you want to live—and then continuously tell yourself that you have what it takes to be that person. The self-image will guide and direct actions and behaviors until the self-image becomes the reality.

40. In the words of Maxwell Maltz, "You will act like the sort of person you conceive yourself to be. More important, you literally cannot act otherwise, in spite of all your conscious efforts or willpower. This is why trying to achieve something difficult with teeth gritted is a losing battle. Willpower is not the answer. Self-image management is."

41. I encourage athletes to create a two-part identity statement. The first part indicates a strength you currently have or want to have. Be sure the strength you choose to accentuate makes achieving success more likely. For example, your strength might be that you are a really hard worker. Whether it is already true or is something that you want to be true, the first part of your identity statement might be "I am the hardest worker on the team."

42. The second part of your identity statement addresses what you want to accomplish. It is OK to stretch a bit here. According to research on affirmations, the more imposing the desired task, the more impact it will have on the self-image. Frame both parts of your identity statement as though the objectives have already been achieved.

43. My own personal identity statement is "I am more motivated than my competition; I am the most effective sport psychology consultant in the world."

44. Examples of Identity Statements

Baseball. I am the hardest worker on the team; I am a dominant major-league hitter.

Basketball. I am intelligent and I know this game better than anyone; I am the most prolific scorer on the court.

45. Remember: the centering breath is a deep breath used to physiologically control heart rate and arousal. Taking a centering breath at the end of the mental workout is necessary for athletes because completing the personal highlight reel may cause the heart rate and arousal state to elevate.

46. I tell athletes that doing the mental workout one time a day is great. Some clients prefer to do it a couple of times a day, and that is OK, but there is no need to do it more than twice a day.

47. Give it a try. Complete the mental workout for two weeks, and judge for yourself if it helps you to improve focus, ability, and consistency.

48. The three levels, or types, of goals that I discuss with clients are ultimate goals, product goals, and process goals

49. Ultimate goals. Ultimate goals are the culmination of what you want to accomplish and how you want to accomplish it. When identifying your ultimate goals, imagine being able to look into the future and witness your retirement dinner. What accomplishments do you want to hang on the wall, and what would you like the speaker to say about you regarding how you played the game and how you conducted yourself?

50. Product goals. Product goals are result-oriented goals. They are clearly measurable and usually are most effective if they emphasize accomplishments in the next twelve months. I have found that the best formula is to assign yourself up to three product goals for the next competitive season in which you will participate and, again, up to three product goals for the upcoming off-season. For example, a basketball player may have the following three product goals for the season:

1. Score at least ten points per game

2. Have a free-throw percentage of 80 percent or better

3. Grab at least six rebounds every game

51. Process goals. Process goals are the "what it takes" to achieve the product goals you set. Process goals also must be specific enough to be measurable. For example, the same basketball player may believe that two of the best ways for her to score ten points per game are by being mentally prepared for each game and by aggressively driving to the hoop (within five feet) at least four times per game.

52.It is important to write your goals down and let others know of your intentions. Writing and talking about your goals will also increase your accountability and motivation for achieving them.

53. "What does Coach Wooden say about excuses?" He replied, "Never make excuses. Your friends won't need them, and your foes won't believe them."

54. For goals to work, they must become a part of daily training.

55. After practices and games, you will take about three to four minutes to fill out a Success Log. The Success Logs ask athletes to answer the following questions:

What three things did I do well today?

Based on today's performance, what do I want to improve?

What is one thing I can do differently that could lead to the desired improvement?

56. Just before doing your mental workout, you will take one minute to review your Success Log entries from the previous day.

57. In my experience, athletes who establish a "personal rewards" system find it easier and more enjoyable to make the sacrifices necessary to achieve their goals.

58. Research strongly suggests that pushing to reach a goal provides more happiness than actually reaching the goal.

59. It is also important to set new goals once a goal is achieved. Remember that you stand to experience more joy and satisfaction from striving to reach your goals than from actually achieving them. So, keep replacing achieved goals with bright, shiny new ones.

60. Edwin Locke and Gary Latham, discovered after years of research on thousands of people is that the most successful men and women in the world use goals as their primary method of motivation.

61. Being solution focused means keeping your thoughts centered on what you want from life and what it takes to achieve what you want, as opposed to allowing thoughts of self-doubt and concern to occupy the mind.

62. "Every time I catch myself feeling angry or scared or depressed it is because I am thinking about what I don't have in my life or about something that is wrong with me. The instant I catch myself feeling uncomfortable I ask, 'What is one thing I can do right now that could make my situation better?,'

63. Bolster yourself. Surround yourself with people who want to scale the same heights. If your training companions are always complaining and wallowing in their problems, your solution focus is sure to waver.

64. Attitudes are contagious; people tend to take on the attitudes and actions of the individuals in their circle. A good attitude is just as contractible as a bad one. If you habitually conduct yourself with a solution focus, those around you will start to follow your model.

65. I consider an individual to be mentally tough when the mind is in control of thoughts that help the body accomplish what is wanted.

66. Let's review how the three phases of 10-MT can help you choose the right mental attitude.

Phase 1: The Mental Workout

Step 1: Centering Breath The centering breath is a fifteen-second breath in which you breathe in for six seconds, hold for two, and then breathe out for seven.

Step 2: Performance Statement After taking your centering breath, repeat to yourself the statement that most effectively focuses you on what it takes for you to be successful in competition.

Step 3: Personal Highlight Reel After reciting your performance statement, spend about three minutes visualizing what it looks like to be successful.

Step 4: Identity Statement Upon completing your personal highlight reel, repeat to yourself your identity statement to help mold your self-image.

Step 5: Centering Breath The mental workout ends the way it begins, with a fifteen-second deep breath. This breath resets your heart rate to a level of controlled arousal and increased mental focus. Use success logs to keep your daily training focused and precise. As best as you can, try to identify just one thing you want to improve prior to each training day.

Phase 2: Goal Setting for Greatness - When you know what you want to accomplish, write it down, and spread the word. Talking about your goals will spring them from your subconscious into your consciousness.

Phase 3: Adopt a Relentless Solution Focus - Anytime you are in the presence of adversity ask yourself, "What is one thing I can do that could make this better?"

Horsey Gap and Horsey Windpump

Sam Hooke — Sat, 16 Dec 2023 14:45:00 GMT

Seals at Horsey Gap

We arrived at Horsey Gap at 12:45pm on a Saturday. There is a dedicated, paid car park, which is the only option for those who have come by car. The fee is reasonable, and although the car park is not very big, cars generally seem to leave frequently enough that you soon get a space.

Following are some of my favourite photos I took of the seals and their pups. While most of the seal pups are down on the beach, a handful were up in the dunes. Some helpful volunteers make sure that everyone keeps a safe distance away so as not to disturb the pups. Additionally, guide ropes help ensure that people stick to the paths, which helps preserve the terrain.

I don't know what i did last summer

Welcome to my weblog on Newbeelearn — Sat, 16 Dec 2023 14:26:23 GMT

TLDR: How author forgot his own code, ended up wasting time and learned importance of documenting the code.

Migrating existing page to hugo static site generator

Welcome to my weblog on Newbeelearn — Sat, 16 Dec 2023 14:26:23 GMT

TLDR: How to migrate existing site to Hugo

How to Use a Bash Script to Count Your Heroku Releases

William Huster — Sat, 16 Dec 2023 13:30:00 GMT

Our Creative Director asked me if I could estimate the number of production releases we have pushed so we could bost about it on our website. I quickly figured out a strategy that would work using just the heroku apps and heroku releases commands, but realized I would need to draft a script to iterate and parse the outputs. I also knew it would be best to do this in bash to make use of those heroku-cli commands.

I’m ashamed to say my bash is not strong, so I turned to ChatGPT 3.5 for help. Here’s a trimmed version of the exchange with ChatGPT.

Me: Please write a bash script that helps me count how total production releases to Heroku. Procedure:

Calls heroku apps -A to list my Heroku apps

Finds all apps whose names contain the substring ‘prod’

For each app in #2, calls heroku releases -a {app_name}

Sums all the latest version numbers to get a total number of production releases

ChatGPT was pretty close on the first try. The overall structure of the script was correct, but it was not correctly parsing the latest release from the output of the heroku releases command. This was the command it drafted:

releases=$(heroku releases -a "$app" | grep -E '^[0-9]+' | head -n 1 | awk '{print $1}')

This is very close, but it won’t yield any matches, because the regex used for grep is matching digits 0-9, but the release numbers are prefixed with a v, e.g. v55. The correct regex is therefore: '^v[0-9]+'. The variable name releases is also not entirely accurate and may indicate a “misunderstanding” of the goal. We’d like this line to give us just the latest release number. That said, the command is using head -n 1 to get the latest release. So I decided to give ChatGPT an example of the output to see if it could correct the error.

Me: This is close. There is just an issue parsing the latest release number from the result of the heroku releases command. Here is an example result:

v88  Deploy b5e486a4                       dev@thinknimble.com  2023/12/15 03:33:29 -0500 (~ 12h ago)
v87  Set REDIS_URL config vars             dev@thinknimble.com  2023/12/15 03:11:25 -0500 (~ 12h ago)
v86  Deploy eb10b67c                       dev@thinknimble.com  2023/12/15 02:48:50 -0500 (~ 13h ago)
v85  Deploy 2ced23ee                       dev@thinknimble.com  2023/12/15 00:41:51 -0500 (~ 15h ago)
v84  Deploy 1f24c745                       dev@thinknimble.com  2023/12/13 02:29:13 -0500
v83  Deploy 58c7705c                       dev@thinknimble.com  2023/12/13 02:08:38 -0500
v82  Deploy 8b80ec50                       dev@thinknimble.com  2023/12/12 21:19:02 -0500
v81  Deploy bb148408                       dev@thinknimble.com  2023/12/12 19:51:02 -0500
v80  Set USE_REDIS config vars             dev@thinknimble.com  2023/12/12 18:47:41 -0500
v79  Set REDIS_URL config vars             dev@thinknimble.com  2023/12/12 18:47:33 -0500
v78  Set REDIS_URL config vars             dev@thinknimble.com  2023/12/12 18:44:43 -0500
v77  Set REDIS_URL config vars             dev@thinknimble.com  2023/12/12 18:36:27 -0500
v76  Update NEW_RELIC by newrelic          dev@thinknimble.com  2023/12/12 10:44:14 -0500
v75  Set AWS_S3_SECURE_URLS config vars    dev@thinknimble.com  2023/12/10 17:29:48 -0500
v74  Set AWS_S3_CUSTOM_DOMAIN config vars  dev@thinknimble.com  2023/12/10 17:29:32 -0500

So in this case, we want to parse out the number 88 and add that to total_releases. Can you regenerate the script with that in mind?

ChatGPT then regenerated the script and changed the line. This time the regex is correct, but we have a regression. For some reason, ChatGPT removed the head -n 1 step. So this will give us a list of the release numbers of all releases in the output. I think it may have done this because I gave it a list of outputs above, instead of a single line.

latest_version=$(heroku releases -a "$app" | grep -E '^v[0-9]+' | awk '{print $1}' | sed 's/v//')

One more nudge from me set this right:

Me: Not quite. Instead of one latest_version this is giving me a list of all the versions. Try again?

And ChatGPT corrected the line. Now the variable name latest_version is good and we are getting a single release number to add to total_releases. Nice!

latest_version=$(heroku releases -a "$app" | grep -E '^v[0-9]+' | sort -rV | head -n 1 | awk '{print $1}' | sed 's/v//')

The Final Script

Here is the final script mainly authored by ChatGPT that will get a list of all production Heroku apps you have access to and sum up their latest release versions to give you a total number of all-time production deployments.

In our case, the number of total production releases is over 2000! And that doesn’t even include apps on AWS or those we have shut down and transferred away to clients. So I’d guess out team has probably pushed to prod almost 5000 times over the past five years. A vanity metric, perhaps, but it’s still cool to think about.

#!/bin/bash

# Step 1: List Heroku apps
apps=$(heroku apps -A)

# Step 2: Find apps with names containing 'prod'
prod_apps=$(echo "$apps" | grep 'prod' | awk '{print $1}')

# Step 3 and 4: Iterate over prod apps and get total releases
total_releases=0
for app in $prod_apps; do
    latest_version=$(heroku releases -a "$app" | grep -E '^v[0-9]+' | sort -rV | head -n 1 | awk '{print $1}' | sed 's/v//')
    if [ -n "$latest_version" ]; then
        total_releases=$((total_releases + latest_version))
        echo "App $app has version v$latest_version as the latest production release."
    else
        echo "No production releases found for app $app."
    fi
done

echo "Total production releases: $total_releases"

👨‍🎨 An Art Tour of Europe

Austin Atchley — Sat, 16 Dec 2023 02:00:00 GMT

While recovering from my second bout of COVID, I have been looking back through the pictures I took on my around-the-world trip this summer (which you can read more about here). On the Europe leg of the trip, we were lucky enough to visit some of the best museums in the world, but we also found some inspiring art in unexpected places. Excluding famous works, here are some selected favorites:

(Suggested 📚) Turn the Ship Around!

er4hn — Sat, 16 Dec 2023 02:00:00 GMT

Then Life Continued - Suggested Reading

SPHINCS+ - Step by Step

er4hn — Sat, 16 Dec 2023 02:00:00 GMT

Then Life Continued

Exposure: Poisoned Water, Corporate Greed, and One Lawyer's Twenty-Year Battle against DuPont

🦉 olshansky 🦁 — Sat, 16 Dec 2023 02:00:00 GMT

Olshansky's review of Exposure: Poisoned Water, Corporate Greed, and One Lawyer's Twenty-Year Battle against DuPont by Robert Bilott

Exposure: Poisoned Water, Corporate Greed, and One Lawyer's Twenty-Year Battle against DuPont

🦉 olshansky 🦁 — Sat, 16 Dec 2023 02:00:00 GMT

Olshansky's review of Exposure: Poisoned Water, Corporate Greed, and One Lawyer's Twenty-Year Battle against DuPont

How to write a JSON Schema

benbrougher.tech — Fri, 15 Dec 2023 14:40:22 GMT

How to write a JSON Schema and where they can be used

The wilderness of Poggio Montironi

Nicola Iarocci — Fri, 15 Dec 2023 08:05:25 GMT

Yesterday was a beautiful, sunny, cold winter Sunday. I felt like going out and enjoying nature, so I took a solo hike in the Foreste Casentinesi National Park, about an hour’s drive from home.

This one marks my very first technology-assisted hiking adventure. It may seem weird for someone who’s been hiking for so long, is a notorious geek and is a professional computer programmer to have never used technology before. A trail map and sometimes a compass were all I was used to, and deliberately. I wanted to avoid technology in this aspect of my life. I welcomed the orientation challenges and superbly looked down at the crowds of phone-smartwatch-compulsive hikers I met on the route. Alas, 2023 was the year I surrendered my motorcycling habits to intercom systems and GPS navigation, and that spoiled me. In a couple of situations yesterday, the app spared me some trouble by warning me of the wrong direction I was going. Nothing major. I would’ve realized the error and backtracked, but being these the shortest and coldest days of the year, I appreciated not risking getting caught by dusk. Also, I found that I can turn on voice-assisted navigation as I do on my motorcycle, and that’s nice (albeit surreal - walking alone in the wilderness, miles from anybody, with a voice coming out of nowhere and whispering when I should take turns on the trail): it avoids looking at the cellphone all the time so I can stay focused on the experience even more, I suspect, than before when the don’t-get-lost alarm bell was constantly ringing in the back of my head.

Balm in GILead

vito.nyc — Fri, 15 Dec 2023 04:00:00 GMT

Python isn’t slow. The core eval loop, while slower than a JIT, is no slouch when it comes to dispatching bytecode. There is no reason that business logic written in Python which orchestrates the operation of highly optimized extension libraries should be a bottle neck.

Lessons from implementing Hurl

ntietz.com blog - technically a blog — Fri, 15 Dec 2023 02:00:00 GMT

I'm proud to announce that Hurl is officially released and done! You can check out the docs on hurl.wtf.

The language itself came out of an interesting question: Python sometimes uses exceptions for control flow, so could we implement a language that eschews normal control flow and only uses exceptions? The answer is yes, and it produces a language that's less bad to use than I expected^[1]!

In the process of implementing it, I learned a lot. Next year I'm going to try to make another language to learn about type systems, and that one should be more normal (but no promises). Here are a few of my main takeaways from building Hurl.

Working without control flow is... fine?

I thought it would be totally mind bending to work without ordinary control flow. The first couple of programs were mind bending but then you just learn the common patterns. If you need to do an if-else, that's a catch (true) ... catch (false). Looping is harder to wrap your head around but it's also not so bad.

And the thing is, this is a general purpose programming language, so we can build this control flow. I ended up with a function called if that takes a condition function and a body function as variables, and it runs those. So you can write code like:

if(func() {
  hurl year == 2023;
}, func() {
  println("Hurl was written in 2023!");
});

It's not as clean as an if in reasonable languages, but it's also cleaner than I expected. Part of this is also because I used dynamic scope, not lexical scope, so functions can operate more easily over outer scopes, but it would be doable either way with minor changes.

This has me really excited to explore things like assembly. I've been really intimidated by it my entire career, feeling inadequate and all the usual impostor feelings. But now I can see concretely that eschewing normal control flow won't be a problem in itself^[2]. I've had The RISC-V Reader on my desk for a while and now it seems more approachable.

An unexpected lesson for me, but I'll take it. So expect to see some RISC-V content next year!

All the nice things are so hard

My ambitions for Hurl were larger than my skills and time allowed for, and I had to pare it back. The things that got cut were any of the tools that would make the language fairly nice to use, and error messages unfortunately went by the wayside.

I realized this would be harder than expected when I started writing the formatter and then I started to rethink some of the other ambitions I had.

In a future language I will come back to some of these things. I'd love to write a slightly more sophisticated formatter that makes things a little prettier (though having one at all is an accomplishment I'm proud of). And I am really interested in exploring writing a language server for a homemade language next year, but this year I just could not work on it.

The big thing I have a lot of appreciation for now is the quality of error messages and debugging support in other languages. Generating error messages that point to where the error happened in the executing program requires that you track all the line information at run time! And that means you have to design it in from the beginning. Guess who didn't realize that and made some mistakes that would've been trouble to fix later?

Yeah, all those things that make a language nice to use are just a lot of work. And my promise (to myself, to you) is that my next language will work at these and the goal will to be fairly pleasant to use as far as educational language projects go.

Writing your own parser/tokenizer can makes sense

After working through Crafting Interpreters in 2022^[3] where we implemented the tokenizer and parser from hand, I left with the impression that this was an intentional choice for instruction but perhaps not how we'd do it for real. Since then I've learned that a great many languages do roll their own tokenizers and parsers, and I still wasn't sure why.

I used pest to generate my parser for Hurl, and in a lot of ways it was pleasant to use. But on the other hand, it felt fairly restrictive and it was a lot to learn. In the end, I'm not sure that I saved time over writing my own parser.

If you write your own tokenizer and parser, you get full control and you avoid adding another dependency. They're also not that difficult to write (but there are a lot of details to get right, so they're tricky to get fully correct).

I would probably use pest again for a real project^[4] and it's used by some quite respectable projects like mdbook. But for for my next language, I'm going to write my own tokenizer and parser again. It's pretty fun, I don't think it'll cost me much time, and yeah why not?

Relying on the OS stack was a big mistake

I implemented a tree-walk interpreter, and recursion^[5] is the only way you get looping, which results in a problem: as you loop, you push onto the stack and you get stack overflows. And it's the OS provided stack for the interpreter which you end up blowing, so this isn't something we can just patch in Hurl itself. This isn't something I thought through critically before deciding to do a tree-walk interpreter. I knew that these are limited in some ways, but forgot how it would impact Hurl.

One solution here would be to migrate to a bytecode interpreter. That's a big project and would be a rewrite of the whole interpreter, so it's not in the cards. There might also be a way to optimize out some of the recursion here and create a loop from something tail recursive, but I don't know.

Another solution would be to add a new language construct. I'm not sure which language construct would help us out here, so it's an undetermined thing at the moment.

In the future, I'll make sure to account for this from the beginning and use a bytecode interpreter approach, or transpile to another language.

Licenses can be fun

Software licenses don't have a reputation for being particularly, uh, exciting^[6]. But they don't have to be boring! They can be an opportunity for play, too.

Part of creating Hurl is art, and the license choice is a big part of that. The best license choice would have been an OSS license and then later do a rug pull and relicense as BSL, but that would imply that this project would get any attention. The second best license choice was to lean into my values intentionally. I ultimately decided to pick a license that would:

permit funny outcomes
allow educational use
reflect my morality and ethics

And to do this I settled on not just a license. Not dual licenses. No, that would make sense.

Hurl is triple licensed.

You can choose which of the licenses applies. You've got the standard AGPL-3.0 (no "or later" here, I don't want to be bound to the FSF). You've also got the choice to buy a commercial license (serious inquiries only 😉). Or you can use it under GAL-1.0 (the Gay Agenda License 1.0). Here's that license, in its full glory^[7]:

# Gay Agenda License - 1.0

Copyright (c) 2023 Nicole Tietz-Sokolskaya <me@ntietz.com>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

  - The above copyright notice and this permission notice shall be included in
    all copies or substantial portions of the Software.
  - The grantee shall actively support rights for all LGBTQ+ people, respecting
    their gender identities.
  - The grantee shall say "be gay, do crime" at least once during use of the
    software.

The license is immediately revoked if the grantee supports restricting the
rights of LGBTQ+ people.

If the grantee is found to not have said "be gay, do crime" during use of the
software, the grantee has thirty (30) days to remediate this violation without
loss of the license. If it is not remediated, then the grantee's grants via
this license are premanently retracted.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Feel free to use this software for your own projects, with a few caveats. It was modified from the MIT license, so most of it is boilerplate, with a couple of additions to it. The main thing to note if you do for some reason decide to use this license is that it is not tested and I would probably be surprised if it's enforceable. No lawyer has been involved or harmed in its creation.

At the end of the day, though, what license is enforceable if you don't have the money to fight Amazon on it?

Playing is very educational

This was the biggest takeaway. It's not new to me, and I've written before that you should write more "useless" software. It was a great reminder of the joy and learning that can come from a long project that's just for fun and that has no practical value.

Along the way, I had a lot of fun. I'm not sure if I made friends or enemies. And I sure did learn a lot. Some of what I learned, I can apply at work starting this week^[8]! And some of it is just added context for why certain things are hard, and makes me more deeply appreciate the tools that our dear language teams give us ❤️.

Go forth and write some playful code!

It is still pretty bad to use, though. ↩
In fact, assembly might get us a little closer to ordinary control flow than Hurl does. ↩
Highly recommend this magnificent tome if you want to learn from a professional language person. And his illustrations are beautiful! ↩
I do have something brewing at work that will possibly use it, or nom. ↩
Despite the relation of names, the Recurse Center has no fault in the creation of Hurl. The people there are quite lovely, and most don't implement languages like Hurl! ↩
To all my lawyer friends out there who are reading this and vehemently disagree, reach out to me, would love to chat. ↩
This license was inspired by boringcactus's post An Anti-License Manifesto. ↩
Jessica, don't worry, I'm not going to actually use Hurl at work. ↩

Offline developer docs: Dash docsets via Zeal

Jiby's toolbox — Fri, 15 Dec 2023 02:00:00 GMT

Since my earliest university days, I’ve spent a lot of time trying to get better at computers, immersing myself in the world of Linux, and generally improving my software development skills.

One of the aspects that was the most helpful to help me grow was attempting to replicate my normal dev workflow, but with less: Learn to operate on Linux not Windows, then learn to do dev via on the terminal instead of GUIs, and eventually learn to do the same but offline, without any network calls.

Lahaina four months after the fire

BillSaysThis — Thu, 14 Dec 2023 22:40:21 GMT

If you want an overview of how things are in Lahaina four months after the fire, check out Jesse Wald’s video. TL;DR: The EPA just completed the hazardous materials removal project and now the main debris removal will start and take about a year. The devastation is just so visible now, makes me so sad. … Continue reading

Phobias Good and Bad

Synystron Synlogica — Thu, 14 Dec 2023 20:16:11 GMT

People should be more into the active *application* of "jerk-o-phobia" than be obsessed with many of the other so-called (and arbitrary!) phobias they like to cite.

hashsets in go

blog — Thu, 14 Dec 2023 15:15:50 GMT

How I learnt to run my own personal AI

I Learnt — Thu, 14 Dec 2023 14:00:00 GMT

By now most people have tried or at least heard of ChatGPT. It is now relatively easy to run your own AI instance on your own machine.

How Many Hobbits?

Nicola Iarocci — Thu, 14 Dec 2023 08:05:25 GMT

For those who don’t know me, I’m a demographer. I study population. And my first love in fantasy was, of course, Middle Earth. How many people live in Middle Earth? Being a demographer, I was mainly interested in the data side of things.

Tolkien is frustratingly vague about population. He almost never gives us estimates of settlement sizes, and many of the larger metropolises of Middle Earth (like Pelargir) never actually appear on the page. Sizable armies make frequent appearances, yet because his adventurers almost exclusively traverse the wilds of Middle Earth, we rarely see where those soldiers are coming from.

The Danger of Metrics

Zac Szewczyk — Thu, 14 Dec 2023 03:32:15 GMT

I got an email a few weeks ago about my article Operational Metrics: The Next Step in the Evolution of Defensive Cyberspace Operations¹ from a researcher studying the same problem. We talked about my article, some of the other writing I have published on this topic, and and then something I briefly touched on in my SOC Metrics series: the dangers of metrics.

Permalink.

Pinecone & Pulumi webinar: Taking AI apps to production

Zachary Proser — Thu, 14 Dec 2023 02:00:00 GMT

I co-hosted a webinar with Pulumi about productionizing AI applications

Malapropism

Studiofreya SSG Site — Thu, 14 Dec 2023 02:00:00 GMT

Ludicrous misuse of a word, especially by confusion with one of similar sound.
An example of such misuse.
The act or habit of misapplying words through an ambition to use fine language.

A malapropism (also called a malaprop, acyrologia, or Dogberryism) is the use of an incorrect word in place of a word with a similar sound, either unintentionally or for comedic effect, resulting in a nonsensical, often humorous utterance.

https://en.wikipedia.org/wiki/Malapropism

Quickdraw Hero

Backendhance — Thu, 14 Dec 2023 02:00:00 GMT

Hi,

Beim Team einer meiner Kunden habe ich den Spitznamen “Revolver-Held” bekommen.

Weil ich so schnell schieße.

Den Ruf bekam ich relativ schnell vom Team. Zu dem Zeitpunkt war er durchaus despektierlich gemeint.

Das Team war mein Tempo nicht gewohnt.

Es hatte es sich gemütlich gemacht. Die Software ist seit mehr als 10 Jahren in Entwicklung. Und über die Zeit haben sich hohe technische Schulden angesammelt. Das war ein schleichender Prozess.

Running Python on AWS Lambda, Revisited

Blog - Keith R. Petersen — Thu, 14 Dec 2023 02:00:00 GMT

It’s been almost a year since I got Python 3.10 and 3.11 running on AWS Lambda before official runtimes for those versions existed. Amazon has since caught up with the Python release cycle, and now has runtimes for those versions, while a 3.12 runtime release is imminent.

Ultimate Beneficial Ownership is Dangerous

Omar Abid - Personal Blog — Thu, 14 Dec 2023 02:00:00 GMT

Examining how transparency laws can affect the balance of power.

Learning to ride

An Ordinary Disaster — Thu, 14 Dec 2023 00:12:12 GMT

The three-mile circuit around the docks where my floating home is tied up takes about an hour. I make my way up to the end of each one and wheel around to the next, sucking in the salt air and the winter light that slants in over the headlands, checking the minus tide, the mud exposed in the flats, listening to the birds out searching for grub, giving each neighbor a nod and a smile as I march along.Passing though the shed at the entrance of Issaquah, I eye the giveaway shelf and snag a secondhand issue of The New Yorker, the front-cover illustration of a family gathered around the holiday table, every face turned screenwards, scrolling.

Dwell and the Tools for Thought Hackathon

swyx's site RSS Feed — Wed, 13 Dec 2023 22:47:31 GMT

I worked on a little hackathon project at AGI House with Sasha. We won!

Riding the Storm Out

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Wed, 13 Dec 2023 22:00:00 GMT

After a successful local meetup for the ‘Learning on Graphs’ Conference, I found myself almost stranded away from home. It had already been snowing a little bit¹ over the previous evening, but upon waking up, I was greeted by a lot of notifications concerning cancelled trains, flights, and the like. A quick glance outside showed that more snow had fallen over night, resulting in a non-Euclidean transformation of the city. Where previously ambient and geodesic distance were at least correlated, the snow had created a metric with singularities that might even stump a Perelman. Public transport was stopped for the time being. Walking was out of the question.²

It looked like I might not reach home after all, but I had a strong incentive to do so since my wife was home alone with our two-month old baby. Fortunately, I love improvising and thought that the situation, while not exactly optimal, could also be much, much worse. I thus walked into the nearest hotel and pretended to be a guest, trying to order a taxi into the inner city. The reason for this non-obvious destination was that I suspected that the airport would be overwhelmed or close to being so anyway in a couple of hours, whereas there might be a chance for trains to run at least in the rough direction of ‘home.’ Seeing taxi after taxi leave for the airport, I started approaching some other stranded guests who were putting their hopes into the train system. Together, we got a taxi van that was willing to drive us into the city.

Along the way, I briefly discussed travel plans with my fellow travellers; upon checking their itinerary, I found that we all shared a rough direction but, alas, no trains were running any more at this point. I thus made the, somewhat jocular, proposition that we should just drive to Zurich. Much to my surprise, this suggestion was taken up by two of my fellow travellers, both execs in a highly-competitive industry. They were motivated to get home as well, home being Nice in this case, and, luckily for me, extremely experienced in such mishaps. Indeed, one of them introduced his job as being a troubleshooter. Our troubleshooter was able to negotiate successfully with the driver, while I served as the translator, and then we were on our way. This was a lot easier than I anticipated!

During the drive we mostly encountered the road in good condition, with some minor traffic jams. Next to the beautiful winter landscape—everything can be beautiful if it does not pose an immediate inconvenience or threat to one’s own plans—the thing I remember most vividly are our conversations. While not exactly like the Canterbury Tales, the second manager, S., turned out to be an outstanding conversationalist, engaging everyone of our small fellowship and drawing some quite interesting stories out.

He also shared some lessons from his long career, which struck a chord in me:

When you are in a problematic situation with imprecise knowledge, try to move as fast as possible in any direction that strikes you as better. You can always adjust course along the way.
The fact that we can charter a taxi at no small cost without batting an eye makes us already highly privileged in comparison to most other human beings.
Our biggest inconvenience is getting home during snowfall; later on, this will make a good story. Many others are fleeing bad economic or climate conditions and might not face such a happy end.
‘Ask and you shall receive:’ We would have never gotten our little fellowship together if we had not asked and improvised.

This was a good trip and, indeed, a little bit of an adventure. I am grateful for the lessons and the shift in perspective. Until next time, stay safe in your travels!

Meaning that all walkways and streets were slightly powdered but no more. ↩︎
I am a pedestrian by nature, but trudging about 20 kilometres through the snow to reach the main station from the conference site did not seem like a viable option. ↩︎

Trader Joe's Parking Lot

Josh Thompson — Wed, 13 Dec 2023 15:00:00 GMT

Hey Trader Joe’s,

This is a bit of an open letter, inspired by a recent visit to the local Trader Joe’s. I just moved to this part of Denver, and now for the first time am living within like a 3 minute scoot of a Trader Joe’s.

I know that some people like to complain about parking at TJs. (Can I call it that? Trader Joe’s -> TJ’s?)

Here’s a sampling from Reddit: https://www.reddit.com/r/traderjoes/search/?q=parking

Lots of people mis-understand parking lots, and so no matter what happens, someone’s going to complain about something. I sort of love that Trader Joe’s has smallish parking lots, and would love to scheme to make them even smaller, while making the users of the parking lot and visitor’s to TJ way happier.

This is something that can be done. It is already done in certain ways around the world. Different countries have common, accepted solutions to different portions of this “parking lot problem”. I’d like to work with TJ’s to cobble together an off-the-shelf solution to all/most of the issues.

Obviously, there’s a lot of money to save, hassle to reduce, delight to be had, money to be made, by better managing business-critical capital, and parking lots are critical to the functioning of businesses.

Lets talk about it.

Email me, we can email/phone/IRL discuss it. joshthompson@hey.com

❤️,

- Josh

Tiktok footage of a local TJ’s parking lot

Here’s footage to whet the appetite.

Sometimes embeds show up strangely, so here’s direct links. I recommend clicking both:

@josh_exists lots to appreciate here. the audio is great I have thoughts on parking lots. I see them as hidden sources of abundance, currently not allowed to function as well as they might. a little love goes a long way. #fyp #dji #traderjoes #toyota #ford #denver #colorado #timelapse #carlessdriver #driverlesscar ♬ Final Speech - From "The Great Dictator" - Charlie Chaplin

@josh_exists I'm gonna send this to someone at trader Joe's. I'll explain the book 'the high cost of free parking'. #traderjoes #parking #drone #dji #timelapse #denver #colorado #freeparkingisnt ♬ original sound - josh

Online comments about Trader Joe’s parking lots

Here’s what some online commentors have said about the parking lot:

I stopped going to Trader Joe’s weekly because of the parking lot situation. I knew it was only a matter of time that I would forced to murder someone in a Prius over a parking spot.

I drive past mine somewhat often and only go about once a month when I see it’s emptyish. Tend to buy more and go less frequently

I live equal distance from two Trader Joe’s. The parking lots are decent compared to the one in Sherman Oaks. One of the local Trader Joe’s has a security guard who will assist the elderly getting out of the parking spots. That makes me return to that specific location.

I drive to a TJ further away because that one has more parking. The closer one is in an extremely dense neighborhood, and there is no guarantee you will ever park anywhere close to it.

source, reddit

what managing a parking lot skillfully looks like

Properly managed, a parking lot will always have about 10% capacity available. This means someone wanting to quickly enter, dash inside, and leave, will be able to do so.

Ensuring availability means using price/cost to mediate demand.

Donald Shoup wrote a book The High Cost of Free Parking, and has a standard recommendation for fixing parking. It would be preferable of american municipal planners properly managed on-street parking, but they don’t, so you, TJ’s, with a bit of my help, have to make up the difference.

Donald Shoup says:

charge appropriately for on-street and off-street parking
Spend the collected money exactly where it’s being collected
Remove parking minimums

TJ’s is obviously operating with different constraints than municipal planners, and I appreciate and can accommodate that. There’s plenty of improvement to be had, and TJ’s is in a better position to make this change than nearly anyone, because:

car-heavy user base
very busy stores
limited parking

[Hμblog] Libreboot 10-year anniversary

devever.net/~hl — Wed, 13 Dec 2023 10:11:09 GMT

Libreboot 10-year anniversary

Leah Rowe has written an interesting article about the history of the Libreboot project on the project's 10th anniversary. Recommended reading if you're interested in the open source firmware ecosystem.

Optimizing Ship Load-outs for Highfleet: Problem Setup

jodavaho.io — Wed, 13 Dec 2023 02:00:00 GMT

Over-engineering Highfleet load-outs using Integer Programming

I built a highfleet ship optimizer by bundling a fast C optimizer, binding to a higher level language, and providing a web API. Then, Altho built a front-end. Try it at hfopt.jodavaho.io/optimizer

This post is about setting up the problem. See related posts.

Dieselpunk warship schematic. Made with Midjourney.

Highfleet is an incredibly fun game. You take control of a small set of ships, raid and pillage and revolt your way to glory. Along the way you must carefully manage your resources to keep your ships repaired, and customize them to the situation at hand. Lovely time.

The most addicting part, as is common in this genre¹, is the ship design and customization. You can fully tailor your fleet to your desired play style! Instead of rave about the gameplay, let’s talk about how you design ships.

Ship Design Problem statement

In Highfleet, you can “pre-design” ships before starting a campaign. You have a limited starting budget, so you need to balance power, weight, engines, guns, ammo types, layout, crew capacity, sensing, etc, to get the desired range, speed, firepower, survivability, and combat time, while minimizing cost. This problem is masterfully posed by the developers, and has hooked a small community for years.

I’m interested in automating this, to a certain extent. Can we at least come up with an optimal module set - and leave the layout / cosmetics to the creative user? Staring at the above paragraph, one might see this is a classic optimization problem². Words like “minimize”, “select”, “all while …”, “such that”, etc are clues.

We can re-state above as:

Select zero or more whole modules from a list
Such that the modules have all their requirements satisfied (e.g., engines have fuel, guns have ammo)
all while minimizing cost
and satisfying some desired final ship stats

Easy. Formulating the problem is half the battle.

Solving

To solve a problem like this you need the defining equations:

A “cost” function. In this case, it’s literally the dollar value of each module - easily obtained by writing down the values from the game. This is also called the “Objective Function”.
The “feasibility constraints”, which essentially answer: Is this a valid ship at all? Without these, we could choose modules that seem to satisfy constraints, but would never fly in-game. For example: “Do we have enough ammo feeds to keep the guns working”, this again we can copy down from in-game stats
The “trade space equations”. In this case, we want to know the following: How does the selection of modules affect the desired final ship stats like Range, Speed, Firepower, etc. This turns out to be very hard, but doable with help.
We also want the “desired stats”, which are just “constraints” we impose on the trade space. We’ll get this from the users as a set of valid ranges (e.g., speed is greater than X, weight is less than Y, blah blah)

Great - all we need to do is reverse-engineer how the in-game stats are calculated. easy right? Instead, I reached out to the game developer, and r/highfleet, to see what they could share. Everyone had some models, and Koshutin didn’t reply. So, I had to do it myself for the most part.

Reverse Engineering the Game Stats

For completeness, here is the list of relevant stats, obtained by parsing the ship files (which deserves its own post).

aavalue, airspeed, ammobox_need, ammobox_total, at_left, at_right, b.x, b.y, combatvalue, crew_capacity, crew_need, dynamic, dynamic_landing, dynamic_map, dynamic_map_repaired, firepower_guns, fss_capacity, fss_total, fuel, fuel_capacity, fuel_need, fuel_need_cr, fuel_on, hp, hp_integral, hp_max, mass, parts, parts_integral, power, power_need, power_real, power_total, power_total_repaired, price, r, signature, signature_ir, signature_rd, thrust_left, thrust_left_landing, thrust_left_map, thrust_left_map_repaired, thrust_right, thrust_right_landing, thrust_right_map, thrust_right_map_repaired, weapons_on

Some we can throw right out. b.x, b.y for example make no sense to me. Some (weapons_on) probably have more to do with the campaign mode than the performance. Some (airspeed, r) are exactly what we want to reverse engineer and optimize for.

My explorations were fairly crude. I’d download many of the fan-made ships from reddit or discord, check the stats, make a spreadsheet, and start plotting to see if the relationships were obvious. Some guesses were obvious (airspeed should be a function of thrust/weight, which it was). Some were not (range (r?), in particular).

For forming the equations, some unit analysis really paid off. For example, range ultimately (sensibly) was essentially:

$$ r \approx C_1 \cdot \frac{ \text{fuel_capacity} }{ \text{fuel_consumption} } \cdot \text{airspeed} + C_2 $$

which has units meters- note the unknown constants though³. Determining the constants is a job for linear regression! I won’t plot out the details, you can see them here.

At this repo you can see how it worked. Note this file with ship data, hand crafted! It has a sparses set of stats manually pulled from ships. Just enough to confirm the relationships. Once I had those I could pull up any ship from reddit or discord, and verify it. It was always close enough. Of course, I have to re-do this every time the game balances, but soon, I’ll automate this - another post to write.

from study notebook, showing combat time regression

I did discover some hidden features this way, did you know generators require fuel? Makes sense, but at the time this was undocumented.

So, that’s how I was able to pin down the relationshps for the “desired stats” from the game files. Next, we can formulate the optimizer.

Mr. Koshutin, if you are reading this, thank you for letting the community enjoy your game in this way. You certainly didn’t have to!

There is a pseudo-genre of such games, and highfleet stands alone for its unique setting and fun combat. ↩︎
There are some softer constraints, too: Does it look cool? Does it land well? These are subjective and not well posed for optimization, so I ignore those (for now!). ↩︎
One might imagine that the constants fall out of the in-game stats, but they do not. I suspect this is because “fudge factors” are applied. ↩︎

Moral Hazard

Craig Pardey — Wed, 13 Dec 2023 02:00:00 GMT

A moral hazard is when an..

…actor has an incentive to increase its exposure to risk because it does not bear the full costs of that risk.

It’s similar to perverse incentives.

A perverse incentive is an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers.

I have spent the last 6 months working closely with a QA team inside a large financial instutition and have become increasingly skeptical of the value they provide - the development team don’t test as thoroughly as they should because they know that QA are playing backstop.

Go Libraries/Packages I Like

Ben E. C. Boyter — Wed, 13 Dec 2023 02:00:00 GMT

In no particular order a list of Go libraries/packages I really like and some reasons why.

zerolog I know that the changes to slog make it far closer to being what zero log is now, but I still prefer it for my own use.
requests Anyone who works with python for a while learns about the requests library for making HTTP calls. This is similar but for Go. It has a few sharp edges when dealing with websites that don’t follow standards, but otherwise is amazing. Coupled with the next library in the list you end up with a pretty solid way to call endpoints.
trip HTTP client middleware that allows automated retries, with backoff’s among other things such as authorization. Coupled with requests it really cleans up your HTTP calls.
lo If you know of the wonderful underscore.js library then you probably have an idea of what this is. Similar idea, providing a lot of helper functions that allow dealing with some of Go’s sharp edges more easily. It does abstract away things that you might need to know about if you need absolute performance, but for most things it just makes life easier.
templ HTML template language for Go. There is nothing you can do in this that you cannot in Go templates, but you get compile time checks of the code rather than actually having to run it. It also means you don’t need to worry about embedding template (although this is fairly easy these days anyway). Works really well with HTMX.
gjson A simple way to get values out of JSON without needing to unmarshall into a struct. Really useful for tests, as well as annoying API’s that have mixed data types.
mux The inbuilt Go router isn’t horrible, but is painful to use. This one supports path variables, and you can specify which http methods it uses.
sqlc Generate type safe Go code to access your database. Of all the DB layer abstractions I have used this is best. Its similar to something I wrote in PHP over a decade ago, although much better.
xurls The number of times I have gone looking for a URL regex only to find it misses some edge case… This solves a lot of that pain, especially when it comes to extracting them out of text.

The below are things I am exploring, but not sold on yet.

Using the BlueSky API to change your Handle to your Domain Name

Pxtl.ca — Wed, 13 Dec 2023 02:00:00 GMT

Personally in the post-Muskening twitter world, I'm pulling for Mastodon, but Bluesky is a somewhat interesting platform too. One feature of Bluesky that stands out is that you can set your name to be based on a domain-name you own. I own Pxtl.ca, so I thought I'd take a crack at leveraging that feature.

What follows is a guide on how to do this in Powershell, which is a programming language you already have installed if you use Windows.

Demystifying OpenID Connect's State and Nonce Parameters in ASP.NET Core

Tore Nestenius — Wed, 13 Dec 2023 02:00:00 GMT

In the world of web application security, OpenID Connect plays a key role in streamlining authentication processes. But what makes it really tick? In this

2023 Recap

Reiterations — Tue, 12 Dec 2023 20:00:00 GMT

I’ve been blogging for three years now. Here’s a look back at what I did for 2023.

Getting Started with Tyranids in Warhammer 40k 10th Edition (Dec 2023)

valbaca’s blog — Tue, 12 Dec 2023 18:03:38 GMT

Intro

Genestealer Cults Rules FAQ (May 2024)

valbaca’s blog — Tue, 12 Dec 2023 18:03:38 GMT

Intro This post goes over the Frequently Asked Questions about Genestealer Cults Rules. I plan to write one about the faction itself later; as in, what models are good etc.

How I learnt about not compromising

I Learnt — Tue, 12 Dec 2023 14:00:00 GMT

I have just finished re-reading the book The Spark, a fictional account of a sports agent and his journey to learn about Cirque de Soleil from the inside.

What I Like About Nix

Davis Haupt's Blog — Tue, 12 Dec 2023 14:00:00 GMT

I got a new computer recently and decided to take the plunge setting it up with Nix¹. I wrote up a snippet on how I set everything up and you can find my whole configuration on GitHub. I’ve only scratched the surface of what Nix can do – In this post I wanted to focus on my subjective impression of Nix so far and why I feel I’ve been enjoying it.

Long-term, shared memory

No need to worry about remembering what switch you flipped in settings six months ago after reading an OSXDaily.com post, or what command you ran in a shell at 3am the morning you unboxed your new computer. Any change you’ve made to, for example, hide the dock by default or allow for repeating keys in VSCode and other editors is there for later review.

On macOS these system preferences are normally either configured in the point-and-click System Preferences interface or specified in loosely documented defaults write invocations that you can find all over the internet. nix-darwin is a Nix tool I’m using that lets you configure virtually all of these settings from a Nix file.

Making your own configuration declarative is good enough, but on top of that people have built up modules and abstractions around more complex configuration like enabling sudo to use TouchID or running daemons through launchd that you can enable with a single line of code.

A complete picture of your environment

All of my dotfiles, editor themes and extensions, and command-line programs are defined in a single repository. It makes it a lot easier to see how it all interacts with each other and hopefully avoid situations where different dependencies conflict. No one wants to end up here, but it happens all too often, at least to me:

Synchronization between machines

I have a desktop and a laptop. They’re now running virtually the same Nix configuration, out of the same repository, with a few tweaks in host-specific overrides. I can work out the kinks in getting Emacs to play nicely with the Yabai window manager on my desktop and have it all work on my laptop in a matter of minutes once I pull down the updated configuration repository and reload my Nix environment.

I should mention these two machines are running different versions of macOS, and virtually everything has worked flawlessly across both.

Reducing manual toil

I’ve used Karabiner Elements for years to remap some keys on my keyboard. More recently I’ve moved my configuration over to Goku, a DSL for Karabiner on top of the JSON configuration format. Every time I wanted to change my configuration I would need to remember to run the goku command. Now that command is run automatically by home-manager whenever my karabiner.edn file changes. When I change my skhd or yabai configuration, Nix will automatically restart those services so they pick up their new settings.

Standardizing the refreshing of all configuration to a single command (darwin-rebuild switch) and running everything only when it’s needed has been a small change that’s significantly reduced the cognitive load of having a lot of subsystems that may require new configuration, occasional restarts, or other babysitting from time-to-time.

Conclusion

If the entirety of Nix was a DSL for a package manager and system configuration system I’d be satisfied. But there’s so much more to Nix, from development environments to production artifacts. I’m excited to dive more into those corners of the ecosystem going forward, and hopefully write about it too both here on the blog and over on the TIL page!

Nix isn’t the easiest thing to introduce since it can refer to a few different distinct components or the whole ecosystem. In case you haven’t heard of Nix before or just want to learn more, I find the conceptual documentation on Zero to Nix to be more approachable than the official docs. ↩

📌 My other ramblings on the world wide web

zufallsheld — Tue, 12 Dec 2023 13:55:00 GMT

Here I list my other writings that I published and talks that I held.

Writings

Visualizing your AAARRP priorities as a way to manage up in your DevRel team

Julien's DevRel corner — Tue, 12 Dec 2023 11:46:10 GMT

In this article, I'm describing how we used the AAARRRP framework to visually describe our DevRel team's strategy and clarify the reason for our day to day activities to our internal stakeholders, and more focus on impact instead.

Mastodon with docker rootless, compose, and nginx reverse proxy

du.nkel.dev — Tue, 12 Dec 2023 02:00:00 GMT

Everyone talks about docker rootless, e.g. for security reasons, but no one actually seems to do it. Very little information is available. Migrations from rootfull systems? File permissions? Reverse-proxy setup? Here I go through the steps to run Mastodon

Capit – AI Social Caption Generator

Swift Fox Software LLC — Tue, 12 Dec 2023 01:45:00 GMT

Announcing my latest app – Capit! The app automatically generates social media captions for Instagram, TikTok, X/Twitter, Reddit, LinkedIn, Pinterest & more. Just pick an image or video, choose your tone, and you will be given five different caption options. I’ve got a lot of features planned for this app, this is just the beginning!... Read more »

The post Capit – AI Social Caption Generator first appeared on Swift Fox Software LLC.

UK to Curtail Legal Recognition of Trans Immigrants

rya.nc — Tue, 12 Dec 2023 00:12:00 GMT

The UK's “Minister for Women and Equality”, Kemi Badenoch, who openly espouses hate for LGBTQ+ people, has said that many of these countries and territories no longer meet British standards. This is because they have followed the overall world-wide trend of improved rights for transgender people, which the UK opposes. The precise line between political scapegoating and bigotry on this issue is unclear.

Building a Clojure CI/CD pipeline of CERTAIN DOOM

Janet A. Carr — Mon, 11 Dec 2023 21:55:21 GMT

When things go wrong

Devops is one of those important positions where nobody cares what you're doing until things stop working. Along with SRE folks, the position may as well be akin to a firefighter. I know this first hand having been a loaded-title myself. You haven&

Note 123

Qubyte Codes — Mon, 11 Dec 2023 19:15:10 GMT

I’ve been solving #AdventOfCode23 with Swift this year, it’s not been completely smooth sailing, but I really enjoy the language.

TIL how to create Files and Commits via the Github-API and Github-CLI

zufallsheld — Mon, 11 Dec 2023 14:55:00 GMT

Recently, I found myself needing to incorporate a CODEOWNERS-file to multiple repositories within our Github organization. The objective was to automatically reviewers for pull-requests. And codeowners are a perfect tool for this. This way every colleague in our company can join our Github organization and get write-access to the …

How I learnt to pick ideas that win

I Learnt — Mon, 11 Dec 2023 14:00:00 GMT

I come up with lots of ideas for potential applications, businesses, stories and random other things.

The simplicity of the modulo operator: how I scaled an inefficient solution on a legacy system

./techtipsy — Mon, 11 Dec 2023 06:00:00 GMT

Sometimes the best solution is the one that works well enough and can be cranked out quickly.

ELI5 Physicists Say That Information Is Never Lost but How the Heck Is It Stored Then and What Theoretical Possibility Could Recover It

Studiofreya SSG Site — Mon, 11 Dec 2023 02:00:00 GMT

https://www.reddit.com/r/explainlikeimfive/comments/18g0nry/eli5_physicists_say_that_information_is_never/

The difference between a pile of ash and a piece of paper with a word written in ink is information. I get that part.

What I don’t get is why they say it is absolutely critical to physics that information can never truly be lost, thus the problem with black holes potentially destroying information.

And what I really don’t get is HOW that information is stored, and what possible theoretical mechanism could be employed to recover it.

Say we have a sheet of notebook paper in a glass box. Someone has written a word on that paper with a black Bic ballpoint pen. The box is sealed and the paper is set on fire burning it to a crisp. We then shake the box and the ashes are powdered and scattered.

Every atom involved is inside the box.

Where is the information and theoretically how we recover it and find out what the word was?

Applied anthropology

nutcroft — Mon, 11 Dec 2023 02:00:00 GMT

On my way to an anthropology lecture today, I learned much more about humans than I could expect of the lecture. To clarify: the content of the lecture was great yet, with respect to understanding humans, engaging in applied anthropology is inherently unsurpassable.

While I was cycling through Upper Street in Angel, there was a couple (or maybe three) high-end SUV automobiles, all of whom driving aggresively. The engine revving scared me. I was in the front. The car propelled ahead and predictably and immediately stopped at the red light 30 meters away.

I’ve always dreamt of facing these people who drive aggressively — and I have a number of times — not with the intention of insulting them — purely out of curiosity and to let them know the effects of their actions. Admittedly, at times I want to swear at them but it wasn’t the case today. I was calm and in a good mood.

“Can you please not drive so aggresively? It’s scary!”, I told the guy in the front passenger seat with zero bad vibes. “We don’t care”, he said, matter-of-factly, almost satisfied that they don’t. I laughed. “You don’t? Why not?”. “We just don’t… Do you want me to come outside?”. The guy at the back opened the door with the intent of restoring justice with bodily interactions. My bike, however, was in the way and I think he didn’t want to traumatise his friend’s glossy car by forcing his way out. The driver delivered his own statement too: “Oh, fuck off!”. I looked deep into his eyes with the intent of knocking on the door of his soul. He didn’t let me in. I smiled and discretely waved goodbye while cycling away.

To start with, I was shocked how fast things escalated. I wasn’t angry at all and wouldn’t want to fight; even though, I must confess, it has occured to me multiple times that it would be satisfying to play punches with aggressive car drivers — and cyclists too.

In retrospect: how should I have responded? Ideally, what should I have done? Practically: what will I do next time there is an aggressive driver behind me revving the car in high traffic central London?

Do I want to punish them? No, there is no point in punishment. Do I want to change them? Maybe, but first: do I understand them?

I think it was some kind of celebration. There were many similar cars honking and revving, all of them fancy SUVs, and I think also people talking with each other. It’s ok to make some noise when one is celebrating something, right? I think society can accomondate that and I hope society can accomondate that. I can understand and I have experienced something like one’s team winning at a sport and everybody shouting because of that. An honest and authentic exclamation of excitement.

Then a random guy on a bike comes up and says “don’t drive aggresively”. Like someone sober at a party at 3 am asking what time everybody is planning to wake up tomorrow.

Ok — we can accept celebration but what is much more common is that people show off their beloved cars by forcing everyone to pay attention to them. Not to mention scaring everybody else around them, either driving, cycling, or, of course, walking. How do we respond to them? Let’s say the same things happens again: I politely ask for less aggression and someone comes out of the car with intentions of touch.

One option would be to accept them as they are. Maybe it’s ok for people to honk and rev their engines once a while — what’s the harm? But it's kind of an egotistical act at everybody else’s expense.

Another option would be to accept that one cannot do anything the moment it happens. The solution is a process that takes place other times. Maybe that’s true and maybe I can conclude to this. But something doesn’t feel right with this conclusion either. There is something missing: the fact that the feeling I’m left with after an engine revving is the feeling of being silenced.

Here’s where I conclude. Next time I encounter a revver, I will come up to their car and politely ask that they stop driving aggressively. Then, they say “fuck off” and proceed to exit the car to fight. At that point I avoid fighting by running around while asking them to consider their actions and reflect on the fact of whether their actions are justified. Does a simple ask warrant punching me? I didn’t even throw an egg to their car, which I have sometimes considered.

However much both the driver and myself would love to fight, this would be a loss for everybody in society except for our ego. It is exactly for this reason that I want to resist punching them. It is exactly this kind of urge one must fight off to achieve democracy. This, along with answering the question: how can we help this person feel they were listened to, that they matter, and that they are loved without the need to prove physical overpowering?

I’m sure one day we’ll figure it out.

A Picture Is Worth a Thousand Words

Backendhance — Mon, 11 Dec 2023 02:00:00 GMT

Hi,

“Ein Bild sagt mehr als tausend Worte”

Dieses Sprichwort habe ich schon früh gehört. Vermutlich das erste Mal in der Schulzeit. Es ist eine Weisheit, die wir schon lange weiter tragen.

Vielleicht verbindest du dieses Sprichwort eher mit Bereichen wie Marketing, Statistik, Politik oder du denkst wie ich an die Weisheiten der Lehrer von damals.

Etwas, das ich in den letzten Jahren immer wieder realisiert habe, ist: Es ist auch für uns in der Softwareentwicklung relevant.

Insights and questions from the original waterfall paper

ntietz.com blog - technically a blog — Mon, 11 Dec 2023 02:00:00 GMT

The waterfall model is probably the most reviled methodology in software engineering. This methodology was first described in a 1970 paper by Dr. Winston Royce. This paper didn't call it waterfall, nor did it endorse the technique, and the paper contains a lot of good insights and raises some interesting questions. Let's take a look at some of those.

Essential steps of software development

Royce says there are two essential steps in all programming: analysis and coding. It's not defined what goes into analysis, but I think we can safely assume it includes thinking about the problem and how to solve it. I think it's pretty clear that these steps are always involved. For extremely small programs, maybe they're all you need, though you probably don't do them serially.

The other steps involved for larger programs are requirements, program design, testing, and operations.

One interesting thing here is that I think these are all done at all sizes of software, they're just not done explicitly and separately. Let's say you write a small program, like a solution to an Advent of Code problem. For this, you need to get the requirements from the problem description, do some analysis on it, design your program, write it up, test it, and then run it for the answer. But these flow together, and code/test/operate get lumped together, and requirements/analysis/design get lumped together—with both of those groups getting intermingled as well. They're not done serially one after another, but each is done at some point.

What are really the essential steps of software development? I'm not sure. I think the breakdown of activities he mentions in this paper is interesting and a nice way to think about the activities we engage in, and I can't really go further than that at the moment.

What is the role of management?

One point that Royce makes is... interesting:

The prime function of management is to sell these concepts [of testing, documentation, analysis, etc.] to both groups [developers and customers] and then enforce compliance on the part of development personnel.

He continues this later on, too:

The first rule of managing software development is ruthless enforcement of documentation requirements.

And this quote is followed by saying that if documentation isn't good enough, then replace management.

So, this makes clear Royce's view on management's role: strictly enforcing rules and proper development practice. If they don't ruthlessly enforce documentation processes, then they'll be fired. And they need to make sure developers do their testing and analysis and design, too.

I mean... I don't know what it was like in the 70s, a couple of decades before my time. So this could be the right take at the time. In the present day, it seems very antithetical to what I've experienced in the teams I work on.

On the teams I'm on, what I've generally seen is:

Engineers advocate for testing, for requirements, for explicit design time
Management pushes for less of these in some instances

This is the opposite of what he says happens! I'll give him this, though: developers sure do like skipping documentation, and customers do want to avoid paying for these things.

For me, the role of management is not as ruthless enforcer but as facilitator. Software engineering is more mature as a field than it was 53 years ago, and we have some established best practices. As practitioners, we take pride in our work and we do push for testing, analysis, all the good stuff. And the role of management is to make sure that everything hangs in balance between technical depth and business needs, and to make sure that the existing processes facilitate that balance.

But if things aren't happening, you don't step in as a ruthless enforcer You look and figure out why, and work with the team together to shift processes to make those things happen.

What are we optimizing for?

He states early on that "[Separate stages of development] must be planned and staffed differently for best utilization of program resources." This describes a world where we have dedicated staff for gathering requirements, different staff for designing the program, yet more staff to write it, another team to test it, and some poor soul has to put our mess into production.

In contrast, most teams today take a much more multidisciplinary approach. Some go to the extreme, and everyone does everything. Most are somewhere in the middle: dedicated testing staff are present, but everyone does some testing; product managers are responsible for requirements, but everyone helps; architects do a lot of design, but each engineer does some architecture.

The key thing though is that last part: "for best utilization of program resources". Here, "program" refers to the project and its staffing, not to the software. And that's the thing, he's optimizing for best utilizing each individual's time and saving money on personnel.

In contrast, modern software development prioritizes other things over direct resource utilization. Time to market, quick validation, all the things to make sure we're going in the right direction. We slow down a little and waste a little bit of each person's time, but we have a lot less backtracking to do.

I could see separate roles making sense in a situation where you do have much clearer requirements. Does something like that exist? Good question. But if it does, separate roles might make sense (I'm not fully convinced, but maybe). For everything else, prioritizing figuring out what we're doing makes more sense than optimizing for full utilization of time.

Write it twice!

One of the best pieces of advice in this paper (not that I'm biased, having written something similar) is to write it twice. The first version should be a fast version to learn what we're doing and gain real-world insight. Then the second version is the final draft that goes to the customer and should meet requirements.

This is great, because it highlights what we run into all the time: we don't know if our solutions work until we try them. We don't really know what the problem is until we try to solve it. Having multiple iterations is a fantastic way to try things, learn those hard lessons, and still have time in the schedule to fix everything.

The recommendation to do it twice, in full, I think is interesting and is something to aspire to. It's easier to advocate for small iterations and small throw-away prototypes, and those are super valuable. If you take nothing else away from this paper, go try doing a throw-away version first.

The gendered language

It's always a little bit of a shock going back and reading a paper from the 70s. Every single pronoun is "he" or "his," and that just really grates^[1]. Not every person on a tech project is a boy, you know.

The norm in tech has pretty well shifted as far as I see. There's still plenty of sexism to go around, but there's less blatant gendered language at least. We've got a long way to go, and sometimes reading papers from the past is nice to remind us of how the norms have changed.

Let's let this remind us that change is possible. Collectively, we've shifted away from default he/him pronouns for anonymous people. We can continue to make a more equitable world. It's going to take a while, it's going to hurt, and it's worth fighting for it ❤️.

As a historical artifact, Royce's paper gives a lot of interesting insights. It's cool to see some of the things he discusses in here be fully realized, and to see other ways in which our field has transcended where it was in the 70s.

The USCF's Official Rules of Chess appear to have previously been this way, too. They've made an effort to update them resulting in some inconsistencies, including at least one time where they switched gender of pronouns midsentence while referring to the same antecedent. ↩

Scientists find exciting new way to talk about nothing in press releases

taylor.town — Mon, 11 Dec 2023 02:00:00 GMT

In a groundbreaking development, a team of renowned scientists unlocked a revolutionary method to discuss absolutely nothing in press releases.

Santa Prefers Wealthy Sick Children

taylor.town — Mon, 11 Dec 2023 02:00:00 GMT

This is the first study, to our knowledge, to dispel the myth that Santa visits children based on behaviour and suggests socioeconomic deprivation plays a greater role in determining a visit.

TIL: macOS’ Hidden Gem – The "caffeinate" Command!

James Brooks — Mon, 11 Dec 2023 02:00:00 GMT

Did you know macOS has a built-in alternative to Caffeine? It’s called caffeinate and it’s awesome.

Handling Process Debt in IT

Vadim Kravcenko — Sun, 10 Dec 2023 18:25:34 GMT

Question:

Answer:

The post Handling Process Debt in IT appeared first on Vadim Kravcenko.

The Winterkeeper

Nicola Iarocci — Sun, 10 Dec 2023 08:05:25 GMT

The Guardian’s The Winterkeeper: A Lifetime Spent Protecting Yellowstone National Park is a beautiful short documentary I truly enjoyed watching.

A little research on Steven Fuller, the protagonist, allowed me to dig out some promising reading material.

The Restaurant at the End of the Universe

Vít Baisa — Sun, 10 Dec 2023 02:00:00 GMT

second book in the Hitchhiker’s Guide to the Galaxy series

TUIs and mouseless computing

Vít Baisa — Sun, 10 Dec 2023 02:00:00 GMT

notes on a presentation about TUIs and mouseless computing for INJECT

Setting Up macOS for a Developer

Salvatore's Blog — Sun, 10 Dec 2023 02:00:00 GMT

In this post, I’ll share the key tools and configurations I use when setting up a new macOS environment for software engineering.

Applications

I’m a fan of Firefox and a bunch of other tools, but I’ll keep this to developer-related tools.

iTerm2

I believe it is better than the built-in Terminal. Probably.

Configure the following shortcuts from this tutorial.

⌥ ⌫ for deleting a word
⌥ ← for going back a word
⌥ → for going forward a word

Set the Login command in profile to open tmux or reuse an existing session. When I accidentally close the window, this keeps me from terminating a long-running program.

tmux attach -t base || tmux new -s base

1Password

Managing passwords and keys is 100x easier with 1Password and they’re more secure than a lot of the other options out there. Their SSH key management and git signing support is a must-use.

Folders

~/Developer

I used to call the folder ~/dev or ~/Development, but if you call it “Developer”, then Apple will reward you with a special folder icon.

~/Screenshots

If you also take a lot of screenshots, you’ll want to see this post about setting up a screenshot folder.

Developer Tools

macOS changed the default from bash to zsh in 2019 so if your previous computer was an Intel-based MacBook you’re in for a very-similar experience.

Oh My Zsh

It is a framework for managing shell plugins.

# .zshrc, line ~73
plugins=(
  asdf
  brew
  git
  z
)

asdf

The one-stop-shop for managing Java/Python/Ruby/whatever versions. If you use Ruby, I recommend enabling the legacy versions so you can use the other version manager .ruby-version files.

# $HOME/.asdfrc
legacy_version_file = yes

Homebrew

It is the most popular way to install command line tools.

Starship

It makes my command prompt look nice.

sal.dev on  main [!?] via 💎 v2.7.8
❯

z

Command line tool for jumping back in to a project you were working on. For example, if I type z ad, it drops me back into the Advent of Coding project.

~
❯ z ad

advent-of-code-2023
❯

Other

I also like to keep this command handy for clearing out git branches that have been merged to main.

git branch --merged main | grep -v "\* main" | xargs -n 1 git branch -d

Addendum

This post was more for me to remember what I use than for others, but I hope some folks found it useful anyway!

Someone interviewed me

Consulting on Adolfo Ochagavía — Sun, 10 Dec 2023 02:00:00 GMT

A few months back I had an email exchange with Drew Capener, from filtra.io, who is interviewing professionals involved in the Rust community. We ended up having a nice chat, which he transcribed and posted online under the title “Contributing to Rust as a novice”. It even triggered some discussion on Hacker News! What did we talk about? The adventure of contributing to Rust in 2014 as a beginner to programming, the crucial role of friendly maintainers, and the challenges involved in working as an independent contractor.

"All foods are either protein or candy."

taylor.town — Sun, 10 Dec 2023 02:00:00 GMT

I've found that protein alone is a suitable proxy for tracking all other micronutrients. When I eat enough protein, I'm probably getting enough vitamins, minerals, etc.

We don't need no stinking bastion host!

Ben E. C. Boyter — Sun, 10 Dec 2023 02:00:00 GMT

So on occasion, I find myself wanting to run SQL against customer databases. Where this gets annoying is that because we are often in AWS using lambda I have no ability to connect to them. Occasionally im able to bully the friendly devops’ and security people to allow me access to a bastion host, which I can then connect to the SQL database.

However this is often a painful process if allowed at all, and so with permission I sometimes add a special route into the application allowing me to run arbitrary SQL as needed.

Ooty: Friendships, Travel and Painting 📍

Playground — Sun, 10 Dec 2023 02:00:00 GMT

I spent the last two weeks of August in Ooty📍, A hill station in Tamil Nadu. Transitioning from Chennai’s heat to Ooty’s cold within a day was drastic. My hoodie was happy to be out from the bottom of my bag.

Things I've Published and Talks I've Given

Sun, 10 Dec 2023 02:00:00 GMT

I’ve been lucky to have the opportunity in my career to speak at conferences and to have my writing published in many written and digital publications.

If you’re interested in having me speak at your conference, or write for your publication, please reach out to me.

Conference Talks/Trainings

Date	Title	Conference	Video	Slides
2025-03-06	OpenStack in the Real World	SCALE 22x / OpenInfra Days 2025	video	slides
2024-06-04	Open Infrastructure: From Monumental to Mundane and Back Again	OpenInfra Days 2024, CERN	video	slides
2023-11-04	Trust in an Open Source Community	SeaGL 2023	video	slides
2023-03-11	Death of Hardware Knowledge	SCALE 20x	video	slides
2015-05-20	Don’t Just Consume; Participate! Operators and the Community	OpenStack Summit	video	N/A
2014-11-05	Hardware in the Cloud; Cleaning up after Bare Metal Tenants	OpenStack Summit	video	N/A
2012-11-12	How to Secure and Route Your Linux Network	USENIX LISA 2012 (Tutorial)	N/A	N/A
2011-12-06	How to Secure and Route Your Linux Network	USENIX LISA 2011 (Tutorial)	N/A	N/A
2010-11-08	How to Secure and Route Your Linux Network	USENIX LISA 2010 (Tutorial)	N/A	N/A

External Blogposts

Date	Title
2025-03-26	Invisible work of OpenStack: Eventlet Migration
2023-06-30	Armada: Six Months in the Sandbox
2023-05-04	Invisible work of OpenStack: Testing Matrices
2023-03-08	What Cloud Developers Need to Know About Hardware
2022-11-29	Invisible work of OpenStack: Security Bugs

Publications (physical)

Date(s)	Title	Publication	Notes
June 2005	Linspire Five-O (review)	Linux Magazine	Physical copy needs to be scanned
2000-2003	(multiple weekly)	Spring Hope Enterprise	Worked for years; ending as Sports Editor

Other Projects/Media

Date(s)	Description
May 2024 - Current	GR-OSS OUT Podcast (audio), (video)
Oct 2022 - Mar 2023	Open Source Office Hours

I hate MFA

Alex Molas Blog — Sun, 10 Dec 2023 02:00:00 GMT

I hate MFA, and in this rant post I explain why. It's basically because (1) is distracts me a lot, (2) it forces me to have a smartphone, (3) it's a leaky abstraction, and (4) it can be replaced with better solutions.

A trekker's best friends

Jelle Smet — Sun, 10 Dec 2023 01:00:00 GMT

Old friends

A little over a year ago, I found myself faced with the inevitable task of parting ways with my beloved Timberland all-leather hiking shoes. They had faithfully accompanied me on countless adventures, but time had taken its toll, and they had begun to show signs of wear and …

Introducing CrankWheel

joisig gone awol — Sat, 09 Dec 2023 23:20:13 GMT

Analyzing Static Website Logs with AWStats

Mike Kasberg's Blog — Sat, 09 Dec 2023 23:00:00 GMT

It’s really common to add Google Analytics to your site to gain valuable insights about who’s visiting. I think most companies use Google Analytics, and because it’s free and easy many solo developers and smaller sites do too. But recently, I think a lot of people have been questioning the value. For many small or independent blogs and websites, Google Analytics is an overly-complicated product that’s difficult to use and navigate. Moreover, using Google Analytics requires loading their relatively large, bloated javascript on your site. When Google Analytics competitor Plausible was starting out a few years ago, they highlighted a bunch of other reasons you might not want to use Google Analytics. In short, there’s gotta be a better analytics tool for small websites right?

This blog is a small website, and I used to use Google Analytics. (I don’t anymore!) I removed Google Analytics from this blog back in 2021. I agree with most of the criticisms in the article linked above – I don’t want to force my visitors to be tracked, I don’t want a bloated script potentially slowing down my site, and Google Analytics is way overkill for what I need. All I really care about is occasionally checking how many people are visiting my site, seeing which pages they look at, and watching for errors. Everything else is bonus, as long as it’s easy to use and understand. For a year or two, I operated without any analytics at all. I used Google Search Console to gain a little insight into which pages people were visiting. Search Console is actually great, and I’m still using it today. It provides a simple view of how people find your site from Google, which is a good approximation for which pages are popular anyway. But Search Console was never a true representation of all my traffic (because some people don’t come to my site from Google), and therefore isn’t a perfect replacement for Google Analytics.

I wanted to find a better tool for analyzing traffic to my website. I started looking for alternatives. Plausible seemed really appealing, and I liked a lot of their selling points. Simple analytics that are easy to understand. Lightweight script. No cookies. Open source. Unfortunately, I didn’t like their price tag. As of December 2023, their cheapest plan is $9/mo. While that’s probably reasonable for most customers, it just doesn’t make sense for me. This blog is a statically generated site that I operate for dollars per year. And while I could afford to pay $9/mo, I just don’t want to. I don’t think I’d get $108 dollars of annual value from analytics on my personal blog that doesn’t make any money. On top of that, I’ve come to enjoy operating this blog really frugally. I take some pride in keeping things simple and cheap, and in knowing that you can run a website really well on a very low budget using open source software.

So, if not Plausible Analytics, where should I turn? I was intrigued by the idea of server-side analytics. I shouldn’t need clients to run javascript to know who’s visiting my website, I should be able to tell who’s connecting to my server and what page they’re requesting! Even though my blog is a static site, my hosting provider uses Apache, and they offer access to the logs. That should be all I need for basic analytics right?!?

AWStats

AWStats is a great tool! Despite the first three letters, AWStats has nothing to do with AWS. In fact, it was created six years before AWS, in May, 2000. A time when PHP, ASP, Perl, ColdFusion, and JSP were the some of the most popular server-side languages. AWStats stands for Advanced Web Statistics. It’s a log analyzer for Apache (and nginx and other) log files. Whether you’re serving static HTML through Apache or using something like mod_php, AWStats will parse your server log files and provide info and statistics about your web requests and visitors. It’s designed to be run on your web server and hosted by Apache – sort of like an admin panel for your web site. It has a separate process that can run on a cron schedule to ingest new log files, and it uses Apache to serve the reports as web pages. I wondered if I could use AWStats to analyze the Apache log files for my static site!

Normally, AWStats expects to be run on a web server. It expects to save its configuration in /etc/awstats/awstats.mysite.conf and it expects to create a database in /var/lib/awstats. I could run AWStats on my hosting provider, but I’d prefer not to. For one, I’m currently only hosting static pages, and don’t want to set up (or pay for) AWStats to run CGI Perl scripts. Also, because I pay (pennies) for (megabytes of) server storage, and in the spirit of doing things frugally, I don’t want to leave all my logs on the server indefinitely. And finally, I’m the only one that needs to see the reports. I don’t want them available on the internet, and don’t want to deal with locking down my configuration to prevent public access. As an alternative to all this, I could install AWStats on my laptop. This would be easy since I use Ubuntu – just sudo apt install awstats and I’m done. But I don’t really like that solution either. AWStats expects to be on a web server, and I don’t want to deal with a bunch of special (and tricky to reproduce) web server config on my laptop. Docker to the rescue!

With Docker, I can mount my log files into a (temporary) AWStats container running on my laptop to parse my log files and serve reports. I mount the log files into the container and run the AWStats script to ingest log files. I can save state between runs by persisting /var/lib/awstats in a Docker volume. And I can run Apache from the container to view my reports! (AWStats can generate static HTML to be viewed without running Apache, but only for a single month at a time so I find it much nicer to let Apache run the CGI script and change the time frame as I please.) I started using this AWStats Docker container to view my server statistics a few weeks ago, and I’m really happy with it! AWStats produces really useful reports, and I have all the info I need about traffic to my site without running any analytics javascript on clients! Perhaps more importantly, it includes a section showing statistics about my error status codes and which URLs produced the error.

AWStats Docker Setup & Configuration

Here’s my current setup. For a long time, I’ve used a Rakefile (a Makefile, but with Ruby) to build and deploy my site. I added a task to my Rakefile to grab the access logs off the server and save them to my laptop (where I keep them and back them up). I also added tasks to ingest the logs into the Docker container (with the AWStats database persisted on a Docker volume) and a task to serve the reports. (I’m using the pabra AWStats Docker image, which appears to be the most well-maintained an up-to-date.) Here’s what that looks like. (Most of this is just bash, so could be easily ported to a Makefile or another script.)

desc 'Fetch logs from the server'
task :logs do
    puts '======== FETCH LOGS ========'.bold
    sh "scp mikekasberg.com:/home/logs/access_log #{__dir__}/logs/access.log"
    size = File.size("#{__dir__}/logs/access.log")
    if size > 10000000 # 10MB
        puts "Size exceeds 10MB. Rotating log file to local only."
        date_string = Time.now.getutc.iso8601
        FileUtils.mv("#{__dir__}/logs/access.log", "#{__dir__}/logs/access_#{date_string}.log")
        sh "ssh mikekasberg.com 'rm /home/logs/access_log'"
    end
end

desc 'Ingest logs for AWStats Docker'
task :awstats_ingest do
    puts '========== INGEST =========='.bold
    sh 'docker volume create awstats-db-mike-jekyll'
    sh "docker run --rm -e AWSTATS_CONF_LOGFILE='logresolvemerge.pl /logs/*.log |' -e AWSTATS_CONF_LOGFORMAT=1 -e AWSTATS_CONF_SITEDOMAIN=www.mikekasberg.com -e AWSTATS_CONF_HOSTALIASES='localhost 127.0.0.1' -e AWSTATS_CONF_ALLOWFULLYEARVIEW=3 -v ${PWD}/logs/:/logs/:ro -v awstats-db-mike-jekyll:/var/lib/awstats pabra/awstats:7.9 awstats.pl -config=awstats  -update"
end

desc 'Run AWStats Interface'
task :awstats => :awstats_ingest do
    puts '========== AWStats =========='.bold
    sh 'xdg-open "http://localhost:8080/awstats.pl"'
    exec("docker run --rm -it -p 8080:80 -e AWSTATS_CONF_SITEDOMAIN=www.mikekasberg.com -e AWSTATS_CONF_HOSTALIASES='localhost 127.0.0.1' -e AWSTATS_CONF_ALLOWFULLYEARVIEW=3 -v awstats-db-mike-jekyll:/var/lib/awstats pabra/awstats:7.9")
end

Grabbing logs from my server is as easy as running rake logs, and I do that at least every couple months. And viewing my website statistics is as easy as running rake awstats. (The awstats task depends on the ingestion task, so it will ingest any new logs first.) I even added a line that will automatically open the reports in my web browser so I don’t need to remember the URL! (xdg-open works on Linux, on macOS you could just use open.) What can you do with AWStats? It does the basics well (visitors and page views). Beyond that, it’s great for looking at the most common error URLs, and it’s also good for looking at referrers.

AWStats is really old technology, but it’s really good at what it does and it’s reliable! There are lots of potential problems with javascript analytics but perhaps the most important is this: Users don’t like it and many try to block it. I think server-side analytics is a great (obvious?) alternative to client-side analytics that’s less intrusive, and could be a great fit for small websites and blogs. Running AWStats in a temporary Docker container is a great way to get web statistics from server logs!

Steam Deck OLED: Wi-Fi, Streaming, Games

Ishan Sharma — Sat, 09 Dec 2023 20:30:00 GMT

I recently got a Steam Deck for my birthday. Ever since its initial launch, I was on the fence about getting one. The first reason was that I already had a handheld in Nintendo Switch. And second, my top played game genre is Real-time Strategy (RTS), which doesn’t really work that well on a handheld.

After using it for a week, I am a big fan of the device. This article is my collection of tips you may find useful and some interesting use cases you’ll want to consider before buying one.

Quoting Andrej Karpathy

Nicola Iarocci — Sat, 09 Dec 2023 08:05:25 GMT

I always struggle a bit with I’m asked about the “hallucination problem” in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.

We direct their dreams with prompts. The prompts start the dream, and based on the LLM’s hazy recollection of its training documents, most of the time the result goes someplace useful.

It’s only when the dreams go into deemed factually incorrect territory that we label it a “hallucination”. It looks like a bug, but it’s just the LLM doing what it always does.

Artificial Intelligence Reading List

Zac Szewczyk — Sat, 09 Dec 2023 05:03:26 GMT

Thanks in no small part to OpenAI’s ChatGPT, the past year has seen an explosion in interest in artificial intelligence in general, and in large language models in particular. That democratization of access turned this niche research area into a common topic of conversation, and has led to a lot of fascinating writing on the subject. Although certainly not comprehensive, this article collects some of my favorite articles, papers, and resources into a single reading list.

Permalink.

Santa Claus is not a god... yet

taylor.town — Sat, 09 Dec 2023 02:00:00 GMT

To dismiss a god of children is unfair to everybody who was ever a child.

Goals for 2024

Oliver Gilan — Sat, 09 Dec 2023 02:00:00 GMT

Taking stock of my progress.

Internet Usage

foreverliketh.is — Sat, 09 Dec 2023 02:00:00 GMT

Image Details & Source

man walking beside man holding umbrella by Frank Holleman on Unsplash

Click Here for the Original, High-Quality Image

This page was last modified: December 13th, 2023

Consequences #

Back in August, I mentioned wanting to “touch grass” more. On the off chance I need to justify that goal for you, reader, consider the following:

The phrase “terminally online” has its own Wikipedia page.
People, myself included, have (ironically) taken to calling the Unabomber: “Uncle Ted” as if, while “eccentric”, he was maybe “onto something” / “looking out for us”.
Anyone unable to recognize that the digital age has got life out of whack– well, as my students might say: “they’re an NPC [#1 & #2]”.

In that vein, I’d like to give my two cents on a prevalent sentiment you’re likely to find on your expeditions across cyberspace:

The thought that “true freedom / peace / happiness / contentment” can only be found in abandoning the Internet entirely. That anything “less” is a “second rate sacrifice”; or a “cope [#1 & #2]”.

Some 96 Lives #

I don’t remember where I read it, but there’s an idea that’s stuck with me for over a decade now. It’s a sort-of criticism directed at monasticism (or “monkhood”), the concept of renouncing “worldly” pursuits to devote oneself fully to spiritual work. You know the trope, visitor: some rando experiencing an existential crisis drops everything to join some monastery on some isolated mountain to search for some… thing.

Now, I don’t mean to repudiate an entire way of life, but the idea challenges that notion and its prerequisites, like:

seclusion from society / modernity
giving things up, renunciation
solitude and silence

It questions whether the people willing to go to such lengths, are, exclusively, the ONLY ones worthy of receiving the fruit it bears. It asks if Joe Nobody, irrespective of what they do, would be undeserving of the same reward. It posits: Which is more difficult? Attaining “spiritual wholeness” in an environment designed for its cultivation? Or attaining it in spite of an “imperfect” situation? That if Mr(s). Nobody can strive towards balance in the midst of their everyday life, with all of its obstacles and struggles, then aren’t they more worthy than the recluse?

Cowardice? #

Isn’t scurrying off to a desolate convent somewhat akin to thinking it’s:

the world’s fault!
my country’s fault!
my city’s fault!
my employer’s fault!
my possessions’ fault!
my friend’s fault!
my family’s fault!
the industrial revolution’s fault!
the Internet’s fault!

Do you lack the courage or will to take responsibility for your own fulfillment? The determination or discipline to face your humanity and its err, your circumstances and its stacked odds… the Internet and its attempts to control you?

Look, there’s many roads to Rome, traveler. I think, on the whole, we’re out of homeostasis; and that you should do whatever you think best to restore your own personal equilibrium. You feel that means “quitting the internet”? Go for it, yo.

Just consider that you may, perhaps, be going from one extreme, to another.

Syndication: Agora Road [#1 & #2] | basement community | IndieNews | Mastodon

This post is part of the basement community 1-Year Anniversary Blog Event [#1 & #2].

This blog post is also part of Agora Road Travelogue - Dec [#1 & #2]. A project attempting to encourage individuals to both write and exercise ownership over their content.

Comments #

Email: hi@foreverliketh.is

IndieWeb h-entry

Internet Usage

Published by foreverliketh.is on 9^th December 2023

All or Nothing

Consequences

Internet Usage

Agora Road [#1 & #2] | basement community | IndieNews | Mastodon

Psychology

A Quick and Brutal Match

Home on Lagomorph — Sat, 09 Dec 2023 00:15:00 GMT

I’ve had the ability to post chess games on this blog for a while but have found few games I have really wanted to highlight - this one is one I’m proud of.

Hopefully I’ll be willing to start posting losses I’m proud of, too.

Advent of Code 2023: Building Intuition with Quadratic Equations

epiccoleman.com — Fri, 08 Dec 2023 20:49:13 GMT

I take a deep dive into some math I haven't touched for a long time, and use it to solve an Advent of Code problem.

Llama Guard leaves things to be desired

Twilight Sparkle's Friendship Castle — Fri, 08 Dec 2023 15:00:00 GMT

Or: it won't filter how to make a pipe bomb

Capturing decisions and those involved

I Learnt — Fri, 08 Dec 2023 14:00:00 GMT

My role has changed recently to include another team. This has also included more interaction with other areas of the business and also the management team.

AWSary on Android

Tiago Rodrigues — Fri, 08 Dec 2023 11:27:00 GMT

AWSary - Apps on Google Play

AWS services dictionary

Finally, AWSary is available on Android.

About 4 Months ago, Luiz Calderaro approached me to help port AWSary to Android, and I couldn't be more thrilled.

Luiz has found some time here and there to sync with me and in

How I decarbonized my home for $3,000

Into the Details — Fri, 08 Dec 2023 02:23:26 GMT

From growing up on a small farm in upstate NY, I have vivid memories of holding the blow-torch for my dad on a plumbing project when I was probably 10 years old. He did all the work on the house himself (with the occasional help from an offspring), and so when it came time to electrify my house, I decided to DIY it as well. Here's how it went.

Working Remotely Using Visual Studio Code

Thoughts by Thomas Derflinger — Fri, 08 Dec 2023 02:00:00 GMT

In many circumstances it is convenient to work remotely using Visual Studio Code. There is a fine VS Code extension that allows you to work on a remote machine as if it was your local machine. This is great for things like embedded programming or developing in the cloud. Read here how to do this.

Cambridge has (at least) two parking spaces per household

Evan Fields — Fri, 08 Dec 2023 02:00:00 GMT

A previous post estimated Cambridge’s non-metered street parking at 31,474 spaces, approximately one per car-owning household. But these spaces constitute a minority of Cambridge’s total parking supply. Here I estimate two other big chunks of our total supply:

Roast #33: My processes

thoughtfulcoffee — Thu, 07 Dec 2023 19:41:00 GMT

Green buying, sample roasting, and production roasting

Breaking things down

I Learnt — Thu, 07 Dec 2023 14:00:00 GMT

A task can often look too big to tackle. It can become very daunting even in deciding where to start.

Cleaning Up Speech Recognition with GPT

Beetle Space — Thu, 07 Dec 2023 10:00:00 GMT

My friends often ask for use cases for GPT. Below I provide one recent example.

I attend a weekly educational seminar where real estate professionals present the work they do. I take notes on paper, and later transcribe them to my computer, which is a pain. As a result, I …

Another Microservice Desaster

Backendhance — Thu, 07 Dec 2023 02:00:00 GMT

Hi,

Microservice Architektur

Dieses Buzzword, ist für mich inzwischen ein Trigger geworden. Darüber habe ich schon oft geschrieben.

Anfang des Jahres trendete mein Artikel Microservices are a Big Ball of Mud auf Hacker News

In den 343 Kommentaren zu meinem Artikel kann man gut sehen, wie sehr das Thema die Gemüter erhitzt 😉 Natürlich ist der rationale Blick darauf ein anderer, als ich in dem Artikel dargestellt habe. Es kommt IMMER auf den Kontext an. AppContinuum ist ein hervorragendes Paper zu einem reflektierten Blick auf die Thematik.

Advent of Code 2023 in Oracle - Day 7: Camel Cards

Philipp Hartenfeller Blog RSS Feed — Thu, 07 Dec 2023 02:00:00 GMT

Advent of Code 2023 in Oracle SQL and PL/SQL. Day 7: Camel Cards.

Getting Started With PyFlink on Kubernetes

Gunnar Morling — Thu, 07 Dec 2023 02:00:00 GMT

Table of Contents

What Is PyFlink and Why Should You Care?
Prerequisites
- Installing the Flink Kubernetes Operator
- Installing Strimzi and Apache Kafka
A Simple PyFlink Job
Building a Container Image With Your PyFlink Job
Deploying a PyFlink Job On Kubernetes

The other day, I wanted to get my feet wet with PyFlink. While there is a fair amount of related information out there, I couldn’t find really up-to-date documentation on using current versions of PyFlink with Flink on Kubernetes.

“We all need our own philosophy”

An Ordinary Disaster — Thu, 07 Dec 2023 00:12:54 GMT

Bowen Dwelle in conversation with Latham Turner of Get Real, Man on writing to explore, how “adventure doesn’t happen by accident,” writing as men and the transition into being an older man, using research in storytelling, how “we all need our own philosophy,” the challenge of positive confrontation and “the Goat Work,” how we relate to our immediate geography, long-distance walking, wayfinding, personal spirituality, and the gods we’re praying as.

What You Missed At My Substack

Stories by Fred Grott on Medium — Wed, 06 Dec 2023 18:34:49 GMT

You missed me solving the missing material design 3 carousel from the flutter SDK problem:

Continue reading on Medium »

Decisions with a cost

I Learnt — Wed, 06 Dec 2023 14:00:00 GMT

Sometimes a decision comes with a cost, a sacrifice.

In the book Leaders Eat Last Simon Sinek says:

Hosting quickly 2 - Dioxus to the web on Fly.io

Martijn Arts — Wed, 06 Dec 2023 13:44:23 GMT

This is the second article in a series called "Hosting quickly". I want to launch Rust apps quickly and often, and I want to have stable, common infrastructure for all of them.

Hosting quickly 1 - Setting up Terraform, Github, and 1Password

Today, let's build a frontend

Firefox focus ring regression

All content on Seirdy’s Home — Wed, 06 Dec 2023 07:53:35 GMT

Firefox 120 appears to have regressed to its older WebKit-like blue focus outlines; it briefly had dual-color white-and-blue outlines.

I’d previously advised certain sites (sites with background colors that have poor contrast against the focus-ring’s blue color) to override the default focus indicators. I planned to temper that advice since Chromium and Firefox adopted focus indicators that appeared visible on a variety of background colors; I suppose I should cancel those plans.

As long as people have to use defaults they didn’t choose (e.g. Tor Browser users are stuck with default settings), we have to override them when they’re inaccessible. We can’t count on users to always have the means to do this: fingerprinting concerns and device-borrowing are facts of life.

Update 2023-12-06: it appears that as of Firefox 121, the issue only appears in forced-colors mode (e.g. when toggling color overrides in the color settings of about:preferences). Input elements are unaffected. With forced-colors disabled, Firefox uses the better two-tone indicator.

Borrowing from big players

All content on Seirdy’s Home — Wed, 06 Dec 2023 07:50:18 GMT

Large organizations’ choices influence my decisions in only one way: by telling me what users are familiar with. For instance: when building a search-results page, it might make sense to borrow the basic semantics of existing search engines (Google, Bing, etc.) so the interface is familiar.

It doesn’t make sense to blatantly violate WCAG (especially at the “A” level!) just because big companies do. The companies you cite know they won’t get sued over link underlines, even though removing them without replacement is an accessibility hazard. Every time someone jumps onto Flutter because Google said it’s accessible, I feel tempted to file yet another accessibility issue.

Recovering data from a failed drive using ddrescue

skouf.com — Wed, 06 Dec 2023 04:09:52 GMT

My first foray into data recovery has been both educational and, blissfully, not too stressful. You always hear tales of damaged hard drives and complete data loss and think “that’ll never happen to me”. Well, it finally has happened to me, and thankfully the outcome has been pretty good (all things considered), and I’ve learned a bit along the way.

Talk @ a16z: Taking AI applications to Production

Zachary Proser — Wed, 06 Dec 2023 02:00:00 GMT

I introduced the new Pinecone AWS Reference Architecture with Pulumi and explained infrastructure as code

An iOS journaling app powered by org plain text

xenodium.com @alvaro — Wed, 06 Dec 2023 02:00:00 GMT

I've been experimenting with building a rich text editing component for iOS, powered by org markup. The idea is to offer a mobile-friendly editing experience, backed by our beloved plain text format.

To make things a little more interesting, I'm introducing a new org-based app to help anyone with regular journaling.

👉 Meet ✨Journelly✨

Plain text is the serialization format. No conversion/import/export needed.

Though it's early days, it's fairly functional. Been using it daily for some time. You can opt in to use an external org file and sync with your beloved Emacs.

Want to give it a try? Want a TestFlight invite? Send me an email address (any would do) at either of these: Mastodon / Twitter / Reddit / Email.

The topic of org being fairly Emacs-oriented, though a strength for someone far down the rabbit hole, it is understandable to call it out for someone in a different position. Lucky for us, org markup is plain text and can be implemented by apps other than Emacs, like Journelly itself for iOS and even more experimentally on macOS:

And like Journelly for iOS, I got other org things available on iOS:

	As an Org mode fan, so I wrote Plain Org for iOS. It's on the App Store.
	Inspired by Atomic Habits, I wrote Flat Habits for iOS. Also on the App Store.
	I needed an Emacs-inspired scratch buffer on iOS (who doesn't?), so I built one.

As an Org mode fan, so I wrote Plain Org for iOS. It's on the App Store.
Inspired by Atomic Habits, I wrote Flat Habits for iOS. Also on the App Store.
I needed an Emacs-inspired scratch buffer on iOS (who doesn't?), so I built one.

Just like the stuff I do or write about? Sponsor me.

How to Observe Buydays

taylor.town — Wed, 06 Dec 2023 02:00:00 GMT

Consolidate non-essential purchases to fewer days.

Make a /wish page!

taylor.town — Wed, 06 Dec 2023 02:00:00 GMT

An honest wishlist lays bare your values, motivations, and desires.

/wish

taylor.town — Wed, 06 Dec 2023 02:00:00 GMT

My wishlist is honest and sometimes embarassing.

Unlocking the Bootloader on the Kyocera 902KC

Home on Lagomorph — Wed, 06 Dec 2023 00:30:00 GMT

There have been mixed reports of people managing to unlock the bootloader on the Kyocera 902KC. I’ve done some extensive research on unlocking and rooting this phone and I’d like to outline all my findings here.

Being able to unlock the bootloader on the Kyocera 902KC would be a huge boon for usability in North America, as it would mean you could sideload the microG services, allowing you to utilize maps properly, as well as any apps that rely on Google Play Services in order to function.

Limiting concurrency with Python Coroutines

Thought Eddies — Tue, 05 Dec 2023 20:08:00 GMT

Limiting concurrency with Python Coroutines

Pseudonyms in American History

Matt Rickard — Tue, 05 Dec 2023 16:30:00 GMT

Debates around the ratification of the Constitution and the early formation of the United States happened through pseudonymous authors. They often used names bo

Simple tips can make a difference

I Learnt — Tue, 05 Dec 2023 14:00:00 GMT

When I grew up I had an old 35mm Praktica camera that was given to me. I purchased some lenses for it and mostly used it to take pictures of the Somerset cricket matches I went to watch.

How to use Visual Studio Code to run and debug code on SLURM compute nodes

Emilio’s Blog — Tue, 05 Dec 2023 12:00:00 GMT

If you’re a developer or data scientist using SLURM to handle your compute workloads, you have surely encountered issues in debugging your code on compute nodes. In this blog post, I share a simple solution for that, allowing you to develop and debug code running directly with the compute resources you need. Although the focus is on Visual Studio Code, the same approach can be applied to other IDEs that support remote development via SSH.

Every time you log out…

English – dotcoma — Tue, 05 Dec 2023 08:58:57 GMT

Every time you log out from GMail, they count you as a YouTube user. So sad to see what rubbish Google has become! Do yourself a favour: Try Tuta !

You Don't Need AI, You Need an Algorithm

Zac Szewczyk — Tue, 05 Dec 2023 05:06:36 GMT

A few weeks ago, I participated in a working group tasked with identifying areas where artificial intelligence and machine learning could improve network utilization. What, specifically, “improve” and “utilization” meant were some of the first questions we addressed. During that conversation, though, one of the participants made an insightful observation that the group promptly ignored. He told a brief story about how someone had asked him for a networking device that could aggregate multiple uplinks and then “use AI to choose the best one.” When he offered a simple WAN failover solution, a technology that has existed for decades and comes built-in to open source firewalls like pfSense, commercial appliances from the likes of Cisco, and every modern mobile phone, the customer told him, “No.” Like our working group, that customer person had a solution (artificial intelligence) driving their requirements (an AI-based WAN failover), not the other way around.

Permalink.

A "Unit" In A Test Is Not The Class Under Test

Backendhance — Tue, 05 Dec 2023 02:00:00 GMT

Hi,

Was ist eigentlich eine “Unit” im Unit-Test?

Dumme Frage, sagst du dir vielleicht.

Aber sie ist interessant. Über viele Jahre war ich, wie die meisten die über das Thema sprechen, der Meinung:

Eine Unit ist eine Klasse, die ich testen möchte.

Und vielleicht hast du das so auch gesehen. Und du hast mit Sicherheit damit ein paar Probleme gehabt.

Zum Beispiel, die Symptome des daraus resultierendem Coupling

Denn was passiert, wenn wir eine UserServiceTest Klasse schreiben? Wir haben - ohne es zu merken - ein semantisches Coupling zwischen der UserService Klasse und dem UserServiceTest hergestellt.

Partner u porodu

Vít Baisa — Tue, 05 Dec 2023 02:00:00 GMT

advice for dads how to be helpful during labour

Về trí nhớ trong việc học, và giáo dục phổ thông

Peter Hung — Mon, 04 Dec 2023 23:29:07 GMT

Gần đây, tôi có một quan sát mới mà trước đây tôi ít khi để ý, đó là về vai trò của trí nhớ trong việc học. Trong bài chia sẻ này, tôi muốn chia sẻ một số suy nghĩ hiện nay của tôi về chủ đề này. Khi học, chúng ta hiểu hay nhớ? … Continue reading Về trí nhớ trong việc học, và giáo dục phổ thông

The post Về trí nhớ trong việc học, và giáo dục phổ thông appeared first on Peter Hung.

Fairchildren

Matt Rickard — Mon, 04 Dec 2023 16:30:00 GMT

In 1956, William Shockley, Stanford professor and winner of the Nobel Prize in Physics for his work on semiconductors, recruited a team of young Ph.D. graduates

Turbo Pascal is 40

I Learnt — Mon, 04 Dec 2023 14:00:00 GMT

Turbo Pascal is 40 years old and it had a big impact on the direction of my career.

My Selfhosting Setup

Tim Bachmann's Blog — Mon, 04 Dec 2023 10:00:00 GMT

A short overview on how I self-host.

#COP28

English – dotcoma — Mon, 04 Dec 2023 09:08:33 GMT

They’re holding #COP28 in this dystopic, fucked-up place.

A Consideration of SQL ORMs

~gallant — Mon, 04 Dec 2023 03:04:13 GMT

Conventional wisdom in modern software development generally endorses the use of an ORM.

"Any project that doesn’t use an ORM will end up with a poorly implemented partial ORM."

Most developers are familiar with the dreaded SQL injection attack - an ORM provides solid protection, while exposing a comfortable OOP interface for application developers:

from blog.models import Blog, Entry
entry = Entry.objects.get(pk=1)
cheese_blog = Blog.objects.get(name="Cheddar Talk")
entry.blog = cheese_blog
entry.save()

Example from the Django framework

But ORMs have a dark side - they hide (but do not reduce) complexity, and the interface they expose doesn’t correspond particularly well to the actual work done by the database. SQL is declarative, already obfuscating execution - ORMs hide even more of the computational work behind class abstractions.

When you get into SQL, it is valuable to spend some time with the query planner. Run enough EXPLAIN statements and you start to see how your declarations are actually executed. Table scans, index-less conditions, expensive joins - SQL has a few typical sources of big costs, and you’ll start developing an instinct for them.

ORMs make it harder to develop that instinct. With an ORM you can retrieve a many-to-many relationship in a single line - the coder might not even know that's what they're doing, much less how it is executed in the database. Put some ORM interactions behind utility methods or decorators, have one route that hits a few of those, and you can end up with hundreds of lines of inefficient SQL executed on every page load.

For many applications, the database is the most critical and heavily-used piece of architecture. Overloading it with inefficient queries (just to let developers use classes and objects instead of SQL) can lead to a very bad place. I’ve seen several ORM-backed applications where typical user flow was significantly slower due to cumbersome data queries generated by the ORM.

There isn’t a quick fix for this either - removing an ORM is usually untenable, and optimizing without reducing functionality is a painstaking process. Adding a few indices can help - but if what you really need to do is change how your schema is arranged, you're in for a lot of work.

Meditating on this, I’m leaning towards a somewhat-unconventional-wisdom perspective - prepared statements should be preferred to ORMs for all but the most straightforward CRUD applications.

import mysql.connector
connection = mysql.connector.connect(database='blog')
cursor = connection.cursor(prepared=True)
get_blog = "SELECT id FROM blog WHERE name = %s"
update_entry = "UPDATE entry SET blog = %s WHERE id = %s"
cursor.execute(get_blog, ("Cheddar Talk",))
blog_id = cursor.fetchone()[0]
cursor.execute(update_blog_entry, (blog_id, 1))
cursor.commit()

Based on the above Django example and the MySQL Python connector

The specific implementation can vary (as it can with ORMs), but the general principle - make your developers actually write SQL. Have strongly typed parameterization, so static checks still prevent SQL injection. But don’t hide the database more than SQL itself already does - encourage developers to interact with it, and to understand their queries. Admittedly this results in more code (at least for this example) - but also, more understanding.

The main positive impact is that developers will better understand the performance impact of the code they write. They'll also be better at debugging and ad hoc fixes, as they'll be able to work directly with the database. And if some terrible and inefficient query makes it into the codebase somehow, factoring it out will be much easier than fixing a whole ORM.

Prepared statements have drawbacks (besides writing SQL, which I’m positioning as a plus) - most notably, they don’t give you a data structure to put your results in, while ORMs involve declaring a class for every table. But there are other excellent ways to specify typed objects - and with prepared statements, you only have to do so for classes that are actually useful (and just let the less frequent or more-blobby things live in key-value structures).

Uncoupling "how it's represented as a business-logic object" from "how it's stored in the database" lets you write classes that are more efficient - taking only what they need, and not forcing tables or relationships in the database to conform to them (which instead can be designed to optimize for the storage and processing of the database).

Of course, you can do these things with ORMs too - but it involves adding more code to your system, rather than less. The happy path of an ORM is when business logic and data storage perfectly align - in reality, this does not occur.

ORMs do offer some perks - they may claim compatibility with different database backends, and they naturally encourage DRY code (which requires more discipline with prepared statements). But in my experience, supposedly-backend-agnostic tools only "just work" for trivial cases - and relying on any tool to ensure code quality is always inadequate. Good code is a process.

So - next time you're building an application, ask yourself - is what I'm doing super-simple CRUD? If so, go with an ORM! If not, give prepared statements a try. It may take some more planning, but hopefully you'll build a better system - and you'll definitely learn something.

A Look Back on 7 Years of Automating Stuff

Jerry Ng — Mon, 04 Dec 2023 02:00:34 GMT

A little bit more than 7 years into my career, I thought it would be fun to pen down a summary post about my adventure in automating various tiny aspects of my life. Most if not all of the stuff here sprouted from my own problems and itches I needed

Building a Homelab, Part 2 - LDAP and Single Sign-On

janissary — Mon, 04 Dec 2023 02:00:00 GMT

Branchless Lomuto Partitioning

orlp.net - Blog Archive — Mon, 04 Dec 2023 02:00:00 GMT

A partition function accepts as input an array of elements, and a function returning a bool (a predicate) which indicates if an element should be in the first, or second partition. Then it returns two arrays, the two partitions:

def partition(v, pred):
    first = [x for x in v if pred(x)]
    second = [x for x in v if not pred(x)]
    return first, second

This can actually be done without needing any extra memory beyond the original array. An in-place partition algorithm reorders the array such that all the elements for which pred(x) is true come before the elements for which pred(x) is false (or vice versa - it doesn’t really matter). Finally by returning how many elements satisfied pred(x) to the caller they can then logically split the array in two slices. This scheme is used in C++’s std::partition for example.

In-place partition algorithms

There are many possible variations / implementations of in-place partition algorithms, but they usually follow one of two schools: Hoare or Lomuto, named after their original inventors Tony Hoare (the inventor of quicksort), and Nico Lomuto.

Hoare

In Hoare-style partition algorithms you have two iterators scanning the array, one left-to-right ($i$) and one right-to-left ($j$). The former tries to find elements that belong on the right, and the latter tries to find elements that belong on the left. When both iterators have found an element, you swap them, and continue. When the iterators cross each other you are done.

def hoare_partition(v, pred):
    i = 0       # Loop invariant: all(pred(x) for x in v[:i])
    j = len(v)  # Loop invariant: all(not pred(x) for x in v[j:])
    while True:
        while i < j and pred(v[i]): i += 1
        j -= 1
        while i < j and not pred(v[j]): j -= 1
        if i >= j: return i
        v[i], v[j] = v[j], v[i]
        i += 1

The loop invariant can visually be seen as such (using the symbols $<$ and $\geq$ for the predicate outcomes, as is usual in sorting):

Doing this efficiently is perhaps an article for another day, if you are curious you can check out the paper BlockQuicksort: How Branch Mispredictions don’t affect Quicksort by Stefan Edelkamp and Armin Weiß, which is the technique I implemented in pdqsort. Another take on the same idea is found in bitsetsort. Key here is that it can be done branchlessly. A branch is a point at which the CPU has to make a choice of which code to run. The most recognizable form of a branch is the if statement, but there are others (e.g. while conditions, calling a function from a lookup table, short-circuiting logical operators).

To cut a long story short, CPUs try to predict which piece of code it should run next and already starts doing it even before it knows if the code it is sending down the pipeline is the right choice. This is great in most cases as most branches are easy to predict, but it does incur a penalty when the prediction was wrong, as the CPU needs to stop, go back and restart from the right point once it realizes it was wrong (which can take a while).

Especially in sorting when the outcomes of comparisons are ideally unpredictable (the more unpredictable the outcome of a comparison, the more informative getting the answer is), it can thus be advisable to avoid branching on comparisons altogether.

Lomuto

In Lomuto-style partition algorithms the following invariant is followed:

That is, there is a single iterator scanning the array from left-to-right ($j$). If the element is found to belong in the left partition, it is swapped with the first element of the right partition (tracked by $i$), otherwise it is left where it is.

def lomuto_partition(v, pred):
    i = 0  # Loop invariant: all(pred(x) for x in v[:i])
    j = 0  # Loop invariant: all(not pred(x) for x in v[i:j])
    while j < len(v):
        if pred(v[j]):
            v[i], v[j] = v[j], v[i]
            i += 1
        j += 1

    return i

This article is focused on optimizing this style of partition.

Branchless Lomuto

A few years ago I read a post by Andrei Alexandrescu which discusses a branchless variant of the Lomuto partition. Its inner loop (in C++) looks like this:

for (; read < last; ++read) {
    auto x = *read;
    auto smaller = -int(x < pivot);
    auto delta = smaller & (read - first);
    first[delta] = *first;
    read[-delta] = x;
    first -= smaller;
}

At the time I was not overly impressed, as it does a lot of arithmetic to make the loop branchless, so I disregarded it. A while back my friend Lukas Bergdoll approached me with a new partition algorithm which was doing quite well in his benchmarks, which I recognized as being a variant of Lomuto. I then found a way the algorithm could be restructured without using conditional move instructions, which made it perform better still. I will present this algorithm now.

Simple version

First, a simplified variant which will make the more optimized variant much more readily understood:

def branchless_lomuto_partition_simplified(v, pred):
    i = 0  # Loop invariant: all(pred(x) for x in v[:i])
    j = 0  # Loop invariant: all(not pred(x) for x in v[i:j])
    while j < len(v):
        v[i], v[j] = v[j], v[i]
        i += int(pred(v[i]))
        j += 1

    return i

This is actually quite similar to the original lomuto_partition, except we now always unconditionally swap, and replace the conditional increment of $i$ with an if statement by simply converting the boolean condition to an integer and adding it to $i$.

To visualize this, the state of the array looks like this after the unconditional swap:

From this it should be pretty clear that incrementing $i$ if the predicate is true (corresponding to $v[i] < p$ for sorting) and unconditionally incrementing $j$ restores our Lomuto loop invariant. The only corner case is when ${i = j},$ but then the swap is a no-op and the algorithm remains correct.

Eliminating swaps

Swaps are equivalent to three moves, but by restructuring the algorithm we can get away with two moves per iteration. The trick is to introduce a gap in the array by temporarily moving one of the elements out of the array.

def branchless_lomuto_partition(v, pred):
    if len(v) == 0: return 0

    tmp = v[0]
    i = 0  # Loop invariant: all(pred(x) for x in v[:i])
    j = 0  # Loop invariant: all(not pred(x) for x in v[i:j])
    while j < len(v) - 1:
        v[j] = v[i]
        j += 1
        v[i] = v[j]
        i += pred(v[i])

    v[j] = v[i]
    v[i] = tmp
    i += pred(v[i])
    return i

This is our new branchless Lomuto partition algorithm. Its inner loop is incredibly tight, involving only two moves, one predicate evaluation and two additions. A full visualization of one iteration of the algorithm (the striped red area indicates the gap in the array):

We can now compare the assembly of the tight inner loops of Andrei Alexandrescu’s branchless Lomuto partition and ours:

.alexandrescu:
    mov     edi, DWORD PTR [rdx]
    mov     rsi, rdx
    mov     ebp, DWORD PTR [rax]
    cmp     edi, r8d
    setb    cl
    sub     rsi, rax
    sar     rsi, 2
    test    cl, cl
    cmove   rsi, rbx
    sal     rcx, 63
    sar     rcx, 61
    mov     DWORD PTR [rax+rsi*4], ebp
    lea     r11, [0+rsi*4]
    mov     rsi, rdx
    add     rdx, 4
    sub     rsi, r11
    sub     rax, rcx
    mov     DWORD PTR [rsi], edi
    cmp     rdx, r10
    jb      .alexandrescu

.orlp:
    lea     rdi, [r8+rax*4]
    mov     ecx, DWORD PTR [rdi]
    mov     DWORD PTR [rdx], ecx
    mov     ecx, DWORD PTR [rdx+4]
    cmp     ecx, r9d
    mov     DWORD PTR [rdi], ecx
    adc     rax, 0
    add     rdx, 4
    cmp     rdx, r10
    jne     .orlp

Half the instruction count! A neat trick the compiler did is translate the addition of the boolean result of the predicate (which is just a comparison here) to adc rax, 0. It avoids needing to create a boolean 0/1 value in a register by setting the carry flag using cmp and adding zero with carry.

Conclusion

Is the new branchless Lomuto implementation worth it? For that I’ll hand you over to my friend Lukas Bergdoll who has done an extensive write-up on the performance of an optimized implementation of this partition with a variety of real-world benchmarks and metrics.

From an algorithmic standpoint the branchless Lomuto- and Hoare-style algorithms do have a key difference: they differ in the amount of writes they must perform. Branchless Lomuto-style algorithms always do at least two moves for each element, whereas Hoare-style algorithms can get away with doing a single move for each element (crumsort), or even better, half a move per element on average for random data (BlockQuicksort, pdqsort and bitsetsort, although they spend more time figuring out what to move than crumsort does - one of many trade-offs). So a key component of choosing an algorithm is the question “How expensive are my moves?” which can vary from very cheap (small integers in cache) to very expensive (large structs not in cache).

Finally, the Lomuto-style algorithms tend to be significantly smaller in both source code and generated machine code, which can be a factor for some. They are also arguably easier to understand and prove correct, Hoare-style partition algorithms are especially prone to off-by-one errors.

Profiling Rust programs the easy way

ntietz.com blog - technically a blog — Mon, 04 Dec 2023 02:00:00 GMT

Performance is one of the big reasons to use Rust. It's not a magic wand for performance, it just gives you the control to eke out whatever performance you need. So if your program is still slow, how do you fix that?

Profiling your program is one of the best options for figuring out why it's slow and where you need to focus your improvement. Without profiling, you're guessing blindly at where the problem may lie. With a profile, you can see where most of the time is spent and focus your efforts.

There are a few ways to profile Rust programs, but my favorite is flamegraph (also called cargo-flamegraph). It's a wonderful tool that wraps around the standard profilers perf (on Linux) and dtrace (on MacOS).

Basic usage

The basic usage of flamegraph is quite straightforward and their docs cover it well, but the amount of options can be daunting. At its most basic, after you install it and dependencies^[1], you can run it as a cargo command. Here are a few of the invocations I use.

# Run your default target with no arguments
cargo flamegraph

# Run your default target with the arguments after --
cargo flamegraph -- arg1 arg2 arg3

# Run the specified bin target with arguments
cargo flamegraph -b mybin -- arg1 arg2 arg3

# Run your default target with arguments and save
# the results to a different filename
cargo flamegraph -o myoutput.svg -- arg1 arg2 arg3

You can mix and match these options to combine them. Running one of these commands will produce a file, named flamegraph.svg unless you overrode the output filename.

After you generate that, you'll want to make sure the results are reasonable to get the results you need. The output will tell you how much data it recorded and how many samples it took, something like this:

[ perf record: Woken up 59 times to write data ]
[ perf record: Captured and wrote 14.706 MB perf.data (925 samples) ]

In this example, we have 925 samples which is probably reasonable to make progress on the big things. How many samples you need will vary depending on your program, but I've not found good results too far below 1,000, and far above that seems to make things really slow. If you have big, sweeping inefficiencies, fewer samples will still catch them. If they're relatively subtle and small gains, you may need many more samples. It's an art to figure out how to tune the sample size.

To control how many samples you get, you have two options: you can change your program, or change the instrumentation. Sampling is done at a particular frequency, so you can control the program duration and you can control the sampling frequency. If you're getting very few samples, but you can make your program run for longer (larger input, multiple repetitions, etc.) then that can increase the samples you get. The same applies in reverse for too many samples. The other option is to change the instrumentation itself: you can use the -F argument to alter the frequency of sampling^[2].

# Sample at a rate of 1997 samples per second
cargo flamegraph -F 1997 -- arg1 arg2 arg3

From here with a good sample, the work is now back to you, the programmer-analyst. I like to open the SVG file in Firefox, which has a convenient viewer that allows you to zoom in and examine individual stacks of events. But you can use any suitable SVG viewer. You should be able to navigate around the flamegraph to see visually where CPU time is being spent and use that to concentrate your efforts. For a stronger introduction to how to read and use a flamegraph, see the flamegraph docs which has a section dedicated to this.

A few gotchas

While doing various profiling of my Rust programs, I've hit a few gotchas that tripped me up. Here are the ones I remember. There are certainly more, so let me know if there's something I should add to this list!

Missing system calls. When the system under test spends a lot of time in system calls, those can lead to a misleading flamegraph if they aren't captured. Since system calls transfer control to the kernel, a standard user typically cannot measure them—and perf is by default running as you! To get around that, you can have it sample as root. In flamegraph you would add the --root flag, which will use sudo to get privileges to sample everything including during system calls. This is especially important when you're doing anything with a lot of disk or network activity, otherwise the code calling those system calls may be missing and you will be on a wild goose chase!
Optimizations hiding information. As stated in the flamegraph docs, "Due to optimizations etc... sometimes the quality of the information presented in the flamegraph will suffer when profiling release builds." To address this, you either set debug = true for your release target, or you can use the environment variable CARGO_PROFILE_RELEASE_DEBUG=true.
Lockstep sampling. As Brendan Gregg points out, sampling frequencies are set off from typical frequencies used by programs. If you use a frequency like 100Hz, you may end up on the same frequency of a repeating event in your program, resulting in sampling from the same point repeatedly instead of sampling from across the entire program. You can experiment with different frequencies and see if any of them result in notably better or worse results; if they're all about the same, then you're probably not in lockstep with your program.

Now go forth^[2] and profile your programs!

On Fedora, I had to install perf with sudo dnf install perf, and I had to downgrade perf (sudo dnf downgrade perf) since the latest version has a regression which results in mangled names appearing in the generated results. If your results don't have the function names you expect, check for that. ↩
How many programming languages can you fit in a relatively normal sounding English sentence? Also, is there a language called "now" yet? ↩ ↩2

Hikaru Nakamura's winning streaks

swaits.com — Mon, 04 Dec 2023 02:00:00 GMT

Chess legend Vladimir Kramnik recently accused Hikaru Nakamura of cheating. He called out one of Hikaru's 46-game winning streaks as supporting evidence, claiming such a streak was highly improbable.

Quoting wikipedia:

In October and November, Nakamura participated in the FIDE Grand Swiss 2023, finishing in second place with 8/11 points (+5-0=6) and thus qualifying for the Candidates Tournament 2024. Vladimir Kramnik made a statement on his Chess.com profile purportedly insinuating that an unnamed high-level player was cheating on November 20. Nakamura believed this post was targeted towards him, and responded with a statement on Twitter reading "Vladimir appears to be referencing my record...is he really accusing me of cheating??? [sic]". Nakamura also expressed disappointment with Ian Nepomniachtchi for reposting Kramnik's claims.

The claim is dubious, at best. But I was curious enough to play with the data myself.

Executive Summary

I started by downloading Hikaru's games from the chess.com API. I wrote code in rust to load these games and re-simulate his chess.com career in order to arrive at probabilities of achieving win streaks. Note that by chess.com career, I mean my analysis is limited to games which happened on chess.com only.

Each game in the series was simulated based on the easily calculated probability of win or loss between two opponents with Elo ratings. I simulated the entire series of 47,407 games one million times. Yes, a million is overkill. But it sounds cool.

My analysis of 47,407 games played by Hikaru on chess.com shows that:

the probability of hitting at least one 46 game winning streak is 100%, with significant confidence
the expected number of 46 game winning streaks is 2
the actual number of 46 game winning streaks in Hikaru's chess.com career is 2
the amount of BS in this cheating accusation is significantly high

For graphs and some discussion, go straight to Results.

Methodology

Hikaru's ability to win streaks completely depends on the sequence of opponents and his skill difference with each opponent. Skill difference is, rather conveniently, quantified in chess with Elo ratings. Given two Elo ratings, we can determine the probability of one player beating another. Given the historical data, which includes opponents and their Elo ratings, we can re-simulate his career many many times and see how often he wins streaks. This approach is a Monte Carlo method.

If you want to see all the details of my model, browse the code here.

Historical Data

In order to start, I needed Hikaru's history from chess.com, sequenced in order. Order is important because, as a popular chess streamer, Hikaru has frequently played batches of opponents in very specific rating ranges, sometimes significantly lower than his actual rating.

To do this, I used curl to hit the API directly, like this:

# Fetch the list of game archive URLs for the player 'Hikaru' from Chess.com
archive_urls=$(curl -Ls https://api.chess.com/pub/player/Hikaru/games/archives | jq -rc ".archives[]")

# Iterate over each archive URL
for url in $archive_urls; do
    # Fetch the games from each archive and extract the PGN
    curl -Ls "$url" | jq -rc ".games[].pgn"
done >> games.pgn  # Append all PGNs to the 'games.pgn' file

I didn't figure this out on my own. Nope. Just found it on this reddit post.

Assumptions

`unwrap()` is ok

I'm using rust to code this simulation. We don't need full blown error handling here. This is just a hacky experiment. So, I'm going to use unwrap() wherever I want and be perfectly okay with it. To the coders out there - this isn't how I'd write production code!

Had to say it before my fellow rustaceans jump on me!

Definitions

A streak of 5 wins is also two streaks of 4 wins. So, if you win games 1 2 3 4 5, it also implies you won 1 2 3 4 and 2 3 4 5. For the purpose of this analysis, a streak only refers to the longest streak. Sub-streaks, as I've shown here, are not counted as streaks.

No Elo Updates

The best simulation we could do of Hikaru's chess.com career includes not only using Elo ratings to simulate game outcomes, but then using those simulated outcomes to update his (and his opponent's) ratings.

However, I decided against doing this because I assume (fairly confidently) that many of the pairings in his sequence of games were based on his and his opponent's ratings. Therefore, we simply take the recorded ratings and assume that the game outcomes are based on them.

This means that if we simulate an unlikely loss, we'd expect Hikaru's rating to go down. And therefore, his next opponent might be different. But, since we don't have access to the pairing methodology used in each game, we must assume that the next opponent is the same.

Draw Probability

Given two Elo ratings, we can deterministically find the probability of white or black winning. However, it doesn't tell us anything about draws. In order to simulate draws, we need to use a heuristic assumption. I came up with this:

$$ P(\text{draw}) = \frac{K_{\text{max}}}{1 + K_{\text{strength}} \times |R_{\text{black}} - R_{\text{white}}|} $$

Where $ K_{\text{max}} $ is the maximum probability of a draw and $ K_{\text{strength}} $ determines how the function scales according to the rating difference.

In my case, I landed on $ K_{\text{max}} = 0.18 $ and $ K_{\text{strength}} = 0.05 $. I admit, I just eyeballed this. I found that about 18% of Hikaru's games were draws. And, I looked at how quickly the probability of a draw goes to zero as the rating difference increases. Feels good enough to me. But, I welcome input on how it can be better.

The resulting rust code looks like:

// estimate probability of draw given two Elo ratings
fn probability_of_draw(white_elo: &Rating, black_elo: &Rating) -> f64 {
    // Constants for draw probability estimation
    let max_draw_probability = 0.18; // Upper limit of draw probability
    let rating_difference_sensitivity = 0.05; // Adjusts sensitivity to rating difference
    let rating_diff = (black_elo.0 as f64 - white_elo.0 as f64).abs();
    max_draw_probability / (1.0 + rating_difference_sensitivity * rating_diff)
}

Adjusted Win Probability

Now that we know the draw probability, we can adjust the win probability. This is done by subtracting 1.0 from the draw probability to arriave at a total (black or white) win probability, and then using the Elo expectation equation to assign that remaining win probability proportionately.

First, find the probability of any player winning:

$$ P(\text{white or black}) = 1.0 - P(\text{draw})$$

Next, compute the expectation of winning for each player according to their Elo ratings:

$$ P(\text{white}) = \frac{1}{1 + 10^{(R_{\text{black}} - R_{\text{white}}) / 400}} $$ $$ P(\text{black}) = 1 - P(\text{white}) $$

And finally, adjust for the overall win probability, which takes draws into account:

$$ P(\text{white}) = P(\text{white}) \times P(\text{white or black}) $$ $$ P(\text{black}) = P(\text{black}) \times P(\text{white or black}) $$

In code, it looks like this:

// estimate probability of white winning, given two Elo ratings
fn probability_of_white_win(white_elo: &Rating, black_elo: &Rating) -> f64 {
    // Expected outcomes for players A and B
    // E_a = 1 / (1 + 10^((b - a) / 400)
    // E_b = 1 - E_a
    1.0 / (1.0 + 10.0f64.powf((black_elo.0 as f64 - white_elo.0 as f64) / 400.0))
}

// estimate probability of black winning given white's probability of winning
fn probability_of_black_win(probability_of_white_win: f64) -> f64 {
    1.0 - probability_of_white_win
}

// given two ratings, compute probabilities of win/loss/draw and return in `ExpectedOutcome`
pub fn calculate_expected_outcome(white_elo: &Rating, black_elo: &Rating) -> ExpectedOutcome {
    // Estimate win probabilities
    let mut p_white = probability_of_white_win(white_elo, black_elo);
    let mut p_black = probability_of_black_win(p_white);

    // Estimate draw probability based on rating difference
    let p_draw = probability_of_draw(white_elo, black_elo);

    // Adjust win probabilities to account for draw probability
    p_white *= 1.0 - p_draw;
    p_black *= 1.0 - p_draw;

    // Sanity check
    assert_approx_eq!((p_white + p_black + p_draw), 1.0);

    ExpectedOutcome {
        // Estimate win probabilities
        white: p_white,
        black: p_black,
        draw: p_draw,
    }
}

Game Simulation

We simulate a game by using the two players' Elo ratings to compute teh win probability of each player and the draw probability, which always sums to 1.0. Then we simply use a random number to choose a simulated outcome. Here's the rust code:

// simulates a single game
fn simulate_game(white_elo: &Rating, black_elo: &Rating) -> Outcome {
    // Get expected outcomes
    let e = calculate_expected_outcome(white_elo, black_elo);

    // Roll the dice
    let mut rng = rand::thread_rng();
    let random_value = rng.gen::<f64>();

    // Determine and return outcome
    if random_value <= e.white {
        Outcome::WhiteWin
    } else if random_value <= (e.white + e.black) {
        Outcome::BlackWin
    } else {
        Outcome::Draw
    }
}

Now to simulate the entire series of Hikaru's games, we just apply this function to the historical games to come up with a new set of outcomes.

Like this:

// simulate a series of games and return a new simulated `Vec<Game>` series
pub fn simulate_games(actual: &[Game]) -> Vec<Game> {
    actual
        .iter()
        .map(|g| Game {
            white: g.white.to_owned(),
            black: g.black.to_owned(),
            outcome: simulate_game(&g.white.rating, &g.black.rating),
        })
        .collect()
}

Once we have all the (one million!) simulated chess.com careers, we build histograms of achieved streaks for each and then merge them into a single histogram for the entire simulation. Here it is all together:

let iterations = 1_000_000;
let histograms: Vec<HashMap<usize, usize>> = (0..iterations)
    .into_par_iter()
    .map(|_| {
        let simulated_games = simulate_games(&games);
        generate_win_streak_histogram("Hikaru", &simulated_games)
    })
    .collect();

// Reduce the histograms into a single histogram
let merged_histogram = histograms
    .into_iter()
    .reduce(|mut acc, h| {
        for (k, v) in h {
            *acc.entry(k).or_insert(0) += v;
        }
        acc
    })
    .unwrap();

Note the use of rayon with into_par_iter(). This is a really nice bit of code which makes it trivially easy to use all my cores.

You can find all of my code here.

Results

I simulated the entire 47,407 game sequence one million times. This section details my findings.

Actual Winning Streaks

Before getting into the model's results, I think it's interesting to look at what we found in Hikaru's data.

My code reports a good number of long winning streaks in Hikaru's chess.com career. Here are his 40-game and longer streaks, along with the number of times he achieved each:

Streak Length:  40, Count: 4
Streak Length:  41, Count: 2
Streak Length:  42, Count: 4
Streak Length:  43, Count: 1
Streak Length:  44, Count: 1
Streak Length:  45, Count: 2
Streak Length:  46, Count: 2  <= OMG DID HE CHEAT WTF 🤡
Streak Length:  47, Count: 6
Streak Length:  48, Count: 1
Streak Length:  50, Count: 1
Streak Length:  51, Count: 2
Streak Length:  52, Count: 1
Streak Length:  54, Count: 3
Streak Length:  55, Count: 2
Streak Length:  56, Count: 2
Streak Length:  57, Count: 1
Streak Length:  58, Count: 1
Streak Length:  59, Count: 2
Streak Length:  61, Count: 2
Streak Length:  62, Count: 1
Streak Length:  69, Count: 1
Streak Length:  72, Count: 1
Streak Length:  77, Count: 1
Streak Length:  78, Count: 1
Streak Length:  82, Count: 1
Streak Length:  85, Count: 1
Streak Length: 100, Count: 1
Streak Length: 117, Count: 1

Yes, you're seeing that right. There are 85, 100, and even 117 winning streaks. You see, these streamers will sometimes play a lower-rated player many times consecutively. If they prevail some pre-chosen number of times (say, 100), then they will colloquially say that they "adopted" that player. It's fun. And this is why it's important to look at the entire, exact sequence of opponents Hikaru played.

Probability of Win Streaks

Based on my analysis, the following chart shows the probability of getting at least one winning streak. For example, if you look at 46 on the x-axis, you'll see its probability is 100% on the y-axis.

As another example, you can see that the probability of hitting at least one 80-game winning streak is just a bit below 30%.

We can even see about a 5% change Hikaru would've hit a 140 game streak. So, if there were 20 parallel Hikaru's playing exactly the same games, maybe one of them would have pulled this off.

Oh, but what about those weird bumps at the 70, 100, 115, and even 140 streaks? Again, remember that this is based entirely on the sequence of opponents Hikaru played. If at some point in his nearly 50k games he played someone rated significantly lower than him for 70, 100, 115, or 140 times consecutively, then that increases the probability of winning that streak.

And when we look at the data, Hikaru does in fact have streaks at 69, 72, 77, 78, 82, 85, 100, and 117.

Herein lies the plainly obvious reasoning that Kramnik seems to be missing.

Expected Number of Win Streaks

Probability is one thing. In this case, it removes all doubt about the possibility of achieving something like a 46-game winning streak.

But what's more interesting when we look at whether someone cheated is how close to the expectations they performed. The chart below plots both the expected number of each winning streak and actual number of winning streaks recorded in Hikaru's career.

The red line is the number of streaks predicted by my modeling. The blue line is the number of streaks Hikaru scored.

For example, we can see that we expect Hikaru to get about 35 streaks of 20 wins. Just look across the x-axis to 20, and go up to find that the red line crosses about about 35. But note that the blue line shows Hikaru only actually got about 28 of those 20 game streaks.

If we look at the 46 streak data point, we see an expectation of 2. And we also see an actual of 2. Hmmmmm....

One note on this chart: the y-axis is logarithmic. This makes it so that we can show a much larger range of data. It might seem a little weird to some, but it's pretty normal.

The data gets a bit noisy to the right of the 25 streak line. This is because such streaks occur less frequently, and thus our sample size gets quite a bit smaller.

How is Hikaru Performing?

Now, let's plot the difference between what Hikaru did and what my simulation expects at each of the streaks of 20 and higher.

To use this chart, look for a winning streak on the x-axis, like 55. We see that the (actual - expected) is 1. This means that Hikaru recorded 1 more 55-game winning streak than expected.

Generally, we can see that for streaks 20 and above, Hikaru is performing just about as expected. If we smoothed this line to account for the noisiness at this end of the data, we'd find that he's well within the margin of expectations.

Closing

Hikaru's win streaks on chess.com are normal. As I said in the beginning, my analysis of 47,407 games played by Hikaru on chess.com shows that:

the probability of hitting at least one 46 game winning streak is 100%, with significant confidence
the expected number of 46 game winning streaks is 2
the actual number of 46 game winning streaks in Hikaru's chess.com career is 2
the amount of bullshit in this cheating accusation is significantly high

Being great at one thing, chess in this case, doesn't mean you're great at everything. Even being literally one of the best chess players in the world is not a reason to think someone is smart. But, our society tells people who achieve greatness in chess that they're smart. And apparently for some folks, especially in this case, that's gone to their head. Now, perhaps Kramnik has some other evidence to Hikaru's cheating. But, what we've seen thus far - the 46 game winning streaks - is not enough. It's not enough for even the most basic of high-school level math and statistics. And so, being chess world champion does not mean you're smart.

That said, many chess players are actually damn smart. It tends to attract the bookish type who are into educating themselves. And, nothing but respect from me for people who can play chess. I am not one of those people.

If you find a mistake in my work that shows I'm wrong, please let me know!

And finally, I'll leave you with this, courtesy of ChatGPT and DALL-E:

A portrait of a sad clown sitting at a chess table, deeply engrossed in a game of chess. The clown has a striking resemblance to a tall, slender man with sharp facial features, reminiscent of a famous chess grandmaster. He wears a traditional clown outfit with a ruffled collar and a small hat, and his face is painted with exaggerated sad expressions. The setting is a dimly lit room, with a focus on the chessboard and the contemplative pose of the clown.

AWS re:invent 2023

Tiago Rodrigues — Sun, 03 Dec 2023 21:52:00 GMT

This was my first re:invent and boy, oh boy, it was overwhelming in a good way. I want more!

It was my first time outside of Europe, and I have to say, jetlag is a real thing, with 8 hours difference I was waking up at 3 am every

Ordering a Cargo field

River Writes - A MediaWiki Blog — Sun, 03 Dec 2023 20:02:20 GMT

How can you enforce a sorting order on a Cargo field when displaying tabular data?

ChatGPT After One Year

Matt Rickard — Sun, 03 Dec 2023 16:30:00 GMT

ChatGPT was released on November 30th 2022. What has changed since then? - **Hundreds of open-source models.** Varying sized models from small to very large. M

Alternatives to (math) academia and how to get there

This Is My True Name — Sun, 03 Dec 2023 09:08:55 GMT

This is a reference post for math PhDs

Comic: Magic spells

Zachary Proser — Sun, 03 Dec 2023 02:00:00 GMT

A comic strip about casting magic spells

You Buy; They Fly

taylor.town — Sun, 03 Dec 2023 02:00:00 GMT

Friends are cool. Spending money sucks; travel is exhausting.

2023 in Review

Jason Eckert's Website and Blog — Sun, 03 Dec 2023 02:00:00 GMT

For me, 2023 turned out to be a year of learning, growth, adventure, and gratitude. And since there’s plenty for me to talk about, I’ve divided my review into two sections: one that summarizes my professional pursuits, and one that summarizes everything else.

The professional stuff:

On the college front, the new technology programs I’ve been pushing since 2020 finally got to market! My favourite one is Mobile Web Developer, as it’s a radical departure from what we did previously. The whole program is taught exclusively on macOS, has two interlaced capstones, and is carefully crafted to allow students to build solid developer skills using modern industry tooling and processes with zero tests/quizzes (students instead perform weekly assignments). These weekly assignments allow instructors to assess and guide skills development for individual students throughout each course, as well as echo the sprint cadence used within industry that students must become accustomed to. We also have a new IT Professional (Infrastructure, Cloud & Cybersecurity) program that restructures and augments many of the topics in our previous IT Professional program to allow for expansion into the areas of cloud and cybersecurity (both of which are key industry focal points today).

Konrad Zuse's Early Computers (Review)

ℤ→ℤ — Sun, 03 Dec 2023 01:39:27 GMT

Although historians are usually pushed to decide questions of “who was first,” the question of impact is more interesting. Konrad Zuse has a claim to the first computer, the first high-level computer language, and the first European commercial computer sale, but his relative isolation working in Germany in the 30s and 40s limited his visibility and impact. It wasn’t until the 1970s for his innovations to become more widely known. Raúl Rojas has long contributed to the historical appreciation of Zuse and the remarkable modern nature of his machines. This volume collects and makes available in English many of his previous articles on the subject.

Raúl Rojas. Konrad Zuse’s Early Computers: The Quest for the Computer in Germany. Springer 2023. Publisher Link

The book is primarily a technical history, discussing how each of Zuse’s early computers (Z1, Z2, Z3, and Z4) were designed and implemented, including the post-war effort to add conditional jumps to the Z4. The author also provides one of the clearest explanations of Plankalkül, Zuse’s high-level, symbolic programming language with an unusual two-dimensional syntax (likely inspired by engineering spreadsheet computation forms). The author also covers the S1 and S2, special-purpose computing machines for finding control corrections caused by aircraft wing manufacturing defects. Zuse’s efforts towards chess playing software are explained. Finally, the book contains two chapters providing historical context on the world-wide efforts to build computers and insight into the politics of Zuse and his close associates.

Rojas' explanations are very clear, improved by a large number of illustrations and diagrams. The tone is measured and he provides evidence for his claims, and the limits of those claims. For instance, the Z4, built between 1942 and 1945, lacked any instructions for jumps, conditional or not. Zuse was aware of the limitation (as noted in his journals of the time), but since the Z4 only had 64 words of memory, he felt it was an acceptable limitation. After the war, the ETH university leased the machine in 1950, but with the proviso that Zuse add conditional jumping. Using only microinstructions, he was able to add test operations (e.g. x equals zero, x is greater than zero) and a subroutine jump instruction. Although the jump was still limited, limiting programs to a single loop, this was sufficient for the use cases of the day. Rojas discusses how this limitation factors into the debate over the first computer, both practically, comparatively (comparing it to the ENIAC and the IBM/Harvard Mark I), and theoretically (how this limitation can be ignored in terms of Turing’s infinite tape).

(If you are interested in how the Z4 was used at ETH, I recommend reading the chapter “Computer Development in Switzerland” in Herbert Bruderer’s Milestones in Analog and Digital Computing, Vol 2.)

In terms of the book’s goal — describe the progression of Zuse’s thinking and the development of his computers through the specification of the Plankalkül language — it succeeds in its purpose. We learn little about Zuse himself, but that is not the point of the book. Due to the after-effects of the war, Zuse was penniless and lost many years before he could restart his efforts and build new computers. In the mean-time, the U.S. and British efforts quickly iterated on new designs and established new markets. While Zuse was not unknown — he met Turing and Plankalkül was referenced by the ALGOL committee — his impact was limited, perhaps partially due to his patents being disregarded by the German government and his doctorate thesis not being published.

However, since the matter of impact is all post-war, I think the book ended too early. Zuse’s story is intertwined with post-war politics and geography, the influence of patents, and the beginning of the commercial computer industry. None of these are technical topics or within the author’s expression of scope, but by not including them, the reader needs to read another volume to get the full story.

Also, the book could be improved by the addition of some color plates as some of the photos and diagrams have more detail than is comfortable in grey-scale. Most of the diagrams are high-quality and easy to read.

Recommended. At $38 for the hardbound edition, this is a good value for an academic work.

An Album for Each Year (2023)

William Huster — Sat, 02 Dec 2023 20:00:00 GMT

I love lists. So here’s a list albums, one for each year of my life. This exercise was inspired by a blog post by the CTO of Amazon, Dr. Werner Vogels.

The rules are simple, Select “a favorite album for every year of your life”. The challenge had two restrictions:

Only one album per year
No repeat artists

I’m probably profiling myself pretty heavily by sharing this list, but who cares! I won’t go through them all, but I have a meaningful memory attached to each of these albums. My 2023 album of choice is a particularly weird and delightful one: “One Wayne G” by Mac Demarco. This album is actually an archive of musical sketches and experiments. The songs are not really complete or cohesive, but they have Mac’s signature style and are all pleasant to listen to. The titles of the songs are mostly just dates, like “20221102.” I found this really inspirational. I ended up doing a few similar musical experiments this year because of this album. Titling song sketches with a date is liberating, because it relieves the pressure of trying to make a complete song.

One of my favorite songs on the album actually does have a title: Proud, True Toyota. It’s hilarious and speaks to me, because I too own a proud, true, Toyota Tundra. The song is ridiculous, and it’s also ridiculous that I live in Washington DC and own an old pickup truck! It’s honestly not very smart or convenient, but it makes me happy.

Finally, the album will also always reminds me of my mom who passed away this year. I was playing the album through my bluetooth speakers at home with her in May, around Mother’s day. That was just a few weeks before she died on June 7.

1987: The Cure, Kiss Me, Kiss Me, Kiss Me
1988: The Pixies, Surfer Rosa
1989: Kate Bush, The Sensual World
1990: Depeche Mode, Violator
1991: Nirvana, Nevermind
1992: Weird Al Yankovic, Off the Deep End
1993: Smashing Pumpkins, Siamese Dream
1994: Pink Floyd, Division Bell
1995: Björk, Post
1996: Barenaked Ladies, Born on a Pirate Ship
1997: Radiohead, OK Computer
1998: Neutral Milk Hotel, In the Aeroplane Over the Sea
1999: Eiffel 65, Europop
2000: Enya, A Day Without Rain
2001: The Strokes, Is This It
2002: Coldplay, Clocks
2003: Belle and Sebastian, Dear Catastrophe Waitress
2004: The Killers, Hot Fuss
2005: Modest Mouse, Good News for People Who Love Bad News
2006: Hot Chip, The Warning
2007: Bright Eyes, Cassadaga
2008: Bon Iver, For Emma, Forever Ago
2009: Grizzly Bear, Veckatimest
2010: Kanye West, My Beautiful Dark Twisted Fantasy
2011: Sufjan Stevens, The Age of Adz
2012: Grimes, Visions
2013: Chvrches, The Bones of What You Believe
2014: St Vincent, St Vincent
2015: Tame Impala, Currents
2016: David Bowie, Blackstar
2017: Kendrick Lamar, DAMN.
2018: Janelle Monáe, Dirty Computer
2019: Aldous Harding, Designer
2020: Phoebe Bridgers, Punisher
2021: Khruangbin, Mordechai
2022: Mighty Brother, Azimuth
2023: Mac Demarco, One Wayne G

McNamara Fallacy

Matt Rickard — Sat, 02 Dec 2023 16:30:00 GMT

The McNamara Fallacy is named after Robert McNamara, the US Secretary of Defense during the Vietnam War. The fallacy describes making decisions using only quant

Assumptions can be dangerous

I Learnt — Sat, 02 Dec 2023 14:00:00 GMT

There is a quote “A backup is not a backup unless you have recently performed a restore from it”; I got to find this out the hard way.

Python Coroutines

Thought Eddies — Sat, 02 Dec 2023 11:44:55 GMT

Exploring accidental synchronous blocking in asynchronous programming in Python with coroutines.

Goodbye Twitter

Reiterations — Sat, 02 Dec 2023 03:52:00 GMT

I’ve removed all Twitter links from my site.

For All Mankind: Season 1

🦉 olshansky 🦁 — Sat, 02 Dec 2023 03:18:01 GMT

Olshansky's review of For All Mankind: Season 1

Retrospective on November 2023 wellness challenge

Ben Overmyer's Site — Sat, 02 Dec 2023 02:00:00 GMT

November's wellness challenge was a success.

I had no alcohol, despite going out with friends and coworkers multiple times. It helps that there are several good non-alcoholic beer options these days. I also kept up the workout regimen. I went to the gym four times a week every week. The only exception was Thanksgiving Day.

Speaking of Thanksgiving, despite eating normal Thanksgiving dinner (twice), I only had a brief bump in weight gain. I'm now the lowest weight I've been since I started tracking in 2019.

I'm planning on continuing the workout regimen and keeping my diet dialed in for muscle gain and fat loss. This morning I weighed in at 219 lbs. I'm hoping to get down to 200 lbs. by mid February, but a more realistic goal is 210 lbs. We'll see where I end up.

I never tried gauging what my one-rep bench press is. I should probably try that next time I'm at the gym.

Advent of Code 2023 in Oracle - Day 1 and 2

Philipp Hartenfeller Blog RSS Feed — Sat, 02 Dec 2023 02:00:00 GMT

Advent of Code 2023 in Oracle SQL and PL/SQL. Day 1 and 2.

Vector search and Retrieval Augmented Generation

Michael Ian Lewis — Sat, 02 Dec 2023 02:00:00 GMT

This is an overview of vector search and Retrieval Augmented Generation (RAG) in the context of text data, illustrated with worked code examples, and some closing thoughts on their usage in practice.

It is primarily to clarify my understanding and for my future reference, and is likely to be updated with further information over time. It may be of interest to others, but those familiar with vector search and RAG can skip the terminology and examples sections.

Advent of Code 2023

Home on mfashby.net — Fri, 01 Dec 2023 23:42:21 GMT

Render Python Encodings Error

Thought Eddies — Fri, 01 Dec 2023 20:04:04 GMT

Render Python Encodings Error

tstate - strongly typed Typescript state machines

Martijn Arts — Fri, 01 Dec 2023 19:12:26 GMT

If you read this blog or my digital garden you'll know I like working with state machines in the frontend. There's plenty of reasons for this, but the two main ones are that user interaction maps really well to state machines, and so do shared background

Data Quality in LLMs

Matt Rickard — Fri, 01 Dec 2023 16:30:00 GMT

Good data is the difference between Mistral’s LLMs and Llama, which share similar architectures but different datasets. To train LLMs, you need data that is:

Document your house

I Learnt — Fri, 01 Dec 2023 14:00:00 GMT

We have manuals and guides for the software we use. We have a manual for the car we drive. Why not the house?

Doctor Who: The Unearthly Child (1963) Audio Description Script

Brett Coulstock - Writer — Fri, 01 Dec 2023 11:00:00 GMT

Unofficial audio-description script for the unbroadcast pilot episode of the 1963 Doctor Who serial The Unearthly Child in CSV, HTML and SRT format.

Control - how to make a game enjoyable for casual audiences

./techtipsy — Fri, 01 Dec 2023 10:00:00 GMT

How one toggle in the game settings allowed me to enjoy a notoriously difficult game.

Ideas for filtering fediverse timelines

All content on Seirdy’s Home — Fri, 01 Dec 2023 09:12:05 GMT

An alternative to controversial recommendation algorithms is timeline-filtering algorithms. Feeling pressured to scroll through too much is unhealthy and lends itself to addiction; filtering things down once a user scrolls down far enough should be a welcome opt-in feature.

For example, a mutuals-only filter combined with a minimum-interaction-count filter could be a good way to “catch up” quickly after a long absence. Only showing posts common across a set of timelines is another; for example, I would like to see a filtered view of Akkoma’s bubble timeline restricted to what’s on my home timeline.

Conditionally showing reblogs is another way to filter. Unread reblogs from mutuals should be the only visible reblogs I have scrolled down far enough.

App Defaults in 2023

Spencer Harston — Fri, 01 Dec 2023 07:21:00 GMT

Inspired by others I've seen across various blogs and on Mastodon, here's a list of default apps I've been using at the end of 2023. I doubt this will change much year-to-year, but we'll see what next year brings.

Email: Gmail
Calendar: Google Calendar
Documents/Cloud Storage: Google Drive
PDF Reader: SumatraPDF
Photos: Google Photos
Password Manager: Bitwarden
Notes/Lists/Tasks: Google Keep, Tasks.org (Android)
Terminal: Windows Terminal, Termux (Android)
Text Editor: Visual Studio Code
Screenshots: ShareX
Browser (Desktop): Firefox Developer Edition, Chrome (Google apps only)
Browser (Mobile): Fennec, DuckDuckGo, Cromite, Chrome (Google apps only)
- I use URLCheck to pick which app to use when tapping a link
Search Engine: DuckDuckGo
Chat: Google Messages, Discord, GroupMe
Music: YouTube Music, MusicBee, Symfonium (Android)
Podcasts: Pocketcasts (Android)
Audiobooks: Audiobookshelf
Ebooks: Calibre
RSS Reader: FreshRSS
Mastodon: Moshidon (Android)
Hacker News: Harmonic (Android)
Twitch: Xtra (Android)
Habits: Loop Habit Tracker (Android)
File Browser: Solid Explorer (Android)
Ad Blocker: uBlock Origin, NextDNS
VPN: Mullvad, Google One, Tailscale

See also my Uses page for hardware and other software.

Startup Reflections

Luke Singham — Fri, 01 Dec 2023 02:00:00 GMT

These are some of the tough lessons learned from my start-up journey. Writing this has been incredibly cathartic.

My top two key-takeaways are:

Validate the problem and willingness to pay upfront i.e. find someone that says 'how can I use this now? Please take my money'
Stress test your co-founder relationship.

Product #

For a little context, our two sentence description of ilumadata was:

ilumadata is Retool for data pipelines. We're building a cloud editor for enterprise data teams that makes it easy to build, test, and launch data transformations in seconds.

Lessons:

Read the Mom test first, it's a quick read and will save you a lot of time. To validate your product, you need to:
- Find and validate the problems your target market has. Ask, what is your biggest problem? Have you tried fixing it? How often does this happen? What is the impact?
- Get some sort of commitment. Will they sign a Letter of Interest (LOI)? Introduce you to their boss? Pay money now? Yes you can do this without a product.
For ilumadata the adoption cost was greater than the value add. See the Collison install for how Stripe reduced their initial adoption cost.
The difference between iterating on your product based on discovery calls and no users vs. paying users is huge.
Ultimately ilumadata was a nice to have - a proliferation of data tools, and a change in the macroeconomy meant no space for nice to haves.
Having domain expertise can de-risk the venture. A lot of seed funding is based on a solid team with a domain insight. On reflection, was our insight shallow? Maybe, there are a million reasons for why startups fail. The big product de-risk is asking, do people want this? No, do they really want it? I thought I de-risked this early on with some discovery calls, but in retrospect they failed the Mom test. To de-risk efficiently, try selling a product that doesn't exist.
Iterate faster - constantly update what you are trying to learn from the next call. What is the hypothesis you are trying to prove/disprove?
There isn't a definitive way to know when to pivot. There are plenty of examples where founders stuck to their initial idea and it worked out e.g. Retool and Airbnb. In our case we did not have enough traction to justify sticking to our initial idea.

"if a company makes a product that people dislike yet still buy, then presumably they'd buy a better alternative if you made one" - pg

👆 When we started, no one liked dbt cloud, the mistake was thinking they were an indicator of success. Yet a year later, based on our user interviews dbt cloud did not appear to have high uptake of their paid cloud product. 7 months into our venture they had laid off 50% of their workforce and doubled their pricing. dbt core solves some real pain points, but as far as I can tell, dbt cloud is a nice to have. dbt has recently overhauled their cloud product, so perhaps they have found a way to make it a must have - time will tell 🤷‍♀️
Does your product need SOC II? We thought we did. We had found an evangelist user at a large US healthcare company who was keen to use ilumadata. Maybe it's going to be our first enterprise sale! In prepping us for the sales process they said we needed SOC II. So we rushed out and got it in two weeks. The lead went cold, completely ghosting us 👻. But we'd just forked out 20k to get our shiny SOC II badge. In hindsight, before getting SOC II, we should have been in front of the buyer and got a commitment from them. SOC II companies sell early stage startups on fear 'Don't let SOC II be the reason you lose your first big deal'. If you are really solving a problem, you should get a commitment from the buyer before you get SOC II.
We were building a data platform, a data operating system, a single pane of glass for data infrastructure. Rather than tackle one single easy to communicate problem, we were solving many smaller problems. We'd tell ourselves certain features were table stakes. For early stage, I'd avoid being a platform. It's hard to communicate, hard to sell and hard to build.
Start quantifying the top problems from each discovery call. Card sorting is a great exercise to do this. Put 5-7 problems you think your persona has on a kanban board, share your screen and ask them to sort them from most painful to least. Several months in, we realised that we weren't addressing the most significant or painful problem. Card sorting also gets people talking about their problems.
In an attempt to breach the chasm between the data team and the rest of the business, we fell into it. The product was too technical still for business folk and too light touch for those tech inclined data people.

Co-Founders #

Here's a guide on: How to find a co-founder

Co-founding a company is the business version of marriage. The first signs we didn't work well together occurred during the YC batch. We had left the happy place of coding as a side project, and now were making all sorts of time pressured decisions to make this a successful company. Our communication styles were very different. Founders are literally the foundation of the company, not working well together always meant a separation was inevitable^[1].

Lessons:

Perhaps the most important factor in an early stage startup, really invest time here building and testing the relationship.
Make sure you go through and write down your answers to First Mark's 50 Founder Questions. It prompts both you and your potential co-founder to reflect on some hard questions, but also helps uncover where there are discrepancies between the two of you. Do this before you start working together. We didn't do this.
It is a marriage, your incorporation documents are your marriage certificate and it takes significant legal fees, stress and heartache to get divorced. Would you get married to someone you haven't really spent time getting to know?
- Work together - starting to code on the idea is unlikely a good stress test. It's just one area of what you need to work on as founders and as tech co-founders you're in your happy place! It tests whether you work as an engineering duo. But what about when you're stressed, tired, and have to make a decision on product direction? I know founders who have purposely set up 12hr days working side by side with a packed agenda to stress test. Others have gone on a two week working 'holiday' together - and then chosen not to continue, a lot better than years later.
- Pick things you disagree on and commit to debating passionately. If you can't do this productively now, how will you in the future?
Listen to any negative hunches you might have - this will likely get worse later on. Would you hang out solo with them, just for fun? If not, why not?
Pre-define areas of ownership. I made the mistake of everyone does everything, which is often true in startups, but everyone doesn't have to agree on everything. Seeking consensus on operational items should perhaps be open to a round or two of feedback, but ultimately the decision to execute and move fast lies with the owner. Your co-founder should be comfortable with this. Areas of responsibility and how you resolve deadlocks is covered in the 50 Founder Questions.

Sales #

We initially had an enterprise sales motion. Following several months of zero traction, we shifted our focus down market, targeting small to medium enterprises. Take the following lessons with a grain of salt, because ultimately we didn't make lots of sales! 😞

Lessons:

Enterprise sales is hard. Super long sales cycle. Lots of touch points. I've had a few emails an entire year later saying they are now 'interested' in buying.
Pre-Product Market Fit both founders should be doing at least some selling. Stick your noses into the market. If you're not selling, you're not learning. And if you haven't found a 'hair on fire' problem, then building may be a waste of time. But sure does feel productive!
Customer onboarding - ensure technical founder/s are present for this. Feel the customer onboarding pain, so it gets prioritised appropriately against other engineering pains.
Read the Mom test first, oh I said this already! Have you read it yet?
You can sell first, then build. Yes, sell a product that doesn't exist. Do outbound "we're building X that solves Y" - this is your hypothesis that you should be able to accept or reject after 5 decent calls. Focus on the person's problems and if what you're building solves a genuine pain point, be transparent about the development progress and get a commitment e.g. a meeting with their boss or an LOI or a design partnership!
There were a couple of hunches I felt when selling to a technical audience. Firstly, they were more interested in how you engineered something - were they wondering how to build it themselves? Secondly, technical teams trust open source more. Which is related to the ease of adoption.
Depending on your product, the user and buyer may be different people. This makes it harder for you to sell. As you need to win both to be successful.

Buyer

We took too long to realise the initial problem we were solving was not the top priority. There was much greater concern with data quality. Here's the results from the card sorting exercise of 88 calls. The problem we started solving placed 6th 🤦‍♀️.
Start with buyer validation, not user validation. Getting a commitment from a buyer is better than the commitment a user can offer you.

User

We spoke to many users and got introductions to their bosses. But initiating contact with users to reach the buyer, involves engaging multiple people within the same company to identify someone willing to intro you. Start with the buyer, at least you can qualify them faster, Please Sir, do you have budget?

Investors #

I'm extremely grateful to everyone who took a bet on ilumadata. We raised a small round ($720k) from a mix of angels and a VC.

Don't be afraid of owning your failure i.e. don't let ego get in the way. I let this get in the way of regular investor updates, I'd think I'll wait for some good news (that never came!). I should have been more open with investors, they are there to help you. I was afraid of being judged, but in retrospect I was judging myself more harshly than they would have.
- Many investors will go into bat for you. If you're not open with them, they can't help you.
We raised a small round but it cost 2 months of time. Did we need to raise? Whilst in YC "The mantra was if you can raise at Demo Day, you should raise" (see another YC founder's experience of this)
- We could have proven this idea didn't work with only the YC investment. In fact, we could have validated it with no money!
Don't be driven by investors, they are not the buyer.
- If they ask for a product roadmap and you are pre-PMF and pre-seed, this is a waste of time. Spend time on getting traction. I remember Siebel during our batch saying "chuck out your Product Roadmap!"
- I've filtered out potential co-founders based on focusing on 'investor traction' rather than customer traction.

Market #

You don't want to be working on a 'nice to have' at the best of times. But if you are and you have a enterprise sales motion, recession fears and major tech lay offs mean most enterprise doors are closed.
One early thought was - 'if we improve productivity even by a tiny fraction like 5% then using ilumadata is an obvious win'. Firstly, your adoption cost needs to be low to prove this, ours was not. Secondly, even this is not enough, you need to solve a problem and 'improving efficiency by 5%' is not a problem, let alone a painful one.
Market Saturation - so many tools in this space, and it only got worse as we were out there. We thought our product was unique or at least sufficiently different, then we started finding many companies doing the same thing.
Outside of the real core of a data platform, cloud data-warehouses and BI, who is getting decent revenue? I had a couple of investors that had invested elsewhere in the modern data stack that many of the big names in that space weren't making much in the way of revenue.

Legal #

Cease and desist - we got one. It scared us and distracted us for at least a month. Don't be scared of this. Unless - you really did break laws!
- Our YC Partner congratulated us, saying it's a sign you're doing something right. It's a badge of honour.
- Big incumbents sometimes monopolize the leading law firms by simultaneously assigning legal matters to all of the top-tier firms. That way, the law firm you're using will have a potential conflict of interest and need the company that sent you the C&D to sign a waiver. Which of course they won't do! It's like a game of monopoly, and they've parked hotels on the top end of town.
- Get referrals to boutique specific law firms - we landed an incredible litigation firm that was very reasonable on the company purse.

Self #

Mental health - I'm not someone who has had any diagnosable mental health issues. Doing a startup is very hard. Everyone tells you this. But the lived experience is very different. Facing hundreds of rejections, many iterations, a hard pivot, co-founder separation and eventually a wind-down is tough. I remember seeing the 'trough of sorrow' graph before going through YC, it's very real!

Sales Led Growth is hard! Not sure if it would be different with better traction - a bit of dopamine goes a long way.
I'm definitely not a perfectionist - if it works, let's ship it! I'm very comfortable with things being rough around the edges.
I learnt more about what I need from a co-founder. Much like your first non-platonic relationships, you learn the hard way what works and doesn't work for you.
I found talking to other founders really helpful, normalising the struggles unique to startups, hearing about what they have tried, what's worked and what hasn't - it's something I do regularly now, but could have done more through the hard times. This is a huge benefit of going through an accelerator, to forge your network of entrepreneurs.
CEO - this role is tough, and it's a role I don't particularly relish as a full-time position. A lot of startups in our batch had the fight over the role, many have co-CEO's until your group partner finally yells at you enough to change! For us, CEO was really who drew the short straw! Early on, I split my time between sales and dev work. After all, we had an MVP to build. 6 months in and the code base was still 40% written by me. This felt productive, but in retrospect validating the problem and willingness to pay would have been time better spent. The last few months of the venture I was pure sales/product, soft pivoting through problem areas in the modern data tooling space. When the business/tech ratio became 100% business for me, that's when I realised I didn't want to be CEO.
There is an inherent duplicity that must be maintained, startups are innately difficult and full of uncertainty. But you must project confidence and optimism when raising funds, when selling to customers and to (potential) employees. All whilst you might be plagued by doubts or recovering from the last bruising 'No' you were banking was a 'Yes'! I found this balance hard to maintain.
I sought consensus. Given the co-founder relationship was already strained, I didn't want to add fuel to the fire. What I should have done, was make clear areas of responsibilities, provide a short time frame for feedback and then execute. The trade-off is speed vs. consensus, and for early stage startups you don't have much time.

Essential reading #

Footnotes #

One exception to this is if you find PMF - it solves everything. I've heard of founders absolutely hating each other and that trickling down into very poor company culture. But when you're on a rocket ship to a big payout, people will stick it out 🥕! ↩︎

How to find a co-founder

Luke Singham — Fri, 01 Dec 2023 02:00:00 GMT

I recently went on the hunt for a new co-founder. This ended up being my process.

Starting principles #

Meet as many as you can in parallel. Like a sales pipeline, you wouldn't only talk to a single customer at a time, nor should you for founders.
Give yourself a deadline, goal setting matters, it also creates urgency so you can close.
It takes longer than you think, several months for me and I know many others that took a lot longer.
Decide on some initial parameters:
- Geography: Initially for me, this was East coast US to Europe for some timezone overlap. This narrowed to UK and western Europe. I realised having the Atlantic ocean between us was going to be too big of a hurdle to de-risk the relationship.
- Profile: I started with 'either a CEO or CTO type'. Typically you'll have a CEO/CTO founding combo. Being technical myself and having reflected on my previous venture, I narrowed this to CEO.
- Idea or not: I was looking for someone with an idea, validated ideally.
- Quit their job: candidates that haven't quit don't have the same urgency. They are comfortable. And sometimes flirting with the idea of startups rather than being ready to commit now.
- Not an existing duo: you want to be on as equal terms with your co-founder(s) as possible. After meeting a few duos, I realised it was unlikely I would feel on the same level as the original duo.

The process #

This is somewhere between dating and hiring someone. You want to keep it natural and a lot of it is about getting to know the other person. But you have a deadline and a business to start!

Top of funnel - get swiping #

In dating the 'old way' was being set up by friends. The 'new way' is dating apps. Both work for co-founder leads, but just like Tinder, you'll get many more leads through Y Combinator's co-founder matching app.

1. The first date #

This is the initial meet and greet. I kept these to a 30min video call or a quick in person coffee. The goals:

✅ Get to know them. Do you vibe? Yep, it's a lot about vibes. If you don't vibe on a first date with someone, you don't go on a second date.
✅ Do you respect them? This is really about whether you see them on eye level.

2. The written assignment #

The aim here is to have the most time efficient process to get to know your potential co-founder. The 50 Founder questions get you to introspect on why you're founding a company, reveals your areas of expertise and raises thorny questions early like how would you resolve a stalemate? The best founders were already aware of the 50 Founder Questions. If someone hadn't heard of them, I'd give them time to get back to me.

✅ Exchange 50 Founder questions - ensure values, expertise and thinking alignment.

3. More dates - discuss the Founder Questions and any other questions #

If the potential co-founder's responses revealed a gap too large, I'd just say so in an email. If you only have minor flags, question marks and areas you want to dig deeper - have as many calls to work through them as required.

✅ Each meeting is increasing your confidence in each other, to the point you are ready for a work trial.
✅ With a view to commencing a work trial, if the co-founder is committed to an idea, get across the idea.^[1]

4. Part time work trial #

By this stage you've worked through values, any possible question marks you've had about each other and are ready to test out what it is like to work with each other. I did this with several people at the same time. Using the dating analogy, this is that tricky and tiring phase where you haven't committed to one person yet. You're switching contexts and trying to not mix up details about each person! The work trial can look quite different depending on each person. For example, with one they hadn't found an idea, so the working together was about making progress through the ideation phase. For another, it was about doing joint discovery calls and validating ideas.

✅ Complete a timeboxed work trial together.

5. Commit #

This doesn't mean you have incorporated. But from step 4 you need to pick one person to move forward. And hopefully you have a strong sense of who the right person is. Having committed to each other, set yourself ambitious goals that are aligned to each of your primary areas of responsibility. For me that looked like developing a prototype and for my co-founder it was securing design partnerships. If this all goes well, you're already a company, you just need to sign the dotted line.

Good luck!

Other inspiration #

Whalesync's trial period template
YCs doc on co-founder matching my highlights:
1. Someone you enjoy spending time with and get along with. Would I want to be friends with this person?
2. An exceptional person - smart and effective
3. If you have hesitancy towards someone, listen to it.
4. Find people you have things in common with - shared interests, hobbies and values
5. The person is more important than the idea
- The median time between when the founders joined the site and when they matched with the person they ultimately ended up working with is 100 days
EF advice
- Both co-founders are technical, or technical with a domain expert
- Both co-founders are involved in everything
- No ‘pity founding’ - we're mates!
- A co-founder isn’t a ‘must-have’, a generic role that needs to be filled... they are a vital value-add
- "Each co-founder needs to have a strong relationship with each co-founder. In our experience, in a three this often isn’t the case. Someone is usually making a compromise and although that may seem ok in the short term, this doesn’t lead to harmonious founding teams in the long term."
- you have, or can quickly acquire the skills needed to build the product you want to build
Paul Graham
- "You haven't seen someone's true colors unless you've worked with them on a startup."
- "the relationship between founders was more important than ability"
- Share startupschool FAQs with them - aim to pick a trial project to work on together
More questions to ask each other from YCs doc here

This is a good opportunity to stress test the relationship too. Really question the idea, poke holes, debate it and see how they respond. What would it take to work on something else? You probably need to pivot in the future, which is a difficult conversation if you don't arrive at it at the same time. ↩︎

Legacy Is Sexy

Backendhance — Fri, 01 Dec 2023 02:00:00 GMT

Hi,

Unpopular Opinion Warnung: Legacy ist sexy!

Um ehrlich zu sein - ich habe das nicht immer so gesehen.

Dir ging es bestimmt ähnlich.

Mein erstes (großes) professionelle Projekt war eine Legacy Anwendung

So alt war sie eigentlich gar nicht. Wenn ich mich richtig erinnere, dann hatte sie nicht mal ein Jahrzehnt auf dem Buckel.

Es fühlte sich aber viel älter an.

Dies lag hauptsächlich an den vielen unerfahrenen Entwicklern, die an der Anwendung arbeiteten.

Tea ratings

Vít Baisa — Fri, 01 Dec 2023 02:00:00 GMT

list of teas with notes and rating

Town Hall #17: Scrimmage

taylor.town — Fri, 01 Dec 2023 02:00:00 GMT

Creation is a toilsome, boring, everyday thing. Historic plays can't be planned in advance.

Getting EC2 Instance Metadata Using IMDSv2

Nelson Figueroa — Fri, 01 Dec 2023 02:00:00 GMT

Curl examples for retrieving EC2 instance metadata using IMDSv2 including IP address, instance ID, and security groups.

Documenting Laravel APIs Talk Video

James Brooks — Fri, 01 Dec 2023 02:00:00 GMT

Watch my Documenting Laravel APIs talk from Laracon AU.

What's new in Seastar - issue 2

Noah Watkins — Fri, 01 Dec 2023 02:00:00 GMT

A summary of recent developments in the C++ Seastar project.

Understanding Objective-C by transpiling it to C++

Juan Cruz Viotti — Fri, 01 Dec 2023 02:00:00 GMT

This article describes how to transpile Objective-C to C++, and use that approach to gain understanding of the Objective-C runtime

December 2023 - Bookmarks

Domenico Luciani — Fri, 01 Dec 2023 01:00:00 GMT

Bookmarks for December 2023: 1 link - BeckDesignRules.

2023-11-30-001

Srijan Choudhary, all posts — Thu, 30 Nov 2023 20:10:00 GMT

Found Samuel's nice post on capturing data for org via email. This is very close to what I was looking for to be able to do GTD capture on-the-go either from phone apps like Braintoss or from any email app. One addition I would like to make is handling attachments in the email by downloading them and attaching to the org entry. This would be useful for voice notes from Braintoss - it does …

Dissapointing Framework Laptop 13 DIY Edition

Paul Cobbaut's blog — Thu, 30 Nov 2023 17:53:18 GMT

What a disaster this is, the DIY framework laptop.

The box...

...contains more boxes.

The parts are few...

...so the laptop is assembled in less than a minute?!

Unpacking the thing took longer than assembling it!

Reading this webpage took longer than assembling it!

Installing the DDR5 memory took 1 second: click and done.
Installing the nvme took 3 seconds: unscrew one screw, click and screw.
Installing the bezel was far more challenging, took almost 30 seconds! (*)
Attaching the keyboard: about three seconds?
Installing the four modules: 1 second per module (or less?)

I'm not that good at irony, but I was hoping for at least 20 minutes of fun assembling this thing... and only got one minute. :-/

(*) This gave fond memories of assembling tower PC's in the nineties; ISA/PCI slots click, CD-ROM atapi click, hard disk and IDE jumpers easy. But closing the tower case... well that was the hard part!

Installing an OS

The guide says to be patient when booting (powering on) the laptop for the first time, it may take several minutes.

I timed it, it took 57 seconds to get to this screen:

And now there is this; it has been a long time since I saw a default XFCE, Firefox and terminal.

Let the fun begin!

Discord and AI GTM

Matt Rickard — Thu, 30 Nov 2023 16:30:00 GMT

Midjourney is the largest discord server, with 16.5 million total users. It accounts for 13% of total Discord invites. Midjourney launched in March 2022 and doe

Clean 架构下的现代 Android 架构指南

张可的博客 — Thu, 30 Nov 2023 14:24:30 GMT

Clean 架构是 Uncle Bob 提出的一种软件架构，Bob 大叔同时也是 SOLID 原则的命名者。

Clean 架构图如下：

这张图描述的是整个软件系统的架构，而不是单体软件，其中至少包括服务端以及客户端。

对于 Android 单体应用开发来说应该还需要一个更贴切更精确的 Clean 架构图。

我大概总结了一下过往的开发经验，找出了应用架构中的重要部分，然后绘制了下面这张 Clean 架构指导下的 Android 应用架构图：

以及一般数据流向图：

依赖关系

Clean 架构基本准则是源码级别的内层不依赖外层，依赖关系永远是单向的，外层向内层依赖。

如上，Model 层是没有任何依赖的，UseCase 可以依赖 Model 和 Repo 等，但绝不能依赖 ViewModel，UI 层依赖 ViewModel，但 ViewModel 绝不能依赖 UI 层。

为了达到这种源码级别的依赖关系，我们必须借助一些工具来实现依赖注入，一般可以使用 Hilt 或者 Koin 这样的框架来实现。

另外，依赖注入不应该被滥用，不是所有的对象都适合用依赖注入，只有那些有明确层次关系的模块，互相有着明确的依赖关系的才需要。对于一些工具类，显然是没必要注入的。

Model（领域模型）

业务模型，或者叫领域模型，是根据软件业务设计出来的具体模型，一般来说会是个 data class，其中不包含任何业务逻辑，只是个单纯的模型对象。

由于是在整个架构的最内层，所以不依赖任何其他模块，并且相对稳定，设计的时候需要考虑这点。如果模型发生变化，那意味着整个上层的依赖方都可能发生变化，需要重新测试。

在命名和包结构上，领域模型不需要带 Entity 之类的后缀，直接命名为像 User 一样即可，但考虑到这是在软件的最内层，可能会被所有模块依赖到，所以要尽可能贴近其设计目标，并且不能太过宽泛。在包结构上，需要被存放在 model 包下面。

Adapter（数据适配器）

数据适配器层主要用来做数据转换，主要有两个职责：

转换网络接口实体数据类和领域模型。
领域模型之间的互相转换。

Adapter 层也比较纯粹，只负责简单的数据转换，而且对外暴漏的函数都是幂等函数。

如果数据转换过程中涉及到复杂的业务逻辑，可以考虑先用 UseCase 处理完成后再交给 Adapter。但因为 Adapter 层比 UseCase 层更靠内，所以 Adapter 不能依赖 UseCase。

习惯上，我们会以待转换类为开头，Adapter 结尾命名，例如我们要把 UserEntity 转换为 User, 那么应该这么写：

class UserEntityAdapter{
    
    fun toUser(entity: UserEntity): User {
        //...
    }
}

Repo

对于我们 Android 开发来说，Repo 层应该是对网络接口或本地磁盘的数据读写的封装，对于 Repo 的使用者来说，不需要关注具体的实现，且 Repo 中一般不具备复杂的业务逻辑，只能包含简单的数据处理逻辑。

Repo 应当隐藏具体的实现细节，不仅包括获取方式是网络还是本地数据，也应该隐藏对应的实体数据类，这意味着 Repo 层对外暴漏的函数的入参和出参不能包含接口返回的实体类，也不应该包含数据库表实体类，只能包含领域模型或者基本类型。我们给 Room 设计的数据库表的 data class 应该限制在 Repo 内部，我们给 Retrofit 设计的接口返回数据 data class 也同样应该限制在 Repo 内部。

data class UserEntity(val name: String, val avatar: String)

interface UserService {

    @GET("/user")
    suspend fun getUserInfo(@query("id") id: String): UserEntity
}

data class User(val name: String, val avatar: String)

class UserEntityAdapter @Inject constructor() {

    fun toUser(entity: UserEntity): User {
        return User(name = entity.name, avatar = entity.avatar)
    }
}

class UserRepo @Inject constructor(
    private val userEntityAdapter: UserEntityAdapter,
) {

    private val userService: UserService by lazy {
        retrofit.create(UserService::class.java)
    }

    suspend fun getUser(id: String): User {
        return userService.getUserInfo(id).let(userEntityAdapter::toUser)
    }
}

除了上面说的相应数据的转换之外，请求数据也需要在 Repo 层转换，对于 Post 请求来说，可能会存在一个请求实体，这个实体数据类最好也不要对外暴漏，可以在 Repo 层的请求方法入参那里做一些转换，最好能让入参更简单友好。

Repo 层还有一个作用就是负责把从接口或者数据库中出来的不友好的数据模型转换成友好的数据模型。

另外，现在由于有了 BFF 的存在，在某些比较简单的业务场景下我们可以为了方便做一些妥协，也就是接口的响应数据实体类可以穿透 Repo 层，直接给到 ViewModel，甚至是 UiState 使用，但应该明白这只是为了方便的妥协，并不是最佳实践，需要严格控制影响范围。

UseCase（用例）

UseCase 一般是指特定应用场景下的业务逻辑，用例引导了数据在模型之间的输入输出，并且指挥着业务实体利用其中的关键业务逻辑来实现用例的设计目标。

因此，一个 UseCase 往往只包含一段具体的业务逻辑，他的输入是基本类型或者领域模型，输出也是，并且是幂等函数，也就是纯函数，所以 Google 建议我们每个 UseCase 只包含一个公开的函数，类似于下面这种写法：

class DoSomethingUseCase {
		operator fun invoke(xxx: Foo): Bar {
				// ...
		}
}

通过利用 Kotlin 特性来使 UseCase 在使用的时候达到直接使用函数的体感。

但考虑到依赖以及管理问题，UseCase 最好还是不要直接使用函数来实现，应当按照上面的方式，定义一个类，然后再暴露一个通过操作符重载的函数。

在使用 UseCase 时可以这么用：

class LoginViewModel @Inject constructor(
    private val doSomething: DoSomethingUseCase,
): ViewModel(){
    
    fun onLoginClick(){
        doSomething()
    }
}

UseCase 的问题

UseCase 的粒度非常细，基本上每个 UseCase 就是一个函数，在复杂的业务背景下将会存在非常多的 UseCase，随着业务的增加，对他们的管理将难以为继。

因此，UseCase 需要一个有效的手段来进行管理，首先，应当按功能对他们的包名进行划分。同一个业务的 UseCase 最好具备相同的包名。

其次，我们不能陷入所有业务都用 UseCase 的极端情况中，很多时候，我们可以将一些极度类似的功能组织在一个类中，其中提供多个公开的方法，这样的写法在以前很常见，比如各种 Manager, Helper, Resolver 等，他们能有效的减少 UseCase 数量，并且相对简单。

UiState

UiState 是用来描述当前 UI 状态的集合类，一般来说应该是个 data class。

UiState 一定是不可变类，如果希望更改其中的某个值，应当重新创建一个对象，直接通过 data class 提供的 copy 方法即可，例如：

data class LoginUiState(
  val name:String,
  val avatar: String,
  val consentAgreed: Boolean
)

fun onAgreeChecked(){
  uistate = uiState.copy(
    consentAgreed = true,
  )
}

UiState 中的数据应当尽可能的方便给 UI 直接使用，因为 UiState 本身就是为了 UI 设计的，例如对于一个需要显示的格式化后的时间，格式化的逻辑最好放在 ViewModel 或者更内层，而不是直接给 UiState 一个时间戳，让 UI 层去格式化。很多时候看起来简单的逻辑也可能犯错误，UI 层没有能力处理异常。

在 ViewModel 中如果需要更新 UiState，可以直接通过 update 方法。

_uiState.update {
    it.copy(
        name = "zhangke"
    )
}

ViewModel

ViewModel 负责管理 UI 状态，执行对应的业务逻辑。

因此 ViewModel 的生命周期与页面是一致的。

一般来说我们会通过直接使用 Jetpack 提供的 ViewModel，但也可以自己创建其他类型的 ViewModel，只要控制好生命周期即可。

ViewModel 主要负责两件事情：

对外提供当前 UI 状态
接收 UI 事件并作出响应

当前 UI 状态我们通过将 UiState 包装在 StateFlow 里对外提供。

private val _uiState = MutableStateFlow()
val uiState:StateFlow<UserUiState> = _uiState.asStateFlow()

接受 UI 事件这点需要注意，ViewModel 需要做的是接收 UI 事件，例如用户手势输入，至于用户点击之后要做什么事情这是 ViewModel 的内部逻辑，不应该对外暴露。

UI 层

我们这里说的 UI 层就是指一个页面，除了常规的 Activity/Fragment 之外，对于 Compose 来说一个页面可能对应的是一个 Composable 函数，这取决于 UI 层的实现。

UI 层应该完全是数据驱动的，UI 层的作用就是百分之百的将 UiState 渲染出来，UiState 发生变化，UI 也跟着变化，这一点声明式 UI 框架做的很好。

UI 层虽然也可以处理一些简单的事件，但大部分的事件都还是要交给 ViewModel 来处理。

以上就是 Android 整洁架构中的一些关键概念的介绍，我已经按照这个架构开发了一年多了，目前看下来确实会让架构很整洁，但对于一些复杂的业务场景，尤其是可能需要穿透多个层级，跨越常规生命周期的模块就需要更精细的设计了。

Oh no second

I Learnt — Thu, 30 Nov 2023 14:00:00 GMT

An oh no second is that moment in time when you realise that you have done something potentially catastrophic and your stomach turns to mush with the dread and realisation.

Comic strip: long day at the office

Zachary Proser — Thu, 30 Nov 2023 02:00:00 GMT

A wordless comic strip about a typically brutal day at work, that nevertheless has a positive ending

cram those ideas into a clown car

taylor.town — Thu, 30 Nov 2023 02:00:00 GMT

To make memorable art, compress arguments/emotions into potent vehicles.

Epithet, Briefly

Brian's Waste of Time — Thu, 30 Nov 2023 02:00:00 GMT

Since Bob asked, I'm cleaning up a system to make using short-lived ssh certificates easy, simple, and secure.

The gist is to use a custom ssh-agent which, in turn, uses some modern authn service (OIDC, Keycloak, Okta, Google Sign-In, etc) to authenticate you. It then passes that auth token up to a CA which relies on a policy service to verify the auth token and respond with cert params (time, principals, extensions, whatever is needed). The CA uses the policy provided attributes to generate a short lived cert, which it gives back to the agent. Voila, you can access things for a couple minutes.

There is a real sequence diagram of what is going on in the repo.

CA

The CA is dirt simple and is the only thing that will ever see the CA private key. The same CA service should be usable by more or less anyone, maybe with some variety in terms of where the private key is stored, but that is it.

The CA receives an opaque auth token + pubkey from the agent and responds with a cert. The existing implementation uses a policy service to decide whether to issue the cert, and what attributes to give it.

The actual CA was kept as simple as possible, because it is a CA. You don't want to muck with it much.

Policy

The policy service consumes the opaque auth tokens from the CA and reponds with basic info, assuming the auth is valid:

{
    "identity":"brianm@skife.org",
    "principals":["oncall_fluffy", "oncall_wibble", "wheel"],
    "expiration":"2m",
    "extensions": {
        "permit-pty":true,
        "permit-port-forwarding":true
    }
}

The policy server (single endpoint, takes a post with the context for the request) is nice as it can be dynamic and make use of risk models and various other signals to decide what to do: eg, "is it a deploy server asking for a key? Is there actually a deploy in progress matching the identity requesting the cert? Okay, give it a 5 minute cert" on the sophisticated side; simply checking an OIDC token for validity on the simpler side.

Agent

The ssh agent generates a new keypair on startup and never lets the private key leave memory. It has a small GRPC API over a domain socket for interfacing with authentication mechanisms, such as a helper to simply feed it tokens over stdin for simple cases. We popped a browser (or used a cli tool) to do the Okta dance.

Michael and I made this at a previous company. We spun up the CA in one lambda function, the policy server in another, and authed against Okta. I want it for my own stuff now and it doesn't look like anyone there is maintaining it anymore, so I have been bringing it back to life.

Windows WPF via VM On Mac

Chris' Full Stack Blog RSS Feed — Thu, 30 Nov 2023 02:00:00 GMT

Start developing and running WPF applications on a Mac in no time!

Conditioning is grouping by

Alex Molas Blog — Thu, 30 Nov 2023 02:00:00 GMT

This insightful exploration draws parallels between mathematical formulations and practical implementations, showcasing how understanding conditioning as grouping can elevate your statistical insights. Discover the parallels between conditional expectations like $\mathbb{E}(y | X=x)$ and popular grouping techniques found in data analysis tools like pandas. Uncover the hidden synergy that exists between statistical conditioning and groupby operations, demystifying complex mathematical concepts with real-world applications. Whether you're a seasoned data scientist or a curious learner, this journey into the interconnected world of conditioning and grouping promises newfound clarity. Elevate your statistical understanding with practical examples, bridging the gap between theory and application. Embark on this enlightening exploration today and revolutionize your approach to statistical modeling. Uncover the simplicity behind complex conditional expressions through the power of grouping, transforming your data analysis skills along the way.

Making noisy SVGs

Daniel Immke — Thu, 30 Nov 2023 02:00:00 GMT

Adding noise texture with only code

OnlineOrNot Diaries 16

Max Rozen — Wed, 29 Nov 2023 19:10:00 GMT

On content marketing, starting a YouTube channel, and improving status pages

Standard Causes of Human Misjudgment (Munger)

Matt Rickard — Wed, 29 Nov 2023 16:30:00 GMT

In 1995, Charlie Munger gave a speech at Harvard on *[The Psychology of Human Misjudgment](https://www.youtube.com/watch?v=Jv7sLrON7QY)**.* It was filled with t

Proposal: an HTML element for spoilers

All content on Seirdy’s Home — Wed, 29 Nov 2023 15:09:39 GMT

An informal proposal for dedicated elements for spoiler tags in HTML: use-cases, syntax, semantics, recommended UA behavior, and comparisons with “details”

Music updates (November 2023)

Blog - Bobbie Chen — Wed, 29 Nov 2023 15:01:19 GMT

The latest and greatest in my life's music:

The Castaways added Jason (guitar), Angel (vocals), and Amolak (trumpet) to play a huge party at Alex's, which must have had over 100 people.

And I attended two Afterglow Chorus concerts: one as a spectator, and one as a participant. It was great, and also found a couple more jazz musicians within the group. I'm hoping to introduce them to the Castaways group eventually, but haven't had the chance yet.

These last couple months have been a little busier (life, work, travel to Asia - and writing this in the airport on the way back), so I'll leave this post here for now.

For a more personal life update, I did get engaged to Yee Aun on September 16th - music to my own ears. With that in mind, I'll share a pair of songs for listening:

DIMENSIONS (Live) - Joshua Milo

Spotify link | Website

This song was playing as we drove into Carmel-by-the-Sea on the night before the proposal. Energetic throughout with ups and downs, and a groove that you can't help but bob your head to - I rarely drive, but this song's fusion feel made for perfect driving music. Or as Yee Aun calls it, "Mario Kart music" (not a bad thing!).

It Only Takes a Taste - Drew Gehling and Jessie Mueller

Spotify link | Waitress the Musical

Going back to the early days, here's a song that is near and dear to my heart. If I remember correctly, I originally heard Waitress from the Sara Bareilles demo album, but then the version with Drew Gehling and Jessie Mueller breathed entirely new life into it. I saw the musical when they toured in Pittsburgh with a couple friends, and this song in particular is one I've always loved.

Post-graduation, I was thinking about making music again, and this became the first song I covered and posted in several years: my recording and blog post here. At the time, I was just scratching the surface of getting to know Yee Aun better, but as it goes, it only takes a taste when it's something special.

Other stuff to look forward to:

Christmas music season :)
Couch concert in SF on December 1 - I'm going, see you there?

As always, recommendations, jamming, gig invites are always welcome!

The unknown future of software developers

I Learnt — Wed, 29 Nov 2023 14:00:00 GMT

The role of the software developer is set for a major upheaval over the coming years.

Bouldering

swyx's site RSS Feed — Wed, 29 Nov 2023 10:16:46 GMT

My first introduction to bouldering was in 2017, in DUMBO Boulders in Brooklyn Bridge Park (now closed):

Seattle Bicycle Commuting Gear

Stories by Zach Wolfe on Medium — Wed, 29 Nov 2023 04:49:00 GMT

Seattle Bicycle Commuting

Seattle’s winter weather can be a bummer to bike through. Here’s a breakdown of the gear I use to make my daily 6-mile commute comfortable.

After the pandemic pushed me away from my work-walkable Seattle downtown apartment to a nearby neighborhood, I worked fully remote from my home office for about 18 months. As Return To Office enforcement began, I found myself needing to plan for a regular 6.6 mile commute.

After a few weeks of testing, I decided to commit to cycling as my main method of commuting for a few reasons:

Consistent, mentally-engaging morning cardio: Needing to sweat and pay attention to get to work makes me not even realize I’m getting exercise. And at ~50 minutes and 500 calories/day, it’s a meaningful amount of cardio.
Quicker than bussing: The bus is great, but very inconsistent especially in the evenings, adding ~20m/day to my total commute compared to bike or car on average, with outliers of > 40m.
Cheap: Parking downtown is $18/day + $6/week in gas, and Uber was ~$28/day, so I’d spend somewhere around ~$3500/year to get to the office.
Low carbon footprint: Using rough estimates, cycling uses about ~21gCO2e/km, compared to driving’s 271gCO2e/km.

Issues

After my test-runs, I found a few problems to solve, some more important than others:

Rain: In Seattle, it rains an average of 156 days per year (42% of days), even though the total rainfall is low. After just one representative rainy commute, I was soaked, dirty, and carried around a wet backpack all day. I knew right away I wanted a few gear upgrades.
Backpack sweat: Cycling with a backpack made me sweaty. I really needed a shower at the office, increasing my total time spent getting from bed-to-desk-to-couch.
Big Laptop, Small Bag: My MacBook Pro is chunky and doesn’t fit in my usual bag. So I was carrying around a 28L hiking pack all day because it’s all I had that worked.
Visibility: Early mornings and late evenings mean low light. I’d like to not be squished, so I wanted some visibility improvements.

The Gear I Picked

I spent a lot of time shopping around, reading reviews, agonizing, talking to cycling friends, etc, so I thought it might be valuable to summarize what I settled on for someone who also wants to commute via bike.

Kona Dew Bike

The Kona Dew is my steed. It’s simple, comfortable, inexpensive, and has many threaded mount-points on the frame to mount the gear described below. I find the relatively upright riding position to be easy on the back, and the cushy slick tires to be a good combination of comfortable riding and road traction.

Portland Design Works Full Metal Fenders — 650 Beast

I bought a set of Portland Design Works Full Metal Fenders — 650 Beast from a local shop, Free Range Cycles, justifying the costly hardware by installing them myself. My only complaint so far is the difficult install, requiring some after-market aluminum spacers to fit around my disc brakes, a hacksaw to shorten the fender stays, and an hour of wiggling and adjusting, but now I’m ready for those rainy days.

Pannier and Bags

To solve my “sweaty back” problem, I opted to ditch the backpack and get a bike-mounted pannier.

However, I use my bag in and around the office many days, occasionally moving around floors and buildings. I wanted a bike-mounted bag that I could use as an office bag, big enough to fit a laptop, clothes, and an extra layer, but not be giant and ungainly.

Arkel Signature H — Urban Pannier

The Arkel Signature H Urban Pannier was my answer after looking at a dozen other options. Made with carbon-neutral waterproof X-Pac X11 fabric, it feels very tough and stayed perfectly dry after a shower test. Plus, no backpack means no sweaty back. This was recommended as the best pannier of 2023 by NY Times, and I can see why.

Topeak Explorer Rack

I got a Topeak Explorer Rack, also being rated best on a bike rack test by the NY Times. Even though I have disc brakes I opted for the non-disc option after looking at the dimensions and reading some reviews. Some disc brake designs have the brake caliper mounted on top of the chain stay, so lots of “disc brake” gear acommodate with spacers, increasing the footprint of the rack. To save some space, knowing my brake calipers wouldn’t interfere at all, I got the more narrow option. This was super simple to install, taking about 30m and typical bike Allen tools.

Tailfin Top Tube Pack — Flip

I use a Tailfin Top Tube Pack for items I regularly want access to while I’m riding, without dismounting: phone, keys, garage door opener, work badge, snacks.

This thing is great, and if you have to pick one position for a small bag, I highly recommend a top tube bag such as this.

Perfectly waterproof (passed a shower test), mounts securely to the top-tube bottle eyelets on my Kona Dew bike (shown below with optional straps I don’t use).

Ortlieb Saddle Bag Two

I use the Ortlieb Saddle Bag Two for the things I need occasionally, but don’t need to-hand while riding: some cash, contact lenses, tube and chain repair items, and gloves. These things I’m OK stopping to get out, as I need them only once in a while or never.

Lights and Tools

Light & Motion VIS Pro 1000 Blacktop

At the recommendation of a friend, I bought a Light & Motion VIS Pro 1000 Blacktop. The side-pointing lights make me feel a bit safer on those dim mornings and evenings, and might avoid a side-swiping.

OneUp Components EDC System

After just a few weeks of biking to work, I saw more than one person repairing a flat on the side of the Lake Union trail. I decided I should get a pump and tools to fix a flat enough for me to limp home.

The OneUp Components EDC Pump seemed perfect. It’s a pump that has a EDC Tool System and EDC Plug Kit that fit within the pump itself, all mounting to the frame of the bike, saving space in my bag.

Wrapping Up

Gear makes a big difference in comfort, especially riding in a rainy city like Seattle. I hope my cycling gear listed above gives you a good starting point for your commute.

-Zach

Building your own bookmark launcher

xenodium.com @alvaro — Wed, 29 Nov 2023 02:00:00 GMT

✨sponsor✨ this content

I've been toying with the idea of managing browser bookmarks from you know where. Maybe dump a bunch of links into an org file and use that as a quick and dirty bookmark manager. We'll start with a flat list plus fuzzy searching and see how far that gets us.

The org file would look a little something like this:

::: captioned-content ::: caption bookmarks.org :::

My bookmarks
- [[https://lobste.rs/t/emacs][Emacs editor (Lobsters)]]
- [[https://emacs.stackexchange.com][Emacs Stack Exchange]]
- [[https://www.reddit.com/r/emacs][Emacs subreddit]]
- [[https://emacs.ch][Emacs.ch (Mastodon)]]
- [[https://www.emacswiki.org][EmacsWiki]]
- [[https://planet.emacslife.com/][Planet Emacslife]]

:::

Next we need fuzzy searching, but first let's write a little elisp to extract all links from the org file:

(require 'org-element)
(require 'seq)

(defun browser-bookmarks (org-file)
  "Return all links from ORG-FILE."
  (with-temp-buffer
    (let (links)
      (insert-file-contents org-file)
      (org-mode)
      (org-element-map (org-element-parse-buffer) 'link
        (lambda (link)
          (let* ((raw-link (org-element-property :raw-link link))
                 (content (org-element-contents link))
                 (title (substring-no-properties (or (seq-first content) raw-link))))
            (push (concat title
                          "\n"
                          (propertize raw-link 'face 'whitespace-space)
                          "\n")
                  links)))
        nil nil 'link)
      (seq-sort 'string-greaterp links))))

The snippet uses org-element to iterate over links to collect/return them in a list. We join both the title and url, so searching can match either of these values. We also add a little formatting (new lines/face) to spiff things up.

(browser-bookmarks "/private/tmp/bookmarks.org")

We can now feed our list to our preferred narrowing framework (ivy, helm, ido, vertico) and use it to quickly select a bookmark. In the past, I've used the likes of ivy-read directly, though have since adopted the humble but mighty completing-read which hooks up to any of the above frameworks.

With that in mind, let's use completing-read to make a selection and split the text to extract the corresponding URL. Feed it to browse-url, and you got your preferred browser opening your bookmark.

(defun open-bookmark ()
  (interactive)
  (browse-url (seq-elt (split-string (completing-read "Open: " (browser-bookmarks "/private/tmp/bookmarks.org")) "\n") 1)))

I remain a happy ivy user, so we can see its fuzzy searching in action.

At this point, we now have our bookmark-launching Emacs utility. It's only an M-x open-bookmark command away, but we want to make it accessible from anywhere in our operating system, in my case macOS.

Let's enable launching from the command line, though before we do that, let's craft a dedicated frame for this purpose.

(defmacro present (&rest body)
  "Create a buffer with BUFFER-NAME and eval BODY in a basic frame."
  (declare (indent 1) (debug t))
  `(let* ((buffer (get-buffer-create (generate-new-buffer-name "*present*")))
          (frame (make-frame '((auto-raise . t)
                               (font . "Menlo 15")
                               (top . 200)
                               (height . 20)
                               (width . 110)
                               (internal-border-width . 20)
                               (left . 0.33)
                               (left-fringe . 0)
                               (line-spacing . 3)
                               (menu-bar-lines . 0)
                               (minibuffer . only)
                               (right-fringe . 0)
                               (tool-bar-lines . 0)
                               (undecorated . t)
                               (unsplittable . t)
                               (vertical-scroll-bars . nil)))))
     (set-face-attribute 'ivy-current-match frame
                         :background "#2a2a2a"
                         :foreground 'unspecified)
     (select-frame frame)
     (select-frame-set-input-focus frame)
     (with-current-buffer buffer
       (condition-case nil
           (unwind-protect
               ,@body
             (delete-frame frame)
             (kill-buffer buffer))
         (quit (delete-frame frame)
               (kill-buffer buffer))))))

Most of the snippet styles our new frame and invokes the body parameter. While I don't typically resort to macros, we get a little syntatic sugar here, so we can invoke like so:

(defun present-open-bookmark-frame ()
  (present (browse-url (seq-elt (split-string (completing-read "Open: " (browser-bookmarks "/private/tmp/bookmarks.org")) "\n") 1))))

Wrapping our one-liner with the present-open-bookmark-frame function enables us to easily invoke from the command line, with something like

emacsclient -ne "(present-open-bookmark-frame)"

Now that we can easily invoke from the command line, we have the flexibility to summon from anywhere. We can even bind to a key shortcut, available anywhere (not just Emacs). I typically do this via Hammerspoon, with some helpers, though there are likely simpler options out there.

function emacsExecute(activate, elisp)
   if activate then
      activateFirstOf({
            {
               bundleID="org.gnu.Emacs",
               name="Emacs"
            }
      })
   end

   local socket, found = emacsSocketPath()
   if not found then
      hs.alert.show("Could not get emacs socket path")
      return "", false
   end

   local output,success = hs.execute("/opt/homebrew/bin/emacsclient -ne \""..elisp.."\" -s "..socket)
   if not success then
      hs.alert.show("Emacs did not execute: "..elisp)
      return "", false
   end

   return output, success
end

function openBookmark()
   appRequestingEmacs = hs.application.frontmostApplication()
   emacsExecute(false, "(present-open-bookmark-frame)")
   activateFirstOf({
         {
            bundleID="org.gnu.Emacs",
            name="Emacs"
         }
   })
end

hs.hotkey.bind({"alt"}, "W", openBookmark)

With that, we have our Emacs-powered bookmark launcher, available from anywhere.

While we used our Emacs frame presenter to summon our universal bookmark launcher, we can likely the same mechanism for other purposes. Maybe a clipboard (kill ring) manager?

What would you use it for? Get in touch (Mastodon / Twitter / Reddit / Email).

Enjoying this content? Find it useful? Consider ✨sponsoring✨.

"MVP" is dead! Long live "FPC".

taylor.town — Wed, 29 Nov 2023 02:00:00 GMT

MVPs are for products. FPCs are for customers. Products aren't people.

SSH Certificate Notes

Brian's Waste of Time — Wed, 29 Nov 2023 02:00:00 GMT

There are lots of tutorials out there, but I want to compile my notes so if I walk away from playing with them again for a few years, I can pick it up again!

SSH Keys for different things are all the same (modulo chosen algo type)

User keys, host keys, CA keys -- they are all the same. Don't let ssh-keygen docs, with all of its options and whatnot, convince you otherwise.

Use `AuthorizedPrincipalsFile` to manage access

Don't go and create a local user account for everyone who may be accessing the host, just have a shared local user, probably even root, where you differentiate who did what using the identity on the cert they authed with.

AuthorizedPrincipalsFile is a file that lists the allowed principals for a given local user. Use groups as principals, not individual users, and list the groups which may use the local user in the file for the given local user. The groups an actual user is in are then added to that user's cert.

AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

The %u is the normal user placeholder, so /etc/ssh/auth_principals/root would control the principals that may log in as root. The file takes one principal per line. Copilot really wants me to add that you can use wildcards in the file, but that idea is making me queasy. Principals as groups sounds better to me.

The overall sshd_config looks something like:

Port {{ port }}
Protocol 2
AcceptEnv LANG LC_*
LoginGraceTime 120

UsePAM no 
PasswordAuthentication no
IgnoreRhosts yes
PubkeyAuthentication yes

HostKey {{ path }}/ssh_host_ed25519_key
TrustedUserCAKeys {{ path }}/ca.pub
AuthorizedPrincipalsFile {{ path }}/auth_principals/%u

Testing with sshd

You can run sshd as a normal user, as long as you only try to log in as that same user.

Invocation looks like, /opt/homebrew/sbin/sshd -d -D -f /tmp/my_ssh/sshd_config. This will start it without forking behavior, process a single login, and give us debug output.

The sshd_config may need StrictModes no depending on the directories in play and the permissions on them.

Various invocations

ssh with weird agent socket

Given I am doing this to test agent + cert muckery, invoking ssh to talk to the sshd mentioned above looks like:

ssh -v \
    -o UserKnownHostsFile=/dev/null \
    -o StrictHostKeyChecking=no \
    -F /dev/null \
    -l $USER \
    -o IdentityAgent=/path/to/agent.sock \
    -p 2222 \
    localhost

Same, but for a local cert instead of agent (assuming user_key is the private key, and has user_key.pub and user_key-cert.pub alongside it):

ssh -v \
    -o UserKnownHostsFile=/dev/null \
    -o StrictHostKeyChecking=no \
    -F /dev/null \
    -l $USER \
    -i path/to/user_key \
    -p 2222 \
    localhost

Given a CA key, generate a user cert locally

ssh-keygen -s  path/to/ca -I wobble -n {{ principals }} .path/to/user_key

Add a comma seperated (no spaces) list of the principals (groups) for {{ principals }} in that invocation (needs to align with the auth principals on sshd). The -I is basically irrevalent for testing purposes, is just a cert identifier.

Creating a Static Media Review Site Backed by Airtable

The David Brownman Blog — Wed, 29 Nov 2023 02:00:00 GMT

My experience building a static site backed by an Airtable database, including some fairly complex TypeScript.

Read the whole thing.

The Unreasonable Effectiveness of Monte Carlo

Matt Rickard — Tue, 28 Nov 2023 16:30:00 GMT

Monte Carlo methods are used in almost every branch of science: to evaluate risk in finance, to generate realistic lighting and shadows in 3D graphics, to do re

The cost of AI

I Learnt — Tue, 28 Nov 2023 14:00:00 GMT

There is a rush at the moment to add AI to everything with a fear that if you don’t then you will be left behind.

Moving My Dotfiles to Nix

trv.rs — Tue, 28 Nov 2023 12:15:00 GMT

I've been experimenting with Nix. I have a PC that is way more powerful than my older Macbook. While experimenting with Ubuntu on WSL, and porting my existing dotfiles over, I remembered my friend Tom talking about Nix and NixOS and decided to give it a whirl.

Two weeks later this article is being written on NixOS, in WSL, using my full dev environment. 🎉

Nix

The orange site and red site are filled with excellent articles about Nix and what it can do. For the sake of this article a brief overview.

The word Nix points to three things:

A functional language¹ for describing configurations of software.
A package manager using the Nix Language to handle installation and path management of software configurations.
A *nix OS (NixOS) that can be configured using the Nix Language.

These three things provide a couple features very desirable to me:

Using a strict configuration for my dev environment means that I can sync my setup between my work laptop, my personal laptop, and now this development WSL PC.
NixOS being a configuration-first OS means I can summon a direct copy of it without having to spend hours setting it up. This is advantageous for virtualisation or if I want to move my WSL setup to a dual-boot someday.

MacOS or WSL

Wezterm works on both my "host" OS. I just run the executable and put my wezterm.lua wherever it needs to be.

I do have a wild little script that runs and installs the wezterm terminfo file. Very un-nix.

On my Macs I can't run NixOS but I can at least install the Nix Package manager. There's a relatively complicated install process involving a separate volume and mounting… but the installer is pretty easy to run.

On the Windows PC I used the WSL2 instructions provided by NixOS. I also added

default_domain = 'WSL:NixOS'

to my Windows .wezterm.lua so that Wezterm will connect to WSL on launch.

Home Manager

After NixOS or Nix Package Manager have been installed we can install Home Manager manage our dev environment.

It's specifically built for this problem and replaces what I was doing with my Brewfile, Makefile, and stow.

I took it slowly over a couple of weeks. At first my home-manager consisted of installing some packages and moving my existing dotfiles to the right place. Very un-Nix-like but it totally worked.

As I explored the options appendix and debugged problems I wound up here today. I chose not to put everything into the Nix configuration and keep some of the configuration in files for backwards compatibility. In a pinch you could stow or ln those folders/files in your home just like I was before.

Let's explore some of the bumps in the road.

Neovim

I eventually moved to using the programs.neovim configuration provided by Home Manager. I really wrestled with getting everything setup correctly. I think part of the problem was existing executables and configuration files still in path on OSX that was confusing the universe but in the end there were two big problems:

Treesitter wants to compile

Treesitter is awesome. But it wants to run scripts and compile things. Nix's store doesn't love that… it wants each folder to be idempotent and sets them to Read Only.

I added pkgs.vimPlugins.nvim-treesitter and eventually found the sequence of settings to make it work:

add config block to configure highlight and indent on, but I got errors… set type = "lua";.
use withAllGrammars so that it automatically gets all of them.
everything is installed but not loading, use packadd to force it to load the package.

All this wasn't working and I realized that…

Lazy.nvim takes over RTP

Turns out my beloved Lazy.nvim completely takes over the RTP. This means that plugins loaded manually (or by Nix!) can't be found by neovim. Someone else pointed out there was a setting, so I was able to disable this and suddenly neovim could see plugins loaded by Nix.

These may be bad and wrong, if so… shoot me an email?

Tmux

Having learned from my Neovim mistakes Tmux was a bit easier. Like Treesitter, TPM wants to modify it's read-only folders. I moved some settings into the configuration, install the plugins using Home Manager, and use my existing config for theming and style.

Fish

I've been using a stripped down Fish install for a while now. I almost wonder if I should use it at all. I basically just install it and use Home Manager to enable Starship.

I still am using asdf in some places. It'll likely phase out as I get better at flakes.

…

I'm starting to play with devenv.sh and flakes to create my development environments. At the moment I'm still making heavy use of asdf to handle ruby/js package management. I am wrestling with how to manage gems that want to build native extensions. I've been warned by kind friends that it's enough to drive one crazy… but I've got the bandwidth presently for the challenge.

I like the idea of only enabling LSP executables and maybe even entire Neovim configurations per project using flakes and overlays. I've got a really simplistic version of this working using direnv at the moment.

All this very much falls under XKCD's warning of automation addiction. For me it's worthwhile to learn a little bit about Nix and to face the puzzle of *nix and pathing problems.

I really love that I can run home-manager switch and know that everything I need to start working gets automatically put in the right place. That's pretty hard to beat.

Now I suppose it's actually time to read Nix's actual documentation. :P

Nix loves the trailing ;. Absolutely required. Really hard for my ruby/JS brain. ↩

Wish

Code & Visuals — Tue, 28 Nov 2023 02:00:00 GMT

Disney Animation’s fall 2023 release is Wish, which is the studio’s 62nd animated feature and also the studio’s film celebrating the 100th anniversary of Disney Animation and The Walt Disney Company. Wish is a brand new story but is also steeped in the past century of Disney storytelling; while Asha and Star and Valentino’s adventure is a new musical story, the themes and setting Wish draws upon are timeless and classic. As part of this theme of modern Disney with throwback elements, Wish also has a unique beautiful visual style that combines the latest of our computer graphics animation with a classic watercolor style. Creating this style presented an interesting set of new challenges for our artists, TDs, engineers, and for Disney’s Hyperion Renderer.

At Disney Animation, every one of our films is a new opportunity for us to push our filmmaking art and technology forward. On most films this advancement takes place across many different aspects of the film, but on Wish, there is one obvious challenge that stood out above everything else: the film’s visual style. Of course we still made large improvements in other areas, such as major pipeline optimizations [Li et al. 2024a], but on Wish a large proportion of technology development was focused on achieving the target visual style. One could be forgiven for thinking that stylization is mostly a rendering problem, but on Wish it really was a challenge that reached into every department and every corner or our production process. Stylization on Wish meant stylization in modeling, lookdev, animation, effects, lighting, everything else in between, and even new pipeline challenges!

The decision to give Wish a unique style came from pretty much the very beginning of the project; the studio wanted to do something special for the 100th anniversary film to tie together our modern way of making animated films with the studio’s rich hand-drawn heritage. The look of Wish is especially influenced by early 20th century Disney traditional watercolor animation, with Snow White and the Seven Dwarfs (1937) and Sleeping Beauty (1959) being the largest guideposts. This influence extends all the way to the very shape of the film, so to speak- Wish is the first CG film that Disney Animation has made in the ultrawide 2.55:1 Cinemascope-style aspect ratio, matching the aspect ratio that was used on Sleeping Beauty and Lady and the Tramp (1955). This aspect ratio choice meant that stylization on Wish even impacted layout, since they had to think about how to frame for such an ultrawide image!

Disney Animation has a long history of stylizing 3D CG to resemble and fit in with hand-drawn animation, going all the way back to the studio’s traditional hand-drawn era [Meier 1996, Daniels 1999, Tamstorf et al. 2001, Odermatt and Springfield 2002, Teece 2003]. In the 3D CG era, Disney Animation has continually experimented with stylizing CG as well with a number of different approaches. Paperman focused on integrating 2D linework with 3D rendered characters [Kahrs et al. 2012 ,Whited et al. 2012], while Feast experimented with driving 2D lighting entirely in compositing on top of flat unlit/unshaded 3D renders [Osborne and Staub 2014]. The studio’s recent Short Circuit experimental shorts program had many shorts that experimented with a variety of different stylized looks, ranging from Chinese ink brush watercolors to graphic 2D illustrations to stop motion and wood carved looks [Newfield and Staub 2020]. Disney Research has also worked closely with Disney Animation in the past decade plus to develop various experimental stylization techniques [Schmid et al. 2011, Sýkora et al. 2014]. Both Strange World and Raya and the Last Dragon had small amounts of stylized sequences as well, with Strange World doing a 1950s pulp scifi comic book look and Raya and the Last Dragon doing a more graphic digital mixed media sort of look. Most recently, the short Far from the Tree utilized a look with cel-shaded characters on watercolor backgrounds. All of these were animated using our standard 3D pipeline, with much of the look being built using a combination of render passes from the renderer (Hyperion for everything except Paperman, which preceded Hyperion’s existence by a few years), various tricks in lighting, and a lot of work in compositing; how much of each was used varied widely per show and per target style.

Wish builds upon all of these predecessors. Wish’s stylization system is a vastly expanded version of what was used on Far from the Tree, which in turn was built on top of everything that was developed for Short Circuit, which in turn drew upon lessons from both Feast and Paperman. One of the biggest challenges came from having to scale up a stylization pipeline from a short film to a full feature length project, while also trying to hit a new target style. Early tests on the show were able to reproduce in CG the target visdev paintings essentially exactly, but through entirely ad-hoc and mostly manual approaches, which we then had to systematically take apart and figure out how to apply to the whole movie. My wife, Harmony Li, was an Associate Technical Supervisor on the show and (among a ton of other things) oversaw the development of the entire technical backend that was built out to support stylization on Wish [Li et al. 2024b]; as a member of the rendering team, I got to work with her on this, which was great fun! Meanwhile, much of the development for the actual techniques used was led primarily by lighting and lookdev artists.

An early breakthrough in achieving Wish’s style was finding that combining Kuwahara filters¹ [Kuwahara et al. 1976] with linework generated from the renderer created a convincing starting point for a line-on-watercolor look that could use the renderer’s physical lighting as a starting point, instead of needing to construct stylized lighting entirely from scratch in comp on top of flat-shaded unlit renders. To help really tie together the watercolor look, early tests put the entire image on a watercolor paper texture background, but once we tested the watercolor paper texture background in motion, some issues became apparent. With just a static watercolor paper texture background, the illusion of motion broke as animation looked like it was “swimming” through the texture, but simply texturing everything with watercolor textures in 3D space looked downright bizarre since it looked less like the frame was a watercolor painting and more like all of the characters and the environment were made out of paper. To solve this problem, the Hyperion team invented a new dynamic screen space texture technique [Burley et al. 2024] where the renderer would project screen space textures onto 3D surfaces while tracking motion vectors. The result is that Wish’s watercolor backgrounds look like just a flat sheet of watercolor paper when still, but under motion convincingly move with the characters while neither looking like they’re actually in 3D space nor looking disconnected from motion.

One interesting question I worked on for Wish was making Hyperion robustly handle shading normals that are really dramatically disconnected from the underlying “physically correct” geometric normal. Extreme bending of normals was used extensively on Wish to simplify shapes and art-direct lighting detail and shadows. In a normal physically based path traced render, shading normals coming from bump mapping and normal mapping usually have at least some relationship with the underlying true geometric normal, meaning that the ways shading normals modify light transport are relatively constrained to a plausible range. However, on Wish, extreme shading normals were used for things like simplifying the lighting on an entire complex tree canopy to match what the lighting would be on a simple sphere. Making Hyperion handle these cases both from an authoring perspective and making Hyperion’s light transport robust against these cases took some work!

There were actually also some more traditional physically based rendering problems to solve on Wish too, which one might not necessarily expect for such a stylized film. For some of Magnifico’s magic, the art-direction called for a sort of prismatic look where white light would get split into different colors. We decided to try to achieve this effect through physically based shading, since the starting point for Wish’s entire stylization pipeline was renders with physical lighting in order to provide consistency. To achieve this effect through physically based shading, I extended the Disney BSDF with spectral dispersion support (retrofitting a spectral effect into a non-spectral renderer was a fun challenge worth discussing on its own someday). Once our lookdev artists had access to dispersion within the Disney BSDF, it was fun seeing all of the other places where they started sprinkling the effect in, such as in various glass objects.

Stylization on Wish didn’t just mean new renderer effects and lighting and compositing work; in order to make characters read correctly in a watercolor look, stylization had to be incorporated into all of the characters at a geometric and design level as well, and had to be incorporated into animation and simulation. As an example: a core story device in Wish is the collective wishes of Rosas, which take the form of orbs containing entire small worlds set inside of swirling volumetrics. Creating these wishes required clever pipeline solutions to embed entire stylized animated scenes inside of the orbs in 3D space, which was used instead of a usual compositing-based insert-shot workflow or the teleport-based solution [Butler et al. 2022] used on Encanto; this approach was taken in order to provide animators with the ability to sync and see fully combined shots interactively and to simplify the rendering setup needed for stylization in lighting and compositing [Karanam et al. 2024]. On top of creating the individual wishes, huge numbers of wishes then had to be choreographed into tight, closely synchronized formations to meet the art-direction and shape language of the songs they were a part of, which required developing new crowd rigs and animation controls. The rendering aspect of the wishes was in a lot of ways actually the easy part! Each wish was also an internally emissive object, so when thousands of wishes are massed together in key sequences in the movie, we initially had some concerns about efficiently rendering all of the wishes, but our long-standing cache points many-light selection strategy [Li et al. 2024c] proved to be more than capable for the task.

Another example of stylization far upstream of lighting is in the project’s entire approach to character stylization. Character hair and fur grooms required a different approach from our usual process; normally in more photoreal Disney Animation films, hair and fur grooms are built to be highly detailed to support the rich detailed look of the film, but Wish’s watercolor style meant using a more simplified and graphic shape language across the board, where detail is traded off for a stronger focus on silhouette and overall massing. Hair and fur grooms had to be adjusted to match, and hair and fur simulation had to be adjusted to keep art-directed shapes intact instead of operating on a more individual strand-based level [Kaur and Stratton 2024]. Asha’s braids, with their North African inspired long box braids, required additional attention to create and simulate [Kaur et al. 2024]. The braids themselves were already a major technical challenge; even using our state-of-the-art in-house grooming system Tonic [Simmons and Whited 2014], the braids still required a final groom two orders of magnitude more complex than our average groom. Once Asha’s groom was figured out, her hair then had to also be put through the same stylized simulation setup mentioned earlier, with extensive 2D drawovers being used to art-direct simulations. Character animation then also had to take into account the fact that Wish does not have motion blur and how that impacts how viewers perceive character performances.

Speaking of 2D drawovers, one particularly interesting use of 2D drawings to art-direct stylization on Wish is in Magnifico’s magic and in various effects like flames and torches. Normal volumetric effects created from simulations tend to be highly physical and detailed, but Wish’s style called for these effects to harken back to the much more graphic shape language of magic effects from Disney Animation’s hand-drawn era. To do this, our effects artists built on top of the neural volume style transfer work from Raya and the Last Dragon [Navarro and Rice 2021] and Strange World [Navarro 2023] to develop a new system where effects animation would begin with hand-drawn 2D elements, which were then projected and extruded into the 3D space to provide a guide for neural volume style transfer on top of volume simulations [Tollec and Navarro 2024]. The result is that Wish’s volumetric effects combine the movement and interactions of physical simulations while retaining the shape and style of traditional hand-drawn effects.

Everything I’ve written about here is just what I was familiar with on this film; vastly more work went into every single frame of Wish than even I know. The final look of Wish is something that I think really is unique and beautiful. Wish’s 3D watercolor look speaks to the entire history of Disney Animation and simultaneously roots itself in the studio’s rich traditional hand-drawn legacy while also exemplifying the studio’s long history of innovating and driving filmmaking craft forward. Walt Disney never stopped seeking to innovate in animation, and 100 years after he founded the studio, the animation studio that carries his name today continues to embody that same bold spirit on every new film. As someone who’s a lifelong fan and student of animation, I feel incredibly humbled and fortunate to get to contribute towards that legacy every day. Wish is also accompanied by the short Once Upon a Studio, which is a wonderful tribute to 100 years of Disney Animation characters.

Below are some frames from Wish, pulled from the Blu-ray and presented in no particular order. As always, I’d highly recommend seeing Wish on the biggest screen you can find!

Here are two credits frames from Wish; the first is the fancy hero-credit card for my wife Harmony Li and her fellow Associate Technical Supervisors, and the second is for the Hyperion team, along with several of the Hyperion’s sister technology teams that all support lighting and lookdev. Also, Wish has a lovely post-credits scene that I’d encourage sticking around for!

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Brent Burley, Brian Green, and Daniel Teece. 2024. Dynamic Screen Space Textures for Coherent Stylization. In ACM SIGGRAPH 2024 Talks. Article 50.

Corey Butler, Brent Burley, Wei-Feng Wayne Huang, Yining Karl Li, and Benjamin Huang. 2022. “Encanto” - Let’s Talk About Bruno’s Visions. In ACM SIGGRAPH 2022 Talks. Article 8.

Eric Daniels. 1999. Deep Canvas in Disney’s Tarzan. In ACM SIGGRAPH 1999 Sketches & Applications. 200.

Avneet Kaur, Jennifer Stratton, David Hutchins, and Nikki Mull. 2024. Art-Directing Asha’s Braids in Disney’s Wish. In ACM SIGGRAPH 2024 Talks. Article 4.

Avneet Kaur and Jennifer Stratton. 2024. Character Stylization in Disney’s Wish. In ACM SIGGRAPH 2024 Talks. Article 5.

John Kahrs, Patrick Osborne, Amol Sathe, Jeff Turley, Brian Whited, and Darrin Butters. 2012. The Art and Science Behind Walt Disney Animation Studios’ “Paperman”. In ACM SIGGRAPH 2012 Production Sessions.

Neelima Karanam, Joel Einhorn, Emily Vo, and Harmony M. Li. 2024. Creating the Wishes of Rosas. In ACM SIGGRAPH 2024 Talks. Article 6.

Michiyoshi Kuwahara, Kozaburo Hachimura, Shigeru Eiho, and Masato Kinoshita. 1976. Processing of RI-Angiocardiographic Images. In Digital Processing of Biomedical Images. 187-202.

Harmony M. Li, George Rieckenberg, Neelima Karanam, Emily Vo, and Kelsey Hurley. 2024a. Optimizing Assets for Authoring and Consumption in USD. In ACM SIGGRAPH 2024 Talks. Article 30.

Harmony M. Li, Angela McBride, Sari Rodrig, and Gregory Culp. 2024b. A Pipeline for Effective and Extensible Stylization. In ACM SIGGRAPH 2024 Talks. Article 51.

Yining Karl Li, Charlotte Zhu, Gregory Nichols, Peter Kutz, Wei-Feng Wayne Huang, David Adler, Brent Burley, and Daniel Teece. 2024c. Cache Points for Production-Scale Occlusion-Aware Many-Lights Sampling and Volumetric Scattering. In Proc. of Digital Production Symposium (DigiPro 2024). Article 6.

Barbara J. Meier. 1996. Painterly Rendering for Animation. In SIGGRAPH 1996: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. 477-484.

Mike Navarro and Jacob Rice. 2021. Stylizing Volumes with Neural Networks. In ACM SIGGRAPH 2021 Talks. Article 54.

Mike Navarro. 2023. Diving Deeper Into Volume Style Transfer. In ACM SIGGRAPH 2023 Talks. Article 39.

Jennifer Newfield and Josh Staub. 2020. How Short Circuit Experiments: Experimental Filmmaking at Walt Disney Animation Studios. In ACM SIGGRAPH 2020 Talks. Article 72.

Kyle Odermatt and Chris Springfield. 2002. Creating 3D Painterly Environments for Disney’s “Treasure Planet”. In ACM SIGGRAPH 2002 Sketches & Applications. 160.

Patrick Osborne and Josh Staub. 2014. Feast – A Look at Walt Disney Animation Studios’ Newest Short. In ACM SIGGRAPH 2014 Production Sessions.

Johannes Schmid, Martin Sebastian Senn, Markus Gross, and Robert W. Sumner. 2011. OverCoat: An Implicit Canvas for 3D Painting. ACM Transactions on Graphics (Proc. of SIGGRAPH) 30, 4 (Jul. 2011), Article 28.

Maryann Simmons and Brian Whited. 2014. Disney’s Hair Pipeline: Crafting Hair Styles From Design to Motion. In Eurographics 2014 Industrial Presentation.

Daniel Sýkora, Ladislav Kavan, Martin Čadik, Ondrej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-Ray: Bas-relief Meshes for Adding Global Illumination Effects to Hand-Drawn Characters. ACM Transactions on Graphics 33, 2 (Apr. 2016), Article 16.

Rasmus Tamstorf, Ramón Montoya-Vozmediano, Daniel Teece, and Patrick Dalton. 2001. Hybrid Ink-Line Rendering in a Production Environment. In ACM SIGGRAPH 2001 Sketches & Applications. 201.

Daniel Teece. 2003. Sable - a Painterly Renderer for Film Animation. In ACM SIGGRAPH 2003 Sketches & Applications.

Marie Tollec and Mike Navarro. 2024. Making Magic with 3D Volume Style Transfer. In ACM SIGGRAPH 2024 Talks. Article 48.

Brian Whited, Eric Daniels, Michael Kaschalk, Patrick Osborne, and Kyle Odermatt. 2012. Computer-Assisted Animation of Line and Paint in Disney’s Paperman. In ACM SIGGRAPH 2012 Talks. Article 19.

Footnotes

¹ I recently learned that the Kuwahara filter originated from completely outside of graphics; it was originally invented at Kyoto University’s medical school and at Shiga University of Medical Science for medical imaging purposes. Specifically, it was invented for reducing noise in radioisotopic heart scans without blurring sharp features, and much later graphics people realized it made for a great edge-preserving blur for painting-like effects. I love when graphics intersects with other fields to produce interesting results! keyboard_return

Creativity: My Top 5 Feelings

Martin Rue — Tue, 28 Nov 2023 02:00:00 GMT

Hearing that someone has enjoyed or found your work useful is an indicator that you're achieving the goal you set out to achieve, and that's an amazing feeling.

Good analogies

Vít Baisa — Tue, 28 Nov 2023 02:00:00 GMT

a collection of analogies from various sources

Lex Fridman Podcast

Vít Baisa — Tue, 28 Nov 2023 02:00:00 GMT

notes on and scores of episodes

graphic design is my passion

taylor.town — Tue, 28 Nov 2023 02:00:00 GMT

🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸🐸

Remembering Charlie Munger

Dan Stroot — Tue, 28 Nov 2023 02:00:00 GMT

Charlie Munger has been Warren Buffet’s partner since he joined Berkshire Hathaway as a vice chairman in 1978. In 1995 Charlie gave a speech at Harvard University about human decision-making and factors contributing to misjudgments. He wanted to share what he had observed and learned; models he used that helped him (and Berkshire Hathaway) succeed.

This post Remembering Charlie Munger first appeared on Dan Stroot's Blog

Missing In Flutter SDK Docs, Your Axis Layout Guide

Stories by Fred Grott on Medium — Mon, 27 Nov 2023 17:08:33 GMT

One of the layout guides not in the Flutter SDK docs.

Continue reading on Medium »

Foundations lead to structure

I Learnt — Mon, 27 Nov 2023 14:00:00 GMT

Foundations are important and need to be correct and strong before you build anything.

Lessons from Career Coaching

rob.sh — Mon, 27 Nov 2023 13:27:01 GMT

In the autumn of 2022, I got an opportunity to attend an external leadership course. One of the things that it involved was some coaching sessions. I didn’t really think that talking to someone about work was something that I needed (and honestly probably didn’t understand what a coach did…), but engaged with it anyway.

Cue the start of 2023 — Google went through some big changes, and unfortunately, my health took another downturn. At the same time, I was personally struggling with balancing the work that I was doing leading a wider technical team (especially with layoffs impacting their direct management), driving an innovation project (which I was mostly doing out of hours), and keeping myself healthy.

`curl` your landing page

www.mootoday.com - RSS Feed — Mon, 27 Nov 2023 08:15:13 GMT

Does your project provide a CLI? Let people install it with `curl` from your marketing landing page.

Announcing the Pinecone AWS Reference Architecture in Pulumi

Zachary Proser — Mon, 27 Nov 2023 02:00:00 GMT

Deploy production-ready systems using Pinecone in minutes with the AWS Reference Architecture

Pinecone AWS Reference Architecture Technical Walkthrough

Zachary Proser — Mon, 27 Nov 2023 02:00:00 GMT

An examination of the Reference Architecture components and functionality

Deploying the Pinecone AWS Reference Architecture - Part 1

Zachary Proser — Mon, 27 Nov 2023 02:00:00 GMT

I deploy the Pinecone AWS Reference Architecture end to end in this three part video series

Deploying the Pinecone AWS Reference Architecture - Part 2

Zachary Proser — Mon, 27 Nov 2023 02:00:00 GMT

I deploy the Pinecone AWS Reference Architecture end to end in this three part video series

Deploying the Pinecone AWS Reference Architecture - Part 3

Zachary Proser — Mon, 27 Nov 2023 02:00:00 GMT

I deploy the Pinecone AWS Reference Architecture end to end in this three part video series

Destroying the Pinecone AWS Reference Architecture

Zachary Proser — Mon, 27 Nov 2023 02:00:00 GMT

I demonstrate how to destroy the Pinecone AWS Reference Architecture using Pulumi

Deploying a jump host for the Pinecone AWS Reference Architecture

Zachary Proser — Mon, 27 Nov 2023 02:00:00 GMT

I demonstrate how to deploy and connect through a jump host in order to interact with Postgres

Creating a Fast Time-Series Graph With Postgres Materialized Views

Dylan Paulus' Blog — Mon, 27 Nov 2023 02:00:00 GMT

Imagine you have a massive amount of time-series data you want to explore and visualize. Seeing the latest trends, the historical patterns, and the outliers in your data can help you gain insights and make decisions. But how do you visualize and analyze time-series data effectively? How do you create graphs, plots, and other visualizations for real-time analytics showing the current state of your data and the historical changes over different time intervals? And how do you do it efficiently without sacrificing performance or accuracy?

In this article with Timescale we dive into practical examples of using materialized views and continuous aggregates to create fast, performant time-series graphs.

🏰 A Day Exploring Berlin

James Carr — Mon, 27 Nov 2023 02:00:00 GMT

Previously I wrote about setting out on my solo journey across Germany and how I prepared. Today I’m going to break down everything about my time in Berlin: places to visit and tons of photos to share. Along the way, I hope to share a few tips and tricks to make your visit enjoyable. Arrival In Berlin After a 12 hour flight I landed in Berlin at 8:20am. Thanks to the sleeping gummies I took, I was wide awake and my legs weren’t sore or stiff thanks to the compression socks I wore.

How a Release Failed

Backendhance — Mon, 27 Nov 2023 02:00:00 GMT

Hi,

Nach 9 Stunden stand die Entscheidung fest: Wir müssen zurückrollen.

Der erste Release seit fast 1,5 Jahren war gescheitert.

Und dabei lief es so gut. Die minutiöse Vorbereitung der letzten Wochen hatte sich ausgezahlt. Alles lief nach Plan. Die notwendigen Änderungen an den VMs verliefen wie erwartet. Die große Migration war nach einer Stunde Laufzeit erfolgreich durchgelaufen. Die neuen Contentfiles konnten erfolgreich mit dem neuen System deployed werden. Die neue CD Pipeline lief zum ersten Mal auf Production – und war erfolgreich.

Carta's 83(b) Oversight: A Tax Trap for Married Startup Employees

Khanna Law Blog — Mon, 27 Nov 2023 02:00:00 GMT

There's a blind spot in Carta's 83(b) election feature you need to know about if you're married and living in California.

Why do companies hire people to be idle a lot of the time?

ntietz.com blog - technically a blog — Mon, 27 Nov 2023 02:00:00 GMT

The biggest tech companies employ a lot of engineers. In 2021, Microsoft employed over 100,000 software engineers. That is just mind boggling scale to me. It's roughly as many people as the whole county I grew up in.

They are paying a lot of engineers. Some of them do very little, with employees saying they "were paid to do little-to-no work"^[1]. So... why are they paying them if there isn't a lot of work for them to do?

There are a couple of theories for this. The one I hear most often is that the big tech companies employ people to keep them off the market. There might be some truth to that, but I think it's a small portion compared to another very reasonable explanation.

Let's take a step back and use everyone's favorite software engineering reference: constructing physical infrastructure, like bridges! I'm not comparing the two fields, but we can observe similar dynamics at play with construction crews. We've all seen the construction crews where one person is operating machinery and four are standing around watching. So the same question comes up here: why are they paying them if there isn't a lot of work for them to do?

In the case of construction crews, I think it's quite obvious that they're not paying them to keep them off the market. So there has to be another dynamic at play. Most of the time, when we see people idle at a job site, they're actually either resting, waiting to be able to do their work, or supervising and inspecting. But some of the time, you really do have a lot of people at the site to do nothing. It's that last part that I think is the key at play here in software, too.

There are multiple metrics you can optimize for in construction projects. Among these are total resource efficiency and total timeline speed. There is a trade-off between these! At the two extremes, you have:

Maximize resource utilization and schedule crews based on when the work for them will be available. This increases timelines, but has better overall resource utilization by sending people only where they can be fully utilized, but that means the job site grinds to a halt when crews are not currently available.
Minimize timeline by having everyone available for their portion immediately. This reduces total utilization of any individual worker, but also dramatically shortens the job because whoever is needed is always already available.

This sort of schedule compression is available for high-priority construction projects where it's essential to get it done as fast as possible. It's also very expensive, so it's not the typical mode.

I think big companies do something similar in tech. It's not a perfect comparison, because companies also do have a lot of inefficiencies and the processes don't maximize for total throughput of features, but I see two modes:

Startups (and other resource-constrained companies) often maximize utilization of employees, so some features and products take different timelines but everyone is at maximum utilization
Big companies (without cash resource constraints) maximize for other metrics so employees have more downtime. This slack allows the company to achieve other metrics by having people available at a moment's notice.

This isn't the full picture^[2]—nothing ever is—but I think it gives a reasonable intuition for why idle engineers at big tech aren't just being kept off the market. It's often the case that there's some other metric being optimized, not individual employee utilization.

These companies have a lot of resources that they can throw at problems. If there is a metric they want to optimize, it's a viable approach for them to hire a lot of people to be only somewhat utilized in order to achieve that other objective. And that's why you see can this trend reverse in leaner times. The big tech companies have started to value full utilization more during this economy, since they are more cash constrained than previously.

From BusinessToday. ↩
Other factors that also lead to idle engineers are prestige and budgets. It's considered prestigious to have a bigger organization reporting to you, so managers have an incentive to grow teams and departments even without work for them to do. And budgets must be used or else they'll be reallocated, so keeping and growing your headcount is a way to defend that budget for the future. ↩

HNTV: play hn yt links locally

taylor.town — Mon, 27 Nov 2023 02:00:00 GMT

Play videos recently submitted to HN via the commandline.

Upgrading to macOS 14 Sonoma broke my network connection

Ellie's Notes — Mon, 27 Nov 2023 02:00:00 GMT

I upgraded my mac to macOS 14, Sonoma earlier today. Upon first using it, my internet wouldn't connect!

Wifi or ethernet, nothing I tried work

Even

ping 8.8.8.8

I did some searching, and it turns out the version of Little Snitch I had installed wasn't supported.

The issue here is that without internet, I couldn't update it

The fix wasn't too difficult. Go to System Settings -> Network -> VPNs and Filters. Disable little snitch.

Default Apps (2023)

Brenton Cleeland — Mon, 27 Nov 2023 01:58:24 GMT

It feels a little no effort November, but I've seen this list pop up on a few blogs and figured I would add mine so I can compare next year. You can see a great list of these from the Hemispheric Views podcast listeners.

Just my personal list (work is sometimes quite different)

📨 Mail Client: Fastmail web and iOS app
📮 Mail Server: Fastmail
📝 Notes: Apple Notes, Scratchpad
✅ To-Do: Reminders
📷 Photo Shooting: Camera (iOS)
🟦 Photo Management: Photos (Mac and iOS)
📆 Calendar: Fastmail via Fantastical
📁 Cloud File Storage: iCloud Drive
📖 RSS: Feedbin via Feedbin web and NetNewsWire
🌐 Browser: Firefox, Safari (iOS)
🔖 Bookmarks: bm2
📜 Word Processing: Google Docs
📈 Spreadsheets: Google Sheets
🛒 Shopping Lists: Apple Notes
🎵 Music: Apple Music
🎤 Podcasts: Overcast
🔐 Password Management: 1password

Currently looking for: a Read it Later approach that works for my scattered workflow.

In the past I've kept ordered lists of apps I use (2023 vs 2019) and it turns out I don't change things up that often.

Resumé

All content on Seirdy’s Home — Mon, 27 Nov 2023 00:36:32 GMT

Detail-oriented, committed, self-motivated, OSS enthusiast proficient in Python, Go, Linux/UNIX systems, and cloud-native computing looking for an internship.

Skateboard 2

Home on mfashby.net — Sun, 26 Nov 2023 22:33:27 GMT

AI-Powered RPGs: The Next Gaming Revolution Inspired by Cicero

Mitrapunk: Engineering Game — Sun, 26 Nov 2023 22:13:12 GMT

AI-powered RPGs, inspired by Cicero's AI breakthrough, are revolutionizing gaming with dynamic worlds and responsive NPCs.

ShareX image effects

River Writes - A MediaWiki Blog — Sun, 26 Nov 2023 18:40:47 GMT

How to use ShareX image effects & spam your friends with rainbows

Remembering Frank H. Lutz

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sun, 26 Nov 2023 18:00:00 GMT

I was deeply saddened to learn about the untimely death of Frank H. Lutz.

I was not closely acquainted with Frank as a person but his work had a big impact on me. In my undergraduate days, I discovered his resource The Manifold Page at a critical time in my studies when I was actively looking for ways to combine my love for algorithms with my love for geometry and topology. At the suggestion of Markus Banagl, I looked up Frank’s website and was immediately hooked: here, then, was someone who provided a glimpse into manifolds and how to ‘operate’ with them algorithmically! I used Frank’s triangulations of 2-manifolds and 3-manifolds to deepen my understanding of simplicial homology, writing a simple program to compute homology groups.

Subsequently, I kept returning to Frank’s resources, both for what one might call ’experimental mathematics,’ but, truth be told, also just for fun. I started looking into some homology statistics, for example. Even later, as my research had expanded more into the realm of computational topology, Frank graciously provided me an opportunity to participate in a Thematic Mini-Conference on Computational Topology and Machine Learning. Being on the job market at this point, I was honoured and excited about presenting my work—and I will be forever grateful for it! For Frank, this was apparently not a big deal, but for me it provided an unprecedented opportunity, and I felt that I had now finally arrived in a nice community.

I had just planned another project that would involve more of Frank’s data and wanted to reach out to him when I learned the sorrowful news of his passing. I am saddened that this opportunity will never come.

I know that Frank’s triangulations are only a small part of his legacy, but I want his friends and family to know that it is still an important part for some people.

May Frank’s memory be a blessing to his loved ones.

How To Write Flutter Adaptive Scaffold Breakpoints Like Were A GDE

Stories by Fred Grott on Medium — Sun, 26 Nov 2023 17:11:37 GMT

Flutter Adaptive Scaffold UX pattern is not in the Flutter SDK, this is the first step in that UX pattern in the form of breakpoints like…

Continue reading on Medium »

How to determine app idea is technically feasible as a non-technical founder?

Vadim Kravcenko — Sun, 26 Nov 2023 14:46:50 GMT

Question:

I'm a non-technical founder with a concept for a productivity app that sets screen time limits and rewards users for staying below these limits. However, I'm struggling to determine if this idea is technically feasible, especially considering the data access restrictions imposed by platforms like iOS. Despite extensive research on Apple's developer forums I'm still not sure. I'm seeking advice on how a non-technical founder like myself should approach verifying the technical feasibility of such an app idea before committing to bringing on technical co-founders or starting development.

Answer:

The post How to determine app idea is technically feasible as a non-technical founder? appeared first on Vadim Kravcenko.

Abstract art

I Learnt — Sun, 26 Nov 2023 14:00:00 GMT

I have never understood abstract art. I have tried.

I am not a big art lover but when I have viewed art I tend to be at the photo-realistic end as, to me, that shows the skill of the artist.

Robot Dad

Untrod — Sun, 26 Nov 2023 10:00:00 GMT

See discussion of this post on Hacker News

Tired of Alexa's feeble "here's what I found on wikianswerspam.com" responses to my eight-year-old son's science questions, I whipped up Robot Dad during my Thanksgiving break. He now runs in the background of our family computer.

Robot Dad sounds like real …

A weekly task list with the Alastair Method

trv.rs — Sun, 26 Nov 2023 08:11:00 GMT

I had never encountered the Alastair system, but I think this could work very well for me and be the secret to a weekly/monthly collection actually working for me.

Rach writes:

The Alastair Method for bullet journaling is when you place some vertical columns next to your list of tasks. The columns could be anything you want to group/organise your tasks by. I choose days. So, I have Monday to Sunday as my columns next to the list. And then, if I plan to do the task on a specific day, I will add a dot to the column. If the task is done, I cross it off in the day I did it and the overall list.

Do I go all in on analog for my tasks next year… tempting…

A weekly task list with the Alastair Method - rachsmith.com

I Made Heston Blumenthal's Chicken

Matteo's Website — Sun, 26 Nov 2023 07:00:00 GMT

Have you ever seen a recipe that looked so wild and out there, you can't help but feel like you need to try it at least once in your life? That's how I felt when I saw Heston's recipe for crispy roast chicken. You start off by brining the chicken …

Some site optimizations

All content on Seirdy’s Home — Sun, 26 Nov 2023 05:18:24 GMT

I continue to work on my site over break. Today I got the build time down to 3.5 seconds for the Hugo build and just under 3 seconds for post-processing and syntax-checking (mostly using xmllint). I’m not parallelizing the post-processing since my laptop just has two cores (no simultaneous multithreading) and CI has one.

Seven seconds is still too slow for changes to feel instant, though. I can’t just rebuild on save. It’s especially slow for a site with only <200 pages.

Hugo’s profiler tells me that a lot of the time is spent searching for Webmentions that match the current page. Perhaps I should do that server-side ahead of time; right now, I download all my Webmentions as one big fat JSON file and load it into memory once. I should also move some post-processing back into Hugo.

I also re-used xmllint to minify my feeds and check their syntax. The minification worked particularly well since my feeds contain raw, un-escaped XHTML. Some clients still fetch my full-text feeds without transparent compression.

Gridnotes - an infinite 2D text editor

sidsite — Sun, 26 Nov 2023 02:00:00 GMT

Gridnotes is an infinite 2D text editor I made.

Measure twice, vise thrice.

taylor.town — Sun, 26 Nov 2023 02:00:00 GMT

Vises reduce movement in undesirable directions. They help you focus your energy in limited axes.

My Book Reviews

taylor.town — Sun, 26 Nov 2023 02:00:00 GMT

Note: There are books listed here whose views I do not endorse. I try to break beyond my bubble and understand others. If you're interested in my current beliefs, feel free to email me.


★★★★		2026-06-22	Thinking in Bets :: Annie Duke
★★★		2026-06-18	The Crowfolk :: Mark Stay
★★★		2026-06-08	Think Faster, Talk Smarter :: Matt Abrahams
★★★★		2026-05-29	Someone Who Will Love You in All Your Damaged Glory :: Raphael Bob-Waksberg
★★★		2026-05-18	Trick Mirror :: Jia Tolentino
★★★★★		2026-05-17	A Wizard of Earthsea :: Ursula K. Le Guin
★★★★		2026-05-16	Heart of Darkness :: Joseph Conrad
★★		2026-05-15	Think Big, Act Small :: Jason Jennings
★★		2026-05-14	First Love, Last Rites :: Ian McEwan
★★★		2026-05-13	I Deliver Parcels in Beijing :: Hu Anyan
★★★★		2026-05-03	The Secret Life of Groceries :: Benjamin Lorr
★★★★★		2026-04-29	The New Me :: Halle Butler
♡		2026-04-27	Useful Not True :: Derek Sivers
★★★★★		2026-04-25	Childhood's End :: Arthur C. Clarke
★★		2026-04-21	The Daydreamer :: Ian McEwan
★★		2026-04-21	Normal People :: Sally Rooney
★★★		2026-04-18	Prey :: Michael Chrichton
★★		2026-04-18	Kindred :: Octavia Butler
★★★		2026-03-23	The Deficit Myth :: Stephanie Kelton
★★★		2026-03-23	Understanding Power :: Noam Chomsky
★★★		2026-03-23	How to Hide an Empire :: Daniel Immerwahr
★★★		2026-03-16	Oranges :: John MacPhee
		2026-03-15	Wittgenstein's Mistress :: David Markson
		2026-03-14	Middle C :: William H. Gass
★★★★		2026-03-12	Careless People :: Sarah Wynn-Williams
★★★★		2026-03-10	The Decadent Society :: Ross Douthat
★★★		2026-03-08	Dawn :: Octavia Butler
★★★		2026-02-20	Uncommon Carriers :: John MacPhee
★★★★		2026-02-20	Beginning of Infinity :: David Deutsch
★★★★		2026-02-18	Sphere by Michael Chrichton
♡		2026-01-19	The Infinite Library :: Alexander Wales
♡		2026-01-12	Building Strongholds :: Alexander Wales
★★		2025-12-27	The Hard Thing About Hard Things :: Ben Horowitz
★★★		2025-12-27	The Man Who Broke Capitalism :: David Gelles
♡		2025-12-24	Trust and Consequences :: Alexander Wales
★★		2025-12-20	Eragon :: Christopher Paolini
♡		2025-12-11	Through Adversity :: Alexander Wales
★★★★★		2025-12-06	House of Suns :: Alastair Reynolds
★★★		2025-11-10	Version Control :: Dexter Palmer
★★★		2025-11-10	Meditations for Mortals :: Oliver Burkeman
★★★		2025-10-13	Matrix :: Lauren Groff
★★★		2025-10-05	The Wizard and the Prophet :: Charles C. Mann
★★★★		2025-10-04	Blindsight :: Peter Watts
		2025-10-03	Vellum :: Hal Duncan
★★★		2025-09-24	Breakneck :: Dan Wang
★★★★★		2025-09-21	Klara and the Sun :: Kazuo Ishiguro
★★		2025-09-20	The Curse of the Mogul :: Jonathan Knee, Bruce Greenwald, and Ava Seave
★★★		2025-09-19	Gideon the Ninth :: Tamsyn Muir
★★★		2025-09-17	Why Greatness Cannot Be Planned :: Kenneth O. Stanley and Joel Lehman
★★★★		2025-09-14	Characteristics of Games :: George Skaff Elias, Richard Garfield, and K. Robert Gutschera
★★★★		2025-09-07	Children of Time :: Adrian Tchaikovsky
★★		2025-09-02	Good to Great :: Jim C. Collins
★★★		2025-09-02	Devil Take the Hindmost :: Edward Chancellor
★★★		2025-09-02	The Grid :: Gretchen Bakke
★★★★		2025-08-30	The Wisdom of Insecurity :: Alan Watts
★★★		2025-08-30	What Makes Sammy Run? :: Buddy Schulberg
★★★★		2025-08-10	The Wager :: David Grann
★★★★		2025-08-05	Black Rednecks and White Liberals :: Thomas Sowell
★★★		2025-08-01	Waybound :: Will Wight
★★★		2025-07-29	Dreadgod :: Will Wight
★★		2025-07-26	Reaper :: Will Wight
★★★★		2025-07-23	Bloodline :: Will Wight
★★		2025-07-23	Can't Hurt Me :: David Goggins
★★★★		2025-07-23	Become What You Are :: Alan Watts
★★★★		2025-07-19	Wintersteel :: Will Wight
★★★		2025-07-11	Uncrowned :: Will Wight
★★★★		2025-07-09	How to Get Filthy Rich in Rising Asia :: Mohsin Hamid
★★★		2025-07-08	Underlord :: Will Wight
★★★		2025-07-07	The Wealth of Humans :: Ryan Avent
★★★		2025-07-06	Ghostwater :: Will Wight
★★		2025-07-06	Skysworn :: Will Wight
★★★		2025-07-05	Blackflame :: Will Wight
★★★		2025-07-01	Soulsmith :: Will Wight
★★★★		2025-07-01	Everything is Tuberculosis :: John Green
★★★		2025-06-25	An Absolutely Remarkable Thing :: Hank Green
★★★		2025-06-23	Unsouled :: Will Wight
★★★★		2025-06-23	Steve Jobs :: Walter Isaacson
★★	review	2025-05-10	Zen and the Art of Happiness :: Chris Prentiss
★★	review	2025-05-10	The The Prime Number Conspiracy :: Thomas Lin
★★	review	2025-05-10	Soulhome :: Sarah Lin
★★★	review	2025-04-30	Good Inside :: Dr. Becky Kennedy
★★★	review	2025-04-29	Factfulness :: Hans Rosling, Ola Rosling, Anna Rosling Rönnlund
★★	review	2025-04-27	Give and Take :: Adam Grant
★★★	review	2025-04-21	On Bullshit :: Harry G. Frankfurt
★★★	review	2025-04-17	An Elegant Puzzle :: Will Larson
★★	review	2025-04-16	How Innovation Works :: Matt Ridley
★★★★★	review	2025-04-16	Musashi :: Eiji Yoshikawa
★★★	review	2025-03-20	The Mom Test :: Rob Fitzpatrick
★★★★	review	2025-03-09	Motivational Interviewing :: William R. Miller and Stephen Rollnick
★★★★★	review	2025-03-06	The Golden Compass :: Philip Pullman
★★★	review	2025-03-04	Selfish Reasons to Have More Kids :: Bryan Caplan
★★★★	review	2025-03-03	Three Laws of Nature :: R. Stephen Berry
★★★	review	2025-02-28	Sirens & Muses :: Antonia Angress
★★★	review	2025-02-14	Creation Lake :: Rachel Kushner
★★	review	2025-02-07	Ministry of Time :: Kaliane Bradley
★★★	review	2025-02-14	Complexity: A Guided Tour :: Melanie Mitchell
★★★	review	2025-02-13	Supercommunicators :: Charles Duhill
★★	review	2025-02-10	How Evil Are Politicians :: Bryan Caplan
★★★★	review	2025-02-10	The Spider's War :: Daniel Abraham
★★★	review	2025-02-09	The Case Against Education :: Bryan Caplan
★★★	review	2025-02-09	Radical Abundance :: K. Eric Drexler
★★★★	review	2025-02-09	How to Talk So Kids Will Listen & Listen So Kids Will Talk :: Adele Faber & Elaine Mazlish
★★★	review	2025-02-08	The Widow's House :: Daniel Abraham
★★★★★	review	2025-02-06	A Naked Singularity :: Sergio de la Pava
★★★★	review	2025-02-01	A General Theory of Oblivion :: José Eduardo Agualusa
★★★	review	2025-01-25	The Impossible Man :: Patchen Barss
★★★	review	2025-01-20	The Tyrant's Law :: Daniel Abraham
★★★★★	review	2025-01-06	The Road :: Cormac McCarthy
★★★	review	2025-01-01	Prisoners of Geography :: Tim Marshall
★★★★	review	2024-12-31	The King's Blood :: Daniel Abraham
★★★★★	review	2024-12-17	Cadillac Desert :: Marc Reisner
★★★	review	2024-12-16	Welcome to the Hyanam-dong Bookshop :: Hwang Bo-Reum
★★★★	review	2024-12-16	Thinking with Type :: Ellen Lupton
★★★	review	2024-12-15	James Acaster's Guide to Quitting Social Media :: James Acaster
★★★	review	2024-12-14	Fab :: Neil Gershenfeld
	review	2024-11-28	Drive Your Plow Over the Bones of the Dead :: Olga Tokarczuk Antonia Lloyd
★★★★	review	2024-11-27	Tyll :: Daniel Kehlmann
★★★	review	2024-11-22	Glucose Revolution :: Jessie Inchauspe
★★★★	review	2024-11-20	The Idiot :: Elif Batuman
★★★★	review	2024-11-19	The Dragon's Path :: Daniel Abraham
★★★★	review	2024-11-13	Oblivion: Stories :: David Foster Wallace
★★★★	review	2024-11-13	The Dream Machine :: M. Mitchell Waldrop
★★★	review	2024-11-04	Heaven :: Mieko Kawakami
★★★★★	review	2024-11-04	The Employees :: Olga Ravn
★★★★	review	2024-11-03	Convenience Store Woman :: Sayaka Murata
★★★	review	2024-11-01	Nexus :: Yuval Noah Harari
★★★	review	2024-10-31	Capitalist Realism :: Mark Fisher
★★★★★	review	2024-10-24	No Country for Old Men :: Cormac McCarthy
★★★	review	2024-10-21	The Strangest Man :: Graham Farmelo
★★★★★	review	2024-10-18	Trust :: Hernan Diaz
★★★★★	review	2024-10-17	Pastoralia :: George Saunders
★★★★★	review	2024-10-16	Atonement :: Ian McEwan
★★★★	review	2024-10-15	When We Cease to Understand the World :: Benjamín Labatut
★★	review	2024-10-11	The Friend :: Signid Nunez
	review	2024-10-10	Hateship, Friendship, Courtship, Loveship, Marriage :: Alice Munroe
★★★★	review	2024-10-09	The Soul of a New Machine :: Tracy Kidder
★★★★	review	2024-10-06	Seeing Like a State :: James C. Scott
★★★	review	2024-09-25	Stay True :: Hua Hsu
★★★★	review	2024-09-16	Evicted :: Matthew Desmond
★★★	review	2024-09-10	Olive Kitteridge :: Elizabeth Strout
★★★★	review	2024-09-04	A Shining :: Jon Fosse
★★★★★	review	2024-09-01	Outline :: Rachel Cusk
★★	review	2024-08-30	The Glass Cage :: Nicholas Carr
★★	review	2024-08-29	Volt Rush :: Henry Sanderson
★★★★	review	2024-08-28	Demon Copperhead :: Barbara Kingsolver
★★★★	review	2024-08-13	The World for Sale :: Javier Blas & Jack Farchy
★★★★★	review	2024-08-11	Roadside Picnic :: Arkady & Boris Strugatsky
	review	2024-08-06	The War Below :: Ernest Scheyder
★★	review	2024-08-06	Frostbite :: Nicola Twilley
★★★★★	review	2024-08-06	War with the Newts :: Karel Capek
★★★	review	2024-08-03	Leviathan :: Paul Auster
★★★	review	2024-08-03	Enchanted Objects :: David Rose
★★★★★	review	2024-07-30	The Long Earth :: Stephen Baxter & Terry Pratchett
★★★★	review	2024-07-29	The Alignment Problem :: Brian Christian
★★★★	review	2024-07-26	How to Have Impossible Conversations :: Peter Boghossian & James Lindsay
★★★★★	review	2024-07-23	The Sellout :: Paul Beatty
★★	review	2024-07-22	Slow Productivity :: Cal Newport
♡	review	2024-07-19	Quantations :: Joseph A. Stirt, M.D.
★★★★	review	2024-07-19	The Sparrow :: Mary Doria Russell
★★	review	2024-07-18	Co-Intelligence :: Ethan Mollick
★★★	review	2024-07-14	The Politics of Design :: Ruben Pater
★★	review	2024-07-13	The Overstory :: Richard Powers
★★★	review	2024-06-27	User-Friendly :: Cliff Kuang & Robert Fabricant
★★★	review	2024-06-19	The Cold Start Problem :: Andrew Chen
★★	review	2024-06-15	Going Infinite :: Michael Lewis
★★	review	2024-06-15	Build :: Tony Fadell
★★★★★	review	2024-06-14	The Rise of Theodore Roosevelt :: Edmund Morris
★★	review	2024-06-07	The Art of Product Design :: Hardi Meybaum
★★★★	review	2024-06-02	Speaker for the Dead :: Orson Scott Card
★★★★	review	2024-06-01	Ender's Game :: Orson Scott Card
★★	review	2024-05-26	Effortless :: Greg McKeown
★★★★	review	2024-05-26	The Fifth Season :: N. K. Jemisin
★★★★	review	2024-05-13	Distrust that Particular Flavor :: William Gibson
★★★	review	2024-05-09	The Player of Games :: Iain M. Banks
★★★★	review	2024-05-03	Bears Discover Fire and Other Stories :: Terry Bisson
★★	review	2024-05-01	Consider Phlebas :: Iain M. Banks
★★★★★	review	2024-04-26	Notes on the Synthesis of Form :: Christopher Alexander
★★	review	2024-04-25	Lawns into Meadows :: Owen Wormster
★	review	2024-04-18	A Court of Thorns and Roses :: Sarah J. Maas
★★★	review	2024-04-07	The MANIAC :: Benjamin Labatut
★★★★	review	2024-03-24	The Midnight Library :: Matt Haig
★★	review	2024-03-22	Birnam Wood :: Eleanor Catton
★★★★	review	2024-03-19	The Night Circus :: Erin Morgenstein
★★	review	2024-03-18	Nine Princes in Amber :: Roger Zelazny
	review	2024-03-12	Little, Big :: John Crowley
★★★	review	2024-03-11	A Soil Owner's Manual :: Jon Stika
★★★★	review	2024-03-10	The Dispossessed :: Ursula K. Le Guin
★★	review	2024-03-07	Book Thief :: Markus Zusak
★★★	review	2024-03-05	Paved Paradise :: Henry Grabar
★★	review	2024-03-03	Revolver: Sam Colt and the Six-Shooter That Changed America :: Jim Rasenberger, Jacques Roy, et al.
★★	review	2024-02-29	What Do You Care What Other People Think? :: Richard P. Feynman
★★★	review	2024-02-29	Genius at Play: The Curious Mind of John Horton Conway :: Siobhan Roberts
★★★★★	review	2024-02-25	The World According to Garp :: John Irving
★★★★★	review	2024-02-20	Educated :: Tara Westover
★★	review	2024-02-18	The Upcycle :: William McDonough and Michael Braungart
★★	review	2024-02-17	From Here to Eternity :: Caitlin Doughty
★	review	2024-02-17	Among Others :: Jo Walton
★★★	review	2024-02-09	Faster :: James Gleick
★★★★	review	2024-02-05	Surely You're Joking, Mr. Feynman! :: Richard Feynman
★★★	review	2024-02-01	How to be Alone :: Jonathen Franzen
★★★	review	2024-01-29	Although Of Course You End Up Becoming Yourself :: David Lipsky
★★	review	2024-01-29	Bringing Nature Home :: Douglas W. Tallamy
★★★	review	2024-01-27	Salvation :: Peter F. Hamilton
	review	2024-01-25	So Long, See You Tomorrow :: William Maxwell
★★★★	review	2024-01-22	Chronicle of a Death Foretold :: Gabriel García Márquez
★★★	review	2024-01-20	The Information :: James Gleick
★★★★	review	2024-01-16	Cradle to Cradle :: William McDonough and Michael Braungart
★★★	review	2024-01-13	One Day in the Life of Ivan Denisovich :: Aleksandr Solzhenitsyn
★★	review	2024-01-13	No Such Thing as Bad Weather :: Linda Åkeson McGurk
★★★★	review	2024-01-13	Hunt, Gather, Parent :: Michaeleen Doucleff
★★	review	2024-01-10	The Joy of X :: Steven Strogatz
★★	review	2024-01-08	Rainbow's End :: Vernor Vinge
★★★★	review	2024-01-06	The Hour of the Star :: Clarice Lispector
★★★★★	review	2024-01-03	The Library at Mt. Char :: Scott Hawkins
★★	review	2023-12-31	Breakfast at Tiffany's :: Truman Capote
★★★★	review	2023-12-30	Working :: Robert A. Caro
★★	review	2023-12-30	Moll Flanders :: Daniel Dafoe
★★	review	2023-12-28	The Gift of Therapy :: Irvin Yalom
★★★★★	review	2023-12-27	Of Mice and Men :: John Steinbeck
★★★★	review	2023-12-26	Kitchen :: Banana Yoshimoto
★★★★	review	2023-12-24	We Have Always Lived In The Castle :: Shirley Jackson
★★★	review	2023-12-21	Radical Acceptance :: Tara Brach
★★	review	2023-12-19	Turn of the Screw :: Henry James
★★★★	review	2023-12-12	Train Dreams :: Denis Johnson
★★★★★	review	2023-12-09	White Teeth :: Zadie Smith
★★★	review	2023-11-26	The Trial :: Franz Kafka
★★	review	2023-11-25	Lucky Jim :: Kingsley Amis
★	review	2023-11-25	High Rise :: J.G. Ballard
★★★	review	2023-11-25	Things Fall Apart :: Chinua Achebe
★★★★	review	2023-11-23	How to Win Friends and Influence People :: Dale Carnegie
★★★	review	2023-11-10	Jonathan Strange & Mr. Norrell :: Susanna Clarke
★★★★	review	2023-11-07	Piranesi :: Susanna Clarke
★★★★	review	2023-10-26	All The Birds In The Sky :: Charlie Jane Anders
★★★★	review	2023-10-23	No One Is Talking About This :: Patricia Lockwood
★★★★★	review	2023-10-20	East of Eden :: John Steinbeck
★★★	review	2023-10-12	The Structure of Scientific Revolutions :: Thomas S. Kuhn
★★	review	2023-10-10	The Body :: Bill Bryson
★★★★	review	2023-10-03	Storm Front :: Jim Butcher
★★★★★	review	2023-10-01	Men, Machines, and Modern Times :: Elting E. Morison
★★★	review	2023-09-16	Breath :: James Nestor
★★	review	2023-09-18	The Nature of Oaks :: Douglas W. Tallamy
★★	review	2023-09-18	Writing Down the Bones :: Natalie Goldberg
★★	review	2023-09-19	The Moon is a Harsh Mistress :: Robert A. Heinlein
★★★★★	review	2023-09-20	The Timeless Way of Building :: Christopher Alexander
★★★	review	2023-09-25	Accelerando :: Charles Stross
★★★	review	2023-09-28	Permutation City :: Greg Egan
★★★	review	2023-09-29	On Having No Head :: Douglas Edison Harding
★★★★	review	2023-09-15	When Things Fall Apart :: Pema Chödrön
★★★	review	2023-09-14	A Walk in the Woods :: Bill Bryson
★★★	review	2023-09-12	Adult Children of Emotionally Immature Parents :: Lindsay Gibson
★	review	2023-09-11	If You Could See The Sun :: Ann Liang
★★	review	2023-09-10	Extra Virginity :: Tom Mueller
★★★★★	review	2023-09-09	Player Piano :: Kurt Vonnegut
★★★★	review	2023-09-05	Restoration Agriculture :: Mark Shepard
★★★	review	2023-09-04	No Logo :: Naomi Klein
★★	review	2023-09-02	The Shadow of What Was Lost :: James Islington
★★	review	2023-09-01	Orbiting the Giant Hairball :: Gordon MacKenzie
★★	review	2023-08-23	So You've Been Publicly Shamed :: Jon Ronson
★★★	review	2023-08-22	The Left Hand of Darkness :: Ursula K. Le Guin
★★★★★	review	2023-08-21	The Omnivore's Dilemma :: Michael Pollan
★★★★	review	2023-08-18	The Design of Everyday Things :: Don Norman
★★★★	review	2023-08-12	The Inner Game of Tennis :: W. Timothy Gallwey
★	review	2023-08-09	Triumph of Seeds :: Thor Hanson
★★	review	2023-08-08	Bird by Bird :: Anne Lamott
★★★★★	review	2023-08-08	The Diamond Age :: Neal Stephenson
★★	review	2023-08-02	Zen in the Art of Writing :: Ray Bradbury
★	review	2023-08-01	How to Read Nonfiction Like a Professor :: Thomas C. Foster
★★★	review	2023-07-29	Last Argument of Kings :: Joe Abercrombie
★★	review	2023-07-24	Before They Are Hanged :: Joe Abercrombie
★★★	review	2023-07-23	The Blade Itself :: Joe Abercrombie
★	review	2023-07-21	Company of One :: Paul Jarvis
★★	review	2023-07-21	Fair Play :: Eve Rodsky
★★	review	2023-07-19	Station Eleven :: Emily St. John Mandel
★★★	review	2023-07-17	We Are Never Meeting in Real Life :: Samantha Irby
★★★★	review	2023-07-15	Story of Your Life and Others :: Ted Chiang
★★	review	2023-07-11	Kafka on the Shore :: Haruki Murakami
★★★★	review	2023-06-29	A Man Called Ove :: Fredrik Backman
★★★★	review	2023-06-25	Paper Menagerie :: Ken Liu
	review	2023-06-11	White Noise :: Don DeLillo
★★★★★	review	2023-06-09	The Grapes of Wrath :: John Steinbeck
★★★★	review	2023-06-08	A Swim in a Pond in the Rain :: George Saunders
★★	review	2023-05-25	The Black Prism :: Brent Weeks
★★	review	2023-05-22	Lucifer's Hammer :: Larry Niven and Jerry Pournelle
★★★	review	2023-05-17	This Is How You Lose the Time War :: Amal El-Mohtar and Max Gladstone
★★★★	review	2023-05-15	The Rosie Project :: Graeme Simsion
★★★★	review	2023-05-13	The 100-Year-Old Man Who Climbed Out the Window and Disappeared :: Jonas Jonasson
★★★★★	review	2023-05-11	Breakfast of Champions :: Kurt Vonnegut
★★★★	review	2023-05-10	The Hitchhiker's Guide to the Galaxy :: Douglas Adams
★★★	review	2023-05-09	Baby-Led Feeding :: Jenna Helwig
★★	review	2023-05-09	Tell Me No Lies :: Bader, Pearson, Schwartz
★★★★	review	2023-05-09	Atlas of the Heart :: Brené Brown
★★★★★	review	2023-05-01	Exhalation: Stories :: Ted Chiang
★★★★	review	2023-04-26	The One-Straw Revolution :: Masanobu Fukuoka
★★★★★	review	2023-04-05	How We Got To Now :: Steven Johnson
★★★	review	2023-04-03	The Walking Man :: Jiro Taniguchi
★★	review	2023-04-03	The Wintringham Mystery :: Anthony Berkeley
★★★★★	review	2023-04-01	Tomorrow, and Tomorrow, and Tomorrow :: Gabrielle Zevin
★★★★★	review	2023-03-27	A Confederacy of Dunces :: John Kennedy Toole
★★	review	2023-03-26	Bridget Jones's Diary :: Helen Fielding
★★	review	2023-03-23	Promise of Blood :: Brian McClellan
★★★★★	review	2023-03-16	Anything You Want :: Derek Sivers
★★	review	2023-03-12	Learned Optimism :: Martin E. P. Seligman
★★	review	2023-03-12	To Say Nothing of the Dog :: Connie Willis
★★★	review	2023-03-07	Get In Trouble :: Kelly Link
★★★	review	2023-03-02	Slade House :: David Mitchell
★★★	review	2023-03-02	The Bell Jar :: Sylvia Plath
★★	review	2023-02-22	The Fall of Hyperion :: Dan Simmons
★★★★★	review	2023-02-13	The Anthropocene Reviewed :: John Green
★★★	review	2023-02-13	Me Talk Pretty One Day :: David Sedaris
★★★	review	2023-02-09	Zero to One :: Peter Thiel
★★★	review	2023-02-05	Artemis :: Andy Weir
★★★★	review	2023-02-02	Hyperion :: Dan Simmons
★★★	review	2023-01-27	The War of Art :: Steven Pressfield
★★★★	review	2023-01-26	God's Debris :: Scott Adams
★★★★	review	2023-01-24	Interview with the Vampire :: Anne Rice
★★★★★	review	2023-01-21	Norwegian Wood :: Haruki Murakami
★★★★	review	2023-01-18	The Way of Kings :: Brandon Sanderson
★★★★★	review	2022-12-25	The Handmaid's Tale :: Margaret Atwood
★★★★	review	2022-12-19	Animal Farm :: George Orwell
★★★★	review	2022-12-04	The Colour of Magic (Discworld #1) :: Terry Pratchett
★★★★	review	2022-11-27	The Remains of the Day :: Kazuo Ishiguro
★★★	review	2022-11-19	The Stranger :: Albert Camus
★★★★★	review	2022-11-13	Metamorphosis :: Franz Kafka
★★	review	2022-10-30	Discipline is Destiny :: Ryan Holiday
★★★★	review	2022-10-16	Ubik :: Phillip K. Dick
★★★★	review	2022-10-09	Triggers :: Marshall Goldsmith
★★★	review	2022-10-02	Book of the New Sun :: Gene Wolf
★★★★	review	2022-09-02	Solaris :: Stanislaw Lem
★★★★★	review	2022-08-18	The Good Earth :: Pearl S. Buck

Zen and the Art of Happiness

This book repeats itself: "convince yourself that you're living in the best possible universe".

The Prime Number Conspiracy

Good variety of math journalism hits with that pleasing Quanta style.

Soulhome

My first foray into progression fantasy. Cool magic system propping up a weak plot.

Good Inside

Dr. Becky is the best! Great guide to navigating big emotions in parenting and life.

Factfulness

A great case that the world is getting better and that human intuition is broken.

Give and Take

Unremarkable inspo porn.

Bullshit

Surprisingly deep philosophical musings on BS.

An Elegant Puzzle

A collection of concrete wisdom about engineering management.

How Innovation Works

This book traces the paths of how tinkerers improve ideas. I think I preferred Isaacson's Innovators, which illustrated many of the same ideas more clearly.

Musashi

Epic.

The Mom Test

Brilliant guide to validating ideas.

Motivational Interviewing

This book contains life-changing strategies for active listening.

The Golden Compass

Awesome. I somehow missed reading this as a child, but it's filled with depth for adults.

Selfish Reasons to Have More Kids

This book makes an excellent case that most people should consider having more kids. In many ways, the central argument is "Kids make people happier. Most parents spend way too much time on individual children. Nature will eventually dominate your nurture, so nurture less and make more nature."

It's written by an economist; the research was comprehensive and convincing. The parenting advice was, uh, obviously written by an econimist.

Three Laws of Nature

Potent primer on energy & entropy. Very focused, no filler. Highly recommended.

Sirens & Muses

A book about art students that seems to both idolize and resent its subject matter. I suspect I would've enjoyed this book more if I were an actual artist.

Creation Lake

There is so much to love about this story. The Neandrathal stuff was very thought-provoking. Unfortunately, the plot lost way too much momentum in its midsection, and the payoffs were not sufficiently setup to my tastes.

Ministry of Time

Charming book. I would've preferred a short story.

Complexity: A Guided Tour

A scattered (but very entertaining) survey of complex systems. Sometimes felt like reading a "greatest hits" of all the coolest ideas in math/science, and that's okay.

Supercommunicators

It's like HTWFAIP written by Malcolm Gladwell! And I really mean that in the best possible way -- I thoroughly enjoyed this book, and feel like I learned a lot from it.

How Evil Are Politicians?

Too much Bryan Caplan for me!

The Spider's War

Satisfying conclusion to a fantastic series.

The Case Against Education

This book mixes flamboyant hubris with intellectual humility -- I think I actually quite enjoy those flavors together: strong opinions, supported with meticulous evidence, but hedged with solid arguments against your case.

But is education really "80% signaling"? Yeah, I think I'm pretty convinced of that. But still unsure what to do with that information though. If it's true, there's a vast and undervalued market gap for talent search/certification.

Radical Abundance

Decent crash course into atomically-precise manufacturing. Great frameworks for thinking at small scales. Drexler seems to have a beef with the science community, which felt like a constant distraction from the brilliant ideas in the book.

How to Talk So Kids Will Listen & Listen So Kids Will Talk

This book really should be called "How to Talk So People Will Listen". It's loaded with actionable advice that I've already started using with my family.

The Widow's House

Okay, I really just need to know what happens next. Good writing, great plot, but just a lot of it.

A Naked Singularity

A "perfect" book. Unbelievably, indescribably good. It's hard not to compare it to DFW's fiction; both the writing style and its subject matter evoked Infinite Jest and Broom of the System, and -- blaspheme incoming -- I much preferred A Naked Singularity. The plot, the themes, the characters, the feels, the research -- what an experience!

A General Theory of Oblivion

A potent literary tangle.

The Impossible Man

This biography must've been exceptionally difficult to write. How does an author explain generational harm without excusing it? How do they cover the pursuit of genius without encouraging some of its most negative stereotypes?

The Tyrant's Law

I am loving this series! …and I want it to move faster.

The Road

A book about life, death, and evil. It'll remain in your memory like a scar. Highly recommended.

Prisoners of Geography

An easy introduction to contemporary geopolitics. I wish I had read this in middle school!

The King's Blood

Great fantasy! Rarely do I read sequels, and even more rarely do I enjoy them this much.

Cadillac Desert

It's one of those non-fiction books that changes everything. Rarely do I read something that needs to take such provocative measures to explain things that should obvious and uncontroversial. Water is essential.

Welcome to the Hyanam-dong Bookshop

This book is the product of many positive social movements. While I didn't enjoy the book's plot all that much, I'm very happy to see the zeitgeist carry us here!

Thinking with Type

A gorgeous, delightful, informative book. This work takes the art of printed words to another level.

Acaster is funny, as usual; the book achieves some surprising depth through absurdism.

Fab

A decent book about a powerful idea: the imminent personal fabrication revolution. So hungry for more details/depth; I'll probably need to watch the related "How to Make Almost Anything" course online.

Drive Your Plow Over the Bones of the Dead

Probably very good, but too much for me for now.

Tyll

Nice, dark folklore.

Glucose Revolution

Practical guide to managing blood sugar, backed by science.

The Idiot

This novel is so detailed, it's hard to believe it's not a memoir. It's sometimes too smart but overall beautifully written.

I admittedly stopped reading this book after ~75% completion. The plot evoked too many memories of longing, humiliation, regret, etc.

The Dragon's Path

Excellent fantasy! I'd put it somewhere between Brandon Sanderson and George R R Martin. Great characters, intriguing plot, well-crafted twists.

Oblivion: Stories

To be honest, it's a bit too much David Foster Wallace. Most of the stories are brilliant; some are extraordinary chores.

The Dream Machine

An opinionated-yet-measured history of personal computing via the lens of JCR Licklider, "Computing's Johnny Appleseed".

Heaven

Cool exploration of good/evil via manic pixie dream girl. Clunky execution overall.

The Employees

Objectification! Of androids, of humans, of women, of men, of employees, of objects, etc. Very well done.

Convenience Store Woman

This tiny book made me feel a lot of things. It made me… wistful? kinder? weirder? resentful?

Nexus

In short: information is neutral, bureaucracy is interesting, and computers are powerful. As usual, Harari is an excellent writer, but I felt somewhat underwhelmed by this book. I suspect that this book was written to convince other people of these things.

Capitalist Realism

Makes a solid case that capitalism is responsible for much of the malaise and misery of our era. I wish the book made clearer descriptions of alternatives to capitalism. I understand that that was not the point of the book, but as the book says, it's really hard to even picture a world without capitalism at this point.

No Country for Old Men

One of those rare books with incredible characters, beautiful (yet unpretentious) prose, gripping plot, clever dialogue, and enough depth to drown yourself.

The Strangest Man

Cool biography about cool dude.

Trust

It's a metafictional novel about capital/power with a deeply compelling plot! Wow!

Pastoralia

One of the best fictional works I've ever encountered. Funny! Painful! Rich! Otherworldly!

Atonement

I generally have no patience for stories like this. I don't know how this differs from other romance/WWII books, but, wow, Atonement hits hard in all the right ways at all the right times -- an expertly crafted plot.

When We Cease to Understand the World

I adore Labatut's willingness to take liberties with history; it's the perfect amount of speculation injected into non-fiction. Stories of mad geniuses are irresistible to me, and this book is full of such true(ish) tales.

The Friend

Well-written literary fiction, but nothing remarkable.

Hateship, Friendship, Courtship, Loveship, Marriage

Tidy, enjoyable short stories.

The Soul of a Machine

A journalist lives in a basement with an engineering team for 18 months while they attempt to build a 32-bit computer from scratch.

Seeing Like a State

After reading Guns, Germs, and Steel, I felt like I was suddenly endowed with a unified framework for understanding history and geopolitics.

For me, Seeing Like a State has induced similar "framework thinking". Governments behave for good reasons; they succeed/fail in predictable ways.

Stay True

It's a memoir. If you like memoirs, you'll like this memoir. It's about growing up, finding identity, friendship, grief -- yep, it's a memoir.

Evicted

An essential non-fiction read. Great storytelling; difficult to stomach at times. Great insights into the struggles of both landlords and tenants.

Olive Kitteridge

Remarkably good short stories, with excellent and subtle commentary on that tiresome brain defect that so many of us are born with.

A Shining

what on e*rth did I just read and why am I barefoot

Outline

Plotless, yet gripping! Somehow reads like second-person POV but obviously isn't. A book that takes listening to oneself and others very seriously.

The Glass Cage

Automation good! Automation bad!

I found Christian's Alignment Problem and Kuang's & Fabricant's User-Friendly more enlightening on this same topic.

Volt Rush

Feels like a timeline that I neither understand nor care about is being forcibly spoonfed to me. I think a commodity-traders narrative (a la The World for Sale), a Michael-Pollan-esque story, or a land-based "first-principles" approach (a la Guns, Germs, and Steel) would've been much more interesting.

Demon Copperhead

Reads like a modern "hillbilly" Anna Karenina (or maybe something more deep and heavy). This book will forever alter the way I see rural America, opioid addiction, and systemic oppression.

The World for Sale

Top-tier historical account of the wide-reaching influence of commodities and their traders.

Roadside Picnic

Read it! It's so, so, so good. The plot, the writing, the characters, the ideas -- it's brilliant.

The War Below

The book somehow moves both too quickly and too slowly for me to absorb its contents.

Frostbite

Refrigeration is cool! I salute the author for a worthy tilt at a difficult topic.

War with the Newts

Peculiar, potent, prescient.

Leviathan

Relentless melodrama with some clever character/plot structures.

Enchanted Objcets

A refreshing plea to build delightful technology with fewer screens.

The Long Earth

One of the most charming and brilliant books I've ever experienced. I'm awestruck by its scope, by its humor, and by its depth.

The Alignment Problem

An excellent history of AI/ML in the context of human goals. If you're looking for a lightly-technical book covering the trajectory of AI/ML, I highly recommend this book.

How to Have Impossible Conversations

A guide to exercising curiosity; lots of great tips on listening, suspending judgement, maintaining common ground, and sowing doubt. These "applied epistimology" folks might be onto something.

My major takeaway is that knowing how people believe is generally more useful than knowing why they believe.

The Sellout

Insanely good lit. Humans will be laughing/crying with this book for centuries.

Slow Productivity

More of the same good Cal Newport advice. I highly recommend his podcast if you want sane and sustainable advice on how to achieve goals.

Quantations: A Guide to Quantum Living in the 21st Century

I'M GOING QUANTUM!

The Sparrow

Thorough exploration of the problem of evil. A brutal tale about faith, science, religion, and culture. The highs are very high and the lows are very low. The cultural worldbuilding was entertaining, but detracted somewhat from the potent narrative.

Co-Intelligence

A practical introduction to integrating AI into your everyday life. I'd definitely recommend it to somebody who doesn't understand how/why ChatGPT is useful. But to me, it was just preaching to the proverbial choir.

The Politics of Design

An appealing fisheye journey through design.

The Overstory

The first ~20% of The Overstory is delightful -- short stories where entire human lifetimes breeze by in a few pages.

But then the author tries to weave together the individual tales (a la Seinfeld?) and completely fumbles the pacing, the characters, and everything I so liked about the introduction.

In the end, it feels like an Ayn Rand's Cloud Atlas about logging. Seems like something that Netflix would try to turn into a series and then cancel it three seasons in.

User Friendly

Good overview of human-centered design in engineering. A surprising amount of anecdotes I hadn't heard before.

The Cold Start Problem

Solid strategic guide for any startup building on networks. Lots of fantastic idea fodder.

Going Infinite

An entertaining biography! I'm curious how the EA community and other proximate parties would tell the story.

Build

A solid startup guide borne from an impressive career.

The Rise of Theodore Roosevelt

The definitive story of the ascent of one of planet Earth's most interesting creatures.

When I was much younger, this book sparked my lifelong pursuit of excitement and power. My lust for bustle persists, much to everybody's chagrin.

The Art of Product Design

An entertaining sales pitch hardly disguised as a book.

Ender's Game saga

I think I'm a sucker for "lonely messiah-complex mastermind" stories, and that is not a good thing. I never thought I'd fall prey to sympathizing with Superman.

Speaker for the Dead is especially good. It's like the cruciform arc coauthored by Borges and Gabriel García Márquez. But again, written by a problematic person.

Effortless

Nothing remarkable; this book was a pleasant reminder to stop trying so dang hard.

The Fifth Season

Very, very good sci-fi/fantasy writing. It's filled with provocative-but-not-preachy themes, which is the hallmark of good sci-fi. Jemisin is unbelievably good at writing characters and making their interactions believable.

One thing I realized from this book is that I may have exhausted my patience for high-fantasy worldbuilding. Sometimes these fantasy books feel like taking a sociology class for fictional cultures, which is expected in the sci-fi/fantasy genre, but incompatible with my shrinking attention span.

Distrust that Particular Flavor

A scattered-but-potent collection of non-fiction writings from a founding father of cyberpunk.

The Player of Games

Clever sci-fi that desparately wants you to know how clever it is.

Bears Discover Fire and Other Stories

Criminally underrated short story collection from the author of They're Made Out Of Meat. Clever, deep, and human.

Consider Phlebas

Well I finally started The Culture series. The first book had excellent sci-fi worldbuilding sandwiched between a lame action/adventure plot and bad writing. I suspect that Banks would be incredible at writing short stories, where his ideas could maintain focus.

Notes on the Synthesis of Form

One of the most profound and underrated works of the twentieth century.

In this short book, Christopher Alexander does for design(?) what Newton did for physics -- he wrestled a whole field out of the realm of the feelings and gods into the domain of math and mankind.

Lawns into Meadows

Felt like reading a very good blog post.

A Court of Thorns and Roses

Solid writing marred by unbearable pacing.

The MANIAC

A one-of-a-kind book. Felt like a (1) BBC radio drama covering (2) the mad scientists of WWII with a (3) brilliant throughline to today.

The Midnight Library

Reminded me of Mitch Albom, Groundhog Day, and maybe Black Mirror… in a good way.

Birnam Wood

The author is skilled at writing believable Sorkin-esque monologues from a ton of different political perspectives. She's also got a talent for setting up a house of cards and knocking it down with style. But I don't know -- I didn't enjoy the story.

The Night Circus

Captivating plot, likable characters, cool magic, fantastic imagery. It's surprising that Hollywood hasn't made an adaptation yet.

Nine Princes in Amber

Begins as a nifty amnesiac action-thriller and then slowly morphs into typical renfair male fantasy.

Little, Big

Extraordinary modern fairy tale with lovely prose. A one-of-a-kind book. Also somewhat boring.

A Soil Owner's Manual

Neat little book about soil health.

The Dispossessed

Marvelous storytelling and worthwhile thought-experiments for anarchist/communist/capitalist sympathizers.

The Book Thief

The author is clearly very talented, but this story was underwhelming. Reads like the worst kind of high-school-English-required-reading.

Paved Paradise

The author makes a surprisingly bipartisan case against parking. He summarizes his recommendations as follows:

The path forward from a policy perspective seemed clear. Abolish parking minimums and let developers build the amount of parking their clients want. Break garage rents apart from apartment rents so carless tenants don't have to subsidize their neighbors driving. Recognize that more parking means less housing, especially affordable housing. Let different uses -- an office and an apartment building, a school and a movie theater -- share parking. Charge for the best street parking, and use parking prices and enforcement not to generate cash and cycles of punishment but to manage city streets. Invest the proceeds in the neighborhood. Let architects design environments where people can walk. Ask drivers to bear some of the externalities of automobile use.

Revolver: Sam Colt and the Six-Shooter That Changed America

I read this book for specific research purposes. I commend the authors for striking a nice balance between entertainment and historical neutrality.

What Do You Care What Other People Think?

Surely your publishers want bonus content, Mr. Feynman!

Genius at Play

I hate reading biographies, but this one was alright. Conway was a character!

The World According to Garp

A book seemingly about death & male feminism, and also the act of writing about death & male feminism. This book started out very funny and then slowly became an artistic chore (in a good way, I guess). A sometimes-annoying-but-admittedly-brilliant work of metafiction.

Educated

It's like Little House on the Prairie, but with graphic abuse and trauma. I wish it wasn't a true story. Sharp words; lasting scars. Unforgettable strength.

The Upcycle

Cradle to Cradle errata and bonus content.

From Here to Eternity

Enlightening survey of problems in the funeral industry, with plenty of good comparisons to historical/worldwide customs. Corpses are not to be feared.

Among Others

Insufferable mix of revisionist-history-memoir, magical realism, and misunderstood bookworm heroine. I know it received great reviews, but I just didn't understand the hype.

Faster

Standard Gleick deep-dive into human-scale time/speed. Lots of good insights on efficiency and "average" time.

Surely You're Joking, Mr. Feynman!

Classic rascal tales!

How to be Alone

Smart essays for smart people. Not for me.

Although Of Course You End Up Becoming Yourself

It's a transcript of a very intimate David Foster Wallace interview, where David Lipsky spends lots of time with Wallace during the final leg of his ascent to infamy. I can't imagine anybody doing as good of a job as Lipsky here; he really knew when to let David talk, and how to highlight the scenery without letting it steal focus from the main show.

It was uncanny to hear DFW voice so many of my own sticky/ineffable anxieties about addiction, fame, media, intelligence, etc. His ability to point at those difficult-to-articulate feelings is probably why he is considered one of the greats.

Bringing Nature Home

Dry book about plant diversity. Would be nice as a 90-minute lecture, but a grueling experience in book form.

Salvation

Starts out as world-class sci-fi and then devolves into cliché. Personally, this book smelled like an incredibly talented author let down by a lack of support from their editor.

So Long, See You Tomorrow

I was completely unsure how to feel about this short work. Such moving storytelling, but at the same time, I felt so distant.

Chronicle of a Death Foretold

Charming story about a rube-goldberg machine drama in a tiny town.

The Information

A history of information theory for the layman. Lots of good tidbits.

Cradle to Cradle

A refreshingly holistic approach to sustainability! Solid foundations on real-world chemistry. Although pretentious in many parts, the authors have receipts to back up their bold opinions.

One Day in the Life of Ivan Denisovich

Expertly crafted book about a very not fun day at the Russian labor camp. While historically significant and well-written, it was hard for me to relate to any of it.

No Such Thing as Bad Weather

Lots of gems, but much too wordy. Children play. Outside good. Cold okay. Scandinavia much advanced.

Hunt, Gather, Parent

A fantastic piece of journalism; full of bold, testable, and convincing claims. This book has already made an impact on how I'm choosing to parent my daughter.

The Joy of X

Nifty little chapterized overview of mathematics. Lots of fun examples and "greatest hits".

Rainbow's End

Lots of neat futurism ideas with a plot stapled to it.

The Hour of the Star

It defies explanation. Just the right amount of crazy -- hardtack biscuits with chocolate chips.

The Library at Mt. Char

Intricate fantasy/sci-fi/horror with great insights into power/god/evil/parenting.

Breakfast at Tiffany's

Fun/sad little story with lots to love.

Working: Researching, Interviewing, Writing

After listening to Conan O'Brien gush over Robert Caro on a design podcast, my curiosity was piqued. After reading this essay collection, I'm now fully convinced that Caro is a national treasure.

Moll Flanders

This book reads like 17th century Domestic Girlfriend. The first half of the book has plenty of juicy twists, but the plot becomes repetetive very quickly.

Moll Flanders was likely progressive for its time, but its attempt at feminism has dated poorly.

The Gift of Therapy

Solid collection of advice/notes from a career therapist.

Of Mice and Men

Tell me about the rabbits!

Kitchen

Excellent insights on the grieving process, i.e. good grief!

We Have Always Lived In The Castle

Sometimes slow and annoying, but overall very clever. The story overflows with crucial details that are easy to overlook.

Radical Acceptance

Solid spiritual advice with good storytelling.

Turn of the Screw

Lackluster gothic horror with plenty of mystery but no fright or feeling.

Train Dreams

Phrenetic, short, and somber.

White Teeth

Well, the Zadie Smith hype is justified. White Teeth is excellent. Solid dialogue. Integrates multiple cultures into one narrative without obvious tokenism. Everything feels extremely intentional (in a good way). Slow beginning, but worthwhile payoffs.

The Trial

A scattered plot with decent depth but generally dislikable characters. Funny at many parts, but no stakes provided.

Lucky Jim

A book whose sole purpose seems to tarnish academia. Somewhat funny, but felt hollow and unconstructive.

High Rise

An almost-interesting story premise embedded in shallow writing.

Things Fall Apart

Solid fiction about tribal Africa. I liked most when underlying motivations for cultural practices were hinted at or explored.

How to Win Friends and Influence People

All the advice about human interaction that you should've learned (but didn't) and need to be reminded of (but don't).

Jonathan Strange & Mr. Norrell

An enthralling and epic story, although sometimes a bit too British for my small brain. I suspect this book would actually be much better as a television series.

Piranesi

A beautiful and unique labyrinth story.

All the Birds in the Sky

Heartwarming apocalyptic story about love, nature, magic, technology, growing up, generational trauma, etc. Some of the romance felt forced. Wouldn't be surprised to see it on the silver screen one day.

No One Is Talking About This

This book was painful. Normally books about internet addiction don't feel applicable to me because I never got sucked into Facebook or Twitter.

But this book felt like a personal attack. The author created a creepy caricature of the habits and values I once espoused. And her takes were completely reasonable.

I am impressed with how the author painted her political counterparts in a reasonable light. The US's alt-left and alt-right seem to be equal products of meme culture.

I didn't care much for the plot. There were quite a few things that felt forced, but they were easy to ignore amidst the excellent writing. All the story beats are relatively predictable, but it doesn't lessen the blow.

East of Eden

Deeply human. Possibly the greatest book ever written.

The Structure of Scientific Revolutions

Dry but incredibly deep work on how paradigm shifts work in the academic/scientific community.

The Body

Excellent writing (as usual) from Bryson, but content felt both too broad and too narrow at the same time. Recommended for middle-school students and adults unfamiliar with human biology.

Storm Front

My new favorite soft-magic fantasy world… embedded in a horny detective thriller. Imagine if Harry Potter was actually a relatively good story; instead of a student, he was a detective trapped in a softcore smut novel. The story would've been a masterpiece if the author swapped the ecchi stuff with some actual character development.

Men, Machines, and Modern Times

The definitive essay collection covering technology/innovation/society. Filled with lovely prose, accurate prophecies, great examples, and useful mental models.

On Having No Head

Entertaining booklet on the difference between your head and other people's heads. Great companion to any meditation or spiritual practice.

Permutation City

Reads like a "consciousness and computers are cool" story written by an engineer. A few incredible ideas padded by weak storytelling and philosophical exposition. Probably would've been better as a short story.

Accelerando

Probably the definitive sci-fi on post-human acceleration possibilities.

The Timeless Way of Building

This book provides a wonderful fractal-like framework for thinking about design. Its focus on objective/falsifiable "patterns" is surprisingly pragmatic. The book is filled with delightful writing and great examples.

The Moon is a Harsh Mistres

Reads like "classic" sci-fi. Explores cool ideas on superintelligence and human organization structures.

Writing Down the Bones

Reflections on zen and writing. Very personable, but didn't feel applicable to me.

The Nature of Oaks

More than you ever wanted to know about oak trees. Excellent demonstration of complex and fragile ecosystem interdependencies.

Breath

This book had lots of bold claims worth investigating.

When Things Fall Apart

Her essays feel like an incredible mix of Alan Watts and Brené Brown. This stuff is life-changing. I'm in love!

A Walk in the Woods

An imperfect-yet-hilarious account of an ambitious hike up the Appalachian Trail. Besides being charming and incredibly well-written, the ecological tangents hit super hard. Species of all kinds are going extinct at unfathomable rates. I've heard the stats before and was unmoved, but his anecdotes somehow seared Nature's impending peril into my arteries.

Adult Children of Emotionally Immature Parents

This book should be required reading for anybody with strained family relations. Although not very dense or academic, there are tons of gems to be found.

If You Could See The Sun

Interesting premise for young-adult-gets-superpower book, but with a cringey execution.

Extra Virginity

And suddenly I'm an olive oil snob.

Player Piano

It's not Vonnegut's funniest nor best-written work, but it's a parable of technology/automation that will have a lasting impact on me.

Restoration Agriculture

One of the most practical takes on permaculture, but somehow integrates a long-view of how biomes develop on Earth. Very convincing arguments for switching to perennial-oriented farming. Strong and reasonable opinions that I haven't heard via other permaculture channels.

No Logo

Thorough dissection of corporations, branding, and activism.

The Shadow of What Was Lost

Unoriginal young adult "chosen one" power fantasy with cool takes on time and fate.

Orbiting the Giant Hairball

Whimsical take on corporate creativity. Seems useful for people trapped in certain types of large organizations.

So You've Been Publicly Shamed

A well-written book on a topic I care surprisingly little about.

The Left Hand of Darkness

Solid science fiction with awesome spiritual insight.

The Omnivore's Dilemma

A rare non-fiction masterpiece. Pollan asks and answers all the right questions about humanity's relationship with food.

The Design of Everyday Things

The definitive guide to design -- a must-read for all makers and managers.

The Inner Game of Tennis

A strange but wonderful book that teaches how to teach/learn through meditation-like techniques.

Triumph of Seeds

Seed biology is a super cool topic, but the author's writing style irked me.

Bird by Bird

Solid writing advice and potentially good memoir smashed into a mediocre package.

The Diamond Age

An incredible exploration of nanotech, education, global borders & ethnic groups, and intuition. Some character arcs were a tad disappointing, but that can be forgiven in the sheer volume of ideas put to paper.

Zen in the Art of Writing

A "zesty" book that is both a disappointing memoir and unhelpful writing guide.

How to Read Nonfiction Like a Professor

Insufferable drivel.

The First Law Trilogy

An unremarkable young-adult fantasy series with a bold ending that somehow captured my attention.

Company of One

I struggled to find anything interesting or actionable from this book. I'd suggest reading Derek Sivers's Anything You Want instead, which delivers many of the same opinions in a potent dose.

Fair Play

Fair Play is a manifesto and guide for balancing domestic labor. I think it made a lot of good points, and I can imagine its clever card game/exercise being a useful tool for couples. I'd recommend this book to couples straining to fairly distribute work. I suspect that clear division of responsibilities with moderate time-tracking would be a great move for all households.

Station Eleven

Well-written book with a premise that didn't hook me.

We Are Never Meeting in Real Life

Funny/sad essays about health and money and maturity and love.

Story of Your Life and Others

Another incredible collection of sci-fi shorts by Ted Chiang.

Kafka on the Shore

I totally get that this book is subtle and clever, but sorry, it didn't really make feel anything. And the sheer volume of media references made me feel like I was reading Ready Player One for self-aggrandized Mensa members.

The 19-year-old-philosophy-student-sex-scene from Kafka on the Shore embodies much of Murakami: pretentious, captivating, and horny.

A Man Called Ove

Heart-warming dark comedy about a grumpy old man who wants to die.

Paper Menagerie

Great short story collection exploring China, technology, and the power of words.

White Noise

This book produces a cool metanarrative about finding meaning amongst noise. It also made me chuckle in quite a few parts. I really liked the characterization of the main couple; despite its humorous takes, I think the story displays a lot of elements of a realistic healthy relationship. I only read one third of this book, because I'm a bit burnt out on modern/postmodern literature right now. It seems worth another try at some point in the future though.

The Grapes of Wrath

This book is a passionate exploration of United States' culture. It brilliantly covers capitalism, hunger, determination, distance, inequality, crime, family, and xenophobia. I also loved the poetic flourishes littered throughout the story. The world would probably be a kinder place if everybody read this book.

A Swim in a Pond in the Rain

A collection of engaging short stories with helpful analyses and writing advice.

The audiobook is read by an incredible cast: George Saunders, Phylicia Rashad, Nick Offerman, Glenn Close, Keith David, Rainn Wilson, BD Wong, and Renée Elise Goldsberry

The Black Prism

Decent fantasy story with a compelling hard magic system based on the light spectrum.

Lucifer's Hammer

An extraordinary number of people hope for the world to end in their lifetimes.

This Is How You Lose the Time War

Borderline pretentious but short, beautiful, and entertaining.

The Rosie Project

Delightful romance story from the perspective of an autistic man.

The 100-Year-Old Man Who Climbed Out the Window and Disappeared

An irreverent Forrest Gump story. Incredible comedic payoffs. Required reading for rascals.

Breakfast of Champions

A wacky book about crazy people for crazy people by crazy people. It's Vonnegut's incisive alien wit at its best.

The Hitchhiker's Guide to the Galaxy

Classic, clever comedy.

Baby-Led Feeding

This book details how and why to transition babies to solids early. It provides helpful guidance on safety and nutrition. The book comprises mostly simple recipes that seem tasty enough for adults and babies to share.

Tell Me No Lies

Wading through pages of filler to find a few worthwhile gems about lies in relationships. Sam Harris's short treatise on lying is a more potent and practical read.

Atlas of the Heart

An engaging encyclopedia of emotions.

Exhalation: Stories

A provocative short story collection about humans and technology. Each story will leave a lasting impact on how I see the world. Highly, highly recommended.

The One-Straw Revolution

Human hubris knows no bounds. This book spurns modernity. It's a charming case against science, technology, and progress. The pursuit of comfort makes life untenable.

How We Got To Now

This book is the best kinds of entertaining and informative. Six threads of technological history demonstrate the interconnectedness of ideas. It's a refreshing take on inevitable innovation.

The Walking Man

Short manga stories about the experience of living. Breathtaking artwork in some parts. Inspired me to get drenched and walk barefoot more often.

The Wintringham Mystery

Clever mystery, but ultimately soulless.

Tomorrow, and Tomorrow, and Tomorrow

Deep and emotional and insightful. A love letter to love in the language of game design. This book oozes "millennial", and that's a surprisingly good thing. It feels like classic literature written yesterday.

A Confederacy of Dunces

Nutty Professor meets Don Quixote meets Infinite Jest. A literary landmark ahead of its time.

Bridget Jones's Diary

An endearing read about the difficulties of young adult life and womanhood in the modern era. Contains great tongue-in-cheek commentaries on health, feminism, egotism, and addiction.

Promise of Blood

It's a charming adventure/fantasy story with good characters and cool magic -- nothing revolutionary.

Anything You Want

I adore this short book. Anything You Want celebrates simplicity and sincerity. It reads like a guide to entrepeneurship for Taoists. I've read a lot of business books, and none come close to hitting this hard. Companies forget their customers and drink bureaucratic kool-aid in the pursuit of profits. As I'm starting my own business, I keep coming back to this book to remind me of why I'm doing what I'm doing.

Learned Optimism

This book has exactly three good observations: (1) optimists attribute permanent causes to good events and temporary causes to bad events, (2) optimism is useful when risk is minimal, and (3) optimism is dangerous in risky situations. Learned Optimism would have been an excellent 250-word book.

To Say Nothing of the Dog

Witty, but distractingly British.

Get In Trouble

The Summer People is one of the best short stories I've ever encountered. The other stories in this collection are well-written but failed to hit me in the feels -- intriguing plot premises and a great literary style wasted on young-adult subject matter.

Slade House

This book is a structural marvel but hollow inside. It gives great pacing with lackluster payoffs.

The Bell Jar

This is a must-read for anybody who wants to learn what it's like to have a mental breakdown, or love somebody in a dark place. This book was well-written but too painfully familiar. This book made me sad without being charming or teaching me anything new. It stole from me and gave me nothing in return, but I imagine it has much to give others.

The Fall of Hyperion

This book seemed to have a lot to say. It wanted to be philosophical, it wanted to be adventurous, it wanted to be suspenseful, it wanted to be mysterious, and it wanted to be thought-provoking. But in my opinion, it didn't do a great job at anything in particular. Or maybe its brilliance was wasted on me.

The Anthropocene Reviewed

Painfully human essays; five stars.

Me Talk Pretty One Day

Witty and deeply personal essay collection. Great insights on parenting, grief, pets, language, and addiction.

Zero to One

Solid book on the how & why of founding a startup.

Artemis

This book is a well-researched sci-fi MacGyver adventure story. It's got great characters and a fast plot. Solid book.

Hyperion

Hyperion is unusually refreshing sci-fi. It's 6-8 interconnected short stories with drama, horror, mystery, and awe. Some sections didn't age well, but the writing is incredible in some parts, and some of its ideas are downright brilliant.

The War of Art

If you need inspiration or a swift kick-in-the-butt, read this book. Since reading, I've been unable to make my usual excuses that divert me from my art and career and health. For me, every page contained a mixture of timeless wisdom and awful advice. This book is overall charming and pragmatic.

God's Debris

From few axioms, God's Debris forms a surprisingly coherent religion, philosophy of science, and practical life advice; a worthwhile ~90-minute read.

Interview with the Vampire

Imagine The Count of Monte Cristo meets Faust meets Twilight -- it's an epic tale with deep philosophical underpinnings… and vampires! This book is worth reading if you want to dramatically ponder death, love, and evil.

Norwegian Wood

This book should be labeled with an extreme trigger warning. It covers death and sex and more death and more sex and a whole lot of depression. Imagine blending together Catcher in the Rye, The Great Gatsby, The Perks of Being a Wallflower, Scott Pilgrim, and a smut novel. I think I learned a lot about selfishness and happiness from this experience. I wish I read this when I was 19.

The Way of Kings

If Sanderson's Mistborn is The Chronicles of Narnia, then The Stormlight Archives is The Lord of the Rings. It's magical, political, and finely crafted. Brandon Sanderson knows how to write payoffs. This book alone contains 383,000 words, so clear your schedule before reading.

The Handmaid's Tale

Much of The Handmaid's Tale is surely beyond my comprehension, but it provided a sobering glimpse into some of the unfairnesses of nature and society. Atwood's general take on United States fundamentalism is outstanding. The book is riveting throughout, but its ending secures it a "modern masterpiece".

Animal Farm

Animal Farm delightfully illustrates how authoritarian regimes usurp control. Although the book seemed to be specifically a critique of Stalin's Russia, there are gems to glean about perverse institutional incentives.

The Colour of Magic (Discworld #1)

This was my first Terry Pratchett book. Oh boy, what a delight! If you like clever authors like Raymond Smullyan and Douglas Adams, you'll adore his writing. The Colour of Magic covers some surprisingly deep ideas about science and religion wearing a witty grin.

The Remains of the Day

The Remains of the Day is a subtle story from the perspective of a charming butler. I highly recommend this book if you're in a wistful mood or receptive to deep introspection.

The Stranger

This book is very highly acclaimed. After reading it, I thought I missed something, so I read a few critical analyses -- nope, I understood it perfectly. The Stranger has a decent plot with great writing. This book was probably ground-breaking in 1942, but existentialism has been beaten to death since then. On my end, this is probably a case of "Seinfeld is unfunny".

Metamorphosis

Metamorphosis is potent. It's witty, horrific, deep, and bizarre -- an unforgettable experience.

Discipline is Destiny

This book was inspiring but forgettable. All the examples of disciplined people got me pumped, but I didn't find much actionable content.

Ubik

Good sci-fi stories explore "cool ideas". Ubik miraculously interleaves 3-5 "cool ideas". And it keeps topping itself with bigger and bigger extensions of its core "cool ideas". Read Ubik if you like "cool ideas".

Triggers

This is a self-help book that delivers. It's an incredibly useful guide on how to enact lasting change in yourself. This book focuses on growth via environmental cues. I particularly liked the concrete advice on daily active questions, accountability partners, and actionable ways to change cognitive contexts.

Book of the New Sun

These books are extremely subversive, yet the author falls prey to the exact tropes he's "subverting". The plot oscillates between mind-bending self-reference and painful self-awarelessness. Some of the twists are incredible. I suspect that this is an excellent book that aged poorly.

Solaris

A sci-fi book that gave me literal nightmares. It's unsettling and thought-provoking.

The Good Earth

A heartbreaking rags-to-riches-to-rags story about wealth, family, work, fairness, kindness, and land. It's easy to see why a book about rural Chinese peasants was the best-selling book in the US during The Dust Bowl and The Great Depression. This book will inspire or haunt you.

Default Apps in 2023

huphtur — Sun, 26 Nov 2023 02:00:00 GMT

Found via a toot by Chris Hoyer, this list of default apps. Started by the folks from the Hemispheric Views podcast in an episode dubbed Duel of the Defaults! to find out which person uses the most default apps. After listening to the episode while Zwifting I decided to jot down my own list.

📨 Mail Client: Mail
📮 Mail Server: iCloud
📝 Notes: Notes
✅ To-Do: Reminders
📷 iPhone Photo Shooting: Camera
🟦 Photo Management: Photos
📆 Calendar: Calendar
📁 Cloud File Storage: iCloud Drive
📖 RSS: Innoreader. Very similar to Feedly, which I ditched after they started using AI for all the wrong reasons.
🙍🏻‍♂️ Contacts: Contacts
🌐 Browser: Firefox because of uBlock ad blocker.
💬 Chat: Discord. Have encountered some great folks on the members only Escape Collective Discord. If you’re into cycling, please join us!
🔖 Bookmarks: Firefox because it has bookmark tags. If only Safari had better bookmarking support…
📑 Read It Later: Pocket
📜 Word Processing: Pages
📈 Spreadsheets: Numbers
📊 Presentations: Keynote
🛒 Shopping Lists: Paprika. Our favorite app to store all our recipes. The bookmarklet to import recipes is fantastic. The option to scale recipes to any portion is life saving (specially when dealing with silly imperial system). All the ingredients for a recipe can be added to a shopping list
🍴 Meal Planning: Paprika.
💰 Budgeting and Personal Finance: Wealthfront for both Cash & Savings and Automated Investing. Set it and forget it.
📰 News: RSS + a bookmark folder in Firefox filled with Dutch and English news sources. Would love to use News but the horrendous ads make it impossible to use.
🎵 Music: Music
🎤 Podcasts: Overcast, but may switch to the Podcasts app. In fact, I listened to the Duel of the Defaults! episode in the Podcasts app.
🔐 Password Management: 1Password, but have been in the process of migrating to Passwords this year already.

So if we go by Hemispheric Views’ extremely scientific scoring system I would end up with ~36 points. Not even close to the winner, Martin Feld, who ended up with 33 points.

For 2024 I’m going to be training hard (by switching from Firefox to Safari), which should put me in the lead with 30 points. I’m coming for that Mr. Default crown Martin!

Roc

Home on mfashby.net — Sat, 25 Nov 2023 23:19:27 GMT

2023 Holiday gift guide

Something About Cats — Sat, 25 Nov 2023 19:22:21 GMT

A compendium of stuff I love that you too can own

After Rustlab

dcz's posts — Sat, 25 Nov 2023 16:00:00 GMT

After RustLab

Last week I attended the RustLab conference for the first time. This report was going to be rather boring, but one of the presentations triggered me, so I'm going to put the (hopefully interesting) hot take right here in the front.

Last-minute talk

The talk about "confidential computing" was not on the schedule. It took the slot of a cancelled one, after some confusion regarding lightning talks in the same spot. Please excuse me that I don't know the exact title or the author (I will update the post once I find the recording).

Same, but different

The speaker asked the audience for a show of hands: who heard the term "confidential computing"? Very few hands went up.

Then he explained that "confidential computing" prevents anyone from accessing confidential data, and even the staff at the data center cannot access your computations. This is due to hardware features like TrustZone, hypervisors, Intel SGX, and similar.

Sounds familiar?

How many people would have put their hands up if he asked about "trusted computing"? Or "DRM"? Was the selection of the term intentional? I can only wonder.

Trusted computing can be used to empower. At Purism, my co-workers used trusted computing to give the user control over who can tamper with their OS. My own team added a smart card reader to the Librem 5 for similar purposes. But this is an exception, rather than a rule. Hardware manufacturers use trusted computing technologies to prevent people from loading their software of choice on the devices they own. Some don't give up, and eventually break into their own devices to gain full(-er) access.

The speaker completely ignored that topic, instead describing the rather tame use case where we pay someone else to do our computations – the classic cloud computing arrangement. Trusted computing then ensures that our contractor doesn't see or mess with our inputs or outputs. That's a quite reasonable take on things.

Except it's like presenting the scientific benefits of the ballistic rocket and staying silent on its applications in war.

Root of trust

For a talk about controlling access to data, there was disappointingly little said about who's wielding that control. Even when someone from the audience (moi) asked about it, the answer didn't really include the core concept of processing secrets: the root of trust.

In very simple terms, the root of trust is the part of the system that the data cannot be hidden from.

As the presenter said, CPU doing the actual processing is one. Translating to more useful terms: you must trust the CPU maker to create a CPU free of bugs and not to intentionally exfiltrate your data. This is the CPU has direct access to your data, and verifying that the CPU is OK after buying it is damn near impossible.

But there's an even more important party here. It's the one who actually wants to process the data. The one who loaded the application. If the application could not access data it wants to confidentially process, it wouldn't be useful for much. So the application loader also holds the keys to the kingdom. Quite literally, because the CPU interface is designed to grant access to the confidential computing facilities to anyone who holds one of the cryptographic keys burned (in)directly into the CPU.

If you load the application, but haven't thoroughly verified its source code, then the authors of the source also become roots of trust – but that's a topic for another day.

It's mine! No, it's mine!

So, who's the key holder in practice?

For Secure Boot on x86, Microsoft is one by default. Linux distributions like Fedora need to beg Microsoft to give them access, or otherwise installation would stop with scary warnings by default.

For Android boot, the phone manufacturer typically holds the keys. The firmware running in TrustZone typically cannot be replaced by anyone else.

But if you're the user, the situation is not hopeless. Many CPU manufacturers on the ARM side keep TrustZone accessible. On the Librem 5, we deliberately left all 4 key slots open, for situations where the owner wants to create their own trusted computing environment.

Some people say that not allowing the owner to access the keys is a good thing. After all, the keys protect the owner's confidential data from attackers, and if the users got the freedom to protect themselves, they would certainly hurt themselves, therefore they should not have the option. I will mercifully not comment on this kind of argument.

Instead, I will encourage you, dear hardware and software maker, to stop for a moment, and think. Are you using confidential computing to empower people who own their hardware, or to split them from the computing power they paid for and own? Are you letting them make decisions, or making it for them?

This talk forgot to place humans in context of confidential computing, even though all computing is one for humans. Please don't make that mistake and don't forget who your work serves.

Favourite books

I Learnt — Sat, 25 Nov 2023 14:00:00 GMT

The top shelf of books in my study has the books that have had the most impact on me or that I have enjoyed reading the most.

[Hμblog] Towards Greater Accountability: A Proposal for CA Issuance Decision Logs

devever.net/~hl — Sat, 25 Nov 2023 13:04:45 GMT

Interesting articles by Ryan Hurst about possible improvements to WebPKI security via more certificate transparency (CT)-like technologies. There's also my own writings on CT.

A practical introduction to kill-safe, concurrent programming in Racket

Nikhil's blog — Sat, 25 Nov 2023 07:35:08 GMT

I’ve written a long tutorial exploring the user of Racket’s Concurrent ML (CML) inspired concurrency paradigm to write an API wrapping git-cat-file. It is meant to serve as an introduction to kill-safety and CML concurrency, while assuming some existing knowledge of Racket or other Schemes. I found that there weren’t a lot of resources beyond the reference documentation that explained how to put these APIs together, and I hope this can fill that hole.

The limited utility of the phrase “GNU/Linux”

All content on Seirdy’s Home — Sat, 25 Nov 2023 07:00:04 GMT

“GNU/Linux” distinguishes Linux distributions primarily by the presence of GNU software, which is seldom the most meaningful or practical distinction.

Native Emacs/macOS UX integrations via Swift modules

xenodium.com @alvaro — Sat, 25 Nov 2023 02:00:00 GMT

Once you learn a little elisp, Emacs becomes this hyper malleable editor/platform. A live playground of sorts, where almost everything is up for grabs at runtime. Throw some elisp at it, and you can customize or extend almost anything to your heart's content. I say almost, as there's a comparatively small native core, that would typically require recompiling if you wanted to make further (native) mods. But that isn't entirely true. Emacs 25 enabled us to further extend things by loading native dynamic modules, back in 2016.

Most of my Emacs-bending adventures have been powered by elisp, primarily on macOS. I also happen to have an iOS dev background, so when Valeriy Savchenko announced his project bringing Emacs dynamic modules powered by Swift, I added it to my never-ending list of things to try out.

Fast-forward to a year later, and Roife's introduction to emt finally gave me that much-needed nudge to give emacs-swift-module a try. While I wish I had done it earlier, I also wish emacs-swift-module had gotten more visibility. Native extensions written in Swift can open up some some neat integrations using native macOS UX/APIs.

While I'm new to Savchenko's emacs-swift-module, the project has wonderful documentation. It quickly got me on my way to build an experimental dynamic module introducing a native context menu for sharing files from my beloved editor.

Most of the elisp/native bridging magic happens with fairly little Swift code:

try env.defun(
  "macos-module--share",
  with: """
    Share files in ARG1.

    ARG1 must be a vector (not a list) of file paths.
    """
) { (env: Environment, files: [String]) in
  let urls = files.map { URL(fileURLWithPath: $0) }

  let picker = NSSharingServicePicker(items: urls)
  guard let view = NSApp.mainWindow?.contentView else {
    return
  }

  let x = try env.funcall("macos--emacs-point-x") as Int
  let y = try env.funcall("macos--emacs-point-y") as Int

  let rect = NSRect(
    x: x + 15, y: Int(view.bounds.height) - y + 15, width: 1, height: 1
  )
  picker.show(relativeTo: rect, of: view, preferredEdge: .maxY)
}

This produced an elisp macos-module--share function I could easily access from elisp like so:

(defun macos-share ()
  "Share file(s) with other macOS apps.

If visiting a buffer with associated file, share it.

While in `dired', any selected files, share those.  If region is
active, share files in region.  Otherwise share file at point."
  (interactive)
  (macos-module--share (vconcat (macos--files-dwim))))

On a side note, (macos--files-dwim) chooses files depending on context. That is, do what I mean (DWIM) style. If there's a file associated with current buffer, share it. When in dired (the directory editor, aka file manager), look at region, selected files, or default to file at point.

(defun macos--files-dwim ()
  "Return buffer file (if available) or marked/region files for a `dired' buffer."
  (if (buffer-file-name)
      (list (buffer-file-name))
    (or
     (macos--dired-paths-in-region)
     (dired-get-marked-files))))

(defun macos--dired-paths-in-region ()
  "If `dired' buffer, return region files.  nil otherwise."
  (when (and (equal major-mode 'dired-mode)
             (use-region-p))
    (let ((start (region-beginning))
          (end (region-end))
          (paths))
      (save-excursion
        (save-restriction
          (goto-char start)
          (while (< (point) end)
            ;; Skip non-file lines.
            (while (and (< (point) end) (dired-between-files))
              (forward-line 1))
            (when (dired-get-filename nil t)
              (setq paths (append paths (list (dired-get-filename nil t)))))
            (forward-line 1))))
      paths)))

I got one more example of a native macOS integration I added. Being an even simpler one, and in hindsight, I prolly should have introduced it first. In any case, this one reveals dired files in macOS's Finder app (including the selection itself).

try env.defun(
  "macos-module--reveal-in-finder",
  with: """
    Reveal (and select) files in ARG1 in macOS Finder.

    ARG1 mus be a vector (not a list) of file paths.
    """
) { (env: Environment, files: [String]) in
  NSWorkspace.shared.activateFileViewerSelecting(files.map { URL(fileURLWithPath: $0) })
}

The corresponding elisp is nearly identical to its macos-share sibling:

(defun macos-reveal-in-finder ()
  "Reveal file(s) in macOS Finder.

If visiting a buffer with associated file, reveal it.

While in `dired', any selected files, reveal those.  If region is
active, reveal files in region.  Otherwise reveal file at point."
  (interactive)
  (macos-module--reveal-in-finder (vconcat (macos--files-dwim))))

My Swift module experiment introduces two native macOS UX integrations, now available via M-x macos-share and M-x macos-reveal-in-finder. I've pushed all code to it's own repo.

I hope this post brings visibility to the wonderful emacs-swift-module project and sparks new, native, and innovative integrations for those on macOS. Can't wait to see what others can do with it.

Enjoying this content? Find it useful? Consider ✨sponsoring✨.

Moon Phases • Art

Muffin Man — Sat, 25 Nov 2023 02:00:00 GMT

It is incredibly hard to take a good photo of these. Colors have metallic reflection which I wasn’t able to capture, but it is very prominent in person. They are plotted using five different color markers. I framed them separately and plan to take a better photo once they are on the wall.

Code is available on my GitHub.

Created:	November 2023
Size:	14.5x14.5cm
Paper:	Fabriano Black Black 300gsm
Pens:	UniBall Signo and Sakura Jelly Rolls

husband, father, obligation

taylor.town — Sat, 25 Nov 2023 02:00:00 GMT

For some folks, family is a debt to be paid. They dole out love-shaped deeds in hope of karmic justice. They oblige to stave off guilt.

Festive offers for books on Python, Linux, Regular Expressions and more

learnbyexample — Sat, 25 Nov 2023 02:00:00 GMT

Hello!

Here are some exciting deals for my programming ebooks as well as from other creators.

My ebooks🔗

Offers valid till 30-Nov-2023:

All 13 Books Bundle — $10 (normal price $32)
Learn by example Python bundle — $4 (normal price $15)
Understanding Python re(gex)? — FREE (normal price $10)

Indie creators🔗

Python Problem-Solving Bootcamp — 40% off or purchasing power parity discount whichever is greater
Python books by Michael Driscoll and Teach Me Python Membership — 33% off with black23 discount code
Ebooks on Django and Git — 50% off, plus purchasing power parity if applicable
- see also author's blog post for links to other Django-related deals
The Python Coding Place Membership — 70% off
Python Morsels Membership — lifetime access for the price of 2 years
- see also author's blog post for links to other Python deals
Python To Projects - 5 Week Online Course — 25% off
Python books by Reuven Lerner — 40% off
wizard zines — 50% off on PDFs, 30% off on print versions
Level up with Tailwind CSS and Complete Guide to CSS Flex and Grid — 50% off

Miscellaneous🔗

NoStarch Press — 35% off with DEALS4DAYS code
The Pragmatic Bookshelf — 40% off on all ebooks and audio books
Manning Publications — save 50% when you buy 2 or more MEAPs, eBooks, pBooks, liveProjects, or liveVideos
InfoSec Hack Friday — InfoSec related software/tools
The Cyber Plumber's Lab Guide and Interactive Access — 33% OFF
Leanpub Monthly Sale and Leanpub Weekly Sale — offers for programming books, bundles and courses
Huge list of awesome deals — tools, productivity, books, courses, etc
blackfridaydeals.dev — Hottest Black Friday Deals for Developers
Black Friday Deals — Savings on Tech Books and Courses

Happy learning :)

Technologies left behind

deadlime — Fri, 24 Nov 2023 19:24:08 GMT

A brief history of the dynamic web from CGI to applications with built-in HTTP servers

The joy and dilemma of reading

I Learnt — Fri, 24 Nov 2023 14:00:00 GMT

I read a lot of books. Last year I read over 50 books and this year I am heading towards 40 books.

RSI Trading Signals in Python with TA-Lib: Generate Buy & Sell Signals

Adnan's Random bytes — Fri, 24 Nov 2023 12:00:42 GMT

This post is the part of trading series. In the past, I gave you a brief intro to Ta-Lib and how it can be used in technical analysis, in this post, I am going to discuss how you can RSI indicator to generate buy or sell signals in Python by using the TA-Lib library. Before I write code about the implementation, let’s discuss a bit about signal generation and RSI. What Are Signals In the world of trading, signals are cues or indicators that are derived from various kinds of analyses and guide investors on when to buy, sell, or hold shares. They help traders to make informed decisions based on market conditions. What Is A Technical Indicator A technical indicator in trading is a tool that helps investors analyze price trends and make decisions. It’s like a financial compass, providing insights into market behavior and helping traders identify potential opportunities or risks. What Is the RSI Indicator The RSI (Relative Strength Index) is a momentum oscillator that measures the speed and change of price movements. It ranges from 0 to 100 and is used to identify overbought or oversold conditions in a market. An RSI level below 30 generates buy signals while an RSI level above 70 generates sell signals. Implementation We discussed the main terminologies, now let’s write code for generating signals based on RSI values. I already discussed the development setup here so you can refer to the previous article about it. One thing, unlike the earlier article where I used the “yfinance” API, I am using AlphaVantage APIs because for some reason yfinance APIs are not working for me. On the other hand, Alphavantage provides awesome APIs which also have a free version. I am going to use the endpoint that returns the daily data of a given symbol. All you have to do is to register for the free API KEY. So… the idea is, to get the daily data of a certain stock using AlphaVantage API and then use TaLib’s RSI function to calculate RSI in a pandas dataframe using the default 14-day window. I am using jupyter notebook as it makes things easier. The very first thing is the import of libraries. # All Imports from datetime import datetime as dt import requests import talib as ta import pandas as pd import matplotlib.pyplot as plt The next thing is using AlphaVantage’s API to get the OHLC data. def get_daily_ohlc_data(symbol): function_name = 'TIME_SERIES_DAILY' base_url = 'https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol={}&apikey={}'.format(symbol,API_KEY) r = requests.get(base_url) data = r.json()["Time Series (Daily)"] return data ohlc = get_daily_ohlc_data("MSFT") # Convert to DataFrame df = pd.DataFrame.from_dict(ohlc, orient="index") # Rename columns df = df.rename(columns={ "1. open": "open", "2. high": "high", "3. low": "low", "4. close": "close", "5. volume": "volume" }) df = df.apply(pd.to_numeric) df = df.sort_index(ascending=True) df The function is pretty simple. Using requests library to call the API and then accessing the relevant dict’s key to fetch OHLC data of Microsoft’s stock. After that, convert the payload into a pandas dataframe. Renaming of columns to make it more readable, after that, I am soring them out in ascending order so that I get the latest records at the bottom. When you do that, you will get something like the below. OK, our required dataframe is available, what is left now to calculate RSI def generate_rsi_signals(data, period=14, overbought_threshold=70, oversold_threshold=30): # Calculate RSI data['RSI'] = ta.RSI(data['close'], timeperiod=period) # Initialize 'Signal' column with 'Neutral' data['Signal'] = 'Neutral' # Generate signals data.loc[data['RSI'] < oversold_threshold, 'Signal'] = 'Buy' data.loc[data['RSI'] > overbought_threshold, 'Signal'] = 'Sell' return data In order to calculate RSI, you need to assign a window period, the default value is 14. In the context of the Relative Strength Index (RSI), the parameter period refers to the number of historical price points used in the calculation. The RSI is a momentum oscillator that measures the speed and change of price movements. The formula for calculating RSI involves comparing the magnitude of recent gains to recent losses over a specified time period. So, calling TALIb’s RSI function for the period of 14, we add another column in our dataframe, named, RSI RSI is calculated but things do not end here, the next thing we have to do is to generate signals: Buy, Sell, and Neutral. Then the threshold’s value is to generate buy or sell signals. df_rsi = generate_rsi_signals(df) df_rsi.tail(17) When the above function is called and fetches the last 15 or 17 values, you will get a dataframe like the one below: As you can see, it generated different signals over the period of time, it generated a sell signal where the RSI was more than 70 otherwise neutral. Usually, when you create a trading bot that runs all the time during trading hours, you pick the very last value to decide whether you wanna buy, sell, or hold. Here, the very last value is Neutral or hold thus we will not take any action. Have a winning trading strategy and want it to trade for you automatically, without lifting a finger? I specialize in turning your strategies into fully automated trading bots—whether for stocks, crypto, or forex. Let your strategy work around the clock and maximize your potential. Ready to bring it to life? Email me at kadnan @ gmail.com Conclusion In this post, you learned how easily you can access stock data and then calculate RSI for generating buy and sell signals and minting money ???? Like always, the code is available on GitHub. If you like this post, you should subscribe to my blog for future updates. * indicates required Email Address *

The post RSI Trading Signals in Python with TA-Lib: Generate Buy & Sell Signals first appeared on Adnan's Random bytes.

Intro to Large Language Models (video)

Nicola Iarocci — Fri, 24 Nov 2023 08:05:25 GMT

Andrej Karpathy has a very well-done Intro to Large Language Models video on YouTube. As a founding member and research scientist at OpenAI and with a two-year hiatus working on Tesla Autopilot, Karpathy is an authority in the field. He is also good at explaining hard things.

As a Kahneman reader, I appreciated the Thinking Fast and Slow analogy proposed at about half-length in the video: “System 1” (fast automatic thinking, rapid decisions) is where we’re now; “System 2” (rational, slow thinking, complex decisions) is LLMs next goal. Also, I suspect Karpathy’s intriguing idea of LLMs as the center of a new “operating system style” is not too far off from what will emerge soon. The final segment on AI security and known attack vectors (jailbreaking, prompt injection, data poisoning) is also super interesting.

Which Japanese SIM to pick? An IT professional's opinion (Ahamo, Rakuten Mobile, Povo)

David Mohl — Fri, 24 Nov 2023 03:00:00 GMT

I was discussing Japanese SIM cards with a friend the other day and noticed that there aren’t really any English resources to help people choose the right SIM card. There is quite a big difference in what you get depending on the provider, so I thought I’d add my 2 cents.

Not all SIMs are equal

If we turn back the clock a couple years - the main carrier SIMs (docomo, softbank, AU) were expensive and predatory. It was easy to get hooked into a 2-year contract costing around 8000 yen a month just for a good data connection. The best you could do was getting something like IIJMO (or a dozen other MVNO options) that would be on the docomo or AU wholesale network, but never be on-par with main-network speed and reliability, often lacking big features like Apple Watch support or 5G.

MVNOs, or “Mobile Virtual Network Operators,” are virtual operators that don’t own their infrastructure or network. These companies, like IIJMO, Mineo, HIS Mobile, have agreements with main network operators (docomo, au, SoftBank, Rakuten) to resell part of their wholesale network, usually at cheaper rates but with varying quality.

But all of that changed when Abe stepped down and Suga took over as prime minister. He made 3 important changes to how the big 3 had to operate:

Providers can’t SIM-lock phones - all phones you buy from providers have to have an option to unlock them at any time without paying a huge amount of money
Providers can no longer charge a contract cancellation fee to spark stronger competition - so switching from, say, docomo to AU is now free of charge
The big providers must offer a cheap plan on their main network that is not MVNOs

This changed the entire market: Carriers can no longer lock customers into contracts for years and need to actually offer something unique, otherwise customers will leave.

The big 3 reacted by first offering cheaper main-network SIMs through their sub-brands (Y!Mobile for Softbank, UQ Mobile for AU). This was already great, because now we had plans that were owned by the big carriers themselves, but the government wasn’t happy about that, and told them again: you have to offer cheaper plans on your main brand!

This gave us 3 new plans: Ahamo from docomo, Povo from AU and LINEMO from SoftBank. The important point here is that all these plans, as I just mentioned, are owned by the big 3, so they are not MVNOs that buy wholesale and piggyback. You get the same speed and network quality as “normal” customers of docomo/au/softbank, with no worse deal. These SIMs will almost always be better than any of the MVNOs available!

So as a first line to choose from, my recommendations would be:

On SoftBank: LINEMO, Y!Mobile
On NTT Docomo: Ahamo
On KDDI/AU: Povo, UQ Mobile
On Rakuten Mobile: Rakuten Mobile

For today’s post I will be focusing on Ahamo, Povo (Promo code: LCOD14FW) and Rakuten Mobile.

A quick look at the networks

Ahamo runs on docomo, Povo on AU, and Rakuten Mobile on its own network.

In general, docomo has the strongest network with the fastest speed, followed by AU. Rakuten is the new kid on the block, and while their network is steadily growing, it’s underwhelming for speed and coverage. There are always patches where one is better than the other, but in general, docomo takes the lead. At the time of writing, the AU network has the best coverage on 5G, with docomo lagging slightly behind.

Rakuten Mobile has a unique roaming agreement with AU, so in places where Rakuten Mobile isn’t available yet, it falls back to some offering from AU. This is NOT the same network as main AU or Povo. It is part of the wholesale network that is being rented out to MVNOs. I couldn’t test this as it happens automatically, but I often noticed that my AU line had reception while Rakuten Mobile did not.

Rakuten Mobile often shows ‘5G’ connection, but the speed and throughput is far from it, often comparable with lower 4G speed. Yes, the connection to the tower is 5G, but that doesn’t mean anything if you don’t get 5G benefits; it just eats more battery. This was the only provider where I actually felt that when I entered a URL, I often had to wait a few seconds before anything actually happened, especially in buses or trains.

Bonus points: Docomo/Ahamo allows you to purchase an extra option (currently costing 0 yen/month) that enables 5G SA (5G standalone) that moves you to a dedicated 5G core network, and gives you ridiculous speeds, currently claimed at 受信実効速度は170Mbps~484Mbpsです。

Only docomo offers 5G SA currently, and the addon is still free for an unspecified amount of time. This is by far the best in terms of speed if that’s what you’re after.

However, while docomo has the best network, it’s also used by a lot of people. So in very crowded areas like at festivals or concerts, it’s usually the worst performing. Ironically, Rakuten is the best for those because it has the smallest customer base.

Data volume and plans

Rakuten Mobile has the most attractive deal here. It’s unlimited, and the price goes up depending on how much data you use, but is capped at 3000 yen for everything beyond 20GB.

Ahamo has 2 options: 20gb for 3000 yen, or 100gb for 5000 yen. If you hit the upper limit you get speed limited to 1Mbps which is still pretty good.

Povo has the most unique offering here: Toppings!

By default, if you pay 0 yen a month, you get 128kbps. That’s very slow, but it’s free and unlimited. Though you won’t be able to do much outside of messaging with it. But did I mention it’s free?

Then you can activate different toppings depending on how much data you need

As with the others, there is a 20GB plan for ~3000 yen, but more interesting options like unlimited for 24h for 330 yen, or 150 GB for 13,000 yen, valid for 6 months.

I’ve been using this SIM a lot on my iPad for its 0 yen option, and then when I have a day of heavy working ahead of me, I buy the unlimited topping and tether to my other devices.

Provider	3GB	20GB	100GB	Unlimited
Rakuten Mobile	980 yen	2980 yen	2980 yen	2980 yen
Povo	990 yen	2700 yen	4000 yen (50GB) * 2 or 7400 (200GB / 2)	330 * days
Ahamo	2970 yen	2970 yen	4950 yen	-

Considering this, Rakuten Mobile appears to be the most cost-effective as it never exceeds 2980 yen. However, its network quality is considerably lower. Using a plan like 150 GB for 6 months with Povo can drastically reduce the cost if you don’t use the full 20 GB each month anyway. It’s also a great sub-SIM because of the 0 yen base plan, and you can always add more data when you need it.

Ahamo is the least complicated - either 20 GB or 100 GB. 20 GB was always enough for me, but having the option to upgrade to 100 GB when needed is very nice, and 1 Mbps is still not bad when it runs out.

Roaming when abroad

Roaming is often inconvenient and expensive. It’s usually better to just get a local SIM or eSIM through something like Airalo and be done with it, but with those new plans we may no longer need to do that.

Ahamo surprised when it got announced that it comes with 20 GB of data roaming free of charge!! That’s honestly bonkers and means that in most countries, you can just leave the SIM active and use free data roaming without any additional steps.

This offer is limited to 14 days of constant roaming. If you are longer than 14 days outside of Japan, the speed will get reduced to 128 kbps. You’ll still have unlimited data abroad for messaging, but yeah, it’s gonna be pretty slow. You also can’t purchase additional data when it runs out. You have to return to Japan for it to refresh.

Next in line is Rakuten Mobile - it also gives you 128 kbps unlimited at slow speeds when abroad, or 2 GB at fast speeds free of additional charge (you can toggle fast/slow within the app). Additional volume is available at 1 GB for 500 yen. Still not bad, especially with unlimited at 128 kbps.

Povo is sadly quite bad. They tried to continue with the topping model but all of the options available are pretty pricey - like 5 GB for 14 days for 7080 yen or 3 GB for 7 days for 4280 yen… yeah don’t use this SIM abroad.

Other special perks (Apple Watch, carrier pay, etc)

Ahamo has:

Carrier pay (pay through your phone bill)
Apple Watch data at 500 yen/month
5G SA (mentioned above)
Free 20 GB international data roaming at no extra cost
5 min of each phone call is free
Free use of docomo wifi hotspots
Synergy if you use d払い

Povo has:

No Apple Watch :(
Carrier Pay (AUかんたん決済)
Unlimited data at 330 yen
Unlimited data at 128 kbps at actual 0 yen base plan
Free use of AU wifi hotspots, Wi2Premium and a few more
Synergy if you use AU Pay

Rakuten Mobile has:

Apple Watch data at 500 yen/month
Free international data roaming for 2 GB, unlimited at 128 kbps
Actual unlimited plan for cheap
Carrier Pay…? at some places (Google Play Store), but most don’t support it
Ability to activate or re-issue an eSIM within ~2 minutes online
Free phone calls, SMS through their apps
Gives you extra points on Rakuten Ichiba, synergy with other Rakuten products like Rakuten Card

*Rakuten Mobile offers “free phone calls and free SMS” but it’s the weirdest thing. SMS only works on Android if you use their Rakuten Link app, which for whatever reason doesn’t offer free SMS on iPhone. Free calls is also only if you use the Rakuten Link app, so it’s VOIP and not true cellular - it’s often very janky.

Rakuten Mobile also just stopped working in the past when a new iPhone got released, and some Android phones aren’t even supported for whatever reason (have you ever seen a physical SIM not supported on a phone??), so it’s by far the buggiest of the providers. They also constantly try to upsell your some other Rakuten product, even their Rakuten Link app is plastered with ads…

So David, which one?

In the end it depends on what you want, no? But if you’re still struggling with deciding which one fits the best, my recommendation would be:

Ahamo if you just want 20 GB at fast speed, have an Apple Watch and/or travel a lot. The 20 GB roaming is unbeatable, so is 5G SA. 100 GB for 5000 yen is still reasonable for when you need it. The best all-around SIM.
Povo for their unique topping mechanism and still very good network, even better than docomo in some areas and crowded places, or with 5G coverage. The 0 yen base plan makes this a no-brainer to have ‘just in case’, or put onto an iPad to always have a 128 kbps data connection on hand. 330 yen for unlimited when needed is damn good for remote working. No Apple Watch option though, and their roaming offering sucks.
Rakuten Mobile if you want something unlimited and don’t care much about network, speed, throughput or coverage. It’s good enough in cities and a great secondary SIM to have for when Docomo or AU are overloaded, or when travelling. The ability to quickly move the eSIM to another device has been a life-saver. But, as you can tell from the post, the network is the most ‘meh’, has high latency, lots of dark spots, and other weirdness like their Rakuten Link app having different behavior on iPhone or Android. Also I hate that they constantly try to upsell me when I’m already a customer.

In the end, just try them and see which one you like the most. There is no contract cancellation fee anymore, so it’s very easy to move a number from plan A to plan B and back.

If you liked this post, please us my referral codes:

Povo: LCOD14FW
Rakuten Mobile: https://r10.to/hN5zpk
Airalo: https://ref.airalo.com/21Qr

Town Hall #16: Bloat

taylor.town — Fri, 24 Nov 2023 02:00:00 GMT

Rock the bloat!

Company Missions are Bullshit

Nelson Figueroa — Fri, 24 Nov 2023 02:00:00 GMT

Every company’s real mission is to make money.

Working with feature flags

Yasha Solutions — Fri, 24 Nov 2023 01:00:00 GMT

some notes about feature flags benefits and implementation

English-only embedding models for multilingual docs

Geography & Coding — Thu, 23 Nov 2023 21:33:13 GMT

Text Embeddings: Speaking languages without learning them?

tl;dr: some *-en models perform well on other languages too.

How to do Alt Data Research

Magis — Thu, 23 Nov 2023 20:54:27 GMT

A practical guide to succeeding in alternative data

In life as in gaming, CHA is the strongest stat

vslira's blog — Thu, 23 Nov 2023 15:01:32 GMT

As I write this post, the board of directors of OpenAI is under extreme pressure to reinstate Sam Altman as CEO and giving him complete control of the company while they're at it. If they don't do that, it's likely that they'll lose the company's financial backers, compute enablers and a significant portion of their most important employees - the proverbial members of technical staff. Oh, and be sued to high Heaven by Satya Nadella.

Far from accusing Sam of being just talk as the title of this post may imply, but however intelligent he is what he brought for the company certainly wasn't his AI development capabilities, but the external and internal support that the company needed to move forward.

Contrary to everything I would like to be true, this skill - convincing people, selling to them - is more important than being able to build what they want. Technically being able to, I mean, because actually forming a team to deliver, and presenting it in a way that convinces people to give you attention, will always be more effective than trying to do it on your own and hoping the interested people just show up waving their money at you.

Now, I'm not implying that the board was morally right or that Mr "I Love You All" programmed GPT-N by himself on his basement or anything like that. I'm saying that EVEN IF that was true, Sam was the face of the company, he was the one with the whole support. Not the board, who technically just did what it could always do, for reasons that might not even be wrong according to its charter. But legal abstractions, while certainly not meaningless, can only do so much.

This is a bitter pill for nerds dreaming about their technical startups to swallow, but doesn't make it any less true. Even if you don't want to open your own company and just want to raise the ranks within your current workplace by technical prowess alone, I assure you it will be a thousand times easier if you manage to be a little bit more like this girl.

And if you nominally have the position and power you want, don't forget to occasionally ask yourself, in Bane's voice: "Do you feel in charge?"

Addendum 2023-11-23: This might be relevant to whoever finds the text above useful

(fake) Football Leagues

infrequent oscillations — Thu, 23 Nov 2023 14:01:32 GMT

In which we fake some leagues for fun.

Published IETF RFCs

rob.sh — Thu, 23 Nov 2023 12:00:00 GMT

Over the years I have published a number of RFCs in the IETF. They are listed below.

As primary author:

RFC7684 - OSPFv2 Prefix/Link Attribute Advertisement
RFC7777 - Advertising Node Administrative Tags in OSPF
RFC7855 - Source Packet Routing in Networking (SPRING) Problem Statement and Requirements
RFC8355 - Resiliency Use Cases in Source Packet Routing in Networking (SPRING) Networks
RFC8370 - Techniques to Improve the Scalability of RSVP-TE Deployments
RFC8402 - Segment Routing Architecture
RFC8660 - Segment Routing with the MPLS Data Plane
RFC8662 - Entropy Label for Source Packet Routing in Networking (SPRING) Tunnels
RFC8665 - OSPF Extensions for Segment Routing

As contributor:

使用子解释器运行Python并行应用

ipfans's Blog — Thu, 23 Nov 2023 11:39:00 GMT

译者注：最近 Python 3.12 引入了子解释器概念，非常火热，更好的消息是已经在FastAPI应用成功了，虽然是很简单的那种。因此顺腾摸瓜，找到了作者的博客，翻译分享给大家。

Python 3.12 引入了一个新的 API 用于“子解释器”（sub interpreters），这是 Python 的一种不同的并行执行模型，提供了真正并行处理和多进程处理之间的良好折中，且具有更快的启动时间。在这篇文章中，我将解释什么是子解释器，为什么它对 Python 中的并行代码执行很重要，以及它与其他方法的比较。

什么是子解释器？

Python 的系统架构大致由三部分组成：

一个包含一个或多个解释器的 Python 进程
一个包含锁（GIL）和一个或多个 Python 线程的解释器
一个包含当前执行代码信息的线程。

要了解更多关于这方面的信息，你可以阅读我的书《CPython 内部实现》中的“并行性和并发性”章节。

自 Python 1.5 以来，就有一个 C-API 可以支持多个解释器，但这个功能由于 GIL 的限制而受到严重限制，没有真正实现真正的并行性。因此，运行并行代码最常用的技术（不使用第三方库）是使用 multiprocessing 模块。

2017 年，CPython 核心开发人员提出改变解释器结构的提议，使它们更好地与拥有它们的 Python 进程隔离，并能够并行操作。实现这一目标的工作相当巨大（6 年后仍未完成），并分为两个 PEP。PEP684 将 GIL 在各个解释器独立开，PEP554 提供了一个创建解释器和在它们之间共享数据的 API。

GIL 是“全局解释器锁”，是 Python 进程中的一个锁，意味着在任何时间点 Python 进程中只能执行一条指令，即使它有多个线程。这实际上意味着，即使你在拥有 4 核 CPU 的电脑上同时启动 4 个 Python 线程，也只有一个线程会在任何时候运行。

Lua Carousel

Kartik Agaram — Thu, 23 Nov 2023 09:00:00 GMT

I finally decided to hang up a shingle on itch.io. My first app there is not a game. Lua Carousel is a lightweight environment for writing small, throwaway Lua and LÖVE programs. With many thanks to Mike Stein who helped me figure out how to get it working on iOS, this is my first truly cross-platform app, working on Windows, Mac, Linux, iOS and Android.

repo

Carousel has its own devlog/notebook. I try to post little scripts that are easy for someone to copy to their clipboard, paste into Carousel and run. Some examples:

Hilbert curve and Lindenmayer tree fractal (70 lines each)
rectangle art (40 lines)
simple example using the physics engine. (100 lines)
Chromacline: a war between colors in Game of Life world (100 lines)
simple UI examples; simple voice recorder (<50 lines each)
sprite editor (100 lines)
Pong wars (70 lines)
equation plotting over an infinite surface with pan and pinch-zoom (150 lines)
Lots of different kinds of charts (200 lines)

I am speaking at the .NET Conference Italia 2023

Nicola Iarocci — Thu, 23 Nov 2023 08:05:25 GMT

In a few days, on Monday, December 4, at 10 a.m., I will present at the .NET Conference Italia 2023, which will be held at the Microsoft House in Milan. My session is titled “C#12 What’s new and interesting” and is part of a rich agenda focused on the recent release of .NET 8.

I presented on the same subject at a DevRomagna meetup a few months ago when C# 12 was in preview and still had several moving parts. Now that the final release is out, it will be interesting to return to the topic to consolidate my knowledge. I suspect that of the many reasons I challenge myself to speak publicly, enriching my knowledge is prevalent. As the adage goes, for learning a subject, nothing beats explaining it in front of a knowledgeable audience.

I ❤️ Releases

Backendhance — Thu, 23 Nov 2023 02:00:00 GMT

Hi,

Nach fast zwei Jahren ohne Release ist es am Samstag endlich so weit.
Bei einem meiner Kunden wurde seit zwei Jahren nicht released - Wahnsinn 🤯

Warum? Viele Gründe.

Das Team hat sich in Komplexität verrannt. Es wurden große (notwendige) Änderungen vorgenommen.

Diese hatten aber viele Seiteneffekte. Und der Anspruch war alles “richtig” zu machen. Es wurde also immer weiter entwickelt. Mit einem abgeschlossenen Thema kamen 5 Neue auf.

Aus diesem Kreislauf kam das Team nicht raus.

Als ich im Juli zum Team gerufen wurde beschlossen wir: Wir müssen diesen Teufelskreis durchbrechen.

wigwam.directory

taylor.town — Thu, 23 Nov 2023 02:00:00 GMT

Support wigwam culture!

Setting up a Raspberry Pi Home Server

Jason Eckert's Website and Blog — Thu, 23 Nov 2023 02:00:00 GMT

The Raspberry Pi has been the most popular single board computer (SBC) among computer hobbyists for the past decade. It can be used to perform a wide variety of very different tasks, such as running graphical apps, streaming media, performing home automation, and monitoring humidity levels in a greenhouse. This is largely made possible by the Raspberry Pi’s many interfaces (including GPIO, USB, HDMI, and wired/wireless Ethernet), as well as its ability to run the highly-customizable Linux operating system.

Five minute hacks: Swapping left and right headphone audio in wireplumber

The Technical Blog of James on purpleidea.com — Thu, 23 Nov 2023 00:25:00 GMT

I’m on Fedora 37 at the moment. My headphone audio is backwards. How do we fix it? I’ve apparently never found the right magic until now.

The Setup:

I have a crappy “couch computer”. It’s used for casual internet browsing and video watching. It has an analog audio out which goes to my speakers. I’d like to use headphones for when I don’t want to annoy my neighbours. I have a pair of wireless headphones that I got for free, but the battery doesn’t last longer than an hour. I also have an old pair of analog headphones that work okay and don’t run out of battery! I got a free USB to analog dongle, but for some reason the left and right audio channels are reversed!

Postmortems in Software Development

Vadim Kravcenko — Wed, 22 Nov 2023 21:40:42 GMT

Question:

Answer:

The post Postmortems in Software Development appeared first on Vadim Kravcenko.

More native than a Web app

All content on Seirdy’s Home — Wed, 22 Nov 2023 09:22:58 GMT

Whenever I discover a new GUI toolkit, my first question is always “is it more native than the Web?” For reference, here are some ways Web apps have better system integration than Flutter:

System fonts, font hinting settings, and font fallback settings.
System text navigation shortcuts.
System scrollbar visibility, thickness, and click settings/behavior.
System highlight menus and controls (on touchscreens).
System display scaling settings.
System reduced-motion settings (An API is documented, but I have never used a Flutter app that actually disables animations. I may be wrong).
System reduced-transparency settings.
System forced-colors mode (Windows High Contrast Mode).
System color palettes (some browsers support setting default colors without necessarily forcing the palette).
System media controls (play/pause/skip, current track, e.g. using MPRIS on freedesktop platforms).

It’s an easier question to answer than my second question: “What WCAG conformance level can I hope for?

I hereby pardon all junior engineers

taylor.town — Wed, 22 Nov 2023 02:00:00 GMT

Shortsighted engineering practices have eroded public trust in technology. We can reclaim that trust by building better things together.

Starfield review

ブラック — Wed, 22 Nov 2023 02:00:00 GMT

Mr. Howard's Mild Ride.

The very first impression I had with Starfield was that the world design looked really good. The art direction of Starfield is great. I'm a big fan of the "NASA punk". However, the further I progressed in the game, the less impressed I got with the world. While the individual aspects of it might look good, it's way too repetitive. You land on a single wasteland planet and you've really landed on all of them. And as far as I can tell, the few points of interest in them are repeated throughout the known universe.

The cities are pathetic too. There's a couple of different cities with vastly different designs. They do however have one thing in common: the scale. All of the cities feel woefully small. It feels like I'm walking through a movie set than an actual city. And despite the lack of scale, there's loading screens absolutely everywhere. Get out of your space ship – loading screen. Go to a small residential area – loading screen. Go to the weapon store inside the small residential area – loading screen. Thankfully the missions aren't as full of loading screens but it seems strange that these small cities are in a big-budget 2023 release.

Traversal between places is also deeply unsatisfying. I had to go from a city on a planet, to a bandit camp on the same planet. I opened the extremely poor planet map, selected the bandit camp, and tried to transport there. Sorry, the game said, you can't fast travel to a place you haven't discovered yet. Then it forced me to fast travel to my spaceship and then land my spaceship at the bandit camp. And what does "landing your spaceship" actually entail? A loading screen and a jump cut to the spaceship landing, which all takes around 10 seconds. Almost all traversal in this game is fast travel and yet it still does this idiotic gatekeeping for what it thinks "fast travel" is.

The space bit in Starfield doesn't really improve the situation at all. You can either talk to whatever NPCs happen to be around you, or engage them in battle. That's it. Despite the lack of anything, you'll still be going to space very often since you need to spend at least a few seconds there between fast travels. Actually, the NPCs either reaching out or attacking got very tiresome since I just wanted to travel to my destination and not play tag with pirates.

The repetition in the world design also comes across in the story campaign. The story had me go look for Andreja, and I found her in an enemy-infested mine, and at the end of the mine, I found an artefact. Then later in the story, I was tasked to go look for more artefacts, one of which was at the end of an enemy-infested mine. A mine with the exact same layout, except this one was on a different planet and lacked one Andreja. Both of these mines also had a locked locker room, with contraband for the taking after you picked open the lock. While I can understand that the planets look similar when you have such a vast number of them, I expected a lot more from the main story locations in a AAA game.

Really feels like Bethesda is just taking the piss with the repetition considering missions are actually named "Into the Unknown" and "Further Into the Unknown". A bit on the nose there Todd.

As for the story itself, I didn't really care for it. First of all, I found the premise of how you became an adventurer quite contrived. And while I think they want to make Constellation feel like some sort of an elite gentleman explorer's club, most of the story missions made me feel like their glorified errand girl instead. For every interesting mission, like Unearthed or Entangled, there were two or three missions where I just had to fly from one remote planet to another, literally digging for quest items. Surely there must've been a more interesting story to write in the Starfield universe, right? And despite the fact that the story involves life and death situations, the writing is just so devoid of any edge or character. The whole thing just feels like such a lifeless and watered down experience.

Guess it didn't help that I couldn't care for any of the characters, including my own. The only (human) member of Constellation that I didn't have any active contempt towards was Walter. Vasco was cool too, and got promoted to my primary companion after Sarah got very angry at me for a heinous murder where I shot a man after he and his friends unloaded every bullet they had towards our direction. I guess Sarah was looking to end her miserable existence and I rudely interrupted it by defending myself.

I've heard that some of the side content is much superior but unfortunately I seem to have never encountered it. I guess I did mainly focus on the main quest so maybe that's on me. However, I did encounter side content that I didn't care for.

One example is a mission where settlers task me to drive out a corp from a planet that they consider their birthright and one that they seem willing to fight for. I go talk to the corp and they want me to either blow up the settlers or convince them to become serfs on their luxury resort planet. I decided to not blow up these poor settlers and instead opt to tell them about this indentured servitude proposal. Considering how defiant the settler captain acted, I expected this to lead into a war where they take the planet by force. Nope, the settlers turn heel and agree to become slaves. They they asked me to bring them a bunch of materials, presumably so they can make their own leg irons to really enjoy their bondage. I considered blowing them up for their own good but settled to just leave instead.

I did enjoy the ending though, at least on a conceptual and storytelling level. Unfortunately the gameplay side of it is extremely herky-jerky, and is not at all enhanced by the space combat or waves of trash mobs you also have to deal with. And right at the last second, the story really just raises more questions than it answers. At least the ending was pretty.

As a game, Starfield is a rather unfocused affair. It's so chock-full of different systems that it feels very overwhelming when you get started. Maybe veterans to Bethesda RPGs have already learned to juggle these systems in prior games, but as a newcomer, you need to ingest a lot of (unhelpful) tutorials at the beginning. You have ground traversal, space traversal, ground combat, ship combat, ship building, outpost building, cooking, crafting, spacesuit upgrading, weapon customisation, mystical powers, lockpicking, persuasion, scanning, mining, stealth, encumbrance, commerce, status effects and so on. "A mile wide and an inch deep" is the best quote I have heard to describe this game. Many of these systems are very badly taught to you, and even when you learn them, they're still not that great. Quite possibly why it also took me around ten hours to actually get into the game properly.

The ground combat feels fine. There's a decent selection of firearms available to you and they do feel pretty distinct from each other. It's definitely not very flashy, and there's very limited amount of specialisation you can do with the combat but I did generally enjoy engaging the enemy. The boost pack also livens up the movement for the combat sections too.

The mystical space powers feel like a complete waste of time, at least for enhancing the combat. I really only ever used the unlimited stamina power and otherwise ignored them. And the way that you obtain all of these various powers is so needlessly repetitive. You have to literally jump through hoops for them, again and again and again. It feels more like an elaborate prank than game design.

Space combat is less fine. First of all, whenever you engage in ship combat, you were most likely just on your way to do something completely different until someone decided to harass you. And when they do, they usually do it with their pals. In three-dimensional space. I feel like most of the time in combat is spent around doing loops in order to just find one of the three hostile ships so you can blast them, while trying to keep their two friends from doing the same to you. Even though I rarely struggled, I still did not enjoy it.

Ship building seemed pretty cool, at least for someone that enjoys that kind of a thing. I mainly used it pretty pragmatically, so instead of trying to replicate famous designs or come up with my own, I just used it to upgrade an existing design. It does take some getting used to, since the game doesn't really teach you how to do it and I had to get supplementary lessons from Google on how I can get ship pieces to connect how I want. And unfortunately changing your ship in any way completely destroys any interior decor you might have going on (for understandable technical reasons), so it definitely kept me from doing it just for fun. Maybe people that are heavily into this would find more to criticise about it but I thought that the ship building was cool overall.

Outpost building is also something that is not extremely well taught to you. And it doesn't really feel like you would ever need to do it. However, once I learned a bit on how it works, I got a frenzy of finding resource-heavy planets, building outposts on them all, and connecting them all together to a master outpost. I got pretty absorbed into it until I had all of my outposts connected. Then I realised that I didn't really have any kind of use for it. It just kinda is there, filling up my inventory and few resource crates with a bunch of minerals that I don't really have an use for. I did my outpost fairly early in my run and had no reason to touch it afterwards.

Inventory management is also subpar. For some reason Bethesda decided that the majority of the screen should be taken by 3D representations of your items instead of useful information. While the gun models look nice, I don't find myself that interested in knowing what copper ore looks like. I'd much rather have a side-by-side view of my inventory and a vendor's inventory instead of having to flip between two or three different inventories as I unload my stuff. The vendors will also instantly run out of money when you go sell things, so you need to go do your tedious inventory management with every single vendor in the game. At least I did get used to the inventory, so the user interface is not a constant source of pain, but I still pine for something better.

The lockpicking minigame is actually rather fun but it takes home the prize for the worst explained system in the game. I had no clue on how to do it and was just randomly trying to get through it. Then I watched a short tutorial video on YouTube and got it. Now I'm pretty good at picking open the locks and can actually enjoy some momentary tinkering with locks. What a shame the game doesn't want to teach you about it.

Not a big fan of character creation either. It feels like some of the aspects of it are very varied and some are barely varied at all. I would've also enjoyed having a voice for my character. There's one part in the story that actually makes me feel like it would've been very reasonable and feasible thing even. And I really don't understand the point of the traits. I remember people criticising the lifepaths in Cyberpunk 2077 for having little impact on the game, but I feel like the traits in Starfield have even less of an impact. I was a Neon Street Rat and it just gave me a couple of new choices in one particular part of the game.

Don't really like the skill trees either, especially how it locks some basic abilities in the game behind a skill point. Or many skill points for things in the advanced tiers of the skill tree. I'd expect to be able to do at least some stealth without investing points to a stealth skill but alas. You also need to level up your skills, so don't even think about reaching rank 4 in Medicine without spending time harming yourself and then ingesting health kits by the dozen.

Gatekeeping parts of the game by hiding them in the skill tree are also why I never really got into the crafting and upgrading aspect of the game. You have a few things open at the start of the game but anything beyond that requires investing your few skill points. A shame, since I would've liked to upgrade my weapons more, but Weapon Engineering is an advanced Science skill, so I'd need to invest and level up my science skills before I even got a single point in.

Technical aspects of the game are a bit of a mixed bag. The game was surprisingly stable and bug-free for a new AAA release, although neither aspect was perfect. My crew sank into my spaceship more times than I remember to count and when the game decided to crash, it screwed me out of a lot of progress. At least Cyberpunk 2077 was polite enough autosave very frequently to make up for its lack of stability, and wouldn't make me lose 30 minutes of gameplay with a single hard crash. The performance seems relatively poor considering the lack of ray tracing. Considering how Cyberpunk 2077 looks a lot better with crazy ray tracing, Starfield doesn't deliver that much of a higher frame rate.

It does seem like they did manage to improve it a bit with the recent patch though, but I've yet to clock any serious amount of time in it to make a judgement. Fingers crossed!

I tend to avoid pre-release marketing since I want to temper my expectations for new stuff, but I'm still left feeling disappointed. It's not that Starfield is an awful game. I did manage to beat it after all. It just feels really mediocre and undaring. Perhaps it's worth taking a stab at if you were already paying for Game Pass, or found it in a bargain bin, but for its pain points and overwhelming sense of mediocrity, it's easy to skip and hard to recommend.

Jack and Jill

one mikro2nd — Wed, 22 Nov 2023 02:00:00 GMT

Jack and Jill, Went up the hill To fetch a pail of water. Who goes up a hill to fetch a pail of water? Streams and wells are not generally found up a hill, and for a very simple reason: Water flows, seeps, trickles… down hills, into dales and hollows and fens, emerging from springs and wells and joining streams that become rivers that wend their way to the sea. You go down the hill to find water.

The Knight Capital Disaster

Speculative Branches — Wed, 22 Nov 2023 02:00:00 GMT

This account comes from several publicly available sources as well as accounts from insiders who worked at Knight Capital Group at the time of the issue. I am telling it second- or third-hand.

On August 1, 2012, Knight Capital fell on its sword. It experienced a software glitch that literally bankrupted the company. Between 9:30 am and 10:15 am EST, the employees of Knight capital watched in disbelief and scrambled to figure out what went wrong as the company acquired massive long and short positions, largely concentrated in 154 stocks, totaling 397 million shares and $7.65 billion. At 10:15, the kill switch was flipped, stopping the company's trading operations for the day. By early afternoon, many of Knight Capital's employees had already sent out resumes, expecting to be unemployed by the end of the week.

Exploring what is inside the ASP.NET Core cookies

Tore Nestenius — Wed, 22 Nov 2023 02:00:00 GMT

ASP.NET Core generates various types of cookies, such as authentication, antiforgery, and session cookies. In this blog post, we’ll take a closer look at

Spoofing Objective-C return values on Apple Silicon using LLDB

Juan Cruz Viotti — Wed, 22 Nov 2023 02:00:00 GMT

This article describes how LLDB can be used to mutate return values of Objective-C methods and functions

Betting on Deep Tech

Coding VC — Tue, 21 Nov 2023 15:36:29 GMT

Is deep tech a good area to build in and invest in? (TLDR: Yes!)

Testing ansible playbooks against multiple targets using vagrant

Srijan Choudhary, all posts — Tue, 21 Nov 2023 08:55:00 GMT

How to test your ansible playbooks against multiple target OSes and versions using Vagrant

What does vehicle congestion tracking mean for privacy?

utf9k — Tue, 21 Nov 2023 07:00:00 GMT

Ironically, I don't actually drive currently so I'll be fine

Introducing Colors

Zac Szewczyk — Tue, 21 Nov 2023 05:35:00 GMT

Over a year ago I read Matthew Strom’s article How to pick the least wrong colors, where he explored the fascinating intricacies of designing good color schemes. Here, “good” meant not just whether the colors looked nice, but also how distinguishable they were from each other and whether or not they accomodated individuals with conditions like protanopia, for example, who cannot see red light. He came up with a great solution and posted the NodeJS code to GitHub. Since I do most of my work in Python, though, I had few chances to take advantage of his research — until now. My latest project, Colors, helps to generate large sets of attractive and accessible color schemes. The current version of this project is, for the most part, a direct NodeJS-to-Python port of Matthew’s code from 2022 with a few minor changes, but I look forward to building on it and keeping pace with Matthew. Colors is currently available as a Jupyter notebook and a regular Python script.

Permalink.

The Sand Mandala

Ben Bolte — Tue, 21 Nov 2023 02:00:00 GMT

A vignette about purpose.

Wild Speculation Regarding the OpenAI Fiasco

Tilting at Windmills — Tue, 21 Nov 2023 02:00:00 GMT

First, I feel I need to make one thing clear: I'm nobody and have no special inside information. I'm just a random terminally-online guy who…

Why Is OOP So Popular?

Backendhance — Tue, 21 Nov 2023 02:00:00 GMT

Hi,

Warum entwickeln wir heute objektorientiert und nicht funktional? Wieso sind 7 von 10 der populärsten Programmiersprachen objektorientiert? Und warum ist keine (rein) funktionale Sprache darunter?

Quelle: Stackoverflow Developer Survey 2023

Die Frage hat sich auch Richard Feldman 2019 gestellt. Und die Hintergründe sind interessant.

Natürlich - wie immer - es ist kompliziert.

Aber einen Aspekt finde ich besonders interessant.

Richard fragt sich, ob es an einzigartigen Features in OOP liegt, die in anderen Paradigmen nicht vorhanden sind.

Visualizing a GroupBy (or a Bipartite Graph)

Playground — Tue, 21 Nov 2023 02:00:00 GMT

Have you ever needed to present the output of a GroupBy or Pivot Table?

"Change Data Capture Breaks Encapsulation". Does it, though?

Gunnar Morling — Tue, 21 Nov 2023 02:00:00 GMT

Table of Contents

CDC—A Quick Primer
Does CDC Break Encapsulation?
Entering Data Contracts
Implementation Approaches For Data Contracts
- The Outbox Pattern
- Stream Processing
Streaming Data Contracts—Beyond the Basics
Handling Schema Changes
Summary

Having worked on Debezium—an open-source platform for Change Data Capture (CDC)—for several years, one concern I’ve heard repeatedly is this: aren’t you breaking the encapsulation of your application when you expose change event feeds directly from your database? After all, CDC exposes your internal persistent data model to the outside world, which may have unintended consequences, e.g. in terms of data exposure but also when it comes to changes to the schema of your data, which may break downstream consumers.

Portable writing setup with Onyx Boox Palma

Liza Shulyayeva — Tue, 21 Nov 2023 00:01:33 GMT

I’ve been using portable writing setups for a while. I started with an AlphaSmart Neo. This was great, but with the screen angle being fixed not as versatile as I had wanted. I wrote a novella on it (said novella would later be extended and become a #1 best-seller in Amazon’s Science Fiction Romance category!)

Consuming RabbitMQ Messages in Go

benbrougher.tech — Mon, 20 Nov 2023 14:40:22 GMT

How to use golang to consume a RabbitMQ queue and correctly ack and reject

French Republican Calendar Tools

Home on Lagomorph — Mon, 20 Nov 2023 09:00:00 GMT

I believe that in all facets of life, anything taken for granted should be examined. For me, this includes how I think about and interface with time. Since beginning research into this topic a few years ago, the French Republican Calendar has continued to fascinate and delight me, so much so to the point that I use decimal time and the republican calendar as my default method of tracking time in my personal life.

2023-11-20-001

Srijan Choudhary, all posts — Mon, 20 Nov 2023 08:45:00 GMT

@rogerlipscombe@hachyderm.io has a nice post on using git with multiple identities. His recommended way (using includeIf to include different config files for different parent folders) also makes sense to me the most.

Open source, locally hosted AI powered Siri replacement

smcleod.net — Mon, 20 Nov 2023 03:00:00 GMT

Using an ESP32 S3 Box 3 with Willow, Home Assistant and Large Language Models (LLMs) create a locally hosted, offline, AI powered Siri / Alexa replacement.

Art of Agile Development in Korean

James Shore — Mon, 20 Nov 2023 02:39:01 GMT

I’m pleased to announce that the Korean translation of The Art of Agile Development is now available! You can buy it here.

Many thanks to 김모세 for their hard work on this translation.

USearch Molecules: 28 Billion Chemical Embeddings on AWS ⚗️

Ash's Blog — Mon, 20 Nov 2023 02:00:00 GMT

Indexing 28 billion chemical embeddings from 7 billion molecules using optimized Jaccard distance kernels, achieving 99% recall at 3,700 QPS on AWS Open Data.

"y'all'n't"

taylor.town — Mon, 20 Nov 2023 02:00:00 GMT

Y'all'n't sophisticated.

Exploring macOS private frameworks

Juan Cruz Viotti — Mon, 20 Nov 2023 02:00:00 GMT

This article describes a series of approaches for reverse engineering macOS private frameworks

What I miss about pandemic life

callmeolive — Sun, 19 Nov 2023 21:10:00 GMT

With the WHO declaring that Covid is no longer a global emergency, we’ve quickly been pushed back to life as it was before the pandemic, with barely a thought given to what things were like for a few years.

Life was in no way a fun jaunty romp, but when I look back I find myself amazed at how we acted back then – and how in some ways we’ve went back.

1) We did away with handshakes for a bit

Does anyone really like handshakes? The awkward dance of meeting eyes, guiding hands without vision with no clue as to how firmly to grasp (and the uncomfortable moment when you squeeze down on the limpest hand known to man). The shaking of hands staining your palm with thick layers of oil and grease and leaving you waiting until it’s socially acceptable to run to the nearest sink to scrub your hands clean… I can’t be doing with it.

With the pandemic came the novel introduction of the Elbow bump, the bumbling alternative that turned you to a drunken duellist, slowly leaning in with a cumbersome thrust. Whilst more time-consuming, it eliminated the problem of oh god I hate touching people’s hands, and the embarrassment of touching elbows was mutual.

For me, the gold standard remains the Fist bump. Quick, efficient, minimal. Keeps respect while acting casual. Perfect.

2) Going the distance (all of 2 metres)

Last week while shopping I noticed something odd by my local’s checkouts – ancient markings 2 metres apart, faded relics from when Covid run rabid guiding us to where to stand. My trance was broken as the bloke behind me coughed on the back of my head. Bit rude. No need to squeeze in so tight mate.

Health aside, the fact that you didn’t have to defend your personal space from some folk was a godsend.

3) The Cleansing of the Ads

For whatever reason, the first year and a half of the pandemic killed all physical advertising in my area, and it was blissful. No longer was I assaulted with a kaleidoscope of jarring colours and bright shapes telling me how my life was shit because I haven’t bought an assortment of products and services – and it’s funny how hunger wasn’t an issue on my commute when massive displays of food suddenly disappeared.

I’d do anything to have that back. Life without advertising is so much calmer on my eyes, and as garish displays returned to plaster every square inch of space I’ve found myself frustrated at how they pry my attention away from the world. We could all do without such visual pollution.

4) Less litter. Littler litter. Words.

There’s nothing as lush as the British outdoors, a grand canvas adorned with every shade of green, adored.

Unfortunately wherever you go there’s a mobs of littering gits who’ll find any excuse to fill hedgerows with whatever they can’t be arsed to put in their pockets. With less of them going out and spoiling the outside world, the state of Britain’s nature pathways were a sight to behold. Who let them know the pandemic was over?

5) All in it together*

Stuck in our fast & fractured worlds, we all run about our daily lives isolated from each other’s struggles. Covid swiftly got in the way of that, and as we were put under lockdown there quickly became one unifying truth about the pandemic: life is shit now, or at least more than usual.

There was no avoiding it, and when the bog roll battle royale simmered down the state of life eventually brought people closer together (safely within 2m of each other). Empathy rode up to new heights as we got past the initial waves of stress, and no longer did people judge one another for the states of their lives or for being behind. You could be living in a tip and it didn’t matter, everyone else might as well have been too.

Unfortunately, not everyone was in it together. But unlike day-to-day life where we’re apathetic towards those in powers flaunting the rules and too burnt out to be outraged, when we found out that prominent politicians were breaking rules (that they themselves had drawn up!), we were rightfully furious. Why should we sacrifice this much if you can’t show the most basic respect for us?

The fury could be heard from the heavens as story after story leaked out to the press, and such cases kicked off the downward spiral of multiple political careers – thinking about those figures sweating like pigs in front of an angry public is still cathartic, but we must be careful. As we’ve returned to normal we’ve slipped back into general apathy at the state of things; and worse, we’ve became complacent at a media industry gifting such disgraced figures free PR via juicy reality TV gigs.

But politicians aside, I’m thankful that the widespread acceptance of mental health issues has managed to stick around.

6) washing hands holy moly

genuine question: how do you forget about washing your hands after doing it for two years straight??? Mental.

Sharing is caring: in defense of ordinary things

This Is My True Name — Sun, 19 Nov 2023 19:53:11 GMT

On foggy mornings and small talk

Design The Data Class Model Instead Of Coding It

Stories by Fred Grott on Medium — Sun, 19 Nov 2023 14:55:42 GMT

Re-align that data class model for the UX driven flutter mobile app design process.

Continue reading on Medium »

Choosing an encrypted DNS protocol

All content on Seirdy’s Home — Sun, 19 Nov 2023 05:33:09 GMT

The three most popular DNS protocols with transit encryption are DNS-over-HTTPS (DoH), DNS-over-TLS (DoT), and DNS-over-QUIC (DoQ). This should help you choose what to use:

Do you actually need to override OS DNS support? If not, or if you’re unsure, go to 6.
Are you ready to implement DNS protocols correctly, or add a dependency that does so? If you’re not, go to 5.
Does the network filter DNS traffic? If it does, go to 5.
Do you already have QUIC support? If not, use DoT. If you do, use DoQ.
Do you have an HTTPS stack? If you do, use DoH.
Give up and delegate to the OS.

Let your HTTPS stack handle HTTP/1.1 vs. HTTP/2 vs. HTTP/3 support; don’t treat DNS-over-HTTP/3 as a separate protocol. I don’t know enough about DNSCrypt to make an informed recommendation about it, but DoQ and DoH meet my needs well enough.

Building a digital vigil for those we've lost

ntietz.com blog - technically a blog — Sun, 19 Nov 2023 02:00:00 GMT

This post is hard to write in a lot of ways. It's more personal than most I've written. This is presumptively a tech blog, and this piece is about so much more than technology. But it's important.

Making things, software or otherwise, is ultimately about people. One of the ways I express love for the people I care about is through making things. Whether that's a hot meal, a picture frame, or a piece of software, it's people who make making matter.

And so I made a Digital Vigil for Transgender Day of Remembrance. The rest of this post explains why and how. I hope you'll join me in this.

TDoR and what it means to me

Tomorrow is Transgender Day of Remembrance (or TDoR). It's a day to memorialize those who have died at the hands of transphobia. Some were murdered, some were lost to suicide.

This day has not been something that I was very conscious of in the past. It was relatively easy to stay detached. This year, that's a lot harder, for two reasons.

The first reason is that someone I know is on the list this time. She was a big influence on me during my internship. We worked on different teams, but she was the epitome of the engineer I wanted to become: she was the badass engineer that you could always turn to, and it felt like she would know the solution to your problem from half a sentence. She was the first engineer I'd worked with who I saw transition, from afar. I saw her continue her successful career, remaining respected by coworkers and thriving during and after her transition. Thank you to the former coworker who let me know about her passing; you mean the world to me, too, and I know her loss hit you hard.

The second reason is that I now hold a fear of someday ending up on this list myself. I'm a trans woman, and the world is not the friendliest to us. My lot in life is still relatively nice. I live in a small accepting town; my parents in another; my employer and coworkers have been grand; and the programming community I'm in is a warm hug. But attacks on all trans people are on the rise, and our access to medical care is at risk with the next general election. If I'm forced to medically detransition, I don't know how my mental health could survive intact. If we're all forced to medically detransition, the list of those we've lost will be much longer.

So this year I learned more about the day. I looked at the list of names that would be read this year, and I cried. And then I started looking to participate. There isn't a candlelight vigil that I can attend in person this year, because I will be traveling for Thanksgiving.

Since I cannot attend a vigil in person, I thought about other ways that I could participate. I toyed with the idea of doing something with software (because of course I did), and wasn't sure what I wanted to do. It crystallized this week after my therapy session, though.

Making my own vigil

It is deeply important to me to participate in TDoR, and making things is my most expressive way of showing love and care. Making something for TDoR would be a way to put my heart into the day, and a way to build the experience I cannot get in person. Plus, it is something that is very visual and user-focused, so a good way to stretch a little outside of my usual areas of focus.

So I started with a simple concept: give people the ability to read the list of names one by one, and experience lighting a candle for each name. Ideally we would add some nice styling, and make the candle look a little animated, but it would depend on time and getting help—I'm not very skilled at CSS.

This is a nice use-case for a frontend application. React was the obvious choice, but configuring React gives me a headache^[1] and it's been a while, so there was a lot of friction to starting. On a whim, I tried Yew instead. Yew is^[2] a framework for writing single-page apps using Rust and WASM. For such a simple application as this one, it was quite easy to get started.

The starter example, a counter that you click to increment, was the foundation of the application. We have a list of names, and each time we press a button it will advance through the list of names. What is this except a counter pointing to the index of which name is next to read? So I took the starter example and connected it up to a list of names, which will then render each one on the page.

After that it was a lot of polishing and tweaking the layout and the look. The primary task that intimidated me was making a candle in HTML/CSS, but we took the easy route and found a permissively-licensed example and adapted it. Trim out the taller candle, then scale the whole thing to the size we want, and we've got it. With some basic styling to make the candle sit with each name in a list, this was the core of the application.

The remaining work was to make it functional as more than a cute demo:

Your state is saved in localStorage, so if you refresh the page you can start reading from where you left off
There are multiple views for the introduction text, the credits, and the vigil itself
You can reset the state if you want to start over

There were a lot of little CSS tweaks, too. The most recent one was making the candles fade in. I'm sure there is more that can be done here, but this is emotional work for me.

Huge thanks to Erika and an anonymous Recurser who helped me with creating the candles, the layout, the application logic... really with all of it. It is much easier to complete an emotionally difficult project when you have friends to help you along the way ❤️.

The code is open-source, and contributions are welcome if you think of ways to improve it. One thing that I think would be nice is to add some more information with each name. This data is available, but I did not have the emotional bandwidth to work through adding it into the application.

Please read their names

Tomorrow is Transgender Day of Remembrance. Please join me in memorializing the day by participating in a vigil. This may be in person, or it may be by using the digital vigil that I built.

This is a very important time to keep all these people in your hearts, to hold them in the light. We are heading into Thanksgiving, and many will be gathering to share meals. Some seats will be empty.

Let's come together to remember those we've lost, and then work to make sure we don't lose more in the future.

And Catherine? You rock. I wish I'd gotten to tell you that.

JavaScript and TypeScript also do not spark joy for me, so if I can avoid them that would be nice. ↩
My brain unhelpfully chimes in "no, you are." ↩

Knuth Airgaps & Knuth Buffers

taylor.town — Sun, 19 Nov 2023 02:00:00 GMT

Knuth knows how to maintain information hygeine.

Social media and social life

Ellie's Notes — Sun, 19 Nov 2023 02:00:00 GMT

Over the past year or so, I've been scaling back how much time and energy I put into social media.

For a long time, I'd keep my Instagram story "up to date", regularly post on TikTok, and spend far too much of my day scrolling various feeds. Lately, I've been scaling that back. I know a number of people who've totally deleted their accounts, but for me that wasn't quite desirable. I didn't want to risk losing touch with people or what's going on. Much as I don't like it, a bunch of social organising has moved to social media.

A number of geographically distant friends I keep in touch with almost exclusively over Instagram
I've found tech community on Twitter + Mastodon
It's useful for meeting new people while travelling
It can be useful for work

Lots of my behaviours towards the consumption of social media were not what I regarded to be healthy. Much as it can be a useful tool, it is a tool that requires control; otherwise, it will use you.

Applying limits

I started off trying to limit things. I figured I could retain some of what I saw as upsides while reducing the downsides if I stuck to say, 15-30 mins a day.

Attempt #1: Screen time limits

I initially setup Apple screen time limits. These did a pretty good job of stopping me from scrolling, for a few days anyway.

After 30 minutes, the screen time limit pops up. Unfortunately, it's easy to bypass - my poor impulse control and big-tech addiction soon got used to dismissing this without even noticing.

Attempt #2: One sec

Next up, I tried One Sec. It essentially delays your access to apps, making you wait for a certain number of seconds before it is possible to open them. This actually worked pretty well! It interrupted the easy-dopamine reach of just opening social media for "a few seconds", and forced me to actually think about what I was doing. Most of the time, it resulted in me remembering that I wanted to spend my life differently and backing out.

After a while, I noticed a different pattern. Sometimes I'd open the app before I anticipated I'd actually want to use it, and other times I'd just use it for longer. My overall usage had gone down a whole bunch though, so I'd call this a partial success.

Just delete it

The main thing holding me back from uninstalling Instagram and Twitter was the messengers. Especially with international friends, it's easier to swap social media than it is to request phone numbers - and often more socially acceptable. Even though I've put a great deal of effort into trying to move people to better alternatives, it's realistically not going to happen any time soon.

If I tried going on a techie rant about how addicting social media is, how it's used to control groups of people, and how "if it's free, you're the product"... most of my not-tech friends would ignore me and think I was losing it. I admire those who stick so strongly to their principals there, but I'm not willing to pay the social cost.

Recently I setup Beeper. It has done an amazing job of totally replacing social media apps for messaging. I can still keep in touch with people where I need to, but I don't end up getting sucked into a time vacuum. Fantastic!

TikTok

The first thing I removed was TikTok. It had an uncanny ability to absolutely entrap my attention, for an embarrassingly long period of time. After an hours doom-scrolling, there was rarely (if ever) anything I could remember. The app is excellent at absorbing your time and contributing little to your life. The worst part was that most of the posts weren't anything at all from my real friends, just internet personalities and memes.

For a while, I actually used to post videos - mostly relating to motorcycles. A few of these ended up with millions of views. TikTok analytics show the total amount of "human time" that has been spent, per video. It showed that videos I had uploaded, putting in very little effort, had been complicit in pointlessly absorbing days of combined human lifespan. Across the platform, it must be measured in decades/centuries.

Some people have said they manage to learn useful things from TikTok, but that wasn't really my experience. I'm also unsure if people convince themselves of this, so that they feel less bad about spending so much time on it.

Instagram

After a while, I uninstalled the Instagram app entirely. I didn't delete my account, as I do still find it useful from time-to-time. I started using Instagram purely through the instagram.com web app, and it hasn't been anywhere near as addictive.

I think the lack of polish on mobile (vs the apps) is just about jarring enough that I don't waste any time at all! I've actually found it interesting how the UI being "smooth" keeps me absorbed, but if it's jerkier/less native the likelihood of me becoming too absorbed decreases dramatically. It's usually the case that after a couple of minutes I just think "wow, there really isn't a lot here huh" and close the page.

I mentioned above, but Beeper has really helped here. I can continue to message people who prefer staying in touch via Instagram, without actually having it installed on my phone.

Twitter / X

I've kept Twitter for now. Much as I'm not a big fan of the platform, I still maintain connections with a bunch of people there, and it has been useful for my work. I'd like to see that change, but for now it is still the case.

The app is still on my phone, but I think soon I'll be banishing it to a Firefox tab just like Instagram.

Facebook

I honestly haven't used this in several years. I have an account, but it's inactive. Nothing to remove!

Mastodon

I've found Mastodon to be very refreshing. The total lack of algorithm means I can very quickly have a quick read of what's been shared since my last visit, and then get back to my day. There's not new content every time the spinner loads.

Generally (and perhaps that's not everyone's experience) I've found the community there to be much more pleasant. Less flamewars, less politics, more genuine humanity.

I'll be keeping this one 😊

Relaxing

I've heard lots of people say that they like checking social media as a way to wind down, or to relax. Obviously I'm unable to speak for everyone, but for me personally there's a huge difference between conscious and intentional winding down, and switching your brain off while the magic box feeds you with whatever the algorithm deems best for engagement. One is restful, while the other burns time.

Conclusion

After a few months, I've found that I have much more time. I've filled it with other things - more time outdoors, reading books much more often, and more time for projects. I'm also trying to learn the guitar.

So far, I've been writing much more too! There's a whole bunch of drafts for my blog, and notes are being published pretty regularly.

I no longer reach for my phone at the first sign of boredom or "empty time", instead using it purely as a tool. I think there's still work to do here, but it's a big improvement.

In an ideal world, tech wouldn't have such a grip on our lives. But there's certainly compromises we can make, without having to go all out and delete everything.

Exploring a Postgres query plan

Notes on software development — Sun, 19 Nov 2023 02:00:00 GMT

I learned this week that you can intercept and redirect Postgres query execution. You can hook into the execution layer so you're given a query plan and you get to decide what to do with it. What rows to return, if any, and where they come from.

That's very interesting. So I started writing code to explore execution hooks. However, I got stuck interpreting the query plan. Either there's no query plan walking infrastructure or I just didn't find it.

So this post is a digression into walking a Postgres query plan. By the end we'll be able to run psql -c 'SELECT a FROM x WHERE a > 1' and reconstruct the entire SQL string from a Postgres QueryDesc object, the query plan object Postgres builds.

With that query plan walking infrastructure in place, we'll be in a good state to not just print out the query plan while walking it but instead to translate the query plan or evaluate it in our own way (e.g. over column-wise data, or vectorized execution over row-wise data).

Code for this project is available on Github.

What is a query plan?

If you're familiar with parsers and compilers, a query plan is like an intermediate representation (IR) of a program. It is not as raw as an abstract syntax tree (AST); it has already been optimized.

If that doesn't mean anything to you, think of a query plan as a structured and optimized version of the SQL query you submit to your database. It isn't text anymore. It is probably a tree.

Check out another Justin Jaffray article on the subject for more detail.

Development environment

Before we get to walking the query plan, let's set up the infrastructure to intercept query execution where we can eventually add in our print debugging of the query plan reconstructed as a SQL string.

Once you've got Postgres build dependencies, build and install a debug version of Postgres:

$ git clone https://github.com/postgres/postgres && cd postgres
$ # Make sure you're on the same commit I'm on, just to be safe.
$ git checkout b218fbb7a35fcf31539bfad12732038fe082a2eb
$ ./configure --enable-cassert --enable-debug CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer"
$ make -j8
$ # Installs to to /usr/local/pgsql/bin.
$ sudo make install

I'm not going to cover Postgres extension infrastructure in detail. I wrote a bit about it in my last post. You need only read the first half, if at all; not the actual Table Access Method implementation.

It will be even simpler in this post because Postgres hooks are extensions but not extensions you install with CREATE EXTENSION. If you want to read about the different kinds of Postgres extensions, check out this article by Steven Miller.

The minimum we need, aside from the hook code itself, is a Makefile that uses PGXS:

MODULES = pgexec

PG_CONFIG = /usr/local/pgsql/bin/pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

The MODULES value there corresponds to the C file we'll create shortly, pgexec.c.

This pg_config binary path is important because you might have different versions of Postgres installed, for example by your package manager. It is important that the extension is built against the same version of Postgres which will load the extension.

Now we're ready for some hook code.

Intercepting query execution

You can find the basic structure of a hook (and which hooks are available) in Tamika Nomara's unofficial Postgres hooks docs.

There is no official central place describing all hooks I could find in Postgres docs. Some hooks are described in various places throughout the docs though.

Based on that page, we can write a bare minimum hook that will intercept queries, log when we've done so, and pass control back to the standard execution path for the actual query. In pgexec.c:

#include "postgres.h"
#include "fmgr.h"
#include "executor/executor.h"

PG_MODULE_MAGIC;

static ExecutorRun_hook_type prev_executor_run_hook = NULL;

static void print_plan(QueryDesc* queryDesc) {
  elog(LOG, "[pgexec] HOOKED SUCCESSFULLY!");
}

static void pgexec_run_hook(
  QueryDesc* queryDesc,
  ScanDirection direction,
  uint64 count,
  bool execute_once
) {
  print_plan(queryDesc);
  return prev_executor_run_hook(queryDesc, direction, count, execute_once);
}

void _PG_init(void) {
  prev_executor_run_hook = ExecutorRun_hook;
  if (prev_executor_run_hook == NULL) {
    prev_executor_run_hook = standard_ExecutorRun;
  }
  ExecutorRun_hook = pgexec_run_hook;
}

void _PG_fini(void) {
  ExecutorRun_hook = prev_executor_run_hook;
}

You can discover the standard_ExectutorRun function from a quick git grep ExecutorRun_hook in the Postgres source which leads to src/backend/executor/execMain.c#L306:

void
ExecutorRun(QueryDesc *queryDesc,
            ScanDirection direction, uint64 count,
            bool execute_once)
{
    if (ExecutorRun_hook)
        (*ExecutorRun_hook) (queryDesc, direction, count, execute_once);
    else
        standard_ExecutorRun(queryDesc, direction, count, execute_once);
}

So our hook will just log and pass back execution to the existing execution hook. Let's build and install the extension.

$ make
$ sudo make install

Now we'll create a new database and tell it to load the extension.

$ /usr/local/pgsql/bin/initdb test-db
$ echo "shared_preload_libraries = 'pgexec'" > test-db/postgresql.conf

Remember, hooks are not CREATE EXTENSION extensions. As far as I can tell they can't be dynamically loaded (without some additional dynamic loading infrastructure one could potentially write). So every time you make a change you need to rebuild the extension, reinstall it, and restart the Postgres server.

And start the server in the foreground:

$ /usr/local/pgsql/bin/postgres \
  --config-file=$(pwd)/test-db/postgresql.conf \
  -D $(pwd)/test-db
  -k $(pwd)/test-db
2023-11-18 19:35:16.680 GMT [3215547] LOG:  starting PostgreSQL 17devel on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1), 64-bit
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on IPv6 address "::1", port 5432
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-11-18 19:35:16.682 GMT [3215550] LOG:  database system was shut down at 2023-11-18 19:20:16 GMT
2023-11-18 19:35:16.684 GMT [3215547] LOG:  database system is ready to accept connections

Keep an eye on this foreground process since this is where elog(LOG, ...) calls will show up.

Now in a new window, create a test.sql script that we can use to exercise the hook:

DROP TABLE IF EXISTS x;
CREATE TABLE x (a INT);
INSERT INTO x VALUES (309);
SELECT a FROM x WHERE a > 1;

Run psql so we can trigger the hook:

$ /usr/local/pgsql/bin/psql -h localhost postgres -f test.sql
DROP TABLE
CREATE TABLE
INSERT 0 1
  a
-----
 309
(1 row)

And in the postgres foreground process you should see a log:

2023-11-19 17:42:03.045 GMT [3242321] LOG:  [pgexec] HOOKED SUCCESSFULLY!
2023-11-19 17:42:03.045 GMT [3242321] STATEMENT:  INSERT INTO x VALUES (309);
2023-11-19 17:42:03.045 GMT [3242321] LOG:  [pgexec] HOOKED SUCCESSFULLY!
2023-11-19 17:42:03.045 GMT [3242321] STATEMENT:  SELECT a FROM x WHERE a > 1;

That's our hook! Interestingly only the INSERT and SELECT statements show up, not the DROP and CREATE.

Now let's see if we can reconstruct the query text from that first argument, the QueryDesc* that pgexec_run_hook receives. And let's simplify things for ourselves and only worry about reconstructing a SELECT query.

`Node`s and `Datum`s

But first, let's talk about two fundemental ways data in Postgres (code) is organized.

Postgres code is extremely dynamic and, maybe relatedly, fairly object-oriented. Almost every entity in Postgres is a Node. While values in Postgres that are exposed to users of Postgres are Datums.

Each node has a type, NodeTag, that we can switch on to decide what to do. In contrast, Datum has no type. The type of the Datum must be known by context before using one of the transform functions like DatumGetBool to retrieve a C value from a Datum.

A table is a Node. A query plan is a Node. A sequential scan is a Node. A join is a Node. A literal in a query is a Node. The value for the literal in a query is a Datum.

Here is how The Internals of PostgreSQL book visualizes a query plan for example:

Every box in that image is a Node.

And all Nodes in code I've seen share a common definition prefix like this:

typedef struct SomeThing {
  pg_node_attr(abstract) // If the node is indeed abstract in the OOP sense.
  NodeTag type;
}

Many Nodes you'll see are abstract, like Plan. But by printing out NodeTag and checking the value printed in src/include/nodes/nodetags.h, you can find the concrete type of the Node.

src/include/nodes/nodetags.h is generated during a preprocessing step. (Don't look if regex in Perl worries you).

We'll get back to Nodes later.

What's in a `QueryDesc`?

Let's take a look at the QueryDesc struct:

typedef struct QueryDesc
{
    /* These fields are provided by CreateQueryDesc */
    CmdType     operation;      /* CMD_SELECT, CMD_UPDATE, etc. */
    PlannedStmt *plannedstmt;   /* planner's output (could be utility, too) */
    const char *sourceText;     /* source text of the query */
    Snapshot    snapshot;       /* snapshot to use for query */
    Snapshot    crosscheck_snapshot;    /* crosscheck for RI update/delete */
    DestReceiver *dest;         /* the destination for tuple output */
    ParamListInfo params;       /* param values being passed in */
    QueryEnvironment *queryEnv; /* query environment passed in */
    int         instrument_options; /* OR of InstrumentOption flags */

    /* These fields are set by ExecutorStart */
    TupleDesc   tupDesc;        /* descriptor for result tuples */
    EState     *estate;         /* executor's query-wide state */
    PlanState  *planstate;      /* tree of per-plan-node state */

    /* This field is set by ExecutorRun */
    bool        already_executed;   /* true if previously executed */

    /* This is always set NULL by the core system, but plugins can change it */
    struct Instrumentation *totaltime;  /* total time spent in ExecutorRun */
} QueryDesc;

The PlannedStmt field looks interesting. Let's take a look:

typedef struct PlannedStmt
{
    pg_node_attr(no_equal, no_query_jumble)

    NodeTag     type;

    CmdType     commandType;    /* select|insert|update|delete|merge|utility */

    uint64      queryId;        /* query identifier (copied from Query) */

    bool        hasReturning;   /* is it insert|update|delete RETURNING? */

    bool        hasModifyingCTE;    /* has insert|update|delete in WITH? */

    bool        canSetTag;      /* do I set the command result tag? */

    bool        transientPlan;  /* redo plan when TransactionXmin changes? */

    bool        dependsOnRole;  /* is plan specific to current role? */

    bool        parallelModeNeeded; /* parallel mode required to execute? */

    int         jitFlags;       /* which forms of JIT should be performed */

    struct Plan *planTree;      /* tree of Plan nodes */

    List       *rtable;         /* list of RangeTblEntry nodes */

    List       *permInfos;      /* list of RTEPermissionInfo nodes for rtable
                                 * entries needing one */

    /* rtable indexes of target relations for INSERT/UPDATE/DELETE/MERGE */
    List       *resultRelations;    /* integer list of RT indexes, or NIL */

    List       *appendRelations;    /* list of AppendRelInfo nodes */

    List       *subplans;       /* Plan trees for SubPlan expressions; note
                                 * that some could be NULL */

    Bitmapset  *rewindPlanIDs;  /* indices of subplans that require REWIND */

    List       *rowMarks;       /* a list of PlanRowMark's */

    List       *relationOids;   /* OIDs of relations the plan depends on */

    List       *invalItems;     /* other dependencies, as PlanInvalItems */

    List       *paramExecTypes; /* type OIDs for PARAM_EXEC Params */

    Node       *utilityStmt;    /* non-null if this is utility stmt */

    /* statement location in source string (copied from Query) */
    int         stmt_location;  /* start location, or -1 if unknown */
    int         stmt_len;       /* length in bytes; 0 means "rest of string" */
} PlannedStmt;

The struct Plan* planTree field in there looks like what we'd want. But Plan is abstract:

typedef struct Plan
{
    pg_node_attr(abstract, no_equal, no_query_jumble)

    NodeTag     type;

So let's try printing out the planTree->type field and find the Node it is concretely. In pgexec.c change the definition of print_plan:

static void print_plan(QueryDesc* queryDesc) {
  elog(LOG, "[pgexec] HOOKED SUCCESSFULLY! %d", queryDesc->plannedstmt->planTree->type);
}

Rebuild and reinstall the extension, and restart Postgres:

$ make
$ sudo make install
$ /usr/local/pgsql/bin/postgres \
  --config-file=$(pwd)/test-db/postgresql.conf \
  -D $(pwd)/test-db
  -k $(pwd)/test-db
2023-11-18 19:35:16.680 GMT [3215547] LOG:  starting PostgreSQL 17devel on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1), 64-bit
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on IPv6 address "::1", port 5432
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-11-18 19:35:16.682 GMT [3215550] LOG:  database system was shut down at 2023-11-18 19:20:16 GMT
2023-11-18 19:35:16.684 GMT [3215547] LOG:  database system is ready to accept connections

And in another window run psql:

$ /usr/local/pgsql/bin/psql -h localhost postgres -f test.sql

And check the logs from the postgres process we just started and you should notice:

2023-11-19 17:46:18.834 GMT [3242495] LOG:  [pgexec] HOOKED SUCCESSFULLY! 322
2023-11-19 17:46:18.834 GMT [3242495] STATEMENT:  SELECT a FROM x WHERE a > 1;

So 322 is the NodeTag for the Plan. If we look that up in Postgres's src/include/nodes/nodetags.h (remember, this is generated after ./configure && make so I can't link you to it):

$ grep ' = 322' src/include/nodes/nodetags.h
        T_SeqScan = 322,

Hey, that makes sense! A SELECT without any indexes definitely sounds like a sequential scan!

Walking a sequential scan

Let's take a look at the SeqScan struct:

typedef struct SeqScan
{
    Scan        scan;
} SeqScan;

Ok, that's not very interesting. Let's look at Scan then:

typedef struct Scan
{
    pg_node_attr(abstract)

    Plan        plan;
    Index       scanrelid;      /* relid is index into the range table */
} Scan;

That's interesting! scanrelid represents the table we're scanning. I don't know what "range table" means exactly. But there was a field on the PlannedStmt called rtable that seems relevant.

rtable was described as a List of RangeTblEntry nodes. And browsing around the file where List is defined we can see some nice methods for working with Lists, like list_length().

Let's print out the scanrelid and let's check out the length of the rtable and see if it's filled out. Let's also restrict our print_plan code to only look at SeqScan nodes. In pgexec.c:

static void print_plan(QueryDesc* queryDesc) {
  SeqScan* scan = NULL;
  Plan* plan = queryDesc->plannedstmt->planTree;
  if (plan->type != T_SeqScan) {
    elog(LOG, "[pgexec] Unsupported plan type.");
    return;
  }

  scan = (SeqScan*)plan;

  elog(LOG, "[pgexec] relid: %d, rtable length: %d", scan->scan.scanrelid, list_length(queryDesc->plannedstmt->rtable));
}

Rebuild and reinstall the extension, and restart Postgres. (You can find the instructions for this above if you've forgotten.) Re-run the test.sql script. And check the Postgres server logs. You should see:

2023-11-19 18:00:34.184 GMT [3244438] LOG:  [pgexec] relid: 1, rtable length: 1
2023-11-19 18:00:34.184 GMT [3244438] STATEMENT:  SELECT a FROM x WHERE a > 1;

Awesome! So rtable does have data in it. There's only one table in this query so its length makes sense to be 1. The scanrelid being 1 also though is weird. Let's fetch the nth value from the rtable list using scanrelid-1 as the index.

For the RangeTblEntry itself, let's take a look:

typedef enum RTEKind
{
    RTE_RELATION,               /* ordinary relation reference */
    RTE_SUBQUERY,               /* subquery in FROM */
    RTE_JOIN,                   /* join */
    RTE_FUNCTION,               /* function in FROM */
    RTE_TABLEFUNC,              /* TableFunc(.., column list) */
    RTE_VALUES,                 /* VALUES (<exprlist>), (<exprlist>), ... */
    RTE_CTE,                    /* common table expr (WITH list element) */
    RTE_NAMEDTUPLESTORE,        /* tuplestore, e.g. for AFTER triggers */
    RTE_RESULT,                 /* RTE represents an empty FROM clause; such
                                 * RTEs are added by the planner, they're not
                                 * present during parsing or rewriting */
} RTEKind;

typedef struct RangeTblEntry
{
    pg_node_attr(custom_read_write, custom_query_jumble)

    NodeTag     type;

    RTEKind     rtekind;        /* see above */

    /*
     * XXX the fields applicable to only some rte kinds should be merged into
     * a union.  I didn't do this yet because the diffs would impact a lot of
     * code that is being actively worked on.  FIXME someday.
     */

    /*
     * Fields valid for a plain relation RTE (else zero):
     *
     * rellockmode is really LOCKMODE, but it's declared int to avoid having
     * to include lock-related headers here.  It must be RowExclusiveLock if
     * the RTE is an INSERT/UPDATE/DELETE/MERGE target, else RowShareLock if
     * the RTE is a SELECT FOR UPDATE/FOR SHARE target, else AccessShareLock.
     *
     * Note: in some cases, rule expansion may result in RTEs that are marked
     * with RowExclusiveLock even though they are not the target of the
     * current query; this happens if a DO ALSO rule simply scans the original
     * target table.  We leave such RTEs with their original lockmode so as to
     * avoid getting an additional, lesser lock.
     *
     * perminfoindex is 1-based index of the RTEPermissionInfo belonging to
     * this RTE in the containing struct's list of same; 0 if permissions need
     * not be checked for this RTE.
     *
     * As a special case, relid, relkind, rellockmode, and perminfoindex can
     * also be set (nonzero) in an RTE_SUBQUERY RTE.  This occurs when we
     * convert an RTE_RELATION RTE naming a view into an RTE_SUBQUERY
     * containing the view's query.  We still need to perform run-time locking
     * and permission checks on the view, even though it's not directly used
     * in the query anymore, and the most expedient way to do that is to
     * retain these fields from the old state of the RTE.
     *
     * As a special case, RTE_NAMEDTUPLESTORE can also set relid to indicate
     * that the tuple format of the tuplestore is the same as the referenced
     * relation.  This allows plans referencing AFTER trigger transition
     * tables to be invalidated if the underlying table is altered.
     */
    Oid         relid;          /* OID of the relation */
    char        relkind;        /* relation kind (see pg_class.relkind) */
    int         rellockmode;    /* lock level that query requires on the rel */
    struct TableSampleClause *tablesample;  /* sampling info, or NULL */
    Index       perminfoindex;

In SELECT a FROM x, x should be a plain relation RTE (to use the terminology there). So we can add a guard that validates that. But we don't get a Relation. (You might remember from my previous post that Relation is where we can finally see the table name.)

We get an Oid for the Relation. So we need to find a way to lookup a Relation from an Oid. And by grepping around in Postgres (or via judicious use of ChatGPT, I confess), we can notice RelationIdGetRelation that takes an Oid and returns a Relation. Notice also that the comment says we should close the relation when we're done with RelationClose.

So putting it altogether (and again, reusing some code from that previous post), we can print out the table name.

static void print_plan(QueryDesc* queryDesc) {
  SeqScan* scan = NULL;
  RangeTblEntry* rte = NULL;
  Relation relation = {};
  char* tablename = NULL;
  Plan* plan = queryDesc->plannedstmt->planTree;
  if (plan->type != T_SeqScan) {
    elog(LOG, "[pgexec] Unsupported plan type.");
    return;
  }

  scan = (SeqScan*)plan;
  rte = list_nth(queryDesc->plannedstmt->rtable, scan->scan.scanrelid-1);
  if (rte->rtekind != RTE_RELATION) {
    elog(LOG, "[pgexec] Unsupported FROM type: %d.", rte->rtekind);
    return;
  }

  relation = RelationIdGetRelation(rte->relid);
  tablename = NameStr(relation->rd_rel->relname);

  elog(LOG, "[pgexec] SELECT [todo] FROM %s", tablename);

  RelationClose(relation);
}

You'll also need to add a new #include for utils/rel.h.

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 18:36:03.986 GMT [3246777] LOG:  [pgexec] SELECT [todo] FROM x
2023-11-19 18:36:03.986 GMT [3246777] STATEMENT:  SELECT a FROM x WHERE a > 1;

Fantastic! Before we get into walking the SELECT columns and the (optional) WHERE clause, let's do some quick refactoring.

A string builder

Let's add a little string builder library so we can emit a single string we build up to a single elog() call.

I wrote this ahead of time and won't explain it here since the details aren't relevant.

Just copy this and paste near the top of pgexec.c:

typedef struct {
  char* mem;
  size_t len;
  size_t offset;
} PGExec_Buffer;

static void buffer_init(PGExec_Buffer* buf) {
  buf->offset = 0;
  buf->len = 8;
  buf->mem = (char*)malloc(sizeof(char) * buf->len);
}

static void buffer_resize_to_fit_additional(PGExec_Buffer* buf, size_t additional) {
  char* new = {};
  size_t newsize = 0;

  Assert(additional >= 0);

  if (buf->offset + additional < buf->len) {
    return;
  }

  newsize = (buf->offset + additional) * 2;
  new = (char*)malloc(sizeof(char) * newsize);
  Assert(new != NULL);
  memcpy(new, buf->mem, buf->len * sizeof(char));
  free(buf->mem);
  buf->len = newsize;
  buf->mem = new;
}

static void buffer_append(PGExec_Buffer*, char*, size_t);

static void buffer_appendz(PGExec_Buffer* buf, char* c) {
  buffer_append(buf, c, strlen(c));
}

static void buffer_append(PGExec_Buffer* buf, char* c, size_t chars) {
  buffer_resize_to_fit_additional(buf, chars);
  memcpy(buf->mem + buf->offset, c, chars);
  buf->offset += chars;
}

static void buffer_appendf(
  PGExec_Buffer *,
  const char* restrict,
  ...
) __attribute__ ((format (gnu_printf, 2, 3)));

static void buffer_appendf(PGExec_Buffer *buf, const char* restrict fmt, ...) {
  // First figure out how long the result will be.
  size_t chars = 0;
  va_list arglist;
  va_start(arglist, fmt);
  chars = vsnprintf(0, 0, fmt, arglist);
  Assert(chars >= 0); // TODO: error handling.

  // Resize to fit result.
  buffer_resize_to_fit_additional(buf, chars);

  // Actually do the printf into buf.
  va_end(arglist);
  va_start(arglist, fmt);
  chars = vsprintf(buf->mem + buf->offset, fmt, arglist);
  Assert(chars >= 0); // TODO: error handling.
  buf->offset += chars;
  va_end(arglist);
}

static char* buffer_cstring(PGExec_Buffer* buf) {
  char zero = 0;
  const size_t prev_offset = buf->offset;

  if (buf->offset == buf->len) {
    buffer_append(buf, &zero, 1);
    buf->offset--;
  } else {
    buf->mem[buf->offset] = 0;
  }

  // Offset should stay the same. This is a fake NULL.
  Assert(buf->offset == prev_offset);

  return buf->mem;
}

static void buffer_free(PGExec_Buffer* buf) {
  free(buf->mem);
}

Next we'll modify print_plan() in pgexec.c to use it, and add stubs for printing the SELECT columns and WHERE clauses.

static void buffer_print_where(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  buffer_appendz(buf, " [where todo]");
}

static void buffer_print_select_columns(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  buffer_appendz(buf, "[columns todo]");
}

static void print_plan(QueryDesc* queryDesc) {
  SeqScan* scan = NULL;
  RangeTblEntry* rte = NULL;
  Relation relation = {};
  char* tablename = NULL;
  Plan* plan = queryDesc->plannedstmt->planTree;
  PGExec_Buffer buf = {};

  if (plan->type != T_SeqScan) {
    elog(LOG, "[pgexec] Unsupported plan type.");
    return;
  }

  scan = (SeqScan*)plan;
  rte = list_nth(queryDesc->plannedstmt->rtable, scan->scan.scanrelid-1);
  if (rte->rtekind != RTE_RELATION) {
    elog(LOG, "[pgexec] Unsupported FROM type: %d.", rte->rtekind);
    return;
  }

  buffer_init(&buf);

  relation = RelationIdGetRelation(rte->relid);
  tablename = NameStr(relation->rd_rel->relname);

  buffer_appendz(&buf, "SELECT ");
  buffer_print_select_columns(&buf, queryDesc, plan);
  buffer_appendf(&buf, " FROM %s", tablename);
  buffer_print_where(&buf, queryDesc, plan);

  elog(LOG, "[pgexec] %s", buffer_cstring(&buf));

  RelationClose(relation);
  buffer_free(&buf);
}

Now we just need to implement the buffer_print_where and buffer_print_select_columns functions and our walking infrastructure will be done! For now. :)

Walking the `WHERE` clause

If you remember back to the SeqScan and Scan nodes, they were both basically empty. They had a Plan and a scanrelid. So the rest of the SELECT info must be in the Plan since it wasn't in the Scan.

Let's look at Plan again. One part that stands out is:

    /*
     * Common structural data for all Plan types.
     */
    int         plan_node_id;   /* unique across entire final plan tree */
    List       *targetlist;     /* target list to be computed at this node */
    List       *qual;           /* implicitly-ANDed qual conditions */
    struct Plan *lefttree;      /* input plan tree(s) */
    struct Plan *righttree;
    List       *initPlan;       /* Init Plan nodes (un-correlated expr
                                 * subselects) */

qual kinda looks like a WHERE clause. (And targetlist kinda looks like the columns the SELECT pulls).

Lists just contain void pointers, so we can't tell what the type of qual or targetlist children are. But I'm going to make a wild guess they are Nodes.

There's even a nice helper that casts void pointers to Node* and pulls out the type, nodeTag().

And reading around pg_list.h shows some interesting helper utilities like foreach that we can use to iterate the list.

Let's try printing out the type of qual's members.

static void buffer_print_where(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  ListCell* cell = NULL;
  bool first = true;
  if (plan->qual == NULL) {
    return;
  }

  buffer_appendz(buf, " WHERE ");
  foreach(cell, plan->qual) {
    if (!first) {
      buffer_appendz(buf, " AND ");
    }

    first = false;
    buffer_appendf(buf, "[node: %d]", nodeTag(lfirst(cell)));
  }
}

Notice any vestiges of LISP?

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 19:17:00.879 GMT [3250850] LOG:  [pgexec] SELECT [columns todo] FROM x WHERE [node: 15]
2023-11-19 19:17:00.879 GMT [3250850] STATEMENT:  SELECT a FROM x WHERE a > 1;

Well, our code didn't crash! So the guess about qual List entries being Nodes seems right. Let's look up that node type in the Postgres repo:

$ grep ' = 15,' src/include/nodes/nodetags.h
        T_OpExpr = 15,

Woot! That is exactly what I'd expect the WHERE clause here to be.

Now that we know qual is a List of Nodes, let's do a bit of refactoring since targetlist will probably also be a List of Nodes. Back in pgexec.c:

static void buffer_print_expr(PGExec_Buffer*, Node*);
static void buffer_print_list(PGExec_Buffer*, List*, char*);

static void buffer_print_opexpr(PGExec_Buffer* buf, OpExpr* op) {
  buffer_appendf(buf, "[opexpr: todo]");
}

static void buffer_print_expr(PGExec_Buffer* buf, Node* expr) {
  switch (nodeTag(expr)) {
  case T_OpExpr:
    buffer_print_opexpr(buf, (OpExpr*)expr);
    break;

  default:
    buffer_appendf(buf, "[Unknown: %d]", nodeTag(expr));
  }
}

static void buffer_print_list(PGExec_Buffer* buf, List* list, char* sep) {
  ListCell* cell = NULL;
  bool first = true;

  foreach(cell, list) {
    if (!first) {
      buffer_appendz(buf, sep);
    }

    first = false;
    buffer_print_expr(buf, (Node*)lfirst(cell));
  }
}

static void buffer_print_where(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  if (plan->qual == NULL) {
    return;
  }

  buffer_appendz(buf, " WHERE ");
  buffer_print_list(buf, plan->qual, " AND ");
}

And let's check out OpExpr!

Walking `OpExpr`

Take a look at the definition of OpExpr:

typedef struct OpExpr
{
    Expr        xpr;

    /* PG_OPERATOR OID of the operator */
    Oid         opno;

    /* PG_PROC OID of underlying function */
    Oid         opfuncid pg_node_attr(equal_ignore_if_zero, query_jumble_ignore);

    /* PG_TYPE OID of result value */
    Oid         opresulttype pg_node_attr(query_jumble_ignore);

    /* true if operator returns set */
    bool        opretset pg_node_attr(query_jumble_ignore);

    /* OID of collation of result */
    Oid         opcollid pg_node_attr(query_jumble_ignore);

    /* OID of collation that operator should use */
    Oid         inputcollid pg_node_attr(query_jumble_ignore);

    /* arguments to the operator (1 or 2) */
    List       *args;

    /* token location, or -1 if unknown */
    int         location;
} OpExpr;

The important fields are opno, the Oid of the operator, and args. args looks like another List of Nodes so we already know how to handle that.

But how do we find the string name of the operator? Presumably there's infrastructure like RelationIdGetRelation that takes an Oid and gets us an operator object.

Well I got stuck here as well. Again, thankfully, ChatGPT gave me some suggestions. There's no great story for how I got it working. So here's buffer_print_opexpr.

static void buffer_print_op(PGExec_Buffer* buf, OpExpr* op) {
  HeapTuple opertup = SearchSysCache1(OPEROID, ObjectIdGetDatum(op->opno));

  buffer_print_expr(buf, lfirst(list_nth_cell(op->args, 0)));

  if (HeapTupleIsValid(opertup)) {
    Form_pg_operator operator = (Form_pg_operator)GETSTRUCT(opertup);
    buffer_appendf(buf, " %s ", NameStr(operator->oprname));
    ReleaseSysCache(opertup);
  } else {
    buffer_appendf(buf, "[Unknown operation: %d]", op->opno);
  }

  // TODO: Support single operand operations.
  buffer_print_expr(buf, lfirst(list_nth_cell(op->args, 1)));
}

And add the following two includes to the top of pgexec.c:

#include "catalog/pg_operator.h"
#include "utils/syscache.h"

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 19:42:52.916 GMT [3252974] LOG:  [pgexec] SELECT [columns todo] FROM x WHERE [Unknown: 6] > [Unknown: 7]
2023-11-19 19:42:52.916 GMT [3252974] STATEMENT:  SELECT a FROM x WHERE a > 1;

And we continue to make progress! Let's look up the type of these two unknown nodes.

$ grep ' = 6,' src/include/nodes/nodetags.h
        T_Var = 6,
$ grep ' = 7,' src/include/nodes/nodetags.h
        T_Const = 7,

Let's deal with Const first.

Walking `Const`

If we take a look at the Const definition:

typedef struct Const
{
    pg_node_attr(custom_copy_equal, custom_read_write)

    Expr        xpr;
    /* pg_type OID of the constant's datatype */
    Oid         consttype;
    /* typmod value, if any */
    int32       consttypmod pg_node_attr(query_jumble_ignore);
    /* OID of collation, or InvalidOid if none */
    Oid         constcollid pg_node_attr(query_jumble_ignore);
    /* typlen of the constant's datatype */
    int         constlen pg_node_attr(query_jumble_ignore);
    /* the constant's value */
    Datum       constvalue pg_node_attr(query_jumble_ignore);
    /* whether the constant is null (if true, constvalue is undefined) */
    bool        constisnull pg_node_attr(query_jumble_ignore);

    /*
     * Whether this datatype is passed by value.  If true, then all the
     * information is stored in the Datum.  If false, then the Datum contains
     * a pointer to the information.
     */
    bool        constbyval pg_node_attr(query_jumble_ignore);

    /*
     * token location, or -1 if unknown.  All constants are tracked as
     * locations in query jumbling, to be marked as parameters.
     */
    int         location pg_node_attr(query_jumble_location);
} Const;

It looks like we need to switch on the consttype (an Oid) to figure out how to interpret the constvalue (a Datum). Remember I mentioned earlier that how to interpret a Datum is dependent on context. consttype is the context here.

In this case, although consttype is an Oid and we had to use Postgres infrastructure to look up the Oid's corresponding object, there are some builtin types and the literals we've queried with are among them.

We can simply check if consttype == INT4OID and the interpret the Datum as an int32 if so. DatumGetInt32 will get us that int32 in that case.

To support the new Const type, we'll add a case in buffer_print_expr to look for a T_Const.

static void buffer_print_expr(PGExec_Buffer* buf, Node* expr) {
  switch (nodeTag(expr)) {
  case T_Const:
    buffer_print_const(buf, (Const*)expr);
    break;

  case T_OpExpr:
    buffer_print_opexpr(buf, (OpExpr*)expr);
    break;

  default:
    buffer_appendf(buf, "[Unknown: %d]", nodeTag(expr));
  }
}

And add a new function, buffer_print_const:

static void buffer_print_const(PGExec_Buffer* buf, Const* cnst) {
  switch (cnst->consttype) {
  case INT4OID:
    int32 val = DatumGetInt32(cnst->constvalue);
    buffer_appendf(buf, "%d", val);
    break;

  default:
    buffer_appendf(buf, "[Unknown consttype oid: %d]", cnst->consttype);
  }
}

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 19:53:47.922 GMT [3253746] LOG:  [pgexec] SELECT [columns todo] FROM x WHERE [Unknown: 6] > 1
2023-11-19 19:53:47.922 GMT [3253746] STATEMENT:  SELECT a FROM x WHERE a > 1;

Great! Now we just have to tackle T_Var.

Walking `Var`

Let's take a look at the definition of Var:

typedef struct Var
{
    Expr        xpr;

    /*
     * index of this var's relation in the range table, or
     * INNER_VAR/OUTER_VAR/etc
     */
    int         varno;

    /*
     * attribute number of this var, or zero for all attrs ("whole-row Var")
     */
    AttrNumber  varattno;

    /* pg_type OID for the type of this var */
    Oid         vartype pg_node_attr(query_jumble_ignore);
    /* pg_attribute typmod value */
    int32       vartypmod pg_node_attr(query_jumble_ignore);
    /* OID of collation, or InvalidOid if none */
    Oid         varcollid pg_node_attr(query_jumble_ignore);

    /*
     * RT indexes of outer joins that can replace the Var's value with null.
     * We can omit varnullingrels in the query jumble, because it's fully
     * determined by varno/varlevelsup plus the Var's query location.
     */
    Bitmapset  *varnullingrels pg_node_attr(query_jumble_ignore);

    /*
     * for subquery variables referencing outer relations; 0 in a normal var,
     * >0 means N levels up
     */
    Index       varlevelsup;

    /*
     * varnosyn/varattnosyn are ignored for equality, because Vars with
     * different syntactic identifiers are semantically the same as long as
     * their varno/varattno match.
     */
    /* syntactic relation index (0 if unknown) */
    Index       varnosyn pg_node_attr(equal_ignore, query_jumble_ignore);
    /* syntactic attribute number */
    AttrNumber  varattnosyn pg_node_attr(equal_ignore, query_jumble_ignore);

    /* token location, or -1 if unknown */
    int         location;
} Var;

It looks like this refers to a relation in the range table list again. So this means we need to have access to the full PlannedStmt so we can read its rtable field again to find the table. Then we need to look up the Relation for the table and then we can use the Var's varattno field to pick the nth attribute from the relation and get its string representation.

However, ChatGPT found a slightly higher-level function: get_attname() that takes a relation Oid and an attribute index and returns the string name of the column.

So here's what buffer_print_var looks like:

static void buffer_print_var(PGExec_Buffer* buf, PlannedStmt* stmt, Var* var) {
  char* name = NULL;
  RangeTblEntry* rte = list_nth(stmt->rtable, var->varno-1);
  if (rte->rtekind != RTE_RELATION) {
    elog(LOG, "[Unsupported relation type for var: %d].", rte->rtekind);
    return;
  }

  name = get_attname(rte->relid, var->varattno, false);
  buffer_appendz(buf, name);
  pfree(name);
}

You'll also need to add another #include for utils/lsyscache.h.

Let's add the case T_Var: check in buffer_print_expr, and also feed the PlannedStmt* through all the necessary buffer_print_X functions:

static void buffer_print_expr(PGExec_Buffer*, PlannedStmt*, Node*);
static void buffer_print_list(PGExec_Buffer*, PlannedStmt*, List*, char*);

static void buffer_print_opexpr(PGExec_Buffer* buf, PlannedStmt* stmt, OpExpr* op) {
  HeapTuple opertup = SearchSysCache1(OPEROID, ObjectIdGetDatum(op->opno));

  buffer_print_expr(buf, stmt, lfirst(list_nth_cell(op->args, 0)));

  if (HeapTupleIsValid(opertup)) {
    Form_pg_operator operator = (Form_pg_operator)GETSTRUCT(opertup);
    buffer_appendf(buf, " %s ", NameStr(operator->oprname));
    ReleaseSysCache(opertup);
  } else {
    buffer_appendf(buf, "[Unknown operation: %d]", op->opno);
  }

  // TODO: Support single operand operations.
  buffer_print_expr(buf, stmt, lfirst(list_nth_cell(op->args, 1)));
}

static void buffer_print_const(PGExec_Buffer* buf, Const* cnst) {
  switch (cnst->consttype) {
  case INT4OID:
    int32 val = DatumGetInt32(cnst->constvalue);
    buffer_appendf(buf, "%d", val);
    break;

  default:
    buffer_appendf(buf, "[Unknown consttype oid: %d]", cnst->consttype);
  }
}

static void buffer_print_var(PGExec_Buffer* buf, PlannedStmt* stmt, Var* var) {
  char* name = NULL;
  RangeTblEntry* rte = list_nth(stmt->rtable, var->varno-1);
  if (rte->rtekind != RTE_RELATION) {
    elog(LOG, "[Unsupported relation type for var: %d].", rte->rtekind);
    return;
  }

  name = get_attname(rte->relid, var->varattno, false);
  buffer_appendz(buf, name);
  pfree(name);
}

static void buffer_print_expr(PGExec_Buffer* buf, PlannedStmt* stmt, Node* expr) {
  switch (nodeTag(expr)) {
  case T_Const:
    buffer_print_const(buf, (Const*)expr);
    break;

  case T_Var:
    buffer_print_var(buf, stmt, (Var*)expr);
    break;

  case T_OpExpr:
    buffer_print_opexpr(buf, stmt, (OpExpr*)expr);
    break;

  default:
    buffer_appendf(buf, "[Unknown: %d]", nodeTag(expr));
  }
}

static void buffer_print_list(PGExec_Buffer* buf, PlannedStmt* stmt, List* list, char* sep) {
  ListCell* cell = NULL;
  bool first = true;

  foreach(cell, list) {
    if (!first) {
      buffer_appendz(buf, sep);
    }

    first = false;
    buffer_print_expr(buf, stmt, (Node*)lfirst(cell));
  }
}

static void buffer_print_where(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  if (plan->qual == NULL) {
    return;
  }

  buffer_appendz(buf, " WHERE ");
  buffer_print_list(buf, queryDesc->plannedstmt, plan->qual, " AND ");
}

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 20:03:14.351 GMT [3254458] LOG:  [pgexec] SELECT [columns todo] FROM x WHERE a > 1
2023-11-19 20:03:14.351 GMT [3254458] STATEMENT:  SELECT a FROM x WHERE a > 1;

Huzzah!

Walking the column list

Let's get rid of [columns todo]. We already had the idea that List* targetlist on the Plan struct was a list of expression Nodes. Let's try it.

static void buffer_print_select_columns(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  if (plan->targetlist == NULL) {
    return;
  }

  buffer_print_list(buf, queryDesc->plannedstmt, plan->targetlist, ", ");
}

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 20:12:48.091 GMT [3255398] LOG:  [pgexec] SELECT [Unknown: 53] FROM x WHERE a > 1
2023-11-19 20:12:48.091 GMT [3255398] STATEMENT:  SELECT a FROM x WHERE a > 1;

Hmm. Let's look up Node 53 in Postgres:

$ grep ' = 53,' src/include/nodes/nodetags.h
        T_TargetEntry = 53,

Based on the definition of TargetEntry, it looks like we can ignore most of the fields (because we don't need to handle SELECT a AS b yet) and just proxy the child expr field.

Let's add a case T_TargetEntry to buffer_print_expr:

static void buffer_print_expr(PGExec_Buffer* buf, PlannedStmt* stmt, Node* expr) {
  switch (nodeTag(expr)) {
  case T_Const:
    buffer_print_const(buf, (Const*)expr);
    break;

  case T_Var:
    buffer_print_var(buf, stmt, (Var*)expr);
    break;

  case T_TargetEntry:
    buffer_print_expr(buf, stmt, (Node*)((TargetEntry*)expr)->expr);
    break;

  case T_OpExpr:
    buffer_print_opexpr(buf, stmt, (OpExpr*)expr);
    break;

  default:
    buffer_appendf(buf, "[Unknown: %d]", nodeTag(expr));
  }
}

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and:

2023-11-19 20:17:51.114 GMT [3257827] LOG:  [pgexec] SELECT a FROM x WHERE a > 1
2023-11-19 20:17:51.114 GMT [3257827] STATEMENT:  SELECT a FROM x WHERE a > 1;

We did it!

Variations

Let's try out some other queries to make sure this wasn't just luck.

$ /usr/local/pgsql/bin/psql -h localhost postgres -c 'SELECT a + 1 FROM x'
 ?column?
----------
      310
(1 row)

$ /usr/local/pgsql/bin/psql -h localhost postgres -c 'SELECT a + 1 FROM x WHERE 2 > a'
 ?column?
----------
(0 rows)

And back in the Postgres server logs:

2023-11-19 20:19:28.057 GMT [3257874] LOG:  [pgexec] SELECT a + 1 FROM x
2023-11-19 20:19:28.057 GMT [3257874] STATEMENT:  SELECT a + 1 FROM x
2023-11-19 20:19:30.474 GMT [3257878] LOG:  [pgexec] SELECT a + 1 FROM x WHERE 2 > a
2023-11-19 20:19:30.474 GMT [3257878] STATEMENT:  SELECT a + 1 FROM x WHERE 2 > a

Not bad!

Next steps

Printing out the statement here isn't incredibly useful. But it establishes a basis for future work that might avoid Postgres's query execution engine and do the execution ourselves, or to proxy execution to another system.

Postscript: On ChatGPT

My recent Postgres explorations would have been basically impossible if it weren't for being able to ask ChatGPT simple, stupid questions like "How do I get from a Postgres Var to a column name".

It isn't always right. It doesn't always give great code. Actually, it normally gives pretty weird code. But it's been extremely useful for quick iteration when I get stuck.

The only other place the information exists is in small blog posts around the internet, the Postgres mailing lists (that so far for me hasn't been super responsive), and the code itself.

I've been on a Postgres roll. Let's dig into interpreting a Postgres query plan in preparation for future work on completely diverting the flow of Postgres query execution using execution hooks!https://t.co/EZrgoIiTuX pic.twitter.com/7S6d6olPX8
— Phil Eaton (@eatonphil) November 19, 2023

An attempt at creating a DIY baby monitor

Gonçalo Valério — Sun, 19 Nov 2023 00:12:39 GMT

Occasionally, “modern” baby monitors hit the news, but not for the best reasons. Sometimes for the lack of security, other times for exploitative practices, etc. This also applies to many other categories of products that we can include in the so-called “Internet of Things”. After reading a recent article about the topic, some comments with […]

Why was Sam Altman fired from OpenAI?

🦉 olshansky 🦁 — Sat, 18 Nov 2023 22:32:16 GMT

tl;dr AGI (Artificial General Intelligence) may be closer than we think.

[TIL] nix develop -c $SHELL: Run zsh with 'nix develop'

Davis Haupt's Blog — Sat, 18 Nov 2023 19:00:00 GMT

I built my development flake for this website with the FlakeHub CLI. That flake uses pkgs.mkShell to load in binary dependencies to the development environment. I run zsh defined and configured by nix-darwin, so it was pretty jarring getting thrown into a plain bash shell when running nix develop. Luckily the fix is simple. Rather than just running nix develop, run:

nix develop -c $SHELL

and your normal shell will be run instead of the default bash shell.

(via this discussion)

Every software is messy and has skeletons

Vadim Kravcenko — Sat, 18 Nov 2023 14:37:12 GMT

Question:

Answer:

The post Every software is messy and has skeletons appeared first on Vadim Kravcenko.

A performant embedding processing pipeline & tutorial for big XML/HTML data dumps

Geography & Coding — Sat, 18 Nov 2023 14:17:29 GMT

Let’s process 1.369.841 single English XML/HTML files and index them with BAAI/bge-base-en-v1.5 embeddings for semantic search! Sounds fun? Let’s go!

Checking semver in the presence of doc(hidden) items

Predrag Gruevski's blog and personal site. — Sat, 18 Nov 2023 02:00:00 GMT

cargo-semver-checks v0.25 squashes nearly all bugs related to doc(hidden) items — its most common source of false-positives. What does doc(hidden) mean in Rust, and why was handling it correctly so hard?

Stack Overflow tags

Vít Baisa — Sat, 18 Nov 2023 02:00:00 GMT

tags from StackOveflow with frequency > 10

Exporting Prometheus metrics from HAProxy

Ellie's Notes — Sat, 18 Nov 2023 02:00:00 GMT

I've recently started using haproxy after a lifetime of nginx, and I actually really like it! I needed to setup some metrics, when I discovered that haproxy supports prometheus out of the box - no exporter needed! 🥳

The documentation shows binding on whatever frontend you're already using, but I wanted to make sure my metrics and stats were not visible to anyone outside of my network

listen stats
	bind *:9000
	mode http
	stats enable
	stats uri /
	http-request use-service prometheus-exporter if { path /metrics }

Pretty simple - we're adding a combined frontend/backend, bound on http 9000. The root shows the HAProxy metrics page, and if we hit /metrics then we show some prometheus metrics. So easy!

Goals for 2024

Chris' Full Stack Blog RSS Feed — Sat, 18 Nov 2023 02:00:00 GMT

There's a new blog post "Goals for 2024" and you can read it online.
undefined
You can also subscribe for weekly emails on my latest posts - if I publish anything that week!

[TIL] nix-darwin and home-manager: Setting up Nix on macOS

Davis Haupt's Blog — Fri, 17 Nov 2023 19:00:00 GMT

I recently got a new computer and am spending some time actually digging into Nix for managing dependencies. My main issue the last time I tried using Nix was how tough it was for me to wade through the documentation, so I was determined to try a different path this time. I started off with the very opinionated Zero to Nix tutorial series, which I thought was a great introduction.

After learning about Nix and Flakes, and getting my website building locally with a nix-powered development environment courtesy of the guidance in Part 6, the next step was to see what else I could do. I’d heard of two different tools, home-manager and nix-darwin. I started off setting up home-manager, but I was confused on how I should be installing graphical applications like VS Code, and system utilities like Karabiner Elements. It turned out nix-darwin was the answer there.

To paraphrase a Reddit thread I found, nix-darwin manages system-level programs and configuration, while home-manager manages user configuration. You can actually get nix-darwin to call into home-manager as part of its own configuration to use both at the same time. This is described in the home-manager reference docs, but like many Nix resources, it’s not very simple for a beginner to figure out how to actually put this snippet to use in their own configuration.

Before following these instructions, you should follow the Zero to Nix guide or install Nix on your system in some other way.

I’m going to lay out the basic steps that I followed below while sharing each configuration file.

Create a nix-darwin flake file. From the README:

mkdir -p ~/.config/nix
cd ~/.config/nix
nix flake init -t nix-darwin
sed -i '' "s/simple/$(scutil --get LocalHostName)/" flake.nix

The default flake puts the configuration inline to the flake.nix. I decided to copy/paste that configuration out into a separate file called darwin.nix:

# darwin.nix

{ pkgs, ... }:

{
    # List packages installed in system profile. To search by name, run:
    # $ nix-env -qaP | grep wget
    environment.systemPackages =
    [ pkgs.vim ];

    # Auto upgrade nix package and the daemon service.
    services.nix-daemon.enable = true;
    services.karabiner-elements.enable = true;
    # nix.package = pkgs.nix;

    # Necessary for using flakes on this system.
    nix.settings.experimental-features = "nix-command flakes";

    # Create /etc/zshrc that loads the nix-darwin environment.
    programs.zsh.enable = true;  # default shell on catalina
    # programs.fish.enable = true;

    # Used for backwards compatibility, please read the changelog before changing.
    # $ darwin-rebuild changelog
    system.stateVersion = 4;

    # The platform the configuration will be used on.
    nixpkgs.hostPlatform = "aarch64-darwin";

    users.users.davish = {
        name = "davish";
        home = "/Users/davish";
    };
}

You should remember to change your username and home directory to match.

Modify flake.nix to work with home-manager. The example code can be found in the home-manager docs. In case that link breaks, here it is:

# flake.nix

{
  description = "Darwin configuration";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    darwin.url = "github:lnl7/nix-darwin";
    darwin.inputs.nixpkgs.follows = "nixpkgs";
    home-manager.url = "github:nix-community/home-manager";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = inputs@{ nixpkgs, home-manager, darwin, ... }: {
    darwinConfigurations = {
      hostname = darwin.lib.darwinSystem {
        system = "aarch64-darwin";
        modules = [
          ./configuration.nix
          home-manager.darwinModules.home-manager
          {
            home-manager.useGlobalPkgs = true;
            home-manager.useUserPackages = true;
            home-manager.users.jdoe = import ./home.nix;

            # Optionally, use home-manager.extraSpecialArgs to pass
            # arguments to home.nix
          }
        ];
      };
    };
  };
}

Create a home.nix file. Here is the one that was generated when I tried to set up home-manager on its own. I’ll replicate it here because the comments were useful as a tutorial on its own:

# home.nix

{ config, pkgs, ... }:

{
  # Home Manager needs a bit of information about you and the paths it should
  # manage.

  # This value determines the Home Manager release that your configuration is
  # compatible with. This helps avoid breakage when a new Home Manager release
  # introduces backwards incompatible changes.
  #
  # You should not change this value, even if you update Home Manager. If you do
  # want to update the value, then make sure to first check the Home Manager
  # release notes.
  home.stateVersion = "23.05"; # Please read the comment before changing.

  # The home.packages option allows you to install Nix packages into your
  # environment.
  home.packages = [
    # # Adds the 'hello' command to your environment. It prints a friendly
    # # "Hello, world!" when run.
    # pkgs.hello

    # # It is sometimes useful to fine-tune packages, for example, by applying
    # # overrides. You can do that directly here, just don't forget the
    # # parentheses. Maybe you want to install Nerd Fonts with a limited number of
    # # fonts?
    # (pkgs.nerdfonts.override { fonts = [ "FantasqueSansMono" ]; })

    # # You can also create simple shell scripts directly inside your
    # # configuration. For example, this adds a command 'my-hello' to your
    # # environment:
    # (pkgs.writeShellScriptBin "my-hello" ''
    #   echo "Hello, ${config.home.username}!"
    # '')
  ];

  # Home Manager is pretty good at managing dotfiles. The primary way to manage
  # plain files is through 'home.file'.
  home.file = {
    # # Building this configuration will create a copy of 'dotfiles/screenrc' in
    # # the Nix store. Activating the configuration will then make '~/.screenrc' a
    # # symlink to the Nix store copy.
    # ".screenrc".source = dotfiles/screenrc;

    # # You can also set the file content immediately.
    # ".gradle/gradle.properties".text = ''
    #   org.gradle.console=verbose
    #   org.gradle.daemon.idletimeout=3600000
    # '';
  };

  # You can also manage environment variables but you will have to manually
  # source
  #
  #  ~/.nix-profile/etc/profile.d/hm-session-vars.sh
  #
  # or
  #
  #  /etc/profiles/per-user/davish/etc/profile.d/hm-session-vars.sh
  #
  # if you don't want to manage your shell through Home Manager.
  home.sessionVariables = {
    # EDITOR = "emacs";
  };

  # Let Home Manager install and manage itself.
  programs.home-manager.enable = true;
}

And there you have it! You can install nix-darwin with:

nix run nix-darwin --experimental-feature nix-command --experimental-feature flakes -- switch --flake ~/.config/nix

You can now run darwin-rebuild switch --flake ~/.config/nix to rebuild all your configuration whenever you make a change. Another useful link is the list of nix-darwin configuration options. I believe this is the best place to go for information on what can be configured with nix-darwin.

Why you should try accountability coaching

Ednevsky Blog — Fri, 17 Nov 2023 18:30:09 GMT

A new practice I've been doing for the last few years

Scooter Crash

denvaar's website — Fri, 17 Nov 2023 05:40:09 GMT

Carving My Own Path

Martin Rue — Fri, 17 Nov 2023 02:00:00 GMT

It's time to pack a small bag, leave my current world behind and begin overusing the word "nomad".

From Terminator To Wezterm

dystroy|Canop / blog — Fri, 17 Nov 2023 02:00:00 GMT

This post doesn't compare Terminator and Wezterm. This is the help I would have loved to find when starting to try Wezterm coming from Terminator.

Latent optima

Alex’s blog — Fri, 17 Nov 2023 02:00:00 GMT

All data is the result of some selection process. The first order manifestation of this is bias: a failure to select a representative sample of the real underlying thing the data is describing. But selection operates at multiple hierarchies. For something to exist such that data can be collected about it, that something must have been created instead of some other thing — it must have survived a selection process. Because of this, any dataset includes latent information about the traits that tend to survive these selection processes. For example, if you counted up all the members of every species on Earth the resulting dataset would give you some information on the types of creatures that tend to do well in Earth’s biosphere. If you then counted how many of those creatures had fins, and how many had wings, you could get partway towards an understanding of how to construct an optimal ‘model’ animal. A mix-and-match book made real. Zooming down to a lower level, Karl Friston (and others) have argued that organisms are embodied statistical models of the states of their environment1. This is a way of thinking about an organism as “a hypothesis of its environment”; in the words of Peter Munz: “The behavior of a fish and the functioning of a theory of water are exactly identical. The fish represents water by its structure and its functioning. Both features define an initial condition (for example, the degree of viscosity of water) which, when spotted or sensed, trigger off a prognosis or behavioral response which, in case of a fish, fails to be falsified. By contrast, a bird does not represent water” In other words, the states that organisms are most likely to be in are the ones they are likely to prefer2. You find organisms where they choose to be. This may just seem like another spin on the familiar process of evolution by natural selection. But I think the selection-eye-view reveals some deeper truth about what is learned by models when they’re trained on real-world datasets. Machine learning models which learn compressed representations of data also naturally end up learning something about the optimal properties of (and selection process acting on) what they are modelling — even if they aren’t provided information about those properties from the outset. Meta’s protein folding model ESMfold illustrates this principle. ESMfold achieved state-of-the-art accuracy on protein structure prediction, even though its language model was trained on protein sequence data alone: the training process involved feeding the model protein sequences with gaps in them, and getting the model to predict amino acids to ‘fill in the blanks’3. Nevertheless, the information about the shape of protein folds was latent in the sequence data. In order to learn better and better predictions to fill the missing amino acids, ESMfold’s language model had to create an internal representation of the structure of the protein and how it folded. This internal language model representation could then be read out by a folding module (which was trained on sequence data) to generate accurate 3D protein structure predictions. Protein folding is a non-random process, and there are ‘best practices’ in how to structure a given protein sequence to achieve a desired structural motif. Evolution has learned these optima and uses them repeatedly, and because they appear frequently, they can be learned. More generally, the set of proteins that exist in the dataset were only able to be sequenced in the first place because they are evolutionary successful. So there is also latent information about the types of sequences and structures that are biologically optimal in the protein sequence dataset. This principle of latent optimality was exploited by researchers who used Meta’s ESM protein models to improve the properties of monoclonal antibodies that were already highly optimised4. The researchers took antibodies already in clinical use — like Regeneron’s COVID antibody — and ran their sequences through the ESM model, tasking the model with predicting where the true amino acid sequence of the antibodies differed most from the model’s predictions. Then, the researchers selectively replaced the amino acids at those divergent sites with the model’s predictions. These substitutions markedly improved the binding affinity, thermostability, and in vitro potency of the antibodies, without a noticeable increase in off-target binding — all without feeding ESM any information but the protein sequence. This suggests that because samples in datasets have themselves undergone a selection process, as long as we are confident that the selection process is producing outcomes we care about (which we can be reasonably certain of in biology), we can use the statistical properties of those datasets as a sort of optimisation function in themselves. Exploration followed by AI-informed selective regression, or annealing, may be a valuable general approach to optimise new designs generally5. One of the challenges of applying machine learning in medicine has historically been that it’s not clear what optimisation function to use. I’ve previously argued that a way around that may be to collect all sorts of data on parameters of biological systems — a detailed description of states — and then use that as an objective function. The evidence from the antibody optimisation experiments using ESM are perhaps a small piece of support in favour of that approach. Biological systems have already undergone billions of years of selection and optimisation — we may as well make use of that latent information. Thanks to Adam Green and David Yang for giving me the idea for this post I wrote about this at length in my post “Intelligence as efficient model building” ↩ Homeostasis is a manifestation of this drive to return to optimal states ↩ Jeffrey Chan pointed out that to me on Twitter that I had originally neglected to mention that while the language model portion of ESMfold doesn’t use structure data, ESMfold’s “folding head” is trained on structure data including PDB and 12 million AlphaFold2 predicted structures. The folding head is fed sequence data and the ESM language model’s internal representations of the protein structure as input and outputs the structural predictions (3D coordinates) ↩ Derek Lowe has a good summary of the paper here ↩ I wrote about a similar approach in this essay about automating scientific discovery ↩

Using More FreeBSD

Brian's Waste of Time — Fri, 17 Nov 2023 02:00:00 GMT

I do some volunteer sysadmining for a local non-profit, mostly network management and related things. Between there, and my home setup, I have gradually moved from defaulting to Ubuntu back towards FreeBSD over the last year. Nothing wrong with Ubuntu, but I kept finding FreeBSD to simply be a little bit easier and a lot better documented. For a dev box, I'll stick with linux—all the tooling optimizes there first, in particular the container ecosystem is too useful to walk away from. For sysadmin'y stuff though, FreeBSD is where it's at. At a minimum, it's rc system is much easier to work with than systemd :-)

My general toolkit for running anything is a pair of refurbished HP Prodesks sharing a CARP virtual interface using devd events if I need to do anything on failover. It's shockingly easy and just works.

(side note: I tested 14.0-RELEASE on a pair of them and discovered that if a server is hosting a VIP and needs to to use that virtual IP when it is the backup, say to find DNS, you need to disable net.inet.ip.source_address_validation in sysctl, but then it works fine. I probably shouldn't do this, but I'll sort that out later.)

(side note 2: With those old HP Prodesks, you need to go into the BIOS and disable UEFI (switch to use "legacy boot" for the FreeBSD install to work). You don't need to for Ubuntu.)

Github CI

Pxtl.ca — Fri, 17 Nov 2023 02:00:00 GMT

Well, it took longer than I would've liked but I got Github CI working with this blog! Finally learned how to use Github workflows.

Workflow files are here

A Day in the Life of a Damn Interesting Guy [1]

Love, Blood & Rhetoric — Fri, 17 Nov 2023 01:32:24 GMT

2 am: Bill Clinton calls, desperate for advice. What tie to wear for inauguration? Chide him for forgetting five-hour time difference, suggest he wear an Aloha shirt, go back to sleep. [2] 8 am: To work up an appetite for breakfast, swim two laps around Oahu towing an Atlantis submarine by a rope clenched between […]

Childhood Bullying III - Things Left Unsaid

callmeolive — Thu, 16 Nov 2023 23:05:00 GMT

Part two

When it comes to writing, if I can’t keep things concise, I like to keep them entertaining. Unfortunately childhood bullying doesn’t lend itself well to either of these goals, much more when it remains such a raw topic to me.

Whilst I’ve already dedicated two posts to the topic, I still have a few things which I either failed to mention, or felt didn’t fit in with either post. With it being a few days since I powered through writing those posts, I’ve had some much needed distance to calm down. Hopefully it makes for an easier read.

I will note though, it’s important to talk about it. Even if this serves no practical use in the real world, I still feel a need to put it out there.

In the real world, I struggle with the thought of being known or remembered. As mentioned before, being bullied is a humiliating stain on your existence, and the fact that others must remember what I became because of it… It hurts. I wish I could hide it, and to erase any knowledge of myself. I hold a lot of shame. Every time I curled up crying, every angry outburst. Even though I had little in the way of emotional control back then, I’m still weighed down with regret.

I’m scared of contact from anyone who might have known me. I despise using my actual name, and I fear for a future where it ends up online attached to my current self. The option of changing my name via deed poll is a valid option, but I can’t begin to think of a different name to go by, let alone change my name on every service I use!

This has also affected what I do online. I see so many developers and artists put out the most amazing work under their real name, and I’m shocked. There’s absolutely no way I could ever attach my real name to something I put out there because of my fear of being found – and I forget that it’s not normal to feel this way.

I try not to pine over what ifs, there’s no better way to break a man’s soul than to show him what he could’ve been, but it’s hard not to think about. How can’t you when you’re that anxious about life and you feel inherently wrong about existing?

I find it hard to deal with media that brings back thoughts about bullying. I refuse to listen to songs on certain albums because it might share the name of someone who made my life hell.

And despite my love of the Back to the Future series, it’s only recently that I noticed why I feel so conflicted watching it as an adult: the whole series is about standing up for yourself. It’s hard to watch those films and feel okay afterwards knowing that I never did when I was younger. That I might as well remain a failure because of it.

There is one film that very briefly touches on the subject of bullying and that’s The World’s End (2013), the final film in Edgar Wright’s Cornetto Trilogy. Spoilers ahead.

Eddie Marsan plays the role of Peter Page, a meek bloke who upon returning to his hometown comes across someone who bullied him relentlessly throughout childhood. He’s immediately anxious when he notices his old bully, quietly confiding in his friends about it. Much like other aspects of the film, it’s sudden, it’s raw, and it gets to me.

Later on in the film, the bully confronts Peter with a smile, apologising for what he did. Peter is immediately petrified by the encounter, but doesn’t remotely consider accepting the apology before taking the chance to knock the bully to the floor, letting out all of his anger with each blow to the head – and eventually he devolves into mindless screaming as he smacks the bully senseless with a tree branch.

Honestly it’s hilarious, but the tragedy is that such a reaction is far too real. I know that if I was confronted with someone I used to know, I’d freeze as all the feelings would rush back to me. If I lashed out at them I doubt I’d come out on top, but I’m sure that I’d be little more than an incoherent screaming mess.

I wish I could conclude all of this better, but I can’t, it’s that much of a mess. Even though it’s a part of my history, it’s made me who I am today, and I have to carry that weight.

Cuddle Up to KDL

Pete's Points — Thu, 16 Nov 2023 17:13:25 GMT

I've been trying out a relatively new config file format called KDL. It's pronounced "cuddle" and here's a very quick run-down of what it's for and why I find it worth experimenting with. It's a general purpose data language that feels mostly like a blend of JSON, command line, and XML. So it would be useable for use cases including configuration files, data interchange, and data storage, but so far I've only been considering it for the use case of a configuration file intended for manual editing by me primarily.

Read the main kdl.dev site for a very nice intro and full docs, but to just give you the gist of it:

It has nice syntax
- Nesting with curly braces works fine even for deep trees
- No need to quote every property like JSON
- More terse than JSON without getting into YAML ambiguity and gotchas
It supports comments properly
It has a nice spec, a test suite, and docs around consistent interchange between JSON and KDL

Don't link to xkcd. Don't do it.

Yes, I've seen the comic. You've seen the comic. Everyone has seen the comic. There are real and painful issues with JSON, YAML, TOML, XML, and bespoke formats that make exploring new formats interesting and compelling to me. If you are hoping for a world where someone created the one true syntax and everyone just used that and lived happily ever after, I'm sorry to disappoint you. You might be best served by closing this browser tab and moving on at this point.

The KDL FAQ addresses the common "Why don't you just use language X?" questions.

Using KDL in my day to day

rofi and launch lag

I am a huge fan of a utility program called rofi which I use to build many pop-up windows in my desktop environment that are absolutely the core of my workflow: switching programs, running scripts, opening bookmarks, grabbing emoji, etc. I ended up building my own variant because I wanted slightly different behavior (stay running permanently instead of do-then-exit) and I couldn't find a way to do it with the popular rofi project. So I set out to write a basic version very tailored to my needs, which ended up being an interesting journey.

false start in go

I started in go because I love the static binaries and not mucking with cloning source repos, installing interpreters, installing dependencies, etc.

My config file started as JSON and has a nested tree structure. I quickly realized how annoying this is to handle in go and struggled to model the types nicely as go structs that could be unmarshaled.

Because I was keen to just get something working to confirm or deny my hypothesis that a custom rofi variant would actually solve my problem, I knew I could do this way faster in node.js. This type of thing is cake in node, so I switched to node using commonjs modules and vanilla javascript.

node.js: nofi and the elm architecture

I got that version working well enough pretty quickly. I named the project "nofi". I haven't done much building of true terminal TUI style apps (and it's been mostly bash scripts), so I wasn't sure whether I'd need a terminal library to get some widgets, screen layout stuff, something to deal with the terminal and treat it essentially like a GUI toolkit. I decided pretty early to try a basic elm architecture which at this point has proven itself to be an amazing balance of utility and beauty/simplicity. I just hand coded a very basic update() & view() API and keystroke event handling. No library for this although I'm sure many exist, I just coded them up loosely, not worrying to much initially about doing things like console.log during the update(model, message) totally in dirty personal project mode.

I got the thing fully functioning way faster than I expected and it was working and clocking in under 200 lines of code. I added some unit tests since the elm architecture makes that so very easy.

replacing JSON with KDL

I think it was at exactly this point that I discovered KDL on mastodon and immediately thought "Hmm. I bet this would be nice for nofi so I could add comments". It would also be way easier to quickly add to my config file without too much friction since my use case maps so nicely to the KDL syntax. So I grabbed the js parsing library and ported my config file from JSON to KDL and swapped out JSON.parse() for the kdl.parse().

A porting frenzy

Because of my current circumstances of keeping my tech skills up to date and job searching, things then got interesting. It went down roughly like this.

I ported nofi to ES modules
I ported nofi to typescript
I ported nofi from node to bun
I ported the unit tests to bun's built-in library

So now I had a nicer config file syntax that was very concise but not something bespoke and specific to nofi, just a generic syntax I could learn once and use a lot.

I also now had modeled the data structure in Typescript types, so maybe I could try porting it to go using that as a reference?

Oops, no go support

At this time in Oct 2023, there were no go implementations of KDL parsing listed on the KDL website. Ah ha! I exclaimed, I will write the first go parser! So I set out to do just that.

Parsing, PEG, and pidgeon

In my brief research about parsing, lexing, recursive descent, hand-tuned parsers, parser combinators, etc, I stumbled upon the Parsing Expression Grammar (PEG) technology and found a go implementation called mna/pigeon. So I played around with that. The basic workflow is you model your language in the special PEG syntax, and then codegen builds you a library that can parse it. And the great thing was that KDL already publishes the PEG syntax in their spec (albeit with cosmetic syntax differences which seems to be a nuisance across the whole PEG ecosystem - they are all similar but not identical). So great - I could in theory take that grammar, tweak the syntax to match what pigeon needs, and get a go parser built? Maybe?

I got a lot of stuff working with this approach. The only other key thing is after the generated code has built an abstract syntax tree, it's basically a sea of []interface{} in go and you need to write little snippets of go code manually to map that to nice proper types which you can design. So I modeled the basic elements of KDL - a Document which is basically a tree of Nodes and nodes have Properties and Args.

I was able to get through the bulk of KDL by porting a bit of PEG, writing some go snippets, and working my way around the language incrementally. I got a lot of stuff working this way:

node lists and node identifiers
key/value properties
positional arguments
scalar data types
- booleans
- numbers
- quoted strings
nested nodes
slashdash comments
node type annotations

This was my night/weekend hobby project for a while, but as I got all the low hanging fruit working, I started to hit issues building the remaining bits of the PEG grammar that I was unable to debug in go. The generated code and sea of deeply nested untyped []interface{} eventually stopped me in my tracks.

Open Source Provides

At this point as I was circling back to the KDL docs to see if I missed anything, I just happened to notice that in just the past few weeks, suddenly instead of zero go implementations, there were now two (2!) separate implementations listed on the KDL site. kdl-go and gokdl. It seems I wasn't the only go developer interested, and these folks had more success. They had both written scanners and parsers by hand, which involves a lot of fiddly unicode/rune stuff. One of them claimed to pass the full suite of tests from the KDL spec. So while I still want to get my pigeon version finished, I could try to use one of these libraries to get unblocked on porting nofi to go. This timing has happened to me before (except an open source library for what I needed literally appeared during my subway commute into Recurse Center that time), and it's such an awesome feeling! It makes me want to high-five these people through the Internet.

gofi and bubbletea

So I added kdl-go and felt some confirmation since the type model almost exactly matched what was in the javascript library as well as what I was working on in my kdlpigeon project.

OK so I had my data loaded in from my config file, but how should I TUI in go? I was aware that there are great libraries for this in go, and aware of bubbletea in particular, but oh boy was I pleased to read the docs on bubble tea and discover that it's actually the elm architecture for TUI programs in go. :chef-kiss:.

I felt immediately at home and blazed through porting my typescript implementation with my home grown elm architecture to go and bubbletea. There was basically no impedance mismatch and everything came over fast and easy. In pretty short order, I had all the basic nuts and bolts of dynamic key mapping across a nested menu structure working in go. I ported over the unit tests and then started using code coverage analysis to add unit tests to cover more paths through the big main switch statement within the update() method which is where the sausage is always made in the elm architecture.

Daily driving gofi

Here's a screenshot of gofi at the moment.

So gofi is now functionally complete and I have updated all my awesomewm keybindings and integrations to use it in my daily workflow. Keep in mind, I run this thing almost every time I switch applications so we're talking for sure hundreds if not thousands of times in a given day. It has to be blazing fast and functionally flawless.

One technical glitch in debugging TUI code in go is so far I don't know of a good way to launch the debugger from with VS Code and have a proper TTY available and way to send keystrokes. So there's some friction there which requires me to launch a dlv debugger process from a terminal with a TTY where the TUI can show and I can use it, then attach to that from VS Code and switch back and forth while debugging. It does work though. It's great to have a breakpoint at the beginning of update() and just examine all the calls there to understand your app state and how bubbletea represents the terminal as messages coming into your program's core logic.

Aside on dynamic and static typing

At the beginning of this project, I was not yet clear on how to structure my configuration file. I didn't know exactly what data shape to use to represent my application. I needed a few cycles of writing a flavor, trying to code against that, realizing some issue, revising the structure to address, and so on. Go being statically typed presents pretty much a hard wall here. Either you have some totally sound representation of all this and you can compile and run your program, or you cannot compile at all. Whereas in dynamic land, I can parse a document into whatever the library gives me and then interactively look at it in the debugger to learn its shape and how bits of syntax in the file get represented within the javascript program. A few sessions later, after I had a working Typescript program, porting it to go was easy and useful, but using dynamic types as a stepping stone was highly useful and effective for me.

Fin

So go check out KDL and play around with it. I would still like to get closure on my kdlpigeon project by squashing the current bug and then completing the remaining bits of the PEG grammar. I'm curious how it fares compared to the hand-coded state machine scanners and parsers in the other open source libraries.

Flutter Stuff From Substack This Week

Stories by Fred Grott on Medium — Thu, 16 Nov 2023 13:07:55 GMT

Sort of the weekly reminder that I moved my Flutter posts to substack.

Continue reading on Medium »

sum-grid.love

Kartik Agaram — Thu, 16 Nov 2023 09:00:00 GMT

A little sudoku-like app for helping first-graders practice addition. This attempt at situated software for schooling got a little more use than spell-cards.love.

video; 25 seconds

SANS Recommendations for Defensive Cyber Analysts

Zac Szewczyk — Thu, 16 Nov 2023 07:38:05 GMT

I keep fielding this question in private, so I finally decided to answer it in public: “What SANS courses should I take?” Although I have a much longer answer to give about training in general, this article answers that specific question based on my own personal experience, having taken several SANS courses over the years.

Permalink.

Dissecting UPI: a DPI by NPCI which we cannot RTI

cpuonfire — Thu, 16 Nov 2023 06:18:46 GMT

Phew, that’s a lot of Is. The terms like Digital Public Infrastructure (DPI) and Digital Public Goods (DPG) have been used liberally at the G20 summit, and UPI is India’s poster child for all of this. This post will be a summary of a rabbit hole I dug about UPI, NPCI, and other things. NPCI is the National Payments Corporation of India which is legally a non-profit controlled and funded by the RBI and a bunch of (mostly public) banks.

Acquiring the skills of growth

An Ordinary Disaster — Thu, 16 Nov 2023 04:17:49 GMT

Change can seem entirely impossible—and, it can be normal, and easy.

Migrating/re-encrypting pass's password store

xenodium.com @alvaro — Thu, 16 Nov 2023 02:00:00 GMT

Note to self, I needed to migrate/re-encrypt someone's password store (aka pass). Straightforward:

Get the new key, probably already in gpg key chain. Try listing it:

gpg --list-keys

To re-encrypt, pass init with new key is enough. It'll prompt for old pass key.

cd path/to/.password-store
pass init NEW-GPG-KEY

Unicorn coloring page with broomstick and Halloween theme

Studiofreya SSG Site — Thu, 16 Nov 2023 02:00:00 GMT

Generated by GPT-4 with prompt:

Can you create a coloring page of a cute, sitting unicorn with wings and large eyes, with a broom ready to fly, with a halloween theme?

GPT-4 description:

A black and white Halloween-themed coloring page featuring a cute, sitting unicorn with wings, large expressive eyes, and holding a broom, ready to fly. The unicorn should have a flowing mane and tail, and a spiraled horn. Its wings are detailed and feathered. The Halloween theme is reflected in the unicorn’s playful costume elements like a witch’s hat, and Halloween decorations like pumpkins and bats around it. The background should be simple, enhancing the whimsical, festive atmosphere, perfect for a children’s Halloween coloring book.

Visual Assist Open Corresponding File

Studiofreya SSG Site — Thu, 16 Nov 2023 02:00:00 GMT

Visual Studio got updated to 17.8, and the shortcut Alt-O to open corresponding file in Visual Assist was unassigned ¹.

Open Visual Studio options, and reassign the VAssistX.OpenCorrespondingFile command to Alt-O.

ChatGPT Killed my Blog

Backendhance — Thu, 16 Nov 2023 02:00:00 GMT

Hi,

Meine Arbeitsweise hat sich im letzten Jahr so stark geändert wie nie zuvor.

Vor einem Jahr hat OpenAI ChatGPT veröffentlicht.

Damals fand ich die Technologie schon cool, aber ich habe nicht erkannt, wie groß der Einfluss auf meine Arbeit sein wird. Und das in nicht mal 12 Monaten.

Heute nutze ich KI für so viele Dinge:

Ich google keine Fragen mehr, es wird ChatGPT gefragt
Github Copilot schreibt Code bevor ich weiß was ich überhaupt schreiben wollte
Mein Content (auch dieser) wird von ChatGPT lektoriert
Stock Images lasse ich mir von Midjourney generieren
Meine Vorträge der letzten Wochen sind mit ChatGPT entstanden
Selbst das Piraten-Rätsel vom Kindergeburtstag generiert ChatGPT

Und diese Liste ist bei weitem nicht vollständig.

Introducing Wikid: A(nother) Wiki System

one mikro2nd — Thu, 16 Nov 2023 02:00:00 GMT

I’m writing a wiki system. Have I lost my mind? Why, when there are already so many wiki platforms languishing out there…? First I have been looking for a good wiki platform to use for collaboration (I have an actual use-case with real people and genuine need), and not finding many I’d rate as suitable. Quite a number of the platforms out there are just way too complex, lacking the sense of elegance and WikiNature I expect and want out of a wiki system.

Buildpacks - 3 months later

Domenico Luciani — Thu, 16 Nov 2023 01:00:00 GMT

Since the beginning of my career in this industry, I've always been fascinated by the open-source world. After so many years of contributing on my own, this year, I finally joined…

Tools for working with JSON

Gaganpreet Arora — Thu, 16 Nov 2023 00:14:52 GMT

I often work with JSON responses and need to process, view or compare objects. I have come to depend on a bunch of command line tools for these tasks. Do you use any other tools? Let me know in comments below.

View (bat) #

bat (better cousin of cat) is perfect for reading syntax highlighted JSON. It’ll use a pager for files that don’t fit on the screen and files can be searched by using the / operator.

CloudFront and the Wild West of the Internet

Tiago Rodrigues — Thu, 16 Nov 2023 00:10:00 GMT

The internet is truly amazing, but it is a Wild West.

The other day I was looking at my CloudFront Popular objects and was supper curious, this is pointing to an S3 bucket with some static images and polly mp3 that are serving the mobile app.

What wp-login.php

A Crash Course on Music Theory

epiccoleman.com — Wed, 15 Nov 2023 22:33:34 GMT

A crash course on music theory that I wrote to help a Reddit user understand a mysterious diagram their guitar teacher drew.

how to rename files without specifying the path twice

blog — Wed, 15 Nov 2023 19:26:47 GMT

rustcred, a git credentials helper

infrequent oscillations — Wed, 15 Nov 2023 15:54:31 GMT

In which we store some credentials with example code.

How to use Weblate

River Writes - A MediaWiki Blog — Wed, 15 Nov 2023 04:34:59 GMT

A guide to using Weblate as a translator

📷 A Photo That Means the World to Me

James Carr — Wed, 15 Nov 2023 02:00:00 GMT

10 years ago, as I was getting ready to take kids somewhere I heard them both laughing wildly in the backseat. I snapped this photo as quick as I could and to me, it just completely captures the companionship these two siblings have shared since the very beginning. The big grins on both of their faces is something I always hope to see every single day.

Wellness challenge halfway point

Ben Overmyer's Site — Wed, 15 Nov 2023 02:00:00 GMT

Here we are at November 15, halfway through the wellness challenge that I set for myself this month. I've kept to it, and while progress has been slow, something interesting is happening.

I started this challenge at 226 lbs. I'm now down to 221. Also, I'm now past the "warmup" workout plan and into the first week of the "phase 1" workout plan. This means I'm going to the gym four days a week instead of three, and alternating between two workout regimens.

I've settled into a routine now. Going to the gym in the morning is something to look forward to, rather than a chore to accomplish. That was unexpected.

I may extend the workouts beyond the end of November. This is turning out to be awesome.

A Kind Introduction

dystroy|Canop / blog — Wed, 15 Nov 2023 02:00:00 GMT

How to leverage the Rust compiler to prevent identifier misuse ?

How to go from ec2ba151-7acf-43a9-bb98-6f5331992f42 in your database to "Cust_ec2ba151-7acf-43a9-bb98-6f5331992f42" in your REST/JSON API ?

How to do it at zero runtime cost, with no boilerplate, with a derive macro ?

CHAOS;HEAD NOAH review

ブラック — Wed, 15 Nov 2023 02:00:00 GMT

Press any key for schizophrenia.

CHAOS;HEAD is a really interesting story. On the surface it appears to be a murder mystery story but it's not really that. There are murders and mysteries, but they don't meet in the same way that they would in a whodunit. I guess I'd rather call it a psychological horror, even if it's not really outright horror. And much like STEINS;GATE, the entry in the series that people are probably the most familiar with, it is rife with conspiracies and larger implications. Really enjoyed myself as I read through it for the first ending.

The meat of the story is really in the common route, which then unlocks the other routes and endings after having completed it once. The character-specific routes aren't that special even if they do flesh out some of the characters a bit more and have some good elements. The most outstanding one of them is the Rimi route, which gives the most exposition for the common route, although I did also enjoy the Sena one too as a Sena fan. The endings also felt a bit lacking compared to the emotions I felt from the common route. Even the true ending didn't really change things up that much or be more detailed than the base ending. Thankfully the common route was interesting so I don't feel soured by these issues.

The protagonist Takumi is a flawed but interesting character. An anti-social, delusional otaku hiding from the world in his shipping container home. And because of those delusions, he becomes an unreliable narrator for the story, making the reader unsure what parts of the story are real and what are delusions. He enters a delusion, then something unexpected happens and you can't know if it's still a delusion or not. Or he encounters a stranger that supposedly knows him and you can't know which is true. The unreliability of his delusions really fits the whole mysterious and disturbing theme of the story very well.

The delusion triggers are pretty interesting system that allow having a positive delusion, a negative delusion, or no delusion. The delusions will then trigger either a small scene with no relevance to the plot, or change the actual routing (at least if you answer a quiz correctly). While the game does have an explanation as to why this would happen, it's still pretty hard to figure out why having a negative delusion in a given scenario would result in a different ending. Not quite as opaque of a routing system as the text message system in STEINS;GATE was, but still mixed up a bit. You'll probably want to look up a route guide for this one.

Art in CHAOS;HEAD is just about alright. Some of the character sprites and CG images have a derpy feeling to them, with the characters looking like their facial features are trying to escape their faces. I didn't find the art fundamentally flawed but it did definitely stick out, even in some quite pivotal parts of the story. The characters also have animated mouths but there's not much animation beyond that and some camera pans. And while the CG did have some quality issues, the amount of it was just fine. I guess this isn't really the most visual of packages.

But what was actually superb was the sound design. The sound design in this game is fantastic. Takumi is fully voiced and Yoshino does a superb job at voicing his uncertainties, fears, delusions, paranoia and other ailments. Maybe his performance was actually too good, since it overshadowed some of the side characters, even though there were also good performances there. Like for example Takahashi Chiaki's performance as Yua. The music and sound effects are also to the point when it comes to producing a creepy, supernatural and mysterious atmosphere. There's very few times when I'm impressed to this point by just the sound design in a visual novel. You'll definitely want to have audio on for this experience. Bonus points for an excellent insert song.

You’ll also want to have audio on because it’s very hard to figure out who’s talking without it. The user interface has no indication on who the speaker at the moment is, or if the text is just narration. If you’re deaf, you can go get fucked as far as CHAOS;HEAD is concerned. The interface is also kinda ugly in general and the text display is kinda rough. This is probably a function of the game’s age, as it’s already 15 years old. Not a dealbreaker but something I did definitely note during it.

For what it's worth, installing the Committee of Zero patch does clean up the text display so maybe consider doing that.

Whilst it's largely overshadowed by its excellent younger sibling STEINS;GATE, CHAOS;HEAD is still an interesting story and recommended reading despite some of its flaws.

Maybe I should now check why everyone hates the anime adaptation…

What the NFL Taught Me About Business

Oliver Gilan — Wed, 15 Nov 2023 02:00:00 GMT

The NFL acts as a microcosm for business and can teach us lessons about building successful organizations from small startups to large companies. Understanding these lessons will shape the choices and effectiveness of both leadership and ICs and can radically alter the outcomes of a business.

Moving to AI safety

This Is My True Name — Tue, 14 Nov 2023 18:44:00 GMT

Kicked up some dust this year, some was settled

Can Debezium Lose Events?

Gunnar Morling — Tue, 14 Nov 2023 16:00:00 GMT

This question came up on the Data Engineering sub-reddit the other day: Can Debezium lose any events? I.e. can there be a situation where a record in a database get inserted, updated, or deleted, but Debezium fails to capture that event from the transaction log and propagate it to downstream consumers?

TypeScript source code generation

wh!le / blog — Tue, 14 Nov 2023 15:33:56 GMT

Examples of different methods to generate valid TypeScript source code via metaprogramming It’s about time for another blog post… been a while! While I still owe it to myself to write the 2nd part of the previous blog post (does anybody have tips for shitty bugs?), I thought it would be time to write about a different topic - Metaprogramming as programming technique. Sometimes writing source code can get a bit repetitive.

My memorable OpenStreetMap edits

unsungNovelty — Tue, 14 Nov 2023 11:51:49 GMT

OpenStreetMap (OSM) is fun and fantastic for so many reasons. One being able to roam around the world from the comfort of your chair. I started mapping on OpenStreetMap in 2015 by coincidence through a friend. And OSM later became part of my work at Mapbox. Where I went through out the world as an armchair mapper. I used to maintain a list of interesting places as I encountered them. Then map it later when I am free. Here are my 3 memorable OpenStreetMap edits.

Valparai, Tamil Nadu, India

Tea plantations in Valparai by Thangaraj Kumaravel

Image license: CC BY 2.0 Deed

I have been to Valparai several times as a kid during summer vacations. It’s surreal and beautiful. My uncle’s place there used to be at the top of the hill, above the clouds itself. You won’t be able to see the road when it is cloudy. We used to play hide and seek in these gigantic guava trees. Or maybe they felt gigantic as I was a kid then. Trees were our targets because leopards were regular visitors and we were not “technically” allowed to go out of house plot.

Valparai hence was one of the places I always wanted to map due to cherished memories. Which is what I did. I have put in hours here to map detailed map features. Rekindling the memories while doing so were worth the time I have put into adding data here.

Screenshot of Valparai in openstreetmap.org

My edit

Screenshot of my edits on Valparai in openstreetmap.org

Pettineo, Sicilia, Italy

Image of pettineo in Wikipedia

I came across Pettineo, Sicilia thanks to end user feedback from Mapbox maps. Amazing place. At least from the satellite imagery, it was. To my surprise, it was not mapped at all when I looked at it in JOSM. The edit I did here was far far more grander than Valparai. Or that’s how I remember it. What I didn’t know was that JOSM somehow hadn’t loaded the data of the place. Almost 4ish hours or so of my time resulting in duplicated data as I realised I have mapped over the original data. My edit was extremely detailed over the original data as I recall it. But it did not connected to any data that was already present. Merging felt far more complex and I had to revert my edit.

After the initial panic and changeset revert, I did edit Pettineo again, it was not as large as my first edit since I didn’t have to map half of it. I was just relieved that I was able to fix it as soon as I messed up.

Screenshot of Pettineo in openstreetmap.org

My edit

Screenshot of my edits on Pettineo in openstreetmap.org

Dierks Lake, Arkansas

Dierks Lake, Arkansas, USA

Over the years, I have created a huge interest in natural features. Especially water bodies. It started with one of my first few edits where I added some of the swimming pools in Delhi which hosted most of the foreign embassies in India.

Dierks Lake, Arkansas came through work as well. It had only extremely bare geometry. Which I hate in a waterbody. Or anything natural. Whether it is a waterbody or a tree, they have distinct shape and geometry which should be celebrated. Editing the entire lake was around ~2ish hours of work IIRC. Dierks Lake’s mapping was the reason I became a fan of waterbody mapping. And hence has a sweet spot in my heart.

My edit

My edit on Dierks Lake, Arkansas, USA

What’s next?

Once you have maps in your life, you will see it everywhere. Apart from some of the map related projects I have been working on, I wanna restart mapping in OSM. Especially natural features like forests, water bodies etc. Here is to creating more map memories.

[Hμblog] Stop deploying web application firewalls

devever.net/~hl — Tue, 14 Nov 2023 09:25:59 GMT

Stop deploying web application firewalls

Excellent writeup by Mac Chaffee about how web application firewalls are universally a terrible idea. I've been meaning to write an article on this subject for some time, but this is a good overview of many issues with them.

Fun with Makefiles - Dynamic Menu Generation

smcleod.net — Tue, 14 Nov 2023 02:00:00 GMT

This reusable snippet will generate a menu of targets from the Makefile. It will use `fzf` if it is installed, otherwise it will use a numbered menu.

Pseudoprose

taylor.town — Tue, 14 Nov 2023 02:00:00 GMT

Pseudoprose is pseudocode for writers.

The least controversial movie

Alex Molas Blog — Tue, 14 Nov 2023 02:00:00 GMT

tldr

📦 Multiple Git Configurations and Identities with Folder-Dependent Includes for GitLab, GitHub et al

Jayson Salazar Rodriguez | @jdsalaro | Blog — Tue, 14 Nov 2023 02:00:00 GMT

People, shout out to some of my friends and collegues, like to configure their development environments in a myriad of ways. Even setting one’s Git author identity, a seemingly boring and mundane task, is sometimes approached in the most creative ways. This, of course, includes not only configuring git, the tool itself, but also any relevant repositories as well as the platforms where those eventually end up hosted. In this brief tutorial I show how to properly handle multiple identities and configurations as well as how to manage Git projects which might be spread throughout multiple Git backends.Please beware this tutorial will likely only be relevant or interesting to you if you already have some experience with Git, otherwise this sort of setup may feel like unnecessary or overly complicated. In case you are new to Git, I highly recommend you go through /tutorial/git and also have a look at /cheatsheet/git.

Restyling with Tailwind CSS

Reiterations — Tue, 14 Nov 2023 01:25:00 GMT

I’ve revamped the styling on the blog, completely replacing the old (and rather dated) theme with Tailwind CSS. That also meant changing the build process to accomodate Tailwind.

Childhood Bullying II – Aftermath

callmeolive — Mon, 13 Nov 2023 23:25:00 GMT

Part one

With it being a decade since I was last at school, you’d think I’d be over it. Sometimes I think I am.

And then I dream.

I’m suddenly back at school, or a face I had forgotten comes back to me, and it all comes back: the powerlessness, humiliation, the anger that someone could hurt me so much for kicks – and finally the horror that I’m unable to stop them. Everything comes back. Not even waking up removes the fear, as I spend the rest of my day trying to keep my composure together.

All it takes is the smallest reminder and the pain rushes back as if I was there again, and it shows how much it destroyed me. It tells me who I am, reminds me that I’m weak, anxious, and ready to crumble at the slightest struggle.

Everyone’s heard the what doesn’t kill you platitude. It’s not true. If I was strong I wouldn’t be bothered by what happened ages ago, and I wouldn’t be left with this lasting fear of existing in the real world.

I’m a shell of a person. When I was young I found out that the best way to cope with everything was to isolate, and a flickering screen was more inviting than the people out there. That preference became a habit, and to this day I struggle with existing in the real world. Every time I’ve taken steps to regain a semblance of normality or confidence, things would swiftly collapse and leave me off in a worse state – and after a short time this became a self-fulfilling prophecy.

When I step out in the world, it’s immediately obvious to others that I’m different – wrong – on some level. Childhood bullying scars you, it paints a target on your back that’s apparent to others and most noticeable to those who can capitalise on it. There’s no escaping it.

If there’s one thing that’s stuck with me over the years, it’s anger. Anger at those who had no right to harm me, at myself for never fighting back, and at adults who neglected their duty of care – especially those who knew was going on yet did nothing.

And finally my anger lands at society, that big amorphous blob which shapes everything around us and how it makes us view bullying. How it’s shameful to have been bullied. How it’s easier to ignore such a problem than deal with it. How we say boys will be boys to cover up, dismiss, and outright excuse pain willingly inflicted on others.

Have you ever stopped and looked at the word bullying?

It’s a silly word. It’s harmless, inane. It doesn’t remotely convey the acts which can constitute it, if anything the word bullying actively ridicules and diminishes the act altogether.

We recognise that assault, threats, stalking, theft, and harassment are extremely harmful acts – we literally have laws against these – yet labelling it as bullying makes it trivial, something minor to be glossed over. If we said that a child committed premeditated assault against a peer, we’d rightly be horrified. Calling it bullying undermines the serious nature of such acts, letting it be swept under the rug.

We trivialise the harms of bullying by calling it that, tacitly telling victims that it’s silly to be in pain over it. It’s the norm to dismiss and humiliate those who who were bullied, and there’s no big push to re-examine this norm.

The effects of childhood bullying are well known: increased rates of depression, anxiety, and suicide. And the effects live long into adult life. Yet there’s no big push to eliminate it or the stigma of being bullied.

When speaking up leads to ridicule, it’s no wonder we choose to suffer in silence.

For the longest time I’ve avoided talking about bullying due to the shame of it. Writing this and putting it up in a public space is daunting, but there’s no other way I can get it out there.

I’m a broken adult, wracked with crushing anxieties about life due to my experiences around others at school. Even if the last years of Secondary got better and College was a blast, it doesn’t make up for an entire childhood where I was never safe around others.

I struggle with existing, with being me, putting myself and my work out there in the wider world. I struggle with being a bloke and fitting around others. I hate having existed and being remembered to the point where I hate to use my real name – the moments where I can forget who I am are bliss. If there was a painless and instant way to give up I would have done it years ago.

At the start of this year, I decided to do something to absolve me from the pain I carry – I mentally forgave everyone who wronged me. It was nice, and a few months I managed to feel comfortable with being me in the world.

And then I dream.

How can I forgive those who hurt me so much?

Part three

Childhood Bullying I – Experiences

callmeolive — Mon, 13 Nov 2023 23:20:00 GMT

This June marked 10 years since I left school.

I was the weird kid. I don’t know if I was innately weird, or if years of bullying turned me that way. Either way growing up wasn’t fun.

When I was 7 years old my family had to move across the country, I landed in a new school at the point where friendships and social dynamics were ironed out – if it wasn’t apparent that there was no room for me, my new classmates would make sure I knew.

I was a sensitive lad, as if being the new kid wasn’t enough to lock me out. It wouldn’t take much for me to cry and when the rest of the class found that out my life was over. Barring two lads, every boy in that class swarmed me in an instant, and daily life became one of punches and bruises – boys goading one another to see who could make me break down the quickest. I’d be lucky if I could go a day without having a small mob actively going out of their way to harass and assault me – I became a valid target of aggression, and nothing was off-limits.

Swiftly it became part of class culture that I wasn’t equal, and the fact I was lesser permeated every aspect of school life. Anything to be shared wasn’t passed on to me, my possessions became anyone’s possessions, and my existence was only acknowledged when I was to be laughed at. There were no qualms about whether it was right to hurt me – to them it was entertainment like any other.

I can’t describe what being in a hostile environment like that does to a young mind, ‘soul-destroying’ doesn’t remotely cover it. Whatever social skills I had were pummelled out of me to the point where during one day of humiliation, all I could scream through confused tears was “Why are you laughing at me?” at a crowd of goading onlookers – that plea as a desperate, damaged boy was mockingly echoed back at me for the rest of my life.

Isolation became my only solace on the playground. The good days consisted of spending my breaks idly pacing with my face pointing down and being left alone – but that would never stop anyone from approaching me to cause harm. There was never a safe moment on the playground, and I don’t have the words to convey what that did to me.

I was only 7 years old. No-one had the right to lay a finger on me.

I was always the kid who followed rules at school, I believed that they were unbreakable tenets of life which would come around to protect me. That, and a fear for getting into trouble meant that when I was told that I shouldn’t get into fights and to tell a teacher if someone was harassing me. As other victims of childhood bullying will tell you, that did nothing solve the problem. On a good day a teacher might get a crowd of lads to pack it in for 10 minutes, but the second backs were turned I’d be kicked to the floor again.

I remember one December day with heavy snow. The staff decided to let us enjoy the magic. A school-wide snowball fight was arranged on the field, and the playground was reserved for those who didn’t want to participate; knowing I’d be chased around and have snow piled in my face and shoved down my clothes, I stayed in the safe zone surrounded by teachers – Not that it stopped anything. One of the teachers had the nerve to say it couldn’t have happened, as if my clothes were bone dry before them.

Staff neglecting their duty of care was common throughout my school life, and I cannot express my pain over it.

I was 9 now. My parents were increasingly distraught over the fresh bruises I’d return home with, and while I can’t remember if I told them what happened, I know that a single dinner lady blew the whistle to them. It wasn’t long before my mum witnessed me one day being punched to the ground moments after dropping me off at the playground – she immediately stormed back in, raising hell at nearby teachers for ignoring it before dragging me back home in tears.

The next day my parents had a meeting with the headmistress, wherein she finally admitted that there was a problem with bullying and said she’d do whatever she could to prevent it – aside from remove those who purposely and repeatedly assaulted me. My parents weren’t buying it, and I was taken out of that school.

I was moved to another primary school a mile away. It would’ve been nice, but I was already a broken, different child. All it took was for a single boy to cotton on to it before it all started again. What didn’t help was that the school was still local, many of the kids knew those from my old school and it wasn’t long before I was asked were you the boy who… Followed by a painful memory.

The bullying here started again. It was smaller in scale, less physical but more social. There was one boy in particular who’d assault me during certain games so that when I’d go to a teacher about what he was doing, the easiest option would be to ban the game altogether – dodgeball in particular. I remember him using that as an excuse to repeatedly pelt me in the face even if I wasn’t part of the game, and he gleeful that he could pin the blame of it being banned on me.

One clear memory is that the new school’s playground was in the shape of a horseshoe around the main building, with a little connecting path off-limits for pupils. One day I was ‘it’ for hide and seek, up against the rest of my class, bar a single girl. It wasn’t long before I spotted them all bundled together – with one of them making eye contact with me – before running off. I gave chase, stopping before the path we were banned to go on. Tracking all the way back around I noticed the same happen, the rest of my class huddling back down the off-limits path, pretending they weren’t spotted. I approached a member of staff about them going down the banned path (again, I was a kid who always followed and respected rules), but they shrugged it off and asked how they could stop so many of them going down that class.

Returning inside and talking to my teacher, he wasn’t having it. He went out of his way to confront the rest of the class when break ended. Funny how their laughs suddenly stopped.

All I can think about was the one girl who was kind enough not to join in.

Secondary school wasn’t fun either. Kids from both the primary schools were part of it, and they spread things about me to the rest of the kids around the school. It wasn’t quick before they all cottoned on. The first few years were hellish.

It was at secondary where I found myself blowing up. I was irritable and quick to anger at the smallest of provocations, and I still hate myself for it.

I find it difficult to talk about what happened. All I’m thankful for was that my head of year changed in year 9, from an English teacher who laughed at my parent’s concerns to a Geography teacher who took my safety seriously. He stuck his head out to get me help, and arranged for in-school counselling and anger management during the rougher times.

Secondary school slowly got better as folks matured and I learned how to cope better. When I moved on to college – far away, with completely new people – I found myself surrounded by other self-prescribed weirdos, many of whom faced the same as I did. They were a wonderful bunch, and for the first time in my life I felt comfortable, safe, and that life could actually be wonderful.

How long did I have to wait for just a moment of peace?

Part two

Rituals Circa 2023

trv.rs — Mon, 13 Nov 2023 19:59:00 GMT

I have a lot of checklists and systems. Definitely too much. My friend Errin asked if he could steal the templates, so some quick thoughts.

Rituals help me by…

Providing a sense of progression and time-pass when working remote.
Holding long-term goals in front of my daily thoughts.
Correctly balancing lead and lag measures.¹
Highlighting Weak Signals² before problems escalate with health or relationships.

I am a crazy man. I have built this up over years. Don't copy this verbatim. Steal one thing that's good, but be inspired but terrified by the rest.

🗒️ Daily: `journal/daily/YYYY-MM-DD.md`

---
follower:
husband:
father:
health:
create:
work:
plan:
---

This is the template for the daily note. I rate myself on my intentions from 1-3 in areas of my life:

1 = Didn't try.
2 = Showed up.
3 = Broke a sweat.

The following becomes a sort of "bullet journal" for what happened that day. Highlights, funny things my kids said, insights from meetings at work… etc.

🌅 Daily Plan (Workdays @ 9a)

# 🧩 Daily Plan

In order to be effective with my day, I design my day with the right balance of focus, time, and effort.

- [ ] 🙏 Pray. Wisdom to know, strength to do.
- [ ] ⭐ Protect the [essential](things:///show?id=today&filter=@Meazure%20Learning,$High).
- [ ] 👨‍👩‍👧‍👦 Prioritize [family projects](things:///show?id=anytime&filter=@Family).
- [ ] 👨 Choose [personal tasks](things:///show?id=anytime&filter=@Me).
- [ ] 📅 Check schedule.
- [ ] 🕊 Make room for margin.

Tags: Rituals: Daily

This ritual gets done 50% of the time. I mostly plan my day the previous night in the Workday Shutdown ritual. This one is designed to help me remember what's important, fill the remaining plan with useful intention, and stay healthy.

🌇 Workday Shutdown (Workdays @ 4p)

# 🗃 Workday Shutdown

As a faithful steward, I put my work in order to be prepared for the next day, identifying the essential task for the next day.

- [ ] Open the [Daily Note](obsidian://advanced-uri?vault=wiki&commandname=Periodic%20Notes%3A%20Open%20daily%20note).
- [ ] 🕰 Review [time logs](https://track.toggl.com/timer) and time blocks. Mark misspent time on Calendar blocks.
- [ ] 📓 [Log tasks/projects](things:///show?id=logbook).
- [ ] 📝 Process notebook.
- [ ] 📨 Process [Drafts inboxes](drafts://).
- [ ] 📥 Dump your brain into the Inbox. Get it all out.
- [ ] 📤 Process [Things inbox](things:///show?id=inbox). (:estimate, $focus, !modality, priority)
- [ ] 🗓 Time block [tomorrow](things:///show?id=tomorrow).
- [ ] 😴 Shut down. (Close. Speak. Shut. Trash. Door. Go home.)
- [ ] 👨‍👩‍👧‍👦 Go live in Coram Deo this evening.

Tags: Rituals: Daily

This ritual is the rock of my whole system. I go through my inboxes, give all the straggling thoughts a home, log everything I intend to log, and leave the desk for the evening.

This I originally installed as a response to overworking after reading Deep Work, but it's expanded to be "manager time" to keep me on track.

🗒️ Weekly: `journal/weekly/YYYY-W[week number].md`

---
Goal1:
Goal2:
---

# Weekly Review: date%3Agggg-%5BW%5Dww

> [!tldr] ![[journal/yearly/2023#Theme]]

## Planning

## Review

### 👍:

### 👎:

### ✅:

---
```dataviewjs
let p = function(b) &lbrace
  switch(b) &lbrace
    case "": return ""; break;
    case " ": return ""; break;
    case 0: return "⚫"; break;
    case 1: return "🔴"; break;
    case 2: return "🟡"; break;
    case 3: return "🟢"; break;
  &rbrace
&rbrace

let f = function(d) &lbrace
  return `[[$&lbraced.obsidianLink()&rbrace|$&lbracemoment(d, 'YYYY-MM-DD').format("dd")&rbrace]]`;
&rbrace

let start = moment(dv.current().file.name, "gggg-[W]ww");
let end = start.clone().add(6, 'd');

dv.header(2, `Dailies: $&lbracestart.format("MM-DD")&rbrace - $&lbraceend.format("MM-DD")&rbrace`)
dv.table(
  ["📆", "🙏", "👫", "👨‍👩‍👧‍👦", "💪", "🖌", "💻", "🎯"], dv.pages('"journal/daily"')
    .filter(d => moment(d.file.name).isBetween(start, end, undefined, '[]'))
    .sort(d => d.file.day, 'asc')
    .map(d => [f(d.file.link), p(d.follower), p(d.husband), p(d.father), p(d.health), p(d.create), p(d.work), p(d.plan)])
)
```

This is the weekly template where all the weekly events show up. It rolls up the daily ratings into something digestible… it's pretty common for me to spot little patterns that I need to address.

The Goals at the top are found in the Quarterly note and roll up there eventually.

🗃️ Weekly Review (Fridays @ 4p)

> [!info] Live _coram deo_. Plan this week in the presence of God.

## 📓 Prep

- [ ] 🙏: Pray. Ask the Lord for wisdom and clarity on what is important eternally.
- [ ] 📓: Review journal and [daily notes](obsidian://advanced-uri?vault=wiki&commandname=Periodic%20Notes%3A%20Open%20weekly%20note).
- [ ] ✅: What did you [accomplish](things:///show?id=logbook)?
- [ ] 💭: What can you learn from last week?
- [ ] 📈: Put logged successes in [OKR system in ADP](https://workforcenow.adp.com/theme/index.html#/Myself/MyselfTabTalentCategoryPerformanceGoals).
- [ ] 📝 Get Clear:
  - [ ] Do a brain dump. Add any tasks or projects you come up with to the Things inbox.
  - [ ] Process your physical inbox. Create tasks in Things for each item in your physical inbox that you want to take action on.
  - [ ] Process your email inbox. Use Mail to Things to forward emails you need or want to take action on to your Things inbox.
  - [ ] Process your Things inbox. Assign each task to an area or to a project. Tag appropriately. (:estimate, $focus, !categorize, priority)
- [ ] Go through each of your projects. Use the checklist[^checklist].

[^checklist]:
    ## 📂 For Each Area:
    - [ ] Are your projects in correct priority order?
    
    ### 📁 For Each Project:
    - [ ] Is this project still relevant?
    - [ ] Can I delegate this project?
    - [ ] Should I move this project to Someday?
    - [ ] Are there are tasks I have already completed?
    - [ ] Are there any tasks I want to delete?
    - [ ] Am I happy with the structure of the project? E.g., should I add or change headings?
    - [ ] Do all tasks with deadlines have the correct deadline set?
    - [ ] Could I add useful notes to any tasks or to the project itself?
    - [ ] Are any new tasks for this project not yet in Things?
    - [ ] Should I convert any tasks to separate projects?
    - [ ] Is there a clear 'next action' for this project? (If not, break down your projects or tasks into smaller tasks until there is a clear next action.)
    - [ ] How can you serve and surprise in this area?

## 📅 Plan

- [ ] Look ahead at the week. Using Deadlines and Upcoming, examine the week and month to find tasks that need to be done, and space them out through the week.
- [ ] Make time for important work. Create 4 hour strategic blocks and identify some buffer blocks.
- [ ] Schedule in the most important task for the day. Using [Upcoming](things:///show?id=upcoming&filter=%40Meazure%20Learning%2CEstimates), schedule the important task for the day.

Besides the Workday Shutdown, this is the other ritual that keeps everything running.

Parts of the list came from Peter Akkies, some of it is automated now.

🫂 Family Weekly (Fridays @ 7p)

# 📖 Weekly Meeting

## Rapport
- [ ] 🙏🏻 Prayer.
- [ ] 👂 How are you in one word?
  - [ ] Screen Time Check

## What's the plan?
- [ ] 🗓 Check [last week’s entry](obsidian://advanced-uri?vault=wiki&commandid=periodic-notes%253Aprev-weekly-note).
- [ ] 📉 Check the [Monthly Plan](obsidian://advanced-uri?vault=wiki&commandid=periodic-notes%253Aopen-monthly-note).
- [ ] 📈 Check our progress against [12WY Scoreboard](obsidian://advanced-uri?vault=wiki&commandid=periodic-notes%253Aopen-quarterly-note).

## What should we change?
- [ ] 🚦 Start / Stop / Continue:
  - [ ] 🟢 What should we start?
  - [ ] 🟡 What should we continue?
  - [ ] 🔴 What should we stop?
- [ ] 🪨 Big Rocks…
  - [ ] Spiritual Leadership?
  - [ ] School?
  - [ ] Next House?
  - [ ] Good Vibes Only?
  - [ ] Leaving and Cleaving?

## Follow through.
- [ ] ❓ Any [questions](things:///show?id=anytime&filter=@Family) to answer?
- [ ] 📆 [What are we committed to next week](fantastical2://show?date=week)?
- [ ] 🧱 Set aside time blocks:
  - [ ] Groceries and errands?
  - [ ] Family-together time?
  - [ ] Fellowship with other families?
  - [ ] Time to work/talk together on the Big Problems?
  - [ ] Projects? (Walking project dictation to unpack and delegate)
- [ ] 💴 Budget Update.

## Reset our purpose.
- [ ] ⛰ What are you going to look forward to this week?
- [ ] 🤗 Express gratitude.

Tags: @Family, Rituals: Weekly

This ritual changes a good bit as my wife and I find new things to work on together. One recent adjustment is to have a 🪨 Big Rocks header that contains nudges to check in on important conversations or projects.

🗒️ Monthly: `journal/monthly/YYYY-MM.md`

(Monthly doesn't have a format at the moment.)

🗓️ Family Monthly (Last Friday of Month)

# 📆 Family Monthly Meeting

The goal is to be through this in less than thirty minutes,[^format] in addition to the [[Family Weekly]].

- [ ] 👈 Review the [last month's plan](obsidian://advanced-uri?vault=wiki&commandid=periodic-notes%253Aprev-monthly-note) and [quarterly plan](obsidian://advanced-uri?vault=wiki&commandid=periodic-notes%253Aopen-quarterly-note).
- [ ] 📆 Look ahead at schedule for the month…
- [ ] 🔨 Identify this month's most important event or project?
- [ ] 🏆 Define success for the month. In this month…
  - [ ] What is a task or routine that will help us work towards best possible week sometime this month?
- [ ] 📈 What micro-adjustment do we want to make to our daily schedule?
- [ ] ⛰ What Real Life Experience are we going to have this month?
- [ ] 🧘 One way we can protect margin this month is…
- [ ] 🏠 Prioritize home maintenance backlog.
- [ ] 🗑️ Pick a declutter space and budget time.
- [ ] Pick a "Deep Cleaning" day.
- [ ] 💸 Create/adjust budget goals.
- [ ] 🛑 Block off days to succeed at our projects…

Tags: @Sarah Travers, Rituals: Monthly

[^format]: Taken in part from Plan Your Year 2020.

Family Monthly is a touchstone for making sure we understand how crammed our current commitments are and make space for weekend projects like painting or tackling a closet.

🗒️ Quarterly: `journal/quarterly/YYYY-Q[N].md`

---
# rate your outcomes from 1 to 5 in the following roles…
roles:
- Follower:
- Husband:
- Father:
- Friends:
- Work:
- Hobbies:
- Emotions:
- Mental:
- Health:
- Rest:
- Finances:
---
## Plan

### Goals

#### Goal 1: _Goal_

#### Goal 2: _Goal_

## Review

### Where are you in each area of life?
```dataviewjs
// we have to sort somehow, so we convert the names to numbers and sort
const s = function(i) &lbrace
  return Number(i.replace('-Q', ''));
&rbrace

const allQuarters =
  dv.pages('"journal/quarterly"')
    .array()
    .sort(function(a, b) &lbrace
      return s(a.file.name) - s(b.file.name);
    &rbrace)

let currentIndex = allQuarters.findIndex((q) => q.file.name == dv.current().file.name);

const quarters = allQuarters.slice(Math.max(currentIndex - 3, 0), currentIndex+1);

let datasets = [];

let trans = function(color) &lbrace
  return color.replace(', 1)', ', 0.5)');
&rbrace

let colors = [
  'rgba(105,105,105, 1)',
  'rgba(169,169,169, 1)',
  'rgba(211,211,211, 1)',
]

const purple = 'rgba(139,108,239, 1)';
let highlight = '#fff';

if (document.body.classList.contains('theme-light')) &lbrace
  colors.reverse();
  highlight = '#000';
&rbrace

colors.push(purple);

for (let i in quarters) &lbrace
  let quarter = quarters[i];
  if (quarter.hasOwnProperty('roles')) &lbrace
    let values = quarter.roles.map(o => Object.values(o)).flat()

    datasets.unshift(&lbrace
      label: quarter.file.name,
      data: values,
      fill: true,
      backgroundColor: trans(colors[i]),
      borderColor: colors[i],
      pointBackgroundColor: colors[i],
      pointBorderColor: highlight,
      pointHoverBackgroundColor: highlight,
      pointHoverBorderColor: colors[i]
    &rbrace)
  &rbrace
&rbrace

let data   = dv.current().roles
let keys   = data.map(o => Object.keys(o)).flat()

const chartData = &lbrace
  type: 'radar',
  data: &lbrace
    labels: keys,
    datasets: datasets
  &rbrace,
  options: &lbrace
    scales: &lbrace
      r: &lbrace
        suggestedMin: 0,
        suggestedMax: 5
      &rbrace
    &rbrace
  &rbrace
&rbrace

window.renderChart(chartData, this.container);
```

### Scoreboard
```dataviewjs
let check = function(bool) &lbrace
  if (bool) &lbrace return "✅" &rbrace;
  return "⛔";
&rbrace

let quarter = parseInt(dv.current().file.name.match(/Q\d/)[0].replace('Q', ''));
let year = dv.current().file.name.match(/\d&lbrace4&rbrace/)[0];
let start = (quarter-1)*13+1

dv.table(
  ["", "Week", "Goal 1", "Goal 2"], dv.pages('"journal/weekly"')
    .filter(function(f) &lbrace
      let num = f.file.name.match(new RegExp(`$&lbraceyear&rbrace-W(\\d+)`));

      if (!num) &lbrace return false; &rbrace
      num = parseInt(num[1]);
      return num >= start && num < start + 12;
    &rbrace)
    .sort(w => w.file.name, 'asc')
    .map(w => [(parseInt(w.file.name.match(new RegExp(`$&lbraceyear&rbrace-W(\\d+)`))[1]) - start)+1, w.file.link, check(w["Goal1"]), check(w["Goal2"])])
)
```

I've previously written about this template… it's pretty complicated. It rolls up the weekly reports on my 12WY goals and provides an opportunity to rate myself on my roles… and see whether my goals effected any change in those roles.

📈 12WY (Every 12 weeks)

# 🎯 12WY Review

- [ ] Open [12WY Note](obsidian://advanced-uri?vault=wiki&commandid=periodic-notes%253Aopen-quarterly-note).
- [ ] Go through week reviews in Obsidian.
- [ ] What can you learn from 12WY?
  - [ ] Rate your intentions on the Wheel of Life.
- [ ] Talk to Sarah.
- [ ] What are you going to do next 12WY differently?
- [ ] What are your new goals for the new 12WY?

Tags: !Planning, :4h, Rituals: 12WY

This is the accompanying check list that goes with the Quarterly template. It's a work in progress. I could probably stand to re-read 12 Week Year and adjust.

🗒️ Yearly `journal/yearly/YYYY.md`

No template at the moment… it keeps changing.

The past few years I've picked a yearly theme and written it in a ## Theme header near the top of the document. This gets referenced in the Weekly template above so that I see it all year long.

🥂 Yearly Review (Year end)

I traditionally journal, do Plan Your Year… just think and journal. It's come naturally to me.

It's funny because writing/journaling about my year is probably my first and oldest ritual. A lot of the ones above exist because I realized that the clarity of review and thinking was helping me get out of trouble… and once a year is too infrequent to stay out of trouble.

I try to focus on gratitude. I also try to count things that are important to me:

good dinners with my wife.
vacations
weekend adventures with the kids.
meaningful conversations.
prayers answered.
books read.

…

I repeat… don't copy my setup. Find one useful idea and run away. 🏃

There is a weird thing about measuring: you should weigh yourself daily, but pay attention to your weekly or monthly average. Most of us pay too much attention to the progress equivalent daily weight and give it more mindshare than we ought. ↩
First encountered in Dan Luu's Normalization of Deviance:

“Pay attention to weak signals” sure sounds like good advice, but how do we do it? Strong signals are few and far between, making them easy to pay attention to. Weak signals are abundant. How do we filter out the ones that aren't important? And how do we get an entire team or org to actually do it? These kinds of questions can't be answered in a generic way; this takes real thought. We mostly put this thought elsewhere.

Weak Signals carry the same information as a Strong Signal but they usually manifest much earlier as little ripples… long before the tidal wave.

Weak Signals are very hard to recognize and learn from. ↩

The cow and the chicken: overcoming resistance to change

CodeKraft — Mon, 13 Nov 2023 17:00:00 GMT

The Cow and Chicken are journeying through the countryside and see a diner with a sign that reads "Steak and Eggs". The chicken nudges the cow and cackles delightfully: "Look! Look, Cow!! We're famous!!!". Whereupon, the cow looks at the same sign and snorts derisively: you're involved! I am committed!!

MBA Washing

smcleod.net — Mon, 13 Nov 2023 11:00:00 GMT

"MBA Washing" refers to the phenomenon where individuals or companies, often with a strong theoretical or academic background but limited recent practical experience, adopt and reinterpret industry-specific terminology and cultural movements.

@Transaction Around Test

Backendhance — Mon, 13 Nov 2023 02:00:00 GMT

Hi,

Mir kribbelte es in den Fingern. Direkt zu Beginn von Thorben Janssens Session “Das große Java Persistence Q&A”.

Die erste Frage war (sinngemäß):

“Im vorherigen Talk (etwas mit JPA) wurde gesagt man solle an jeden Test @Transactional schreiben. Dadurch werden die Daten nach dem Test automatisch zurückgerollt. Wir haben das diskutiert und sind uns uneinig. Was ist deine Empfehlung?”

Thorbens Antwort machte mich nervös. Er antwortete (auch sinngemäß) mit einer Gegenfrage:

Back to Edinburgh for a mini zookeeper experience

Jonathan Jones homepage: planetjones.net — Mon, 13 Nov 2023 02:00:00 GMT

My second trip report to Edinburgh - this time it was for the mini zookeeper experience at Edinburgh Zoo, a visit to Edinburgh Castle and the adjacent Camera Obscura. Edinburgh really loses none of its appeal over time and is a must visit.

Patterns for Personal Web Sites

Vít Baisa — Mon, 13 Nov 2023 02:00:00 GMT

a list of good practices for personal web sites

My first beg bounty

Look mum, I have a blog on tsak.dev — Mon, 13 Nov 2023 02:00:00 GMT

This morning, still a bit sleepy, morning coffee in hand, I came across a Hacker News post about Beg Bounties. Intrigued by the title, having heard about Bug Bounties before, I learned about its next logical conclusion and the people that “randomly” discover security vulnerabilities and then expect to be paid for their “ethical disclosure”.

Putting this aside as a Today I Learned kind of thing, I logged into my work email, only to be greeted by one of such “disclosures”, aimed at my employer and subsequently becoming my problem. Formatting exactly as sent, sender and my employer’s $domain redacted.

Introducing Yet Another Rust Resource (or YARR!)

ntietz.com blog - technically a blog — Mon, 13 Nov 2023 02:00:00 GMT

Rust is a hard language to learn, in the scheme¹ of things. I've previously talked about why the learning curve is hard and what we could do about it. Today, I'm proud to say that there's another resource to help people learn Rust in a more approachable way.

Introducing Yet Another Rust Resource² , or YARR. (Yes, many examples are pirate themed.) YARR is a short introductory course on Rust which is designed to be completed in just a few days. The goal is to get you some foundational knowledge and the lay of the land so you can go forth and deepen your knowledge through real-world programming and other books/courses. When you complete YARR, you should be able to write simple Rust programs and you should have enough familiarity to pair with someone on a bigger Rust program.

I've linked to YARR previously from my projects page and soft-launched it with some friends, but never officially announced it. Whoops! So here it is, announced and ready to use.

What follows are some usage suggestions, how to contribute feedback and help, and why this exists in the first place.

How to use YARR!

YARR is written to get people up to speed quickly in an environment where they will be able to continue working with an experienced Rust programmer. This may be a work environment, it may be the best community for programmers, or it could be a hobby project where you can pair a lot. People can also learn entirely independently and use deeper resources after or alongside YARR.

Here's my suggestion on how to complete the course:

Read/skim through it once without doing the exercises. You won't retain a lot of the material or understand it well on the first read. The purpose of the first pass is to start to load terms into your head and start building familiarity. You'll also get a little lay of the land. If you don't understand something, skip it and move on.
Find another Rust programmer to help you when you have questions or are stuck. The goal with getting comfortable with Rust quickly is to avoid some of the hard parts, and an experienced Rust programmer will be able to get you unstuck and move past some of the tricky things. You may want to reach out to people at work, in your communities (RC is great for this), friends, or internet strangers. If you don't have someone to pair with, feel free to email me; I can pair with a few people who are working through this and I can also pair volunteers.
Read it carefully and do the exercises (get your friend). Your second pass through it should be a more careful reading. Do the exercises as you go, and try to see if you generally get the concepts. This is a great time to work with your Rust partner on this. As a more experienced Rust programmer they should be able to help you through the tricky bits and help with some of the concepts. And don't be too hard on yourself, though: You will probably not understand lifetimes and a lot of the other concepts your first time through. Those sure took me a long time to grok, too. Just see what you understand, see if you can do the exercises, and ask your partner for help!
Bookmark it as a resource. Once you've finished the course but before you move on to something else, make sure you save it! (You can actually do this whenever you like.) It's a handy resource to come back to for a quick refresher on things, especially after you've gone through something deeper.
Explore more deeply! Now that you've finished YARR, you can move on to other resources and other learning paths. There are a lot of great books out there on Rust, each with a different flavor, so it is worth looking at multiple for which suits you best. And you can also dive in more deeply through pair programming: this is how I helped a coworker get more comfortable with Rust (he also did YARR and has read some of a Rust book). And as you go through these paths, revisit YARR occasionally to get an overview again.

Feedback and help wanted

This is just the first version of YARR, which is very much a living document. I wrote this version by myself with very gracious feedback from friends and coworkers, but it needs more to be even better.

Here's what you can do to help YARR be even better:

Send me feedback! What worked well for you? What are you still confused on? Is something wrong? Do you have a better pirate example I should include?
Volunteer to pair with learners! It can be hard to find an experienced Rust programmer to pair with so I'm going to see if I can do those pairings. If you want to be paired with someone learning or you want to be paired with a mentor, email me!
Share this post and YARR! If you found this material helpful or you think someone else would, please share it with friends, on your blog, wherever you think someone who can use it will find it. Resources are only as helpful as they are discoverable. I think sharing this post would be the best introduction into YARR, but do as you feel best.
Send a patch! If you want to directly contribute improvements to the content, you can also submit patches to it. Instructions are in the repo and you can also email me if you want any pointers on contributing, since email contribution workflows are uncommon.

Why YARR exists

I wrote the content for the first version of YARR because I wanted—no, needed—it to exist. There were a dearth of training materials for quickly getting people up to speed in Rust, and that's a big gap. We were considering introducing Rust at work, and one of the big challenges with Rust has always been the time to onboard new programmers into it. When we used Go, we could get someone up to speed in under a week (although mastery takes far longer), so this was a big drawback for Rust. If we could make this faster, it would aid adoption of Rust at work.

When Google released the content for Comprehensive Rust, I saw an opportunity to make a similar training course that's run asynchronously. I can't run a 3-day workshop at work, but I can write the material ahead of time and help people when they get stuck! So I wrote this material to help my coworkers quickly get up to speed on Rust so that we could use Rust in production.

The full version here is public and shared because the material was written once, but can be shared many times. There's no sense in writing good training materials and then keeping them closed off. This exists so that everyone can learn Rust and can get up to speed as quickly as possible.

It isn't intended as a full, comprehensive course in Rust. That's not doable in a condensed timeframe. It's just intended to be a pragmatic introduction to get people going and to make it a whole lot less scary.

🦀

Towards Fearless Macros

Lambda Land — Mon, 13 Nov 2023 02:00:00 GMT

Macros are tricky beasts. Most languages—if they have macros at all—usually include a huge “here there be dragons” warning to warn curious would-be macro programmers of the dangers that lurk ahead.

What is it about macros that makes them so dangerous and unwieldy? That’s difficult to answer in general: there are many different macro systems with varying degrees of ease-of-use. Moreover, making macros easy to use safely is an open area of research—most languages that have macros don’t have features necessary to implement macros safely. Hence, most people steer clear of macros.

There are many ways to characterize macro systems; I won’t attempt to cover them all here, but here’s the spectrum I’ll be covering in this post:

A spectrum of how easy macro systems are to use safely

C macros: advanced search-and-replace

If you’ve done any C programming, you’ve likely run into things like:

#define FOO 42

printf("The answer is: %s\n", FOO);     /* prints "The answer is: 42" */

That #define bit is a macro—albeit a C macro. These operate just after the lexer: they work on token streams. It’s a bit like textual search-and-replace, though it knows a little bit about the structure of the language (not much: just what’s a token and what’s not) so that you won’t run into problems if you do something like this:

#define FOO 42

printf("It says FOO");     /* prints "It says FOO" not "It says 42" */

because that FOO in the string is not a token—it’s just part of a string.

C macros can’t do very much: you scan the token stream for a macro, then fill in the variables to the macro, and then replace the macro and the arguments its consumed with the filled-out template that is the macro definition. This prevents you from doing silly things like replacing something sitting inside of a string literal, but it’s far, far from being safe, as we’ll see in the next section.

Lisp macros: operating on ASTs

In contrast to C’s macros, Lisp’s macros are much more powerful. Lisp macros operate after the lexer and the parser have had a go at the source code—Lisp macro operate on abstract syntax trees—or ASTs, which is what the compiler or interpreter works with.

Why is this a big deal? The ASTs capture the language’s semantics around precedence, for instance. In C you can write a macro that does unexpended things, like this:

#define DOUBLE(x) x + x

3 * DOUBLE(5); /* Does 3 * 5 + 5 = 20, not 3 * (5 + 5) = 30 */

The DOUBLE macro didn’t know anything about precedence and we computed the wrong thing. This means that, to use a macro in C, you have to have a good idea of how it’s doing what it’s intended to do. That means C macros are leaky abstractions that prevent local reasoning: you have to consider both the macro definition and where it’s used to understand what’s going on.

In contrast, Lisp macros are an improvement because they will rewrite the AST and the precedence you’d expect will be preserved. You can do this, for example:

(defmacro double (x) `(+ ,x ,x))

(* 3 (double 5))                        ; returns 30

Lisp macros are also procedural macros, meaning you can execute arbitrary code inside of a macro to generate new ASTs. Macros in Lisp and its descendants are essentially functions from AST → AST. This opens up a whole world of exciting possibilities! Procedural macros constitute a “lightweight compiler API”. [4]

Scheme macros: hygiene

“Same except for variable names” is also called alpha-equivalence. This comes from the λ-calculus, which states that the particular choice of variable name should not matter. E.g. and are the same function in the lambda calculus, just as and are the same function in algebra.

Lisp macros aren’t without danger—many a Lisp programmer has shot their foot off with a macro. One reason is that Lisp macros are not hygienic—variables in the macro’s implementation may leak into the context of the macro call. This means that two Lisp programs that are the same except for different variable names can behave differently:

(defmacro swap (x y)
  `(let ((tmp ,x))      ;(tmp-leaky)
     (setq ,x ,y)
     (setq ,y tmp)))

(let ((foo 1)
      (bar 2))
  (swap foo bar)
  (list foo bar)) ; produces '(2 1)

(let ((tmp 1)
      (baz 2))
  (swap tmp baz)        ;(tmp-capture)
  (list tmp baz)) ; produces '(1 2) --- no swap!

The fact that the macro implementation uses a variable named tmp (tmp-leaky) has leaked through to the user of the macro. (tmp-capture) This phenomenon is called variable capture, and it exposes this macro as a leaky abstraction! There are ways to mitigate this using gensym, but those are error-prone manual techniques. It makes macro writing feel like you’re writing in an unsafe lower-level language.

Scheme’s macros introduce a concept known as hygiene, which prevents variable capture automatically:

(define-syntax swap
  (syntax-rules ()
    ((swap a b)
     (let ((tmp a))      ; (tmp-intro-macro)
       (set! a b)
       (set! b tmp)))))

(let ((foo 1)
      (bar 2))
  (swap foo bar)
  (list foo bar)) ; produces '(2 1)

(let ((tmp 1)            ; (tmp-intro-let)
      (baz 2))
  (swap tmp baz)
  (list tmp baz)) ; still produces '(2 1)

In this case, the variable tmp that the swap macro introduces (tmp-intro-macro) is not the same thing that the variable tmp from the calling context (tmp-intro-let) refers to. This separation of scopes happens automatically behind the scenes, so there’s now no chance of accidental variable capture.¹

If you’d like to know more about hygiene, [1] is an excellent resource.

Racket macros: phase separation and scope sets

Since Scheme macros (and Lisp macros more generally) allow running arbitrary Scheme code—including code from other modules—the dependency graph between modules can get so tangled that clean builds of a Scheme codebase are impossible. Racket solves this problem with its phase separation, which puts clear delimiters between when functions and macros are available to different parts of the language. This detangles dependency graphs without sacrificing the expressive power of macros. I wrote a little bit about phase separation; you can read more on the Racket docs as well as Matthew Flatt’s paper [3] on phase separation.

Racket also has a robust system for reasoning about where a variable’s definition comes from called a scope set. This is a notion makes reasoning about where variables are bound sensible. See a blog post as well as [NO_ITEM_DATA:flattBindingSetsScopes2016b] by Matthew Flatt for more on scope sets.

Phase separation and scope sets make Racket macros the safest to use: Racket macros compose sensibly and hide their implementation details so that it is easy to write macros that are easy to use as if they were built-in language constructs.

Racket also goes beyond the syntax-rules form that it inherited from Scheme; Racket’s syntax-parse macro-building system makes generating good error messages easy.

There’s a little bug in the swap macro we used earlier, and that is the set! form only takes an identifier (i.e. a variable) as its first argument. We don’t have any error checking inside the macro; if we were to call swap with something that wasn’t an identifier, we’d get an error in terms of the set! the macro expands to, not the macro call itself:

(let ((foo 1)
      (bar 2))
  (swap (+ foo 1) bar)
  (list foo bar))


> (swap (+ foo 1) bar)
  set!: not an identifier
  at: (+ foo 1)
  in: (set! (+ foo 1) bar)

This isn’t good because there’s no set! in our code at all! We could add some error handling in our macro to manually check that a and b are identifiers, but that’s a little tedious. Racket’s syntax-parse helps us out:

(require syntax/parse/define)

(define-syntax (swap stx)
  (syntax-parse stx
    [(_ a:id b:id)
     #'(let ([tmp a])
         (set! a b)
         (set! b tmp))]))

(let ([tmp 1]
      [baz 2])
  (swap tmp baz)
  (list tmp baz))            ; returns '(2 1) as expected

(let ([foo 1]
      [bar 2])
  (swap (+ foo 1) bar))

swap: expected identifier
  at: (+ foo 1)
  in: (swap (+ foo 1) bar)

Much better! Now our error is in terms that the macro user will recognize. There are lots of other things that syntax-parse can do that make it easy to write correct macros that generate good error messages—a must for macros that become a part of a library.

Other languages

Many modern languages use macros; I’ll only talk about a few more here. If something’s missing, that’s probably because I didn’t want to be exhaustive.

Julia

Julia macros have a lot of nice things: they operate on ASTs and they’re hygienic, though the way hygiene is currently implemented is a little strange: all variables get gensym’d automatically{{% sidenote sn2 %}}Meaning, they all get replaced with some generated symbol that won’t clash with any possible variable or function name.{{% /sidenote %}} whether or not they come from inside the macro or they originated from the calling code.

Part of the problem is that all variables are represented as simple symbols, which [1] shows is insufficient to properly implement hygiene.

Evidently there is some ongoing work to improve the situation. This is a good example of research ideas percolating into industry languages I think.

Elixir

Elixir has robust AST macros, and its standard library makes heavy use of macros; many “core” Elixir constructs like def, if, raise, |>, and others are actually macros that expand to smaller units of Elixir.

Elixir actually gets hygiene right! Unlike Julia, variables in Elixir’s AST have metadata—including scope information—attached to them. This and other aspects of Elixir’s macro system open it up to lots of exciting possibilities. The Nx library brings support for numerical and GPU programming to Elixir, and it works essentially by implementing a custom Elixir compiler in Elixir itself, and macros play a big role in this.

Me thinking that Elixir is a big mainstream language should tell you something about the languages I spend my time with in my job as a PhD student.

I think Elixir macros are really neat—they’re the most powerful I’ve seen in a “big mainstream” language.

Rust

Rust supports two kinds of macros: macros-by-example, and procedural macros.

Macros-by-example are a simple pattern-to-pattern transformation. Here’s an example from The Rust Book:

#[macro_export]
macro_rules! vec {
    ( $( $x:expr ),* ) => {     // (pattern-repeat)
        {
            let mut temp_vec = Vec::new();
            $(
                temp_vec.push($x);
            )*
            temp_vec
        }
    };
}

This macro takes a pattern like

vec!(foo, bar baz)

and expands it to a pattern like

{
    let mut temp_vec = Vec::new();
    temp_vec.push(foo);
    temp_vec.push(bar);
    temp_vec.push(baz);
    temp_vec
}

Notice how the * marks a part of the template that can be repeated. (pattern-repeat) This is akin to Racket or Scheme’s ... repetition form. Macros-by-example work on AST, but you can’t perform arbitrary computation on the AST. For that, you need procedural macros.

Rust’s procedural macros (called “proc macros”) work on a token stream, and you can perform arbitrary computation, which puts them in a bit of a funny middle ground between C and Lisp. There is a Rust crate that you can use to parse a Rust token stream into Rust AST, but you don’t get any nice source information from the AST nodes, which makes producing good error messages a challenge.

I personally find Rust macros to be disappointing.

Conclusion

There’s a wide variety of macro systems. The best macro systems:

Operate on the AST rather than on a stream of tokens
Avoid leaking implementation details through inadvertent variable capture by being hygienic
Produce good error messages that are in terms of the caller’s context
(Bonus) have good phase separation to enforce clear separation between complex macro systems

Different languages have different features in their macro systems; some languages make it easy to use macros sensibly, while for others macros are a formidable challenge to use properly—make sure you know what your language provides and the trade-offs involved.

Why shouldn’t you use macros?

Turns out you can do a lot with functions. Powerful function programming languages let you do so much with first-class functions. If you can get access to first-class continuations, as you can in Racket and Scheme, then you can create powerful new programming constructs without having to resort to macros.

I came across the JuliaCon 2019 keynote talk, where Steven Johnson explains how many of the things that you can do with macros can be solved just with Julia’s type dispatch.

If you can do something with functions, you probably should: functions are first-class values in most languages these days, and you’ll enjoy increased composability, better error messages, and code that is easier to read and understand by your peers.

Macros introduce little languages wherever you use them. For simple macros, you might not have any constraints on what you may write under the scope of a macro. As an example, consider a macro that adds a while-loop construct to a language by rewriting to another kind of looping mechanism: you shouldn’t have any restriction on what you can write inside the body of the while loop.

However, more complex macros can impose more restrictions on what can and cannot be written under their lexical extent. These restrictions may or may not be obvious. Examples: accidental variable capture limits what can be safely written, and grammatical errors (e.g. using an expression where an identifier was expected) can lead to inscrutable errors.

Better macro systems mitigate these problems. It’s not enough to just have a macro system that uses ASTs; you need a macro system that makes it easy to write correct macros with clear error messages so they truly feel like natural extensions of the language. Few languages do this right.

Why should you use macros?

Macro systems have improved since the 1960s. While Lisp excluded many of the pitfalls of C macros by construction, you still had to use kluges like gensym to manually avoid variable capture. Scheme got rid of that with hygienic macros, and Racket improved matters further by improving macro hygiene through scope sets and introducing phase separation. It is so much easier to build robust macro-based abstractions.

Macros are good—anyone can write macros and experiment with new syntactic constructs. This makes development and extension of the language no longer the sole domain of the language designer and maintainer—library authors can experiment with different approaches to various problems.

We see this a lot with Elixir: Elixir’s core language is really rather small; most of the magic powering popular libraries like Ecto or Phoenix comes from a choice set of macro abstractions. These and other libraries are free to experiment with novel syntax without fear of cluttering and coupling the core language with bad abstractions that would then need to be maintained in perpetuity.

Macros can be powerful when used correctly—something made much easier by modern macro systems.

References

[1]

Adams, M.D. 2015. Towards the Essence of Hygiene. Proceedings of the 42nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (Mumbai India, Jan. 2015), 457–469.

[2]

Barzilay, E., Culpepper, R. and Flatt, M. Keeping it Clean with Syntax Parameters.

[3]

Flatt, M. 2002. Composable and compilable macros: You want it when? Proceedings of the seventh ACM SIGPLAN international conference on Functional programming (New York, NY, USA, Sep. 2002), 72–83.

[4]

Krishnamurthi, S. 2006. EDUCATIONAL PEARL: Automata via macros. Journal of functional programming. 16, 03 (May 2006), 253. DOI:https://doi.org/10.1017/S0956796805005733.

NO_ITEM_DATA:flattBindingSetsScopes2016b

Breaking hygiene has some utility in some cases—for example, one might want to add a break form inside the body of a loop. There are ways around hygiene, but these are not without some problems. For more details see [2]. ↩

A short bleep...

Fabian's public notepad — Sun, 12 Nov 2023 23:00:00 GMT

As my family and I are looking forward to move into a new home in January, the next weeks will be filled by throwing out junk that accumulated over the years, packing up our belongings into boxes, looking for new furniture etc.pp. and I'll likely be hard-pressed for time to write — therefore, as a short bleep before I go radio silent, I want to cluster some ideas that I'd like to pick up, once I have a little more time at my disposal again.

I'd like to finish a draft about the implementation of the little web component that I mentioned previously. This one has the strongest chance to still get done this year.
There's a pile of scattered notes on microfrontend architectures, that have accumulated over the nearly eight years that I'm working on that topic professionally. I'd like to consolidate them with my personal experience and opinions on it. As that might get out of hand and maybe deserves a longer treatment, I'm thinking about experimenting with some kind of preplanned article series.
Earlier this year I mused about the great books of software engineering, but the article was a bit unconclusive. I will have to pack up my library soon and thought, when I unpack the books, I'd like to do some reviews and summaries on a selected few items.
Then, and this one is a bit of a moonshot topic, earlier this year, I took some interest in the current state of rapid application development. I'm still occasionally asking myself why, a quater of a century after the final release of Visual Basic 6, nothing has emerged that lives up to its developer experience, when it comes to making UIs and wrapping them up with a datasource. I have a some hypotheses about that, though...
The quality of the major search engines have deteriorated over the last years to the point where their results are basically garbage. The rise of LLMs has, if anything, accelerated their enshittification. Plus ça change, plus c'est la même chose... I'd like to dive into the topic of curated web directories and explore whether there are lessons to be learned from the demise of Yahoo and DMOZ and if smaller directories could somehow become (again) a backbone for finding information on the web.

If any of you'd like to talk about such stuff, feel free to shoot me a mail. Until next time, take good care :).

The most commonly imported module

ThatXliner's Blog — Sun, 12 Nov 2023 18:09:22 GMT

Not all modules are created equal

The Lean Startup

🦉 olshansky 🦁 — Sun, 12 Nov 2023 02:00:00 GMT

Olshansky's review of The Lean Startup

Unix Stackexchange Tags

Vít Baisa — Sun, 12 Nov 2023 02:00:00 GMT

a list of tags from unix.stackexchange.com

Town Hall #15: Artifacts

taylor.town — Sun, 12 Nov 2023 02:00:00 GMT

101,721 words of cryptic nonsense.

Home DNS with Unbound and NSD

Brian's Waste of Time — Sun, 12 Nov 2023 02:00:00 GMT

I recently redid the DNS on my home network, moving from dnsmasq to Unbound and NSD. Unbound acts as the DNS server the network uses, and Unbound hosts losts local zones for my search domains. For reasons, this is really across two sites: our home, and a barn we own a few miles away. Because I am a dork, I have things at both sites :-)

My network controller has a built in DNS server which assigns a local domain, a la brians-laptop.local to anything which gets a DHCP lease. I wanted to be able to respect these, but also assign a diferent domain to statically assigned things on the network, such as printers and our NAS. For these I set up a .home and .barn respectively, for things in the home and in the barn. Those zones are hosted on NSD, with a config like:

# /usr/local/etc/nsd/nsd.conf
server:
    ip-address: 127.0.0.1
    port: 53530
zone:
    name: home
    zonefile: "home.zone"
zone:
    name: barn
    zonefile: "barn.zone"

Note that NSD is only listening on 127.0.0.1 and on port 53530. It should only ever be queried from the unbound instance on the same host (which is using port 53).

The zone files referenced are just stubs, not really correct, but they don't need to be :-)

; /usr/local/etc/nsd/home.zone
$ORIGIN home. ; 'default' domain as FQDN for this zone
$TTL 600 ; default time-to-live for this zone

home.   IN  SOA     ns.home. noc.dns.icann.org. (
        16          ;Serial
        7200        ;Refresh
        3600        ;Retry
        1209600     ;Expire
        3600        ;Negative response caching TTL
)

; The nameserver that are authoritative for this zone.
                        NS      ns.home.
nas.home.       A       192.168.2.114
printer.home.   A       192.168.2.140
m0001.home.     A       192.168.2.101
m0002.home.     A       192.168.2.102

and the barn:

$ORIGIN barn. ; 'default' domain as FQDN for this zone
$TTL 600 ; default time-to-live for this zone

barn.   IN  SOA     ns.barn. noc.dns.icann.org. (
        17          ;Serial
        7200        ;Refresh
        3600        ;Retry
        1209600     ;Expire
        3600        ;Negative response caching TTL
)

; The nameserver that are authoritative for this zone.
                NS      ns.barn.
m0003.barn.     A       192.168.81.101
m0004.barn.     A       192.168.81.102
dvr.barn.       A       192.168.81.110

With NSD up and running, I set up Unbound to make use of those local zones. It runs on the same instance as NSD, and is configured ot use it for those domains:

# /usr/local/etc/unbound/unbound.conf
server:
    # This is a CARP interface, which apparently 
    # requires explicitely listing
    interface: 192.168.2.100
    interface: 0.0.0.0
    do-not-query-localhost: no
    access-control: 192.168.0.0/16 allow

    local-zone: "home" nodefault
    domain-insecure: "home"

    local-zone: "barn" nodefault
    domain-insecure: "barn"

stub-zone:
    name: "home."
    stub-addr: 127.0.0.1@53530

stub-zone:
    name: "barn."
    stub-addr: 127.0.0.1@53530

forward-zone:
    name: "local."
    forward-addr: 192.168.1.1

Unbound is set up to listen on all interfaces, but because I am using a CARP interface as well, it seems to require seperately listing that one to avoid confusions. The two local zones are configured to be insecure (no DNSSEC) and to forward to the NSD instance as a stub-zone for each domain, respectively.

The dynamically assigned .local domain is forwarded to the router to pick up the names of things which get DHCP leases via the forward-zone block.

Finally, to make it all work, I hand out three search domains on DHCP, .local, .home, and .barn. This is DHCP code 119 with a value local,home,barn, for future reference.

Deployment wise, this is running on two servers in each location, with each server running both NSD and Unbound. The CARP interface is used to provide a single IP address for the DNS servers, and the DHCP server is configured to hand out that IP address as the DNS server for the network.

Fun fact along the way: I learned emacs has a mode for zone files which automatically increments serial when you save. Handy!

Making Transmitic Faster...By Doing Less!

Andrew Shay's Blog and Digital Garden — Sun, 12 Nov 2023 02:00:00 GMT

What is Transmitic & The Prototype Phase

Transmitic is an encrypted, p2p, file transfer and sharing program. I created it because, while there are many file transfer programs out there, there was nothing quite like Transmitic.
The main difference is in the UI. You add users you want to share with and which files and folders each user can download from you. Then users can download those files whenever they want. Completely p2p.

A few years ago, I wanted to learn a new programming language and decided on Rust. I then decided my first project with Rust should be Transmitic.
Learning Rust was quite difficult at first, and when combined with creating a new p2p application, it was a lot to try and handle after work.
Therefore, I could not try to optimize, cache, or really architect anything.
In order to not get overwhelmed by the project, I decided that I would just learn, and code, little bits at a time, without being concerned about code quality (except for the crypto) or performance.

This worked out well because I have been able to get Transmitic to a very functional (yet still Beta) state. It now includes all the primary features I wanted it to have.
There is still much more I want to do though.

Until very recently, it was really more in an alpha/prototype state. There was wasted bandwidth and things that could be cached, but weren't.
I will discuss some of the enhancements that helped make Transmitic faster and use less bandwidth, which can be summarized as "do less!"

None of this is novel. It's all obvious, and if you're able to truly focus on a project and plan it, you won't hit these.
I just couldn't think about it at the time. Once the feature worked, ship it!

Overview of Features

A quick overview of the features that will be discussed.

The protocol is simple. It sends u16 messages to instruct the action to be performed, with a u32 payload size, followed by the payload itself.
File Listings are JSON strings. They contain all file paths, and their sizes, that are shared with an individual user.
To determine when a file download is complete, the client sends the file path and current file size on disk to the server. The server sends more chunks of data or a download complete message.
"Reverse Connection" is a feature, that when enabled, Transmitic will periodically connect to each user (outgoing) to establish a connection for sharing. This feature is for users with networking limitations that cannot accept incoming connections.

Cache File Listings

When a user refreshes files that are shared with them, they download JSON data containing the file paths shared with them and the file sizes.
This is the "File Listing".
Initially, when a user started downloading files, the file listing was requested again. This ensured they had the latest, valid, list of files shared with them.
The client could then determine locally if the file was invalid or not (of course, the server would still reject any invalid files that were requested).
However, this is wasted bandwidth. Especially if the file listing was large, like 50MB. A user downloads 50MB worth of JSON to see what's available, and once they start downloading, they immediately download 50MB again.
Wasteful!

The same scenario occurred with the Reverse Connection feature.
A user with Reverse Connection enabled, periodically connects to everyone they share with, and "hands off" that connection to allow file listings and downloads to occur.
Because the File Listing was not cached, every Reverse Connection would trigger the file listing to be downloaded.
Therefore, if you were sharing with anyone who used Reverse Connection, they would continually trigger your client to download the File Listing even if you already had the latest File Listing!
Wasteful!

Let's cache it!

Reducing this waste is simple. We just need to cache the File Listing, ensure the appropriate threads have access to it, and determine when we need a new copy.

When a client starts downloading files, instead of requesting a new File Listing immediately, first it checks if it already has one. If it does, it attempts to download the file.
Now, it will only get a new File Listing if the server responds with the file choice being "Invalid" i.e. Not shared with you.

The Reverse Connection logic now maintains a list of users who have the latest File Listing. This list is reset when the user changes which files they're sharing.
When the Reverse Connection is established, there is now a new message that can be sent, MSG_REVERSE_NO_LIST.
This tells the client a Reverse Connection is being established, but the server thinks you have the latest File Listing.
The client does not need to request a new one.

If the Client Can Do It Alone, Do It Alone

When a user downloads a directory, the client loops through every sub file in the File Listing and requests it from the server.
If the download is interrupted, it can continue where it left off.
It would NOT redownload every file, but it would start from the first file in the directory and loop through them all again.
Originally, the logic was simple. The next file is pulled from the file listing, check its size on disk, and send it to the server which would respond.
This would happen for files that were already finished. The client would tell the server the current size and the server would respond, download finished.
Wasteful!

We have the File Listing locally, which already contains the expected file sizes!
Now, when the client loops through the File Listing again, if the file size on disk matches the File Listing, it just moves onto the next file.
The client does not contact the server for finished files anymore!

After that change, a bug was introduced. I noticed it when transferring some test directories.
0 byte files.
When getting ready to process a file for download, the client sets the current size on disk, but if the file does not exist, it sets it to 0.
That, combined with "if the size on disk matches the File Listing, move on to the next file", meant 0 byte files were not being created anymore. Oops!
That was an easy fix though. If the File Listing says a file's size is 0 bytes, the client just creates the file locally. There is no need to ask the server anything!

Redundant Server Messages

Originally, the server would send a MSG_FILE_CHUNK message which contains a chunk of file as the payload.
When the file completed, the server would then send a MSG_FILE_FINISHED message with no payload.
This means every file download ended with an extra message, MSG_FILE_FINISHED. And it has no data...
Wasteful!

So, I extended MSG_FILE_FINISHED to also contain a payload.
Now, the last chunk of file would be MSG_FILE_FINISHED instead of MSG_FILE_CHUNK.
I was able to remove the pointless extra message of MSG_FILE_FINISHED with no payload.

I also uncovered another wasteful message.
The code looked like this

bytes_read = -1
while bytes_read != 0:
    bytes_read = read_file_into_buffer(buf)
    send_to_client(MSG_FILE_CHUNK, buf)

The intent is to keep sending chunks until nothing left was read from the file, i.e. Everything has been sent.
However, while bytes_read != 0 means loop UNTIL bytes_read == 0, which means we need a pass through the loop that has read nothing.
This causes the server to send a MSG_FILE_CHUNK with a payload of nothing!
The server tells the client, "Here's a 0 byte chunk of your download."
Wasteful!
So, I refactored the loop and the inner logic to not send this useless message.

In Summary

Cache data where you can so you do not have to rerequest it
Look for useless communication. i.e. Data with no payloads
If you can operate with only local data, do not contact the server

It's always fun to make these optimizations! Big or Small.
Not worrying about them from the beginning allowed me to keep working and deliver value to users.
Thinking about everything from the beginning would have been too much, burned me out, and I would have given up. I was already close to giving up as is, but I'm so glad I didn't!

https://transmitic.net/
https://github.com/transmitic/transmitic/

Essential keyboard shortcuts (Windows/macOS/Linux/UNIX)

Jason Eckert's Website and Blog — Sun, 12 Nov 2023 02:00:00 GMT

Following are some shortcuts that I believe all desktop users should be familiar with. It is not a complete list by any means, and does not include virtual desktop or app pinning shortcuts as I felt those were beyond the scope of common desktop shortcuts. If you feel I omitted a common shortcut, email me (jason.eckert@trios.com) and I’ll add it to the list!

1. Windows shortcuts

The Windows (⊞) key on your keyboard has the Windows logo printed on it. Modern keyboards use the newer Windows logo with four equally-sized squares, while older keyboards use a Windows logo with four slanted or curved squares.

Using a Markov chain to generate readable nonsense with 20 lines of Python

Ben Hoyt's technical writing — Sat, 11 Nov 2023 18:30:00 GMT

Describes a simple Markov chain algorithm to generate reasonable-sounding but utterly nonsensical text, and presents some example outputs as well as a Python implementation.

Roast #32: Coffee Club V5

thoughtfulcoffee — Sat, 11 Nov 2023 17:21:09 GMT

Orange (Awesome) Blossom, Peaches, and Funk.

A list of the best bug write-ups I've read

Blog by Jade Michael Thornton — Sat, 11 Nov 2023 17:00:00 GMT

A modest list of interesting blog posts containing tech mystery stories exploring how a bug or mistake came about, how it was discovered, and how it was fixed.

Javascript Promises

Thought Eddies — Sat, 11 Nov 2023 13:50:58 GMT

Javascript Promises

Kubernetes Liveness Check

Thought Eddies — Sat, 11 Nov 2023 13:38:52 GMT

Kubernetes Liveness Check

Java agent to unfinalize class

Random Notes by agilob — Sat, 11 Nov 2023 02:00:00 GMT

That one boring Saturday I wanted to learn something more about agents and thought it would be a really cool idea to “unfinalize” java.lang.String. I started working on the project, developed simple transformer: if (name.equals("java/lang/String")) { ClassPool classPool = ClassPool.getDefault(); final CtClass ctClass; try { ctClass = classPool.get(name.replace("/", ".")); } catch (NotFoundException e) { throw new RuntimeException(e); } int modifiers = ctClass.getModifiers(); if (Modifier.isFinal(modifiers)) { System.out.println(name + " modifiers: " + modifiers); ctClass.

Native memory leak in a cloud environment

Random Notes by agilob — Sat, 11 Nov 2023 02:00:00 GMT

A Java project with published container image that contains intentionally leaky native code to observe symptoms of a memory leak in Java in podman/docker or Kubernetes. Native code intentionally “leaks” provided number of megabytes in a loop. The project runs by default with -XX:NativeMemoryTracking=summary enabled. I wanted to observe how JVM will report native memory, crash and what pod and JVM metrics will look like. Java doesn’t allocate many objects, almost none.

20 Years of Wasting Time!

Brian's Waste of Time — Sat, 11 Nov 2023 02:00:00 GMT

I cannot let 2023 go past without writing anything given it is 20 years of this blog, but an update to Hugo broke things, and getting it to work was just annoying. So I got frustrated enough making Hugo do what I wanted that I ported over to Zola and here we go, I am able to update again!

Along the way I took a moment to add all the posts from my original blosxom based blog to the archive, which makes me happy. Really, I want to post about the home DNS setup I did, but that will have to wait until tomorrow :-) For now, I want to push this and see if the deploy works!

Happy 2023!

Cynical PM Framework, Live at Lean Culture

Frank Tisellano | ft.io — Sat, 11 Nov 2023 02:00:00 GMT

Thanks to Sean Murphy and Theresa Shafer for inviting me to speak on the Cynical PM FrameworkThe Cynical PM Framework, a business-first approach to productAs PMs, of course we have a responsibility to our users — but we’re primarily responsible for advancing the goals of our business. In this post, I posit a simple framework for identifying the different strategic roles your product might play and how to prioritize, position, and measure success once you’ve identified that role.. Had a blast sharing and discussing it live for the first time. Fantastic insights from Sean & co, and great questions from the whole Lean Culture crew.

Video and transcript inside.

Stop deploying web application firewalls

Mac's Tech Blog — Sat, 11 Nov 2023 02:00:00 GMT

I wanted to write this because I don't hear enough real people discouraging the use of Web Application Firewalls (WAFs). Probably because the search results for "Web Application Firewall" are all written by WAF vendors. Anyone reading just that could conclude that WAFs are a good idea. I'm here to offer another perspective, after having suffered through using a WAF for two years.

Web Application Firewalls were created early in the Internet's history, especially popularized by the ModSecurity project in 2002. WAFs essentially work by intercepting every single HTTP request (and sometimes responses too) and evaluating several hundred regular expressions over the URI, headers, and body, sometimes aided by machine learning. If the request kinda looks like SQL, shell code, etc., the server may block your request.

In the infancy of the cybersecurity field, WAFs seemed like a good idea. HTTP requests were tiny, infrequent, and mostly contained mundane form data. But today, WAFs have overstayed their welcome in the security toolbelt. There are better techniques you can use that make even the most advanced WAFs entirely obsolete.

Disclaimer update 2024: In this article, I'm objecting to using WAFs for their original purpose which was to mitigate common exploits like remote code execution, SQL injection, etc. With the rise of DDoS attacks, some cloud WAFs such as AWS WAFv2 or Cloudflare's WAF are now the recommended way of implementing rate limits or fingerprinting and blocking denial of service attacks, which is a use-case I fully support.

WAFs have Horrible Performance

Since WAFs run hundreds of regular expressions on every request, you may ask, "isn't that super inefficient?" Yes, very.

	WAF	No WAF
Average time taken to upload 9,462 text files	7.36	4.55
Average requests per second	1285	2079
Number of requests blocked erroneously	5	0
Peak nginx CPU during trial	73%	8%

Specifics about the benchmark

The easiest way I know to get modsecurity + CoreRuleSet installed is through ingress-nginx, which I've installed in a Kind cluster.

# https://kind.sigs.k8s.io/docs/user/quick-start/
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 32080
    hostPort: 32080
    protocol: TCP
  - containerPort: 32443
    hostPort: 32443
    protocol: TCP
EOF

# https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/modsecurity/
helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=NodePort \
  --set controller.service.nodePorts.https=32443 \
  --set controller.service.nodePorts.http=32080 \
  --set controller.ingressClassResource.default=true \
  --set controller.allowSnippetAnnotations=true

For the test, I'll be uploading files to MinIO using these values:

replicas: 1
mode: standalone
resources:
  requests:
    memory: 512Mi
persistence:
  enabled: false
rootUser: rootuser
rootPassword: rootpass123
buckets:
  - name: bucket1
    policy: none
    purge: false
ingress:
  enabled: true
  hosts: [minio-waf.localhost]
  annotations:
    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
    nginx.ingress.kubernetes.io/enable-owasp-core-rules: "true"
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
      SecRuleEngine On
      # Even the core rules are ridiculous, blocking PUT requests, certain content-types, or any body with "options" in it
      SecRuleRemoveById 911100 920420 921110

helm upgrade --install minio minio/minio -f values.yaml -n minio --create-namespace
helm upgrade --install minio-waf minio/minio -f values-waf.yaml -n minio-waf --create-namespace
# Verify the WAF is working (should get a 403)
curl 'http://minio-waf.localhost:32080/?q=../../etc/passwd'

We'll be uploading just the "Documentation" folder of the v6.6 Linux Kernel, which contains 9462 files for a total of 65MB.

curl -LO https://github.com/torvalds/linux/archive/refs/tags/v6.6.zip
unzip v6.6.zip 'linux-6.6/Documentation/*'

Configure the minio client:

# You may need to add these hosts to /etc/hosts
export MC_HOST_nowaf='http://rootuser:rootpass123@minio.localhost:32080'
export MC_HOST_waf='http://rootuser:rootpass123@minio-waf.localhost:32080'

Run the benchmark (5 times each):

time mc cp -r linux-6.6/Documentation/ waf/bucket1/
time mc cp -r linux-6.6/Documentation/ nowaf/bucket1/

In addition to slowing down every request, you also need significant additional RAM for buffering requests. Since not a single byte in the buffer can be flushed to the backend server until the WAF completes its analysis, you need several gigabytes of RAM to store request bodies. Servers like nginx buffer requests by default, but enough large concurrent requests (like pushing a container image) can make a buffering web server run out of RAM. When using a WAF, every server becomes a buffering web server, which is simply incompatible with many types of applications.

I know computers are fast and hardware is cheap, but we shouldn't be spending that kind of CPU and RAM on WAFs unless they're a really effective security tool. But they aren't, as you'll see next.

WAFs are Easily Bypassed

WAF vendors and attackers are locked in a constant arms race, but it seems attackers are much better armed. How could they not be? Many of the attacks that a WAF purports to block involve complex grammars like SQL, shell code, and entire programming languages. They often include comments, character escaping, encoding issues, and more oddities. These oddities mean that attackers always have a significant advantage and can typically bypass any WAF rule if they are clever enough.

For example, you might think Log4shell is pretty easy to catch: just check for ${jndi, right? Unfortunately, Log4J supports nested "lookups", including ones that convert letters to upper/lower case like ${lower:J}

That means an attacker can insert an arbitrary number of nested lookups around each letter and still perform the attack, like this: ${${lower:J}ndi:.... This lead CloudFlare to say "WAF vendors need to be looking at any occurrence of ${ and treating it as suspicious", which is just another hilarious example of how WAFs can never live up to the expectations placed on them.

I just discussed the fairly simple grammar that is Log4J Lookups, but you can imagine how many more evasion tactics you could use in a language as complex as SQL or PHP, especially when considering encoding tricks. For an in-depth description of specific WAF bypass techniques, check out this awesome post.

Another way to bypass a WAF involves just padding your attack string to appear >8KB or so into the request body. Like I mentioned in the section on performance, request bodies must be buffered into RAM for analysis, so WAFs must choose some cut-off point to avoid spending infinite CPU and RAM on a single request. For some WAFs like AWS's, that cutoff point is around 8KB. So if you just put 8192 innocuous characters before your Log4Shell attack string, you've rendered the WAF worthless.

WAFs are an Attack Vector

In 2019, CapitalOne experienced a breach of 100 million credit applications that was allegedly caused by a WAF misconfiguration. The attacker allegedly tricked the WAF into sending requests to the EC2 Metadata Service, which handed out a credential that allowed reading sensitive files from S3.

While this is just one example, it illustrates the curious fact that WAFs actually have a large attack surface.

Most WAFs are giant, complex codebases that are usually closed-source and written in memory-unsafe languages. Since they're expensive "enterprise" products, companies stuff them full of unnecessary features to make them stand out more than competitors. All of this adds up to make WAFs yet another example of a dangerous "security" tool, just like SolarWinds.

No security officer would approve taking such a risky piece of software, putting it directly on the internet, making it parse mountains of untrusted input, and giving it access to all your backend servers, logging infra, SIEM, alerting systems, and even JIRA for some reason UNLESS it's covered in security buzzwords and costs 5-6 figures per year.

Somehow, companies that sell security products have gotten a pass on implementing foundational security principles like secure by default, secure by design, attack surface reduction, and the principle of least privilege. Don't let them keep getting away with that.

WAFs have a High False Positive Rate

Over the last twenty years, open-source WAF rulesets have expanded considerably to detect more-recent types of attack. Apparently all those proprietary WAFs are doing the same. That means there are more and more possible strings that could trigger a WAF to block your request. If you want to write a comment on an article discussing Log4shell, you might be blocked for including the string ${jndi in your comment. So naturally the false positive rate continues to rise with every new rule, and it's already quite high based on my experience maintaining a giant list of ModSecurity rule exceptions.

So-called "next-generation" WAFs claim to solve this problem by looking at multiple requests or by using IP reputation systems. While these can improve false positive rates, they can never truly solve the problem. In some ways, less false positives can increase the impact of particular false positives since neither users nor support teams have a clear procedure for fixing it. CloudFlare's algorithm can randomly decide to block you and you will have no recourse. Imagine that happening to someone less tech-savvy.

This is the classic problem with using an outdated security tool like a WAF: defenders have to configure the tool absolutely perfectly to be safe and avoid false positives, but attackers just need to find a single weakness. Those are horrible odds. You should use alternatives that don't require perfection from imperfect humans.

Alternatives to WAFs

Since WAFs are resource-hungry, inneffective, unsafe, and noisy, how do I convince an auditor to not make me use one? The technical term would be to use "compensating controls", but that sounds like such a weak term to describe the powerful and simple alternatives to WAFs I'm about to describe:

Isolation: Isolation involves ensuring that a breach in one component can not affect the rest of the system, and there are many technologies that provide isolation.
- Browsers do this by executing all code inside special sandboxed processes that don't have carte blanch access to cookies, saved passwords, other tabs, etc. Imagine how slow the web would be if every piece of JavaScript needed to be analyzed by hundreds of regexes before being executed!
- Microservices are designed with isolation in mind, but you can also do it in a monolith with a variety of libraries and languages.
Immutability: Entire classes of attack can be eliminated by removing a few assumptions, like having a readOnlyRootFilesystem, a package manager that requires rebooting, or append-only/immutable backups.
Static Analysis: SQL injection has a miracle cure called "prepared statements". The problem is that devs forget to use them. Static analysis checks in a CI pipeline can all but ensure that zero SQL injection vulnerabilities are in your codebase, at which point there is no need for any SQL injection WAF rules. No, "defense in depth" is not a valid excuse to use a WAF anyway, because it provides no real defense! Like surrounding Fort Knox with an army of guard guinea pigs.
Capability-based security: Not every API endpoint needs to have unrestricted read/write access to your entire database and file system, but that is the normal way people build APIs today. By using capabilities, you can express exactly that "GET /api/v1/books" only needs read access to the "books" table. Or that "POST /api/v1/imageupload" needs write access to a specific folder, but doesn't need the ability to spawn processes.

Now I'll admit these ideas are quite broad; you'll need to adapt them to your particular app. WAF vendors offer a one-WAF-fits-all fantasy that I can't match. But these secure-by-design strategies are the way that the security industry needs to be heading. Unfortunately, it's a lot harder for the security industry to profit off of design-based techniques, so don't hold your breath.

Envoy Gateway a new Gateway API for Kubernetes

Fabrice Aneche — Sat, 11 Nov 2023 01:01:00 GMT

Envoy Proxy is a well known proxy load balancer offering HTTP, gRPC, TCP, customizable with Lua, Go, WASM … It’s often the core component used to build gateways/proxies for Kubernetes. For example Istio is using it as a proxy load balancer for ingresses but also as a sidecar to create a “mesh network”. Envoy Gateway is a more recent project to manage Envoy proxies inside Kubernetes, it is meant to be a common foundation to build on top of it, but it can be used directly as it is.

2023-11-10-001

Srijan Choudhary, all posts — Fri, 10 Nov 2023 15:55:00 GMT

My new #IndieWeb enabled website is now live! You can follow my microblog posts or blog articles by entering @srijan.ch@srijan.ch in your Fediverse app search bar or reply and interact with any post using #Webmention or your Mastodon/Fediverse app. Site Features: Indieauth Webmentions Federation with the fediverse (via Bridgy Fed) Structured author, posts, and feeds using microformats Microsub …

Cheapskate beginners guide to switching from hands-on web development to WordPress

Lasse Laursen — Fri, 10 Nov 2023 09:28:46 GMT

This post is the third in a three-part series about migrating from a self-built PHP-based Laravel backend to a managed WordPress backend, in an attempt to minimize on-going maintenance and custom code. Part one discussed why moving to WordPress felt like a visit to the Ferengi homeworld, and part two detailed what motivated the switch. […]

Monsters, Aliens, and Holes in the Ground (Review)

ℤ→ℤ — Fri, 10 Nov 2023 04:26:29 GMT

For those interested in the history of tabletop roleplaying game design, Monsters, Aliens, and Holes in Ground is an excellent, near encyclopedic treatment. The author, Stu Horvath, documents major and minor game systems, how they innovated or were influenced by other game systems, and how the systems expanded with settings and adventures. Horvath covers the 1970s through the 2010s and, reading through the entries, chronicles the major changes in design theory from simulation, to forms of narrative, to today’s many experimental systems.

Stu Horvath and Kyle Patterson. Monsters, Aliens, and Holes in the Ground: A Guide to Tabletop Roleplaying Games from D&D to Mothership. MIT Press 2023. Publisher Link

I bought the standard edition and the book is weighty and beautiful. In fact, the book is a bit too heavy and can be uncomfortable to read in a chair. Over 400 pages long of heavy paper stock, almost all entries are annotated with at least one full-color image of the book being discussed. Each decade is introduced by a thematic two page illustration of the major game design thrusts by Kyle Patterson.

Organized by the publication date, the text supports a linear read-through as game systems influence later books. Horvath is fairly opinionated about what works and didn’t work within a design. This improves the readability, although, as an academic imprint, some of his arguments deserve greater support. (Horvath does pull from game journalists and many interviews with game designers.) I was pleased to finally learn the background of titles like Skyrealms of Jorune, which was advertised consistently in Dragon Magazine, but was sufficiently weird to remain obscure.

Although the book documents an extensive number of systems, it is not exhaustive. The author is open about this fact in the introduction. First, a system may not be included if he has never heard of it. Second, there are many systems (particularly foreign language) that he has not been able to acquire. Third, he excludes some systems if due to their subject matter or his disagreements with the creators. Of these, he only specifies Empire of the Petal Throne as eliminated. Thus, for some of the rare games in my collection that are not mentioned in the book, I do not know if their absence is intentional or inadvertent.

Additionally, the main threads of game design are described over many individual game entries. I think the book, and his arguments, would have been improved by a summary tying the major milestones together.

I recently visited a half-dozen game shops across three states and I was disappointed in the small RPG selection at the stores. If it’s not D&D 5e, they are unlikely to have it. In comparison, growing up, every store stocked multiple major systems and many had a deep catalog of used titles. So, I appreciate books like this that show the sheer range of the hobby and how designers have influenced and built upon each other’s innovation and experiments. Recommended.

(Suggested 📚) Elon Musk

er4hn — Fri, 10 Nov 2023 02:00:00 GMT

Then Life Continued - Suggested Reading

Getting Things Done

Vít Baisa — Fri, 10 Nov 2023 02:00:00 GMT

a personal productivity system

2023-11-09-001

Srijan Choudhary, all posts — Thu, 09 Nov 2023 21:25:00 GMT

New #Coffee #BlueTokai This new packaging from BlueTokai looks nice. And the coffee tastes amazing. Chocolatey flavour with very little bitterness.

ZMK Added the Apple Globe Key

trv.rs — Thu, 09 Nov 2023 19:09:00 GMT

I recently noticed that ZMK added the "globe" key from the Apple ecosystem to their key codes.

I threw it on my Corne-ish Zen and now I have access to a bunch of useful power user shortcuts on iPad and iOS. I particularly like starting dictation or going to the previous app.

I am using it as a sticky modifier, so I press it once and then press the next key. That lets me tuck it up in a layer, but you can use it however you want.

Here's my commit.

Tech Layoffs 2023

Mitrapunk: Engineering Game — Thu, 09 Nov 2023 14:34:06 GMT

Billionaire property developer Tim Gurner said that "We need to see unemployment rise. We need it to rise 40-50%.

Prioritizing Features: A Game That Teaches Real-World Product Management

Mitrapunk: Engineering Game — Thu, 09 Nov 2023 14:27:07 GMT

Explore 'FeatureFront,' where gaming meets product management. Prioritize features, balance stakeholder demands, and use data to drive decisions, all within a game that simulates real-world tech strategies

How to use XmlWriter along with StringWriter to properly serialize a UTF-8 string

Nicola Iarocci — Thu, 09 Nov 2023 08:05:25 GMT

Today, I (re)learned how to serialize an XML to a UTF-8 string. Like all the other times I did this, I got backstabbed by StringWriter, which only supports UTF-16. A simple code snippet like this:

    await using var sw = new StringWriter();
    await using var w = XmlWriter.Create(sw, new() { Async = true });
    ...
    await w.FlushAsync();
    return sw.ToString();

Will emit this output:

    <?xml version="1.0" encoding="utf-16"?><...

There’s nothing inherently wrong with UTF-16, but XML is usually UTF-8, so one must do something about it. StringWriter exposes an Encoding property, but it is read-only for unknown reasons. One might think that given that the XmlWriter allows setting its own Encoding value, something like this would work:

My go-to 'make code go faster' ideas

jodavaho.io — Thu, 09 Nov 2023 02:00:00 GMT

I’m not an optimization guru by any means. It’s never been something I’ve been allowed to focus on at work, sadly. At some jobs, performance is secondary to correctness and robustness, and at others, it’s secondary to flashy features.

But, I’ve used the following tricks in hotloops

Dimension reduction (esp via convolution)
Branchless calculation
SIMD

SIMD gets a lot of love, but it’s a constant-factor improvement and can be tough to coax out of the compiler (unless you use a library).

Branchless calculation is getting less relevant as a “trick”. A really dumb example is if you have to do this:

// Example 1
if (x>=0) {
    return x*y;
} else {
    return x*b;
}

Then you can speed it up by doing this:

// Example 2:
int c = (x>0);
return x*(c*y + (1-c)*b); // either x*y or x*b;
//better: x*(c*(y-b) + (b));

And if you can combine this with unrolling and SIMD, then you’re really well off. The second example works just as well if those are Eigen::VectorXd length 1000, and now you’re really cranking out the speed.

Something like this:

// Example 3
for ( i .. N )
{
  if (x[i]>=0) {
      out[i] = x[i]*y[i];
  } else {
      out[i] x[i]*b[i];
  }
}

Becomes trivial, and looks exactly like //Example 2

But even more fundamentally, those usually need to come after:

Using the right data structures
Using the right algorithms
Reframing the problem for speed, e.g., giving up a little to go a lot faster
Memoization/caching/pre-calculating

These can usually get me 80% of the way there. When they all come together, you’re staring at your machine wondering how it is done already.

I rarely see great returns on multi-threaded / multi-process, unless you’re doing rote calculations on a large dataset in a batch.

Monotonic Attention

Ben Bolte — Thu, 09 Nov 2023 02:00:00 GMT

Idea for an implementation of monotonic attention using a probabilistic graphical model.

Easily Generate Mock Data with PostgreSQL

Dylan Paulus' Blog — Thu, 09 Nov 2023 02:00:00 GMT

import { Accordion } from "../../../../components/Accordion";

When writing articles about PostgreSQL there are times I need to generate large swaths of data to test and demonstrate a topic (for example, these Timescale articles here and here).
It can be a pain to manually insert data or create a CSV and import it into PostgreSQL--which I always have to StackOverflow.
Lately, I ran into a much simpler solution to generating data: the generate_series function.
It's perfectly suited for integers and timestamp data types, streamlining various tasks such as populating test databases or setting up date ranges for reports.

Generating Mock Data

generate_series can be thought of as a for-loop or generator to produce a range of numbers or dates.

To generate a range of numbers, we call generate_series with a start and end value:

SELECT * FROM generate_series(1, 10);

We can specify a step value by providing generate_series as an optional third argument.

SELECT * FROM generate_series(1, 10, 2.5);

Inserting Mock Data

generate_series' power comes to fruition when we need to insert mock data into a table.
We can easily insert a range of numbers into a table by using generate_series in a SELECT statement and then inserting the results into a table.

For the rest of this article, we'll use users table that looks like this for our examples:

CREATE TABLE users (
 id serial PRIMARY KEY,
 name text NOT NULL,
 created_on timestamptz
);

Then, if we wanted to insert five users into the table we could run:

INSERT INTO users (name)
SELECT
    'Dylan'
FROM generate_series(1, 5);

This pattern can be used to insert any number of rows into a table.
If you're feeling brave, try inserting 1,000,000 users into the table by changing generate_series(1, 5) to generate_series(1, 1000000)!

Up to this point we are creating a lot of duplicate data.
What if we wanted to create a bunch of users with all distinct names?
We can access the current index of the loop (using as) to generate a unique name for each user.

That would look like this:

INSERT INTO users (name)
SELECT
    'Dylan number ' || i
FROM generate_series(1, 5) as i;

Creating Time-Series Data

generate_series is incredibly effective at generating time series data.
This is where I get the most use out of it because it can become extremely tedious to create a lot of data around time ranges.

Similar to the integer example, generate a range of timestamps by providing a start and end value followed by an optional step or interval.

For example, to generate a user for every hour in a week, we could run the following SQL:

INSERT INTO users (name, created_on)
SELECT
    'Dylan',
    time_hour
FROM generate_series(
    TIMESTAMPTZ '2023-11-01',
    TIMESTAMPTZ '2023-11-07',
    INTERVAL '1 hour'
) as time_hour;

This query creates 146 rows with a user named Dylan every hour between November 1st and November 7th, 2023.

You may be asking yourself, can we use multiple generate_series to populate multiple fields? Yes!
But, we need to use joins to combine the data from the two series.
To quickly create a grid of users and created_on dates we can take the cartesian product of two series, otherwise known as a cross join, by adding another generate_series to the FROM clause.

A cartesian product is the result of joining two tables where the result is the combination of every row in the first table with every row in the second table.

For example, let's say we have two pants (blue jeans and sweatpants) and three shirts (t-shirt, polo, and sweatshirt).

The cartesian product of our pants and shirts would look like:

| Pants       | Shirts      |
| ----------- | ----------- |
| Blue Jeans  | T-Shirt     |
| Blue Jeans  | Polo        |
| Blue Jeans  | Sweatshirt  |
| Sweatpants  | T-Shirt     |
| Sweatpants  | Polo        |
| Sweatpants  | Sweatshirt  |

INSERT INTO users (name, created_on)
SELECT
    'Dylan number' || i,
    time_hour
FROM
    generate_series(1, 5) as i,
    generate_series(
        TIMESTAMPTZ '2023-11-01',
        TIMESTAMPTZ '2023-11-07',
        INTERVAL '2 days'
    ) as time_hour;

In the INSERT statement above, we'll create four time_hour timestamps for every two days in the week for every iteration of i (there are five of them).
Resulting in 4 * 5 = 20 twenty rows created.
To better grasp what's happening here, it'll be easier to look at the SELECT output of the users table.

Using multiple generate_series functions combined with a cartesian product can be a powerful combination for creating unique, time-series data.

Conclusion

PostgreSQL's generate_series is a game-changer for developers like me, who frequently dive into the deep end of data generation for testing and optimizing queries. It's a powerhouse that can effortlessly create anything from a handful to millions of rows, transforming a potentially arduous task into a few simple lines of SQL.
So, when you need to mock up data, let generate_series be your go-to solution.

Happy data generating!

Need a realistic dataset to test something more interesting than a flat table? Postgres: The Graph Database You Didn't Know You Had uses generate_series to populate nodes and edges, then walks through recursive CTEs against the same data. For performance work on bigger tables, Explaining PostgreSQL EXPLAIN covers how to read query plans once the rows pile up.

Binding a C++ Library to 10 Programming Languages 🔟

Ash's Blog — Thu, 09 Nov 2023 02:00:00 GMT

Binding USearch to 10 languages reveals the friction in each ecosystem—from Python's simplicity to Java's complexity—and why C++ still rules performance-critical code.

Panic

Backendhance — Thu, 09 Nov 2023 02:00:00 GMT

Hi,

Die Nervösität zerschnitt den Raum. Ich sah es in den Augen aller Teilnehmer.

Eigentlich war es nur ein Regeltermin.

Einmal die Woche kommt die Entwicklung mit den Admins zusammen und bespricht aktuelle Themen.

Der Termin wird seit Jahren so gemacht.

Gleiche Zeit, gleiche Menschen, (fast) gleiche Themen.

Aber etwas war anders.

Es nahm zum ersten Mal ein neuer Kollege Teil. Ein Externer.

Wir haben ihn zu uns geholt, um an “Devops” zu arbeiten. Also, hauptsächlich Automatisierung. CI/CD, Provisionierung, Tooling, Compilezeit-Optimierungen. Alles was die Developer Experience verbessert.

GopherConSyd 2023 - All slides, notes and links

Ben E. C. Boyter — Thu, 09 Nov 2023 02:00:00 GMT

Welcome. If you are reading this you either follow me on RSS, watch my blog or were directed here at the start of my GopherConSyd 2023 talk about “Bloom Filters: Building a Cutting Edge Go Search Engine to Explore the World’s Source Code”. Whatever the case may be, welcome!

For the slides visit https://boyter.org/static/gophercon-syd-presentation-2023/ then just use the space bar to advance, the arrow keys to move around or the ESC key to see everything and then enter to select a slide. Press S to see the speaker notes, which are more or less the script I will be sticking to.

Pain is information

Steph Ango — Thu, 09 Nov 2023 02:00:00 GMT

As a child, you touched something hot, and it burned you. That pain gave you a piece of information: be careful touching hot things.

When you sign up to run a marathon, you are signing up for pain. But whether or not you keep running is up to you. It’s been said that “pain is inevitable but suffering is optional”. You can choose pain without choosing suffering.

Pain is information, and information is painful. Rewiring your brain is not a frictionless process. Some knowledge can only be discovered the hard way.

The sooner you can convert pain into knowledge, the sooner you can experience the next useful pain. Don’t put yourself through the same pain too many times. To access new information you must experience new pains.

They say “knowledge is power”. If pain is information, then pain can be converted to power. To do so, you must learn to control your suffering, accept that scars are beautiful.

If you felt pain, ask what information it gave you. If that information was useful, seek the next pain. You are learning.

A virtual machine approach to fuzzing interfaces

Noah Watkins — Thu, 09 Nov 2023 02:00:00 GMT

Clang’s libfuzzer can design adverserial programs.

Love as a place

An Ordinary Disaster — Wed, 08 Nov 2023 23:49:54 GMT

a clutch of roses dropped in a Capp Street crosswalk

A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox

trv.rs — Wed, 08 Nov 2023 20:23:00 GMT

The title says it all.

The essay says more… and rings true with my experience.

I had written an essay about Ivan Illich and systems thinking, a topic I had never found anyone else intrigued by, and which magazines thought below a rejection letter—and the internet had suddenly reshaped itself so that my keyboard hooked up to the screens of a bunch of people who wanted to talk about these topics, and a little later, their keyboards hooked up to mine.

I had written for 15 years, but never before had this happened. I had conjured a minor conference! And I hadn’t even known that you could do that.

Write and publish. The more specific and true to you the better. If you have a same path of conversation twice with different friends… that would probably be a good blog post.

A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox - www.henrikkarlsson.xyz

Where To Go For Answers To Flutter Questions

Stories by Fred Grott on Medium — Wed, 08 Nov 2023 14:35:13 GMT

AI generated stuff has flowed into several QA forums including stackoverflow.

Continue reading on Medium »

🇩🇪 My Solo Journey Through Germany

James Carr — Wed, 08 Nov 2023 02:00:00 GMT

I recently returned from a week-long solo adventure in Germany, a trip that came about quite unexpectedly. Earlier in 2023, in my eagerness to reengage with speaking at conferences, I submitted a series of talk proposals to various events and one of my submissions was accepted at RabbitMQ Summit. However, it was only after this exciting acceptance that I realized the conference was in Berlin, not in the United States.

The Hitchhiker's Guide to the Galaxy

Vít Baisa — Wed, 08 Nov 2023 02:00:00 GMT

a classic sci-fi comedy novel dealing with deep philosophical questions

You shouldn’t use Lambda layers

AJ Stuyvenberg — Wed, 08 Nov 2023 02:00:00 GMT

AWS Lambda layers can help in certain, narrow use cases. But they don't help reduce overall function size, they don't improve cold starts, and they leave you vulnerable to a particularly nasty bug.

Note 122

Qubyte Codes — Tue, 07 Nov 2023 21:50:39 GMT

Hello #IndieWeb enthusiasts. I’m holding an unofficial Homebrew Website Club at the Lord Nelson Inn near Brighton station tomorrow evening. I’ll be there around 17:30, wrapping up around 20:00. #ffconf folk most welcome!

I’ll post more information when I’ve found a spot, but if you look for a balding dude with a beard and a dubious waxed moustache the odds are good that you’ll find me.

Snyk Language Server in Emacs

Blog by Jade Michael Thornton — Tue, 07 Nov 2023 17:00:00 GMT

Setting up the Snyk Language Server for use in Emacs by defining a custom server connection in LSP Mode

ChatGPT Wrote my Shell Prompt

Zac Szewczyk — Tue, 07 Nov 2023 05:04:30 GMT

I have, for years now, wanted a cool shell prompt — something worth showing off on r/linux or even r/unixporn. Not enough to dig into the weeds and figure out how to write one myself, but enough that it came up every once in a while. Then, the other day, I had a great idea: why not just let ChatGPT make it for me?

Permalink.

Anchor Tags

Weblog on marginalia.nu — Tue, 07 Nov 2023 02:00:00 GMT

I’ve been working on getting anchor tag keywords into the search engine, basically using link texts to complement the keywords on a webpage.

The problem I’m attempting to address is that many websites don’t really describe themselves particularly well. As Steve Ballmer’s stage performance once illustrated, merely repeating a word doesn’t on its own make what you’re saying relevant to the term.

Another good example of how it falls short is PuTTY’s website, which will be used as a pilot case to improve.

Openai Assistant Example

rrich.io — Tue, 07 Nov 2023 02:00:00 GMT

Video

Flutter Docs Missing DevOPS

Stories by Fred Grott on Medium — Mon, 06 Nov 2023 18:28:54 GMT

Google GDE’s are not telling you this but their team builds flutter apps on fast non Apple hardware but then the final release builds…

Continue reading on Medium »

Fix iPhone not sending/receiving text and photos with Android devices

Mobile Software Engineer in NYC. — Mon, 06 Nov 2023 10:06:33 GMT

I recently helped a family member fix an issue they were having with their iPhone 15 Pro, they were unable to send or receive any photos (SMS/MMS) if they were texting a person that has an Android device. If they disabled iMessage, then the photos were being sent as HTML code to the Android device.

I was one such person affected, I would text them photos, and on my end, it shows up as being sent. They never received it and I was left wondering why they weren’t responding to me. If they were trying to send a photo, then iMessage just showed a “not sent” error but no details as to why.

This family member wasted a bunch of time talking to Apple Customer Support who pointed their fingers at Verizon, saying it is a carrier issue. They would then give Verizon a call and explain things over again, only for Verizon to say it is an Apple bug and to wait for a new iOS update. New iOS updates would come and the issue would still exist.

Spoiler alert, it ended up being a Verizon issue.

After Googling various phrases related to the issue and following the usual troubleshooting steps:

Disable iMessage and Facetime: This didn’t solve the issue. This is when photos would arrive as HTML to Android devices.
Reset network settings: This didn’t solve the issue.
Double check iMessage and Facetime contact information, ensuring their phone number was selected: This didn’t solve the issue.
Remove old devices (phones and Macbooks) from their Apple ID: This didn’t solve the issue.

Since this family member uses Verizon’s service, what finally fixed the issue of sending and receiving photos with Android devices was to install the My Verizon App.

Install the My Verizon App
Log into your account (might need to log in with the owner account)
Scroll down to manage the iPhone that is having issues and select it
Scroll down to find block specific services
Make sure Block Picture & Video Messaging is not enabled, as that won’t allow sending/receiving the photos/videos from Android devices.

For an unknown reason, the Block Picture & Video Messaging was enabled on this family member’s device. So once we disabled it, forced quit the app, and then double checked it was still disabled, everything started working as expected.

If you are having this issue with AT&T or T-Mobile as your carrier, I would check to see if their apps have a similar functionality.

Maybe don't call someone an asshole?

Greater Dan Or Equal To — Mon, 06 Nov 2023 09:05:36 GMT

Arguments that feel good to make generally aren't very persuasive.

Kids Treasure Hunt

Backendhance — Mon, 06 Nov 2023 02:00:00 GMT

Hi,

Für eine Schatzsuche für einen 5 Jährigen möchte ich einen Text schreiben. Der Text soll von einem Piraten geschrieben sein, der ein Rätsel beschreibt wo der Schatz des Piraten versteckt ist. Er soll an einem Ort versteckt sein wo gruselige Bäume sind. Kannst du mir den Text schreiben und ihn reimen lassen?

Mit diesem Prompt habe ich ChatGPT gefüttert. Ich war gerade aus Berlin von der RabbitMQ Summit zurück .

Accessibility is a requirement, not a feature

ntietz.com blog - technically a blog — Mon, 06 Nov 2023 02:00:00 GMT

Stop me if you've heard this one before: "We're putting accessibility (features) on the roadmap." Or this one: "We don't need to make it accessible since we don't have any blind users¹ ."

It belies an attitude that's all too common in the software industry: That accessibility is something you can build once and be done with. That it's an extra feature, not something core to a product. That it's optional, a business decision, to make your product accessible.

Just as with security, this is a misunderstanding of the nature of accessibility. Security is something you have to always think about, and always work on; you are never "done". And the same with accessibility.

Learning firsthand about accessibility

For most of my life, I had not needed accessibility tech² . I could use products on the "happy path" of good mobility, vision, and hearing. Almost everything was accessible to me, because I had the abilities that designers expected of users.

Then my arms and hands failed me. In the summer and fall of 2022, I developed nerve pain in my arms. If I typed for more than a couple of sentences, I would get this strong pain in one of my arms and hands, and I always had weird nerve sensations. I didn't recognize what it was at first, because it first felt like I'd just scraped the skin. The pain developed fairly quickly: my arm felt funny on a walk with the kids, then in the evening I was typing and it hurt, and by the morning I couldn't drive because the steering wheel vibrations were intensely painful.

I'm a software engineer, so typing has always been core to my ability to do my job. I had previously said that my hands were my most valuable body part, probably after my brain. So when I was unable to type, I was pretty scared. What would happen to me, and my family, if I couldn't type and couldn't do my job?

Enter: accessibility. I invested in learning to use Talon so that I could write code again, write Slack messages again. It was slow, it interfered with my thinking since it wasn't natural, but it was so empowering to be able to produce something without physical pain.

For a little while it was just the one arm, but when the second one developed pain as well I could no longer use my mouse without pain. Now I would have to use the keyboard to navigate everything. I'm about to find out firsthand just how inconsistent accessibility is. Some software worked very well for me, including terminals and our own product. Other software, like a well-known issue tracker, was all but unusable without a mouse and required very different workflows to work around it. Most software was somewhere in the middle, with a lot working but some commonly used features just failing; I still don't know how to go back and edit a specific message in Slack without a keyboard, only the most recent one.

Accessibility affects everyone

I'm not alone in my brush with accessibility. If you've ever broken an arm or a leg, you know how hard it can be to interact with the world with limited mobility. If you've lost your glasses and stumbled around at dawn on the running trail³ , you know vision impairment can make everything harder to use.

There are plenty of statistics out there on how many people are disabled, and how many will be temporarily disabled. That's there, a search away. Right here, though, is an argument from our humanity.

We all go through life with a tenuous relationship with our abilities. As my loss (and regaining) of typing ability showed me, we can lose some of our abilities on a moment's notice, and this loss can be temporary. Each of us may become disabled at any point in the future, without warning. Sometimes it's temporary; sometimes it's permanent.

Accessibility isn't optional

"People with disabilities" isn't a demographic you can choose to ignore. In many ways, it's not a demographic⁴ , but a shifting subset of the population. But if you do choose to ignore people with disabilities, you're hurting everyone.

If you think you don't have users with disabilities, you... might be right, in the worst way. Your software might be so inaccessible that users with any sort of disability cannot use it. That was nearly my experience with an issue tracker; it was almost impossible to use, I had people move issues for me sometimes. But that doesn't mean that no one with a disability tries to, wants to, or should be able to use your software.

There's a famous story about planes in World War II where we were armoring the wrong parts of them. We were seeing only the ones that survived, so we assumed that where there weren't bullet holes, we didn't need to add armor. But that's exactly where we needed more armor, because a hit in that area would down the plane entirely.

This is the same situation in software. If you're looking at the population as a whole and a particular segment of it is not represented in your userbase, it might be your fault. It means that something about your software isn't working for those people. If you have no users with disabilities, then... they probably can't use your software if they wanted to.

It's kind of silly to say "we have no blind users, so we don't need to make the software screen-reader friendly." If you never do the work, they'll never be able to use it.

On top of all this, accessibility is required by law for a great deal of software. I'm not a lawyer, so that's just about where I'll leave it, but be aware that in the US and other countries, you may be open to lawsuits if you don't make your software reasonable accessible.

A requirement, not a feature

Accessibility is something that may affect each of us. We have brushes with it throughout our lives. And it's something that isn't optional.

It's not something we can put on our roadmaps once and be done with. It demands continued effort throughout the entire process of software development, just as with security and performance. Accessibility is something you work on throughout the life cycle of your product.

Don't get me wrong, accessibility can have features. There are sometimes features you can add to make software much more accessible in particular ways. Just, accessibility as a whole isn't a feature, but a requirement for development. It's part of each feature you work on.

Making a feature accessible is just one of many aspects of the work to complete a feature. You have to make that feature secure, and performant, and accessble. It's not something you can delay or choose not to do; it's something you must do as part of routine development.

If you want to do the right thing, make sure you add accessibility requirements as part of the completion criteria for things you work on.

Testing against every Next.js canary release

Articles by François Best — Mon, 06 Nov 2023 02:00:00 GMT

How to run a GitHub Actions workflow when a new pre-release version of Next.js is published.

Catitecture

Home on Lagomorph — Sun, 05 Nov 2023 21:46:00 GMT

Contra Posturing

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sun, 05 Nov 2023 17:00:00 GMT

With recent events in the Middle East escalating and spiralling further, I need to voice some thoughts that I held back for some time now. It all started with Russia’s invasion of Ukraine. Back then, I had already seen my social media feeds light up in frantic activity. Having endured a lot of self-appointed virology experts during the pandemic,¹ I now had to endure armchair warfare experts and historians. Given how wildly off the most of their statements turned out to be, I have the feeling that their claim to fame was probably copious experience with Civ and Risk…

Nevertheless, the battle lines were drawn viciously and with a lot of vitriol. Profile pictures were changed, tag lines were added, and the Twitter feeds were full of statements of solidarity. So far, so good. Amidst all that noise, there were few people willing to listen, and consider what type of help would be really helpful. This is to some extent just par for the course, and I am guilty of that myself. It is so much easier to ‘stand in solidarity with someone’ if I do not have to change my views or my behaviour! I feel reminded of a quote by the young Augustine of Hippo:

O Lord, make me chaste, but not yet!

In that sense, social media has unfortunately driven many of us to be posturing and signalling instead of actually doing something. For the events in Ukraine, I am somewhat personally affected,² and I found myself going for a social media hiatus because I was well fed up with people taking cheap discussion pot-shots and spitting at their screen while somewhere else, real human beings were dying.

(And the worst thing is, I am not one iota better; for every day, people are killed in conflicts all over the world, but I ignore most, if not all, of these conflicts.)

Now, with the vicious Hamas terror attack on Israel, a new ugly set of discussions is cropping up. There is one difference this time: the battle lines are way more murky; at the same time, the convictions of those that have no skin in the game run even deeper. I see many appalling, dehumanising statements by folks that are neither affected in any sense nor—or so it seems to me—do they understand this conflict as well as they think they do. I have no such delusions and know for a fact that I lack understanding of the context so I do not feel compelled to duke it out with others that are as ignorant as I am.

Now, the prevailing mores of social media being what they are, a lack of understanding never prevented a hot take, and outrage drives a lot of the engagements. Maybe, just maybe, now would be a good time for a little bit of listening and reflecting on what is said. These complex topics cannot be addressed by a good-or-evil narrative alone. That’s too much like ‘if you are not an apple, you are a banana!’³

I believe that a genuine contribution to any discussion requires some understanding that transcends feelings of outrage. If you are not one of the affected groups, listen first and think, lest you be taken advantage of. None of us want to end up like the vaccine-hating shill who peddles Ivermectin to people scared of a pandemic.⁴

I hope peace will soon be restored, but it does not look like it. One thing is sure, though: For any conflict to end peacefully, there is no need for armchair experts and keyboard warriors. I think it is all right to leave certain things to the pros. Heeding the ancient wisdom ne supra crepidam might not be the worst thing to do.

Which, by the way, was far from being over for many folks at the time of the invasion. ↩︎
The details are not important here. ↩︎
A wonderful quote from a wonderful book, which, despite its fantastical setting, shines a great light on our human affairs. ↩︎
I deliberately used that comparison because only the most untruthful folks were denying the actual existence of the virus, whereas in geopolitical conflicts, the lines are often very, very murky. ↩︎

Fixing Tor Browser accessibility

All content on Seirdy’s Home — Sun, 05 Nov 2023 15:34:53 GMT

It looks like the Tor Browser is finally addressing some of the accessibility issues inherent to its fingerprinting resistance, starting with Issue 42226: Reduce Customisability of default fonts and colours. The propsed fix of curating common “sets” of settings would add some entropy to everybody’s fingerprint but allow more people to use the browser overall, as it would support some degree of personalization (e.g. dark mode).

I welcome the change. Anonymity should be accessible.

Design

Home on Lagomorph — Sun, 05 Nov 2023 08:00:00 GMT

Inspired by Gwern Branwen’s design outline, this page documents the design philosophy behind not only this website, but also every other project I work on.

Lagomor.ph is implemented as a static website, compiled via Hugo from Standard Markdown, and hosted on a cloud server in Toronto.

“Indifference towards people and the reality in which they live is actually the one and only cardinal sin in design.”

Dieter Rams

It is designed to be readable, pleasing to the eye, and easy to navigate. The logo, which is often the singular image on each page, was drawn by Rek Bell.

No Hard Feelings

🦉 olshansky 🦁 — Sun, 05 Nov 2023 03:16:21 GMT

Olshansky's review of No Hard Feelings

Always get Google search results in English including Incognito mode

Gourav Goyal — Sun, 05 Nov 2023 02:00:00 GMT

keep Google searches in English while traveling, with a straightforward tip that works in any browser, incognito included

First week of wellness challenge

Ben Overmyer's Site — Sun, 05 Nov 2023 02:00:00 GMT

The first week of my November 2023 wellness challenge went well! Going without alcohol has given me back a ton of calories I would otherwise have gained, and I'm feeling more energetic every morning, too.

My first workout was on Wednesday morning at 5:00. It took me a bit to figure out where the stations and machines were that I needed for my workout plan. Some things I had to improvise a bit on; there was no cable row, so I just used the rowing machine.

My second workout on Friday morning was better. After blundering my way through the first workout, I took time on Thursday to put together a gym bag. I brought a log book and made sure to note down what weights I used for each exercise. I was still a bit sore from Wednesday's workout, but by Saturday morning the soreness from both sessions was gone.

This week coming up will be the last week of the "warmup plan" specified by my program. The weeks following will be more involved, and will be four days a week instead of three.

I've already started to lose a little weight. I'm looking forward to seeing how far I can get this month.

what are authors even trying to do?

taylor.town — Sun, 05 Nov 2023 02:00:00 GMT

Rather than saying "bad movie lol", Big Joel deconstructs The Room and asks the difficult question: what are authors even trying to do?

This is a fundamental question that your high-school teachers failed to teach you. While they quizzed you on Symbolism™ and Metaphor™ and Foreshadowing™, they fumbled the underlying lesson: people sometimes make stuff for reasons.

Can they understand the cat in Stewart Little?
Dig for authorial intent in Grown Ups 2.
Overanalyze Avatar: The Last Airbender.
Why did we watch a 2-hour documentary about a video game we've never heard of?
Listen for leitmotif in Lord of the Rings.
Become a film conspiracy theorist.
Engage in terrible writing.
Study idiot plots.
Is 300 facist?
Meet the Robinsons was underrated.
Look for inivisible choices in video games.
Avoid holes that were made for you.
Skin Theory in Spongebob Squarepants.
Understand the temporal pincer maneuver.
Soulja Boy provides his thoughts on Braid.
Experience visual comedy.
Find ludditism and neophilia in Jurassic Park.

Russian Woman Achieves 9x "Mythical" Man Month

taylor.town — Sun, 05 Nov 2023 02:00:00 GMT

I have 22 children at the age of 26 and most of them were born in the space of a year.

How to use Dev Tools

River Writes - A MediaWiki Blog — Sat, 04 Nov 2023 22:52:47 GMT

Learn how to edit CSS interactively

Creating an OpenAPI generator from scratch : From YAML to JetBrains HTTP Client

Julien's DevRel corner — Sat, 04 Nov 2023 19:40:33 GMT

In this article, I'll be implementing an openapi generator from scratch so you can too! We'll be creating a very simple generator for the Jetbrains HTTP Client

Habits of great software engineers

Vadim Kravcenko — Sat, 04 Nov 2023 13:08:19 GMT

Question:

Answer:

The post Habits of great software engineers appeared first on Vadim Kravcenko.

Big Mouth: Season 7

🦉 olshansky 🦁 — Sat, 04 Nov 2023 03:16:21 GMT

Olshansky's review of Big Mouth: Season 7

A Market Research Colossus

Magis — Fri, 03 Nov 2023 22:29:13 GMT

Nielsen in A.C. Nielsen's own words

Being a minor AI public figure

swyx's site RSS Feed — Fri, 03 Nov 2023 22:21:53 GMT

I was recently involved in moderating a chat with Kanjun Qiu of Imbue at the MIT AI conf: https://photos.google.com/share/AF1QipNJ9i78ICeg2YuemyAXmtStKvqr9l0Tao3xQWxmeAVjBALHnNnFvPXFlNSkdMfqA?pli=1&key=dTFRRHBTLVRZTEVCem0zal8tNVkxblh0V3k4VXhR

Testing The PHASE M645-GFX E with Mamiya, Phase One, and Schneider Lenses

9μm Pixels — Fri, 03 Nov 2023 11:11:07 GMT

TL;DR: Works perfectly on newer Silver Ring and Blue Ring lenses, less perfect on older Mamiya lenses. Flare is another issue. Overall, it is really nice to use these lenses with the GFX100s, and get metadata and aperture control.

Recently, Kipon released a new adapter for controlling the apeture

Wrangling Table Bloat in Large PostgreSQL Tables

Dylan Paulus' Blog — Fri, 03 Nov 2023 02:00:00 GMT

I worked with Timescale again to write an article on how table bloat effects large tables, and how we can fix it.

We cover a range of topics including:

What is table bloat?
Why is it an issue?
How to monitor table bloat?
What Vacuum and AutoVacuum is
Tips with tuning AutoVacuum
Table Partitioning
How Timescale chunks makes partitioning easier
And more!

Rust for Dummies

Rajeev Singh Naruka — Fri, 03 Nov 2023 02:00:00 GMT

Unstructured notes to self on Rust. Peculiar stuff or just basics.

Compressing Codebase Collocates

taylor.town — Fri, 03 Nov 2023 02:00:00 GMT

Replacing collocates with simpler substitutes is intuitively called "compression".

Digital Desire Paths

taylor.town — Fri, 03 Nov 2023 02:00:00 GMT

Good architects anticipate nature. They don't blame water for dripping. They don't blame people for peopling.

The technical problems with LLMs

Macleod Sawyer | Web Portal Feed — Fri, 03 Nov 2023 02:00:00 GMT

I think my biggest (technical, long-term) issues with current generation AI, primarily through LLMs such as ChatGPT, are the following:

You cannot readily replicate responses, which is the entire point of the scientific process. To ensure correct results for discovery and use is the ability to easily replicate results.
They are primarily cloud based, which means you will not be able to stand them up for testing in 1-5 years and replicate any sort of results or research papers that may be discovered now. Technical, direct, or anything else.
They are primarily cloud based, which means any thing you throw at them, will not be able to be tuned with similar characteristics without massive manual testing as they are internally changed all the time.
They are primarily cloud based, which means you cannot have any ownership or security in the knowledge, that they are safely utilizing your data.
They are primarily cloud based, which means they can price you out of their system at any time.
They are primarily cloud based, which means that they will price you out of their system at any time.
They are primarily cloud based, which means they will break your/their API, and there will be no recompense or ability to change providers.
They are all primarily a private company, and if you build on their platform, then you have no platform of your own. Your business, is dependent on, and is their business.
You own nothing. They own all of it. No matter the legal contract you have with them will protect you from that.
They are primarily cloud based, they are primarily private businesses, and they are anti-scientific development as they are black boxes of inputs, controls, and outputs.

Anything you develop on them now, will not work the same, if at all, in the future. You will have virtually, tangentially, literally, no legacy or agency.

Postgres HBA with a Tailscale network

Ellie's Notes — Fri, 03 Nov 2023 02:00:00 GMT

Following on with my postgres on zfs setup, I needed to configure the auth so that my PostgreSQL replica could securely connect to the primary.

The constraints here are that I'm not using a cloud private network, so need a VPN of some kind! I'm using Tailscale, which is pretty much just wireguard made easy.

Firewall

First up, following Tailscale's docs. Make sure you don't lock yourself out! I'm also using tailscale SSH with 2fa.

ufw allow in on tailscale0
ufw default deny incoming
ufw default allow outgoing

ufw reload

systemctl restart sshd

postgresql.conf

Next, we need to tell postgres to listen on all addresses. In your postgres config:

listen_addresses = '*'

I wish I could just provide tailscale0, the tailnet CIDR or... anything else. Alas. It's not that flexible, but at least we can lock things down with the firewall + hba. Don't skip the other steps!

pg_hba

Next up we just need to setup pg_hba.conf to allow the login!

host    all             all             100.64.0.0/10           scram-sha-256

Where 100.64.0.0/10 is the CIDR range used by Tailscale. Read more here.

Bunny Sync: Handy Upload Helper

Sergey's Blog — Thu, 02 Nov 2023 21:52:04 GMT

Sync local files effortlessly with Bunny Sync, a simple CLI tool designed for Bunny Storage users. No FTP fuss, no waiting for APIs - just quick and straightforward syncing.

The Baseline Fallacy in Defensive Cyber Operations

Zac Szewczyk — Thu, 02 Nov 2023 20:25:05 GMT

Both civilian cybersecurity practitioners and military defensive cyber operations personnel continue to push a similar idea, that to identify malicious activity one must “just get a baseline and then flag anything different.” If only! I call this the Baseline Fallacy:

Permalink.

We decided to open a cafe for a day. This happened

Ednevsky Blog — Thu, 02 Nov 2023 15:01:00 GMT

If you're a knowledge worker and ever dreamed of doing something like this, read along. This is a full story/guide of how we approached this

Unstable: Season 1

🦉 olshansky 🦁 — Thu, 02 Nov 2023 03:52:38 GMT

Olshansky's review of Unstable: Season 1

Inspiration from Fall 2023—and a reader survey!

An Ordinary Disaster — Thu, 02 Nov 2023 02:15:15 GMT

Sophie Strand, Ben McGrath, John and Monica Chapman, Jessica Carew Kraft, Kendra Atleework, Paul Kingsnorth, Stewart Brand, Holly Payne, Bari Weiss, Scott Galloway, Richard V. Reeves, Scott Lyons

Add Cloudflare Web Analytics to Next.js site

Gourav Goyal — Thu, 02 Nov 2023 02:00:00 GMT

Add Cloudflare Web Analytics to Next.js site

Návod na prežitie pre bejby

Vít Baisa — Thu, 02 Nov 2023 02:00:00 GMT

souhrn poznatků od youtubera

How to opt-out of ChatGPT training without clearing conversation history

taylor.town — Thu, 02 Nov 2023 02:00:00 GMT

OpenAI provides a Google Form for opting out of data collection on ChatGPT.

My Podcast Subscriptions

taylor.town — Thu, 02 Nov 2023 02:00:00 GMT

Podcasts are great! Except when they're not.

I highly recommend overcast.fm for listening to podcasts.

My Current Rotation


★★★★★	99% Invisible
★★★★★	Articles of Interest
★★★★	Conversations with Tyler
★★★★	Derek Sivers
★★★★	Radiolab
★★★★	The Memory Palace
★★★	Cortex
★★★	What's Your Problem?
★★★	Software Unscripted

Bingeworthy series that are completed or on indefinite hiatus:


★★★★★	S-Town
★★★★★	The Anthropocene Reviewed
★★★★	On the Metal
★★★	Björk: Sonic Symbolism
★★★	My Year in Mensa
★★★	The Trojan Horse Affair
★★★	Hello Internet

Support these cool tech podcasts produced by my friends:


♥	devtools.fm
♥	The Changelog
♥	Elm Town
♥	Future of Coding
♥	Hest
♥	Software Unscripted
♥	TODEPOND PODCAST

Honorable Mentions

Series that I enjoyed at some point in the past:


★★	Against the Rules with Michael Lewis
	Ahead of its Time
	Akimbo: A Podcast from Seth Godin
	American Innovations
	Brains
★★★	Cautionary Tales
★★★	Chemistry For Your Life
★★★★	Common Sense with Dan Carlin
	Conan O'Brien Needs A Friend
★★★★	Dan Carlin's Hardcore History
	Darknet Diaries
	Dear Hank & Jon
★★★	Deep Questions with Cal Newport
★★	Endless Thread
★★	Epic Gardening
★★★★	Everything Everywhere Daily
★★★	Everything is Alive
	Experimental History
★★★	Freakonomics
	Harmontown
	Hey Riddle Riddle
	Land by Hand
	Monday Morning Podcast with Bill Burr
	More Perfect
★★★★	Nice Try!
	Oxford University Computer Science
★★	Oxide & Friends
	Patented: History of Inventions
★★★	Planet Money
	Reasonably Sound
	Reconcilable Differences
	Reply All
★★	Revisionist History
	Stuff You Should Know
★★	The Economics of Everyday Things
	The Joy of Why
	The Permaculture Podcast
	The Peter Attia Drive
	The Strong Towns Podcast
	The Tim Ferriss Show
	Trailblazers with Walter Isaacson

CDC Use Cases: 7 Ways to Put CDC to Work

Gunnar Morling — Thu, 02 Nov 2023 02:00:00 GMT

Table of Contents

What is CDC?
CDC Tools
Analytics Data Platforms
Application Caches
Full-Text Search
Audit Logs
Continuous Queries
Microservices Data Exchange
Monolith-to-Microservices Migration
Summary

Change Data Capture (CDC) is a powerful tool in data engineering and has seen a tremendous uptake in organizations of all kinds over the last few years. This is because it enables the tight integration of transactional databases into many other systems in your business at a very low latency.

Jobs, swords, and doors

Jordan Webb's blog — Wed, 01 Nov 2023 22:33:37 GMT

What have I been up to lately?

What is the average length of a queue of cars?

Emilio’s Blog — Wed, 01 Nov 2023 12:00:00 GMT

Some time ago I was driving on a twisty mountain road, stuck in a slow-moving queue of cars as it was impossible to overtake safely. Out of boredom, I was wondering how many cars were in the queue, and, more generally, what would be the average length of queues in this road. Let’s find out!

E-Bike to Lenovo Power Supply

transistor-man.com — Wed, 01 Nov 2023 10:38:38 GMT

Does your CAD laptop run it's battery dry in less than 3 hrs? Do you like working off in the hills and want access to a longer runtimes on your power hungry desktop-replacement machine?

Inspiration

Love, Blood & Rhetoric — Wed, 01 Nov 2023 05:16:54 GMT

“A little warm for a fire, don’t you think?” “It’s for inspiration, not warmth.” “Well, it’ll burn better if you stop poking it.” “I’m tickling it. To amuse it. You know, Shakespeare? Oh, to amuse a fire?” “I don’t think you’re remembering that right.” “And if tickling doesn’t work, I’ll try poetry or knock-knock jokes.” […]

Data Sovereignty In Aws Rds

rrich.io — Wed, 01 Nov 2023 02:00:00 GMT

In the era of global data flows, maintaining compliance with data sovereignty laws and managing cross-border data transfers can be challenging. Cloud infrastructures often span multiple geographies, and as data moves across these boundaries, it becomes subject to diverse legal frameworks and regulations (such as the General Data Protection Regulation or GDPR in Europe). This technical blueprint provides a guide on using RDS to address these challenges.

if it talks like a therapist...

taylor.town — Wed, 01 Nov 2023 02:00:00 GMT

They speak fearlessly but fairly. They're strong, yet nonthreatening. They push back while maintaining peace. I yearn for this powerless power.

Writing a storage engine for Postgres: an in-memory Table Access Method

Notes on software development — Wed, 01 Nov 2023 02:00:00 GMT

With Postgres 12, released in 2019, it became possible to swap out Postgres's storage engine.

This is a feature MySQL has supported for a long time. There are at least 8 different built-in engines you can pick from. MyRocks, MySQL on RocksDB, is another popular third-party distribution.

I assume there will be a renaissance of Postgres storage engines. To date, the efforts are nascent. OrioleDB and Citus Columnar are two promising third-party table access methods being actively developed.

Why alternative storage engines?

The ability to swap storage engines is useful because different workloads sometimes benefit from different storage approaches. Analytics workloads and columnar storage layouts go well together. Write-heavy workloads and LSM trees go well together. And some people like in-memory storage for running integration tests.

By swapping out only the storage engine, you get the benefit of the rest of the Postgres or MySQL infrastructure. The query language, the wire protocol, the ecosystem, etc.

Why not foreign data wrappers?

Very little has been written about the difference between foreign data wrappers (FDWs) and table access methods. Table access methods seems to be the lower-level layer where presumably you get better performance and cleaner integration. But there is clearly overlap between these two extension options.

For example there is a FDW for ClickHouse so when you create tables and rows and query the tables you are really creating and querying rows in a ClickHouse server. Similarly there's a FDW for RocksDB. And Citus's columnar engine works either as a foreign data wrapper or a table access method.

The Citus page draws the clearest distinction between FDWs and table access methods, but even that page is vague. Performance doesn't seem to be the main difference. Closer integration, and thus the ability to look more like vanilla Postgres from the outside, seems to be the gist.

In any case, I wanted to explore the table access method API.

Digging in

I haven't written Postgres extensions before and I've never written C professionally. If you're familiar with Postgres internals or C and notice something funky, please let me know!

It turns out that almost no one has written how to implement the minimal table access methods for various storage engine operations. So after quite a bit of stumbling to get the basics of an in-memory storage engine working, I'm going to walk you through my approach.

This is prototype-quality code which hopefully will be a useful base for further exploration.

All code for this post is available on GitHub.

A debug Postgres build

First off, let's make a debug build of Postgres.

$ git clone https://github.com/postgres/postgres
$ # An arbitrary commit from `master` after Postgres 16 I am on
$ git checkout 849172ff4883d44168f96f39d3fde96d0aa34c99
$ cd postgres
$ ./configure --enable-cassert --enable-debug CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer"
$ make -j8
$ sudo make install

This will install Postgres binaries (e.g. psql, pg_ctl, initdb, pg_config) into /usr/local/pgsql/bin.

I'm going to reference those absolute paths throughout this post because you might have a system (package manager) install of Postgres already.

Let's create a database and start up this debug build:

$ /usr/local/pgsql/bin/initdb test-db
$ /usr/local/pgsql/bin/pg_ctl -D test-db -l logfile start

Extension infrastructure

Since we installed Postgres from scratch, /usr/local/pgsql/bin/pg_config will supply all of the infrastructure we need.

The "infrastructure" is basically just PGXS: Postgres Makefile utilities.

It's convention-heavy. So in a new Makefile for this project we'll specify:

MODULES: Any C sources to build, without the .c file extension
EXTENSION: Extension metadata file, without the .control file extension
DATA: A SQL file that is executed when the extension is loaded, this time with the .sql extension

MODULES = pgtam
EXTENSION = pgtam
DATA = pgtam--0.0.1.sql

PG_CONFIG = /usr/local/pgsql/bin/pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

The final three lines set up the PGXS Makefile library based on the particular installed Postgres build we want to build the extension against and install the extension to.

PGXS gives us a few important targets like make distclean, make, and make install we'll use later on.

`pgtam.c`

A minimal C file that registers a function capable of serving as a table access method is:

#include "postgres.h"
#include "fmgr.h"
#include "access/tableam.h"

PG_MODULE_MAGIC;

const TableAmRoutine memam_methods = {
  .type = T_TableAmRoutine,
};

PG_FUNCTION_INFO_V1(mem_tableam_handler);
Datum mem_tableam_handler(PG_FUNCTION_ARGS) {
  PG_RETURN_POINTER(&memam_methods);
}

If you want to read about extension basics without the complexity of table access methods, you can find a complete, minimal Postgres extension I wrote to validate the infrastructure here. Or you can follow a larger tutorial.

The workflow for registering a table access method is to first run CREATE EXTENSION pgtam. This assumes pgtam is an extension that has a function that returns a TableAmRoutine struct instance, a table of table access methods.

Then you must run CREATE ACCESS METHOD mem TYPE TABLE HANDLER mem_tableam_handler. And finally you can use the access method when creating a table with USING mem: CREATE TABLE x(a INT) USING mem.

`pgtam.control`

This file contains extension metadata. At a minimum, the version of the extension and the filename for the extension where it should be installed.

default_version = '0.0.1'
module_pathname = '$libdir/pgtam'

`pgtam--0.0.1.sql`

Finally, in pgtam--0.0.1.sql (which is executed when we call CREATE EXTENSION pgtam), we register the handler function as a foreign function, and then we register the function as an access method.

CREATE OR REPLACE FUNCTION mem_tableam_handler(internal)
RETURNS table_am_handler AS 'pgtam', 'mem_tableam_handler'
LANGUAGE C STRICT;

CREATE ACCESS METHOD mem TYPE TABLE HANDLER mem_tableam_handler;

Build

Now that we've got all the pieces in place, we can build and install the extension.

$ make
$ sudo make install

Let's add a test.sql script to exercise the extension:

DROP EXTENSION IF EXISTS pgtam CASCADE;
CREATE EXTENSION pgtam;
CREATE TABLE x(a INT) USING mem;

And run it:

$ /usr/local/pgsql/bin/psql postgres -f test.sql
DROP EXTENSION
CREATE EXTENSION
psql:test.sql:3: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:test.sql:3: error: connection to server was lost

Ok, so psql crashed! Let's look at the server logs. When we started Postgres with pg_ctl we specified the log file as logfile in the directory where we ran pg_ctl.

If we look through it we'll spot an assertion failure:

$ grep Assert logfile
TRAP: failed Assert("routine->scan_begin != NULL"), File: "tableamapi.c", Line: 52, PID: 2906922

That's a great sign! This is Postgres's debug infrastructure helping to make sure the table access method is correctly implemented.

Table access method stubs

The next step is to add function stubs for all the non-optional methods of the TableAmRoutine struct.

I've done all the work for you already so you can just copy this over the existing pgtam.c. It's a big file, but don't worry. There's nothing to explain. Just a bunch of blank functions returning default values when required.

#include "postgres.h"
#include "fmgr.h"
#include "access/tableam.h"
#include "access/heapam.h"
#include "nodes/execnodes.h"
#include "catalog/index.h"
#include "commands/vacuum.h"
#include "utils/builtins.h"
#include "executor/tuptable.h"

PG_MODULE_MAGIC;

const TableAmRoutine memam_methods;

static const TupleTableSlotOps* memam_slot_callbacks(
  Relation relation
) {
  return NULL;
}

static TableScanDesc memam_beginscan(
  Relation relation,
  Snapshot snapshot,
  int nkeys,
  struct ScanKeyData *key,
  ParallelTableScanDesc parallel_scan,
  uint32 flags
) {
  return NULL;
}

static void memam_rescan(
  TableScanDesc sscan,
  struct ScanKeyData *key,
  bool set_params,
  bool allow_strat,
  bool allow_sync,
  bool allow_pagemode
) {
}

static void memam_endscan(TableScanDesc sscan) {
}

static bool memam_getnextslot(
  TableScanDesc sscan,
  ScanDirection direction,
  TupleTableSlot *slot
) {
  return false;
}

static IndexFetchTableData* memam_index_fetch_begin(Relation rel) {
  return NULL;
}

static void memam_index_fetch_reset(IndexFetchTableData *scan) {
}

static void memam_index_fetch_end(IndexFetchTableData *scan) {
}

static bool memam_index_fetch_tuple(
  struct IndexFetchTableData *scan,
  ItemPointer tid,
  Snapshot snapshot,
  TupleTableSlot *slot,
  bool *call_again,
  bool *all_dead
) {
  return false;
}

static void memam_tuple_insert(
  Relation relation,
  TupleTableSlot *slot,
  CommandId cid,
  int options,
  BulkInsertState bistate
) {
}

static void memam_tuple_insert_speculative(
  Relation relation,
  TupleTableSlot *slot,
  CommandId cid,
  int options,
  BulkInsertState bistate,
  uint32 specToken) {
}

static void memam_tuple_complete_speculative(
  Relation relation,
  TupleTableSlot *slot,
  uint32 specToken,
  bool succeeded) {
}

static void memam_multi_insert(
  Relation relation,
  TupleTableSlot **slots,
  int ntuples,
  CommandId cid,
  int options,
  BulkInsertState bistate
) {
}

static TM_Result memam_tuple_delete(
  Relation relation,
  ItemPointer tid,
  CommandId cid,
  Snapshot snapshot,
  Snapshot crosscheck,
  bool wait,
  TM_FailureData *tmfd,
  bool changingPart
) {
  TM_Result result = {};
  return result;
}

static TM_Result memam_tuple_update(
  Relation relation,
  ItemPointer otid,
  TupleTableSlot *slot,
  CommandId cid,
  Snapshot snapshot,
  Snapshot crosscheck,
  bool wait,
  TM_FailureData *tmfd,
  LockTupleMode *lockmode,
  TU_UpdateIndexes *update_indexes
) {
  TM_Result result = {};
  return result;
}

static TM_Result memam_tuple_lock(
  Relation relation,
  ItemPointer tid,
  Snapshot snapshot,
  TupleTableSlot *slot,
  CommandId cid,
  LockTupleMode mode,
  LockWaitPolicy wait_policy,
  uint8 flags,
  TM_FailureData *tmfd)
{
  TM_Result result = {};
  return result;
}

static bool memam_fetch_row_version(
  Relation relation,
  ItemPointer tid,
  Snapshot snapshot,
  TupleTableSlot *slot
) {
  return false;
}

static void memam_get_latest_tid(
  TableScanDesc sscan,
  ItemPointer tid
) {
}

static bool memam_tuple_tid_valid(TableScanDesc scan, ItemPointer tid) {
  return false;
}

static bool memam_tuple_satisfies_snapshot(
  Relation rel,
  TupleTableSlot *slot,
  Snapshot snapshot
) {
  return false;
}

static TransactionId memam_index_delete_tuples(
  Relation rel,
  TM_IndexDeleteOp *delstate
) {
  TransactionId id = {};
  return id;
}

static void memam_relation_set_new_filelocator(
  Relation rel,
  const RelFileLocator *newrlocator,
  char persistence,
  TransactionId *freezeXid,
  MultiXactId *minmulti
) {
}

static void memam_relation_nontransactional_truncate(
  Relation rel
) {
}

static void memam_relation_copy_data(
  Relation rel,
  const RelFileLocator *newrlocator
) {
}

static void memam_relation_copy_for_cluster(
  Relation OldHeap,
  Relation NewHeap,
  Relation OldIndex,
  bool use_sort,
  TransactionId OldestXmin,
  TransactionId *xid_cutoff,
  MultiXactId *multi_cutoff,
  double *num_tuples,
  double *tups_vacuumed,
  double *tups_recently_dead
) {
}

static void memam_vacuum_rel(
  Relation rel,
  VacuumParams *params,
  BufferAccessStrategy bstrategy
) {
}

static bool memam_scan_analyze_next_block(
  TableScanDesc scan,
  BlockNumber blockno,
  BufferAccessStrategy bstrategy
) {
  return false;
}

static bool memam_scan_analyze_next_tuple(
  TableScanDesc scan,
  TransactionId OldestXmin,
  double *liverows,
  double *deadrows,
  TupleTableSlot *slot
) {
  return false;
}

static double memam_index_build_range_scan(
  Relation heapRelation,
  Relation indexRelation,
  IndexInfo *indexInfo,
  bool allow_sync,
  bool anyvisible,
  bool progress,
  BlockNumber start_blockno,
  BlockNumber numblocks,
  IndexBuildCallback callback,
  void *callback_state,
  TableScanDesc scan
) {
  return 0;
}

static void memam_index_validate_scan(
  Relation heapRelation,
  Relation indexRelation,
  IndexInfo *indexInfo,
  Snapshot snapshot,
  ValidateIndexState *state
) {
}

static bool memam_relation_needs_toast_table(Relation rel) {
  return false;
}

static Oid memam_relation_toast_am(Relation rel) {
  Oid oid = {};
  return oid;
}

static void memam_fetch_toast_slice(
  Relation toastrel,
  Oid valueid,
  int32 attrsize,
  int32 sliceoffset,
  int32 slicelength,
  struct varlena *result
) {
}

static void memam_estimate_rel_size(
  Relation rel,
  int32 *attr_widths,
  BlockNumber *pages,
  double *tuples,
  double *allvisfrac
) {
}

static bool memam_scan_sample_next_block(
  TableScanDesc scan, SampleScanState *scanstate
) {
  return false;
}

static bool memam_scan_sample_next_tuple(
  TableScanDesc scan,
  SampleScanState *scanstate,
  TupleTableSlot *slot
) {
  return false;
}

const TableAmRoutine memam_methods = {
  .type = T_TableAmRoutine,

  .slot_callbacks = memam_slot_callbacks,

  .scan_begin = memam_beginscan,
  .scan_end = memam_endscan,
  .scan_rescan = memam_rescan,
  .scan_getnextslot = memam_getnextslot,

  .parallelscan_estimate = table_block_parallelscan_estimate,
  .parallelscan_initialize = table_block_parallelscan_initialize,
  .parallelscan_reinitialize = table_block_parallelscan_reinitialize,

  .index_fetch_begin = memam_index_fetch_begin,
  .index_fetch_reset = memam_index_fetch_reset,
  .index_fetch_end = memam_index_fetch_end,
  .index_fetch_tuple = memam_index_fetch_tuple,

  .tuple_insert = memam_tuple_insert,
  .tuple_insert_speculative = memam_tuple_insert_speculative,
  .tuple_complete_speculative = memam_tuple_complete_speculative,
  .multi_insert = memam_multi_insert,
  .tuple_delete = memam_tuple_delete,
  .tuple_update = memam_tuple_update,
  .tuple_lock = memam_tuple_lock,

  .tuple_fetch_row_version = memam_fetch_row_version,
  .tuple_get_latest_tid = memam_get_latest_tid,
  .tuple_tid_valid = memam_tuple_tid_valid,
  .tuple_satisfies_snapshot = memam_tuple_satisfies_snapshot,
  .index_delete_tuples = memam_index_delete_tuples,

  .relation_set_new_filelocator = memam_relation_set_new_filelocator,
  .relation_nontransactional_truncate = memam_relation_nontransactional_truncate,
  .relation_copy_data = memam_relation_copy_data,
  .relation_copy_for_cluster = memam_relation_copy_for_cluster,
  .relation_vacuum = memam_vacuum_rel,
  .scan_analyze_next_block = memam_scan_analyze_next_block,
  .scan_analyze_next_tuple = memam_scan_analyze_next_tuple,
  .index_build_range_scan = memam_index_build_range_scan,
  .index_validate_scan = memam_index_validate_scan,

  .relation_size = table_block_relation_size,
  .relation_needs_toast_table = memam_relation_needs_toast_table,
  .relation_toast_am = memam_relation_toast_am,
  .relation_fetch_toast_slice = memam_fetch_toast_slice,

  .relation_estimate_size = memam_estimate_rel_size,

  .scan_sample_next_block = memam_scan_sample_next_block,
  .scan_sample_next_tuple = memam_scan_sample_next_tuple
};

PG_FUNCTION_INFO_V1(mem_tableam_handler);

Datum mem_tableam_handler(PG_FUNCTION_ARGS) {
  PG_RETURN_POINTER(&memam_methods);
}

Let's build and test it!

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE

Hey we're getting somewhere! It successfully created the table with our custom table access method.

Querying rows

Next, let's try querying the table by adding a SELECT a FROM x to test.sql and running it:

$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
psql:test.sql:6: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:test.sql:6: error: connection to server was lost

This time there's nothing in logfile that helps:

$ tail -n15 logfile
2023-11-01 18:43:32.449 UTC [2906199] LOG:  database system is ready to accept connections
2023-11-01 18:58:32.572 UTC [2907997] LOG:  checkpoint starting: time
2023-11-01 18:58:35.305 UTC [2907997] LOG:  checkpoint complete: wrote 28 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=2.712 s, sync=0.015 s, total=2.733 s; sync files=23, longest=0.004 s, average=0.001 s; distance=128 kB, estimate=150 kB; lsn=0/15F88E0, redo lsn=0/15F8888
2023-11-01 19:08:14.485 UTC [2906199] LOG:  server process (PID 2908242) was terminated by signal 11: Segmentation fault
2023-11-01 19:08:14.485 UTC [2906199] DETAIL:  Failed process was running: SELECT a FROM x;
2023-11-01 19:08:14.485 UTC [2906199] LOG:  terminating any other active server processes
2023-11-01 19:08:14.486 UTC [2906199] LOG:  all server processes terminated; reinitializing
2023-11-01 19:08:14.508 UTC [2908253] LOG:  database system was interrupted; last known up at 2023-11-01 18:58:35 UTC
2023-11-01 19:08:14.518 UTC [2908253] LOG:  database system was not properly shut down; automatic recovery in progress
2023-11-01 19:08:14.519 UTC [2908253] LOG:  redo starts at 0/15F8888
2023-11-01 19:08:14.520 UTC [2908253] LOG:  invalid record length at 0/161DE70: expected at least 24, got 0
2023-11-01 19:08:14.520 UTC [2908253] LOG:  redo done at 0/161DE38 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2023-11-01 19:08:14.521 UTC [2908254] LOG:  checkpoint starting: end-of-recovery immediate wait
2023-11-01 19:08:14.532 UTC [2908254] LOG:  checkpoint complete: wrote 35 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.010 s, total=0.012 s; sync files=27, longest=0.003 s, average=0.001 s; distance=149 kB, estimate=149 kB; lsn=0/161DE70, redo lsn=0/161DE70
2023-11-01 19:08:14.533 UTC [2906199] LOG:  database system is ready to accept connections

This was the first place I got stuck. How on earth do I figure out what methods to implement? I mean, it's clearly one or more of these methods from the struct. But there are so many methods.

I tried setting a breakpoint in gdb on the process returned by SELECT pg_backend_pid() for a psql session, but the breakpoint never seemed to be hit for any of my methods.

So I did the low-tech solution and opened a file, /tmp/pgtam.log, turned off buffering on it, and added a log to every method on the TableAmRoutine struct:

@@ -12,9 +12,13 @@

 const TableAmRoutine memam_methods;

+FILE* fd;
+#define DEBUG_FUNC() fprintf(fd, "in %s\n", __func__);
+
 static const TupleTableSlotOps* memam_slot_callbacks(
   Relation relation
 ) {
+  DEBUG_FUNC();
   return NULL;
 }

@@ -26,6 +30,7 @@
   ParallelTableScanDesc parallel_scan,
   uint32 flags
 ) {
+  DEBUG_FUNC();
   return NULL;
 }

@@ -37,9 +42,11 @@
   bool allow_sync,
   bool allow_pagemode
 ) {
+  DEBUG_FUNC();
 }

 static void memam_endscan(TableScanDesc sscan) {
+  DEBUG_FUNC();
 }

 static bool memam_getnextslot(
@@ -47,17 +54,21 @@
   ScanDirection direction,
   TupleTableSlot *slot
 ) {
+  DEBUG_FUNC();
   return false;
 }

 static IndexFetchTableData* memam_index_fetch_begin(Relation rel) {
+  DEBUG_FUNC();
   return NULL;
 }

 static void memam_index_fetch_reset(IndexFetchTableData *scan) {
+  DEBUG_FUNC();
 }

 static void memam_index_fetch_end(IndexFetchTableData *scan) {
+  DEBUG_FUNC();
 }

 static bool memam_index_fetch_tuple(
@@ -68,6 +79,7 @@
   bool *call_again,
   bool *all_dead
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -78,6 +90,7 @@
   int options,
   BulkInsertState bistate
 ) {
+  DEBUG_FUNC();
 }

 static void memam_tuple_insert_speculative(
@@ -87,6 +100,7 @@
   int options,
   BulkInsertState bistate,
   uint32 specToken) {
+  DEBUG_FUNC();
 }

 static void memam_tuple_complete_speculative(
@@ -94,6 +108,7 @@
   TupleTableSlot *slot,
   uint32 specToken,
   bool succeeded) {
+  DEBUG_FUNC();
 }

 static void memam_multi_insert(
@@ -104,6 +119,7 @@
   int options,
   BulkInsertState bistate
 ) {
+  DEBUG_FUNC();
 }

 static TM_Result memam_tuple_delete(
@@ -117,6 +133,7 @@
   bool changingPart
 ) {
   TM_Result result = {};
+  DEBUG_FUNC();
   return result;
 }

@@ -133,6 +150,7 @@
   TU_UpdateIndexes *update_indexes
 ) {
   TM_Result result = {};
+  DEBUG_FUNC();
   return result;
 }

@@ -148,6 +166,7 @@
   TM_FailureData *tmfd)
 {
   TM_Result result = {};
+  DEBUG_FUNC();
   return result;
 }

@@ -157,6 +176,7 @@
   Snapshot snapshot,
   TupleTableSlot *slot
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -164,9 +184,11 @@
   TableScanDesc sscan,
   ItemPointer tid
 ) {
+  DEBUG_FUNC();
 }

 static bool memam_tuple_tid_valid(TableScanDesc scan, ItemPointer tid) {
+  DEBUG_FUNC();
   return false;
 }

@@ -175,6 +197,7 @@
   TupleTableSlot *slot,
   Snapshot snapshot
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -183,6 +206,7 @@
   TM_IndexDeleteOp *delstate
 ) {
   TransactionId id = {};
+  DEBUG_FUNC();
   return id;
 }

@@ -193,17 +217,20 @@
   TransactionId *freezeXid,
   MultiXactId *minmulti
 ) {
+  DEBUG_FUNC();
 }

 static void memam_relation_nontransactional_truncate(
   Relation rel
 ) {
+  DEBUG_FUNC();
 }

 static void memam_relation_copy_data(
   Relation rel,
   const RelFileLocator *newrlocator
 ) {
+  DEBUG_FUNC();
 }

 static void memam_relation_copy_for_cluster(
@@ -218,6 +245,7 @@
   double *tups_vacuumed,
   double *tups_recently_dead
 ) {
+  DEBUG_FUNC();
 }

 static void memam_vacuum_rel(
@@ -225,6 +253,7 @@
   VacuumParams *params,
   BufferAccessStrategy bstrategy
 ) {
+  DEBUG_FUNC();
 }

 static bool memam_scan_analyze_next_block(
@@ -232,6 +261,7 @@
   BlockNumber blockno,
   BufferAccessStrategy bstrategy
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -242,6 +272,7 @@
   double *deadrows,
   TupleTableSlot *slot
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -258,6 +289,7 @@
   void *callback_state,
   TableScanDesc scan
 ) {
+  DEBUG_FUNC();
   return 0;
 }

@@ -268,14 +300,17 @@
   Snapshot snapshot,
   ValidateIndexState *state
 ) {
+  DEBUG_FUNC();
 }

 static bool memam_relation_needs_toast_table(Relation rel) {
+  DEBUG_FUNC();
   return false;
 }

 static Oid memam_relation_toast_am(Relation rel) {
   Oid oid = {};
+  DEBUG_FUNC();
   return oid;
 }

@@ -287,6 +322,7 @@
   int32 slicelength,
   struct varlena *result
 ) {
+  DEBUG_FUNC();
 }

 static void memam_estimate_rel_size(
@@ -296,11 +332,13 @@
   double *tuples,
   double *allvisfrac
 ) {
+  DEBUG_FUNC();
 }

 static bool memam_scan_sample_next_block(
   TableScanDesc scan, SampleScanState *scanstate
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -309,6 +347,7 @@
   SampleScanState *scanstate,
   TupleTableSlot *slot
 ) {
+  DEBUG_FUNC();
   return false;
 }

And then in the entrypoint, initialize the file for logging.

@@ -369,5 +408,9 @@
 PG_FUNCTION_INFO_V1(mem_tableam_handler);

 Datum mem_tableam_handler(PG_FUNCTION_ARGS) {
+  fd = fopen("/tmp/pgtam.log", "a");
+  setvbuf(fd, NULL, _IONBF, 0); // Prevent buffering
+  fprintf(fd, "\n\nmem_tableam handler loaded\n");
+
   PG_RETURN_POINTER(&memam_methods);
 }

Let's give it a shot!

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
psql:test.sql:6: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:test.sql:6: error: connection to server was lost

And let's check our log file:

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_estimate_rel_size
in memam_slot_callbacks

Now we're getting somewhere!

I later realized elog() is the way most people log within Postgres/within extensions. I didn't know that when I was getting started though. This separate logging was a simple way to get the info out.

`slot_callbacks`

Since the request crashes and the last logged function is memam_slot_callbacks, it seems like that is where we should concentrate. The table access method docs suggest looking at the default heap access method for inspiration.

Its version of slot_callbacks returns &TTSOpsBufferHeapTuple:

static const TupleTableSlotOps *
heapam_slot_callbacks(Relation relation)
{
    return &TTSOpsBufferHeapTuple;
}

I have no idea what that means, but since it is defined in src/backend/executor/execTuples.c it doesn't seem to be tied to the heap access method implementation. Let's try it.

While it works initially, I noticed later on that TTSOpsBufferHeapTuple turns out not to be the right choice here. TTSOpsVirtual seems to be the right implementation.

@@ -19,7 +19,7 @@
   Relation relation
 ) {
   DEBUG_FUNC();
-  return NULL;
+  return &TTSOpsVirtual;
 }

 static TableScanDesc memam_beginscan(

Build and run:

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
psql:test.sql:6: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:test.sql:6: error: connection to server was lost

It still crashes. But this time in /tmp/pgtam.log we made it into a new method!

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan

`scan_begin`

The function signature is:

TableScanDesc heap_beginscan(
  Relation relation,
  Snapshot snapshot,
  int nkeys,
  ScanKey key,
  ParallelTableScanDesc parallel_scan,
  uint32 flags
);

Since we just implemented stub versions of all the methods, we've been returning NULL. Since we're failing in this function, maybe we should try returning something that isn't NULL.

By looking at the definition of TableScanDesc, we can see it is a pointer to the TableScanDescData struct defined in src/include/access/relscan.h.

Let's malloc a TableScanDescData, free it in endscan, and return the TableScanDescData instance in beginscan:

@@ -30,8 +30,12 @@
   ParallelTableScanDesc parallel_scan,
   uint32 flags
 ) {
+  TableScanDescData* scan = {};
   DEBUG_FUNC();
-  return NULL;
+
+  scan = (TableScanDescData*)malloc(sizeof(TableScanDescData));
+
+  return (TableScanDesc)scan;
 }

 static void memam_rescan(
@@ -87,6 +87,7 @@

 static void memam_endscan(TableScanDesc sscan) {
   DEBUG_FUNC();
+  free(sscan);
 }

Build and run (you can do it on your own). No difference.

I got stuck for a while here too. Clearly something must be filled out in this struct but it could be anything. Through trial and error I realized the one field that must be filled out is scan->rs_rd.

@@ -34,6 +34,7 @@
   DEBUG_FUNC();

   scan = (TableScanDescData*)malloc(sizeof(TableScanDescData));
+  scan->rs_rd = relation;

   return (TableScanDesc)scan;
 }

We build and run:

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
 a
---
(0 rows)

And it works! It doesn't return anything but that's correct. There's nothing to return.

So what if we actually want to return something? Let's check our logs in /tmp/pgtam.log.

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan
in memam_getnextslot
in memam_endscan

Ok, I'm getting the gist of the API. A full table scan (which this is, because there are no indexes at play) starts with an initialization for a slot, then the scan begins, then getnextslot is called for each row, and then endscan is called to allow for cleanup.

So let's try returning a row in getnextslot.

`getnextslot`

The getnextslot signature is:

bool memam_getnextslot(
  TableScanDesc sscan,
  ScanDirection direction,
  TupleTableSlot *slot
);

So the sscan should be what we returned from beginscan and the interface docs say the current row gets stored in slot.

The return value seems to indicate whether or not we've reached the end of the scan. However, the scan will still end even if you return true if the slot is not filled out correctly. If the slot is filled out correctly and you unconditionally return true, you will crash the process.

Let's take a look at the definition of TupleTableSlot:

typedef struct TupleTableSlot
{
    NodeTag     type;
#define FIELDNO_TUPLETABLESLOT_FLAGS 1
    uint16      tts_flags;      /* Boolean states */
#define FIELDNO_TUPLETABLESLOT_NVALID 2
    AttrNumber  tts_nvalid;     /* # of valid values in tts_values */
    const TupleTableSlotOps *const tts_ops; /* implementation of slot */
#define FIELDNO_TUPLETABLESLOT_TUPLEDESCRIPTOR 4
    TupleDesc   tts_tupleDescriptor;    /* slot's tuple descriptor */
#define FIELDNO_TUPLETABLESLOT_VALUES 5
    Datum      *tts_values;     /* current per-attribute values */
#define FIELDNO_TUPLETABLESLOT_ISNULL 6
    bool       *tts_isnull;     /* current per-attribute isnull flags */
    MemoryContext tts_mcxt;     /* slot itself is in this context */
    ItemPointerData tts_tid;    /* stored tuple's tid */
    Oid         tts_tableOid;   /* table oid of tuple */
} TupleTableSlot;

tts_values is an array of Datum (which is a Postgres value). So that sounds like the actual values of the row. The tts_isnull field also looks important since that seems to be whether each value in the row is null or not. And tts_nvalid sounds important too since presumably it's the length of the tts_isnull and tts_values arrays.

The rest of it may or may not be important. Let's try filling out these three fields though and see what happens.

Datum

Back in the Postgres C extension documentation, we can see some simple examples of converting between C types and Postgres's Datum type.

For example:

Datum
add_one(PG_FUNCTION_ARGS)
{
    int32   arg = PG_GETARG_INT32(0);

    PG_RETURN_INT32(arg + 1);
}

If we look at the definition of PG_RETURN_INT32 in src/include/fmgr.h, we see:

#define PG_RETURN_INT32(x)   return Int32GetDatum(x)

So Int32GetDatum() is the function we'll use to set a Datum for a cell in a row.

@@ -54,13 +54,26 @@
   DEBUG_FUNC();
 }

+static bool done = false;
 static bool memam_getnextslot(
   TableScanDesc sscan,
   ScanDirection direction,
   TupleTableSlot *slot
 ) {
   DEBUG_FUNC();
-  return false;
+
+  if (done) {
+    return false;
+  }
+
+  slot->tts_nvalid = 1;
+  slot->tts_values = (Datum*)malloc(sizeof(Datum) * slot->tts_nvalid);
+  slot->tts_values[0] = Int32GetDatum(314 /* Some unique-looking value */);
+  slot->tts_isnull = (bool*)malloc(sizeof(bool) * slot->tts_nvalid);
+  slot->tts_isnull[0] = false;
+  done = true;
+
+  return true;
 }

 static IndexFetchTableData* memam_index_fetch_begin(Relation rel) {

The goal is that we return a single row and then exit the scan. It will have one 32-bit integer cell (remember we created the table CREATE TABLE x (a INT); INT is shorthand for INT4 which is a 32-bit integer) that will have the value 314.

But if we build and run this, we get no rows.

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
 a
---
(0 rows)

I got stuck for a while here. Plugging my getnextslot code into ChatGPT helped. One thing it gave me to try was calling ExecStoreVirtualTuple on the slot. I noticed that the built-in heap access method also called a function like this in getnextslot.

And I realized that tts_nvalid is already set up and the memory for tts_values and tts_isnull is already allocated. So the code became a little simpler.

@@ -66,11 +66,9 @@
     return false;
   }

-  slot->tts_nvalid = 1;
-  slot->tts_values = (Datum*)malloc(sizeof(Datum) * slot->tts_nvalid);
   slot->tts_values[0] = Int32GetDatum(314 /* Some unique-looking value */);
-  slot->tts_isnull = (bool*)malloc(sizeof(bool) * slot->tts_nvalid);
   slot->tts_isnull[0] = false;
+  ExecStoreVirtualTuple(slot);
   done = true;

   return true;

Build and run:

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
  a
-----
 314
(1 row)

Fantastic!

Creating a table

Now that we've proven we can return random data, let's set up infrastructure for storing tables in memory.

@@ -15,6 +15,41 @@
 FILE* fd;
 #define DEBUG_FUNC() fprintf(fd, "in %s\n", __func__);

+
+struct Column {
+  int value;
+};
+
+struct Row {
+  struct Column* columns;
+  size_t ncolumns;
+};
+
+#define MAX_ROWS 100
+struct Table {
+  char* name;
+  struct Row* rows;
+  size_t nrows;
+};
+
+#define MAX_TABLES 100
+struct Database {
+  struct Table* tables;
+  size_t ntables;
+};
+
+struct Database* database;
+
+static void get_table(struct Table** table, Relation relation) {
+  char* this_name = NameStr(relation->rd_rel->relname);
+  for (size_t i = 0; i < database->ntables; i++) {
+    if (strcmp(database->tables[i].name, this_name) == 0) {
+      *table = &database->tables[i];
+      return;
+    }
+  }
+}
+
 static const TupleTableSlotOps* memam_slot_callbacks(
   Relation relation
 ) {

Based on what we logged in /tmp/pgtam.log it seems like memam_relation_set_new_filelocator is called when a new table is created. So let's handle adding a new table there.

@@ -233,7 +268,16 @@
   TransactionId *freezeXid,
   MultiXactId *minmulti
 ) {
+  struct Table table = {};
   DEBUG_FUNC();
+
+  table.name = strdup(NameStr(rel->rd_rel->relname));
+  fprintf(fd, "Created table: [%s]\n", table.name);
+  table.rows = (struct Row*)malloc(sizeof(struct Row) * MAX_ROWS);
+  table.nrows = 0;
+
+  database->tables[database->ntables] = table;
+  database->ntables++;
 }

 static void memam_relation_nontransactional_truncate(

Finally, we'll initialize the in-memory Database* when the handler is loaded.

@@ -428,5 +472,11 @@
   setvbuf(fd, NULL, _IONBF, 0); // Prevent buffering
   fprintf(fd, "\n\nmem_tableam handler loaded\n");

+  if (database == NULL) {
+    database = (struct Database*)malloc(sizeof(struct Database));
+    database->ntables = 0;
+    database->tables = (struct Table*)malloc(sizeof(struct Table) * MAX_TABLES);
+  }
+
   PG_RETURN_POINTER(&memam_methods);
 }

If we build and run, we won't notice anything new.

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
  a
-----
 314
(1 row)

But we should see a message in /tmp/pgtam.log about the new table being created.

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator
Created table: [x]


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan
in memam_getnextslot
in memam_getnextslot
in memam_endscan

And there it is! Creation looks good.

Inserting rows

Let's add INSERT INTO x VALUES (23), (101); to test.sql and run the SQL script.

$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
INSERT 0 2
  a
-----
 314
(1 row)

And let's check the log to see what method is called when we try to INSERT.

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator
Created table: [x]


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_slot_callbacks
in memam_tuple_insert
in memam_tuple_insert
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan
in memam_getnextslot
in memam_getnextslot
in memam_endscan

tuple_insert seems to be the method! Looks like it gets called once for each row to insert. Perfect.

The signature for tuple_insert is:

void memam_tuple_insert(
  Relation relation,
  TupleTableSlot *slot,
  CommandId cid,
  int options,
  BulkInsertState bistate
);

We can get the table name from relation, and instead of writing to slot we can read from slot->tts_values instead.

@@ -141,7 +141,38 @@
   int options,
   BulkInsertState bistate
 ) {
+  TupleDesc desc = RelationGetDescr(relation);
+  struct Table* table = NULL;
+  struct Column column = {};
+  struct Row row = {};
+
   DEBUG_FUNC();
+
+  get_table(&table, relation);
+  if (table == NULL) {
+    elog(ERROR, "table not found");
+    return;
+  }
+
+  if (table->nrows == MAX_ROWS) {
+    elog(ERROR, "cannot insert more rows");
+    return;
+  }
+
+  row.ncolumns = desc->natts;
+  Assert(slot->tts_nvalid == row.ncolumns);
+  Assert(row.ncolumns > 0);
+
+  row.columns = (struct Column*)malloc(sizeof(struct Column) * row.ncolumns);
+  for (size_t i = 0; i < row.ncolumns; i++) {
+    Assert(desc->attrs[i].atttypid == INT4OID);
+    column.value = DatumGetInt32(slot->tts_values[i]);
+    row.columns[i] = column;
+    fprintf(fd, "Got value: %d\n", column.value);
+  }
+
+  table->rows[table->nrows] = row;
+  table->nrows++;
 }

 static void memam_tuple_insert_speculative(

Build and run and again we won't notice anything new.

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
INSERT 0 2
  a
-----
 314
(1 row)

But if we check the logs, we should see the two column values we inserted, one for each row.

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator
Created table: [x]


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_slot_callbacks
in memam_tuple_insert
Got value: 23
in memam_tuple_insert
Got value: 101
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan
in memam_getnextslot
in memam_getnextslot
in memam_endscan

Woohoo!

Un-hardcoding the scan

The final thing we need to do is drop the hardcoded 314 we returned from getnextslot and instead we need to look up the current table and return rows from it. This also means we need to keep track of which row we're on. So beginscan will also need to change slightly.

@@ -57,6 +56,14 @@
   return &TTSOpsVirtual;
 }

+
+struct MemScanDesc {
+  TableScanDescData rs_base; // Base class from access/relscan.h.
+
+  // Custom data.
+  uint32 cursor;
+};
+
 static TableScanDesc memam_beginscan(
   Relation relation,
   Snapshot snapshot,
@@ -65,11 +72,13 @@
   ParallelTableScanDesc parallel_scan,
   uint32 flags
 ) {
-  TableScanDescData* scan = {};
-  DEBUG_FUNC();
+  struct MemScanDesc* scan;

-  scan = (TableScanDescData*)malloc(sizeof(TableScanDescData));
-  scan->rs_rd = relation;
+  DEBUG_FUNC();
+
+  scan = (struct MemScanDesc*)malloc(sizeof(struct MemScanDesc));
+  scan->rs_base.rs_rd = relation;
+  scan->cursor = 0;

   return (TableScanDesc)scan;
 }
@@ -89,23 +97,26 @@
   DEBUG_FUNC();
 }

-static bool done = false;
 static bool memam_getnextslot(
   TableScanDesc sscan,
   ScanDirection direction,
   TupleTableSlot *slot
 ) {
+  struct MemScanDesc* mscan = (struct MemScanDesc*)sscan;
+  struct Table* table = NULL;
   DEBUG_FUNC();

-  if (done) {
+  ExecClearTuple(slot);
+
+  get_table(&table, mscan->rs_base.rs_rd);
+  if (table == NULL || mscan->cursor == table->nrows) {
     return false;
   }

-  slot->tts_values[0] = Int32GetDatum(314 /* Some unique-looking value */);
+  slot->tts_values[0] = Int32GetDatum(table->rows[mscan->cursor].columns[0].value);
   slot->tts_isnull[0] = false;
   ExecStoreVirtualTuple(slot);
-  done = true;
-
+  mscan->cursor++;
   return true;
 }

Let's try it out.

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
INSERT 0 2
  a
-----
  23
 101
(2 rows)

And there we have it. :)

Awesome SQL power

So we tried one table and we tried a SELECT without anything else.

What happens if we use more of SQL? Let's create another table and try some more complex queries. Edit test.sql:

DROP EXTENSION IF EXISTS pgtam CASCADE;
CREATE EXTENSION pgtam;
CREATE TABLE x(a INT) USING mem;
CREATE TABLE y(b INT) USING mem;
INSERT INTO x VALUES (23), (101);
SELECT a FROM x;
SELECT a + 100 FROM x WHERE a = 23;
SELECT a, COUNT(1) FROM x GROUP BY a ORDER BY COUNT(1) DESC;
SELECT b FROM y;

Run it:

$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to 2 other objects
DETAIL:  drop cascades to table x
drop cascades to table y
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
CREATE TABLE
INSERT 0 2
  a
-----
  23
 101
(2 rows)

 ?column?
----------
      123
(1 row)

  a  | count
-----+-------
  23 |     1
 101 |     1
(2 rows)

 b
---
(0 rows)

Pretty sweet!

Next steps

It would be neat to build a storage engine that reads from and writes to a CSV a la MySQL's CSV storage engine. Or a storage engine that uses RocksDB.

It would also be good to figure out how indexes work, how deletions work, how updates and DDL beyond CREATE works.

And I should probably contribute some of this to the table access method docs which are pretty sparse at the moment.

I've been working this week to understand Postgres Table Access Methods for alternative storage engines.

Especially challenging because the documentation is pretty sparse and few minimal implementations exist.

I wrote up my approach!https://t.co/LQGglRkev5 pic.twitter.com/v0MeOu4Hbr
— Phil Eaton (@eatonphil) November 2, 2023

Ruthless prioritization

swaits.com — Wed, 01 Nov 2023 02:00:00 GMT

I feel overwhelmed these days. My TODO list has spiraled out of control and I feel immobilized. I need to get myself organized, and I need to ruthlessly prioritize. Someone smart at Amazon once said that you'll have 100 things you need or want to do; but you can only do one or two of those and need to throw the rest away. That's tough, but important!

Here I describe an approach I was using in the recent past, and an approach I'm going to move to today.

Contents:

My most recent pen and paper productivity system

I like productivity tools. And I always find myself coming back to pen and paper stuff. Writing things on paper is very satisfying. Something about the physicality of it just feels nice.

For some time, I used this ink+volt Dashboard Deskpad weekly planner and tracker. It's very nicely designed. If you like the looks of it, please consider ordering it from them and giving it a try. I don't regret buying and using it myself.

When I actually used this, I found it really useful. However, I've been missing a crucial ruthless prioritization step. Over time, my TODO list has exploded. Now today, I feel buried under an impossible amount of work that while I'm committed to finishing, is unrealistic.

Today I am using this planner less and less. I fell off the wagon.

My not-so-new, new pen and paper system

I decided it's time to simplify things. I've used the Eisenhower Matrix quite a bit in the past to prioritize work. It's definitely not new.

But my previous use of it was always in a digital form, such is on a Trello board or similar proprietary tools.

Now I'm reviving it as a pen and paper system. This combines the simplicity of a prioritization system I've used extensively with the pleasure of it being a pen and paper system.

So, without further ado, here it is. First, the screenshot (click for a printable PDF):

Using the system

It's pretty simple. If you're already familiar with the Eisenhower Matrix, you can probably skip this section. But, in case you're not, here's how it works: w Add tasks: As I add each task, I think about its importance and its urgency. Then I write it in the matching box.

Follow the priorities:

Execute: These are important and urgent. They're things I need to complete now.
Schedule: These are important, but not urgent. I need to schedule or otherwise plan for how I'll get to them soon. But not today.
Redirect: Tasks that are urgent, but not important. I should find ways to distribute these to teammates, friends, or family members (depending on the context) to see if they can help. Not in a "here, do my stuff" sort of way. More in a "hey, are you interested in helping with this" way.
Abandon: Not important and not urgent. In practice, I've found that I'll never get to these. So, cut to the chase. Delete them.

In summary, we only do work we deem important, and we do it or plan to do it in order of urgency. All other work is delegated or discarded.

Do the work: That's the point of this whole thing! Whenever I'm ready to start a new task, I pay attention to these priorities and let them guide my choice.

Remember, there are dopamine hits coming your way as you complete tasks and physically check them off.

Create a ritual: Each week I print a new blank sheet and transfer any incomplete tasks over. As I copy each task, I reassess its priority and importance. These things do change over time.

It's also a good opportunity to commit to the Abandon step and actually drop the tasks I'll never get to. That's why it's ruthless prioritization.

Download the PDF or copy the Google Sheet template

If you want to try this for yourself, please feel free to grab the PDF and print it, or copy the Google Sheet using the File -> Copy menu and personalize it.

Download and print the Prioritization Matrix (US Letter, PDF).
Copy the Prioritization Matrix (Google Sheet).

Enjoy!

November 2023 - Bookmarks

Domenico Luciani — Wed, 01 Nov 2023 01:00:00 GMT

Bookmarks for November 2023: 13 links - Confusing git terminology; In a git repository, where do your files live?; Working without mocks - Learn Go with tests, and more.

Random

xorvoid — Tue, 31 Oct 2023 21:20:16 GMT

Random

Real name policies

All content on Seirdy’s Home — Tue, 31 Oct 2023 20:11:51 GMT

The primary, hopefully-unintended function of a “real-name policy” is to exclude people and make people less genuine. Many aren’t at home with the name deemed by society to be “real”.

There’s plural systems. And trans users who haven’t made a full break from their deadname(s) yet. And at-risk users who would be doxed by such a policy. And folks who simply feel more comfortable with online handles than meatspace names, for no special reason in particular.

What incredible benefits could be worth shafting all these people? Assuming you even manage to pull it off; see Falsehoods Programmers Believe About Names by PatrickMcKenzie.

WCAG 2.2 and validation

All content on Seirdy’s Home — Tue, 31 Oct 2023 19:56:05 GMT

WCAG 2.2 removed SC 4.1.1, Parsing (Level A). I maintain that valid markup has important benefits despite no longer being required. We may find it possible to write good software without static analysis, construct a building without blueprints, or make an accessible website without validation. They remain good practices.

WCAG’s normative text describes outcomes, not how to achieve them. Validation is but one part of the process of achieving the necessary outcomes.

Let’s not interpret the removal of SC 4.1.1 as permission to ship known errors.

The Thinning of the Hair

callmeolive — Tue, 31 Oct 2023 19:10:00 GMT

I had a lovely conversation with my mother this week. I can’t remember what on Earth we were blabbering on about, but as I knelt down to give one of the dogs a hug, she shrieked something unforgettable: Your hair is thinning!

I couldn’t tell who was more shocked: my mum at the sight of my head, or myself. Especially when I’ve been buzzing my head for years now. It’s not like I still have the wild mop I had as a teen!

Growing up, baldness was always my very own sword of Damocles. My old man is bald, every male on his side of the family is bald, and as my brothers hit their twenties it became apparent they wouldn’t have their cranial insulation for long, and I was soon to follow.

In 2019¹ I got the clippers out and took the plunge for some daft reasons: I was fed up of maintaining my hair, and going to the barber’s is a pain. A weekly one and done took less effort, was much cheaper, and hey it turns out I look good without hair. Hell, I even picked up a safety razor for when I want to get closer.

Now I get it, baldness is scary. Despite the amount of bald folk out there, there’s still a ton of prejudices against it. It also sucks to have a core part of your identity and the way people perceive you be snatched from you without control. It’s okay to feel vulnerable about your hair, and I know how useless it is for me to say I just buzzed it mate simple.

Despite my hair not getting longer than 4mm these days, there’s still a hint of worry about my hair thinning. While I have no interest in returning to my teen mop, there’s something disconcerting about the fact that eventually I won’t be able to do anything with it, even if I don’t really want to. Could settle for a monk haircut at least.²

At the end of the day I’m not an authority on hair loss, not having to faff about with it is good enough for me. The only downsides I can come up are that my ears are marginally colder in winter and sticking your head out a car window doesn’t feel as satisfying.

And with all that said and done, if the fates wish to take my goatee away they can pry it from my cold, dead ~~hands~~ chin.

For the sake of my ego I have to make it clear that this was January 2019, way before the Covid pandemic. I was so unique, and then everyone else did the same. ↩︎
During the writing of this, I found out that the monk haircut is called a Tonsure.`` ↩︎

What Is Going On With Youtube Ads?

Matteo's Website — Tue, 31 Oct 2023 06:00:00 GMT

I've recently found myself watching a lot of Youtube videos on my flip phone because I have no audio on my new desktop (thanks, Linux). This means getting hit with ads designed to be seen by people with no viewer history, since my phone doesn't have enough RAM for cookies …

Hunting a spooky ethernet driver bug

Matt Keeter — Tue, 31 Oct 2023 02:00:00 GMT

Is Scrum Democratic?

Backendhance — Tue, 31 Oct 2023 02:00:00 GMT

Hi,

Ist Scrum demokratisch?

Diese Frage kommt immer wieder auf.

Immer wieder bin ich in Teams in denen es auf diese Frage hinausläuft.

Das Stichwort ist immer wieder “selbstorganisiertes Team”

Wie weit entscheidet ein selbstorganisiertes Team?

Unumschritten: Das Team organisiert sich die Arbeit selbst.

Der Product Owner entscheidet welche Aufgaben in welcher Reihenfolge abgearbeitet werden.

Und er ist ein Teil des Teams.

Auch unumschritten: Die Entwickler planen einen Sprint und definieren wann ein Thema “fertig” ist.

Wellness challenge for November 2023

Ben Overmyer's Site — Tue, 31 Oct 2023 02:00:00 GMT

Last month, I went over my goals for this year to remind myself what they were and see how I'm doing. It was an unpleasant surprise; I'd failed to do any of them. Then, earlier this month, I read Arnold Schwarzenegger's "Be Useful" book. His stories about growing up and getting into bodybuilding made my brain start working.

After a few days of reflection, I decided to do a wellness challenge for myself over the course of November. There are two components to this challenge:

No alcohol
Gym workouts at least four days a week

The goal with the first is to eliminate a source of empty calories and to give my body a break from the toxic effects.

The goal with the second is threefold - first, to establish a gym habit; second, to lose useless weight; and third, to try out bodybuilding as a hobby.

Giving up alcohol is not new to me. I gave it up for two months last November. I felt pretty great even just a couple weeks into it, so I'm looking forward to this.

For the workouts, I intend to go in the mornings before work. The gym that I go to is empty then, and it might be a good wake-up routine. I've chosen this workout plan from Muscle and Strength. The first couple weeks use this schedule:

Squats, 2 sets, 12 reps
Leg curls, 1 set, 12 reps
Calf raises, 2 sets, 12 reps
Bench press, 2 sets, 12 reps
Pulldowns, 2 sets, 12 reps
Cable rows, 2 sets, 12 reps
Overhead press, 2 sets, 12 reps

Tomorrow morning I'll need to figure out what weight I should be using for each. I will be logging my workouts so I can figure out how I'm doing over time. I know how to do all of the above exercises except for the overhead press, so I'll need to make sure I get the proper form for that one before I really push it.

How I Do Things: Chronic Fatigue Edition

taylor.town — Tue, 31 Oct 2023 02:00:00 GMT

I feel like a battery that can't keep its charge.

Running bare metal PostgreSQL on ZFS

Ellie's Notes — Tue, 31 Oct 2023 02:00:00 GMT

I'm setting up new postgres servers for Atuin! We're going with a hot replica this time, and making things much more reliable. Atuin has had no outages or database issues in a couple of years, but I don't want to push my luck.

You might also be interested in the hetzner k3s setup I did for the Atuin api images

I'm going to be doing a fairly minimal setup to begin with, and tune things later. I'm not sure which options will best suit my workload, so will keep things simple.

Note that the Atuin queries are not complicated. We mostly just store a pretty high volume of data, and read sequentially (ish). There are minimal joins, and minimal complex queries.

We will often have bursts where 10s/100s of thousands of rows need to be written or read as quickly as possible - however this is also pretty sequential and not at all "complex".

Ideally, we will compress the data as much as possible. While Atuin mostly stores encrypted data (which does not compress well), there is a decent amount of unencrypted data in the form of JSON padding + timestamps etc. Compress them! This will nicely reduce disk usage, but also IO - at the cost of some CPU.

As our queries are simple, the CPU cost is fine.

Setting up

I'm running this on a couple of hetzner machines I bought in an auction. They have Ryzen CPUs, and 4x 1tb nvme SSD.

Two of the SSDs will be running a simple RAID mirror, and will store the OS and logs. The other two will have my ZFS filesystem + postgres on them.

This costs me 50% of my storage in mirroring, however as this is bare metal hardware there's a chance that a disk could fail. I'd like to ensure my database can continue running until I failover to the replica + a technician can replace the disk.

# Install zfs
apt install zfsutils-linux

# check version
zfs version

Check your disk layout with lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme1n1     259:0    0 953.9G  0 disk
├─nvme1n1p1 259:1    0    32G  0 part
│ └─md0       9:0    0    32G  0 raid1 [SWAP]
├─nvme1n1p2 259:2    0     1G  0 part
│ └─md1       9:1    0  1022M  0 raid1 /boot
├─nvme1n1p3 259:3    0   128G  0 part
│ └─md2       9:2    0 127.9G  0 raid1 /var
├─nvme1n1p4 259:4    0     1K  0 part
└─nvme1n1p5 259:5    0 792.9G  0 part
  └─md3       9:3    0 792.7G  0 raid1 /
nvme0n1     259:6    0 953.9G  0 disk
nvme3n1     259:7    0 953.9G  0 disk
nvme2n1     259:8    0 953.9G  0 disk
├─nvme2n1p1 259:9    0    32G  0 part
│ └─md0       9:0    0    32G  0 raid1 [SWAP]
├─nvme2n1p2 259:10   0     1G  0 part
│ └─md1       9:1    0  1022M  0 raid1 /boot
├─nvme2n1p3 259:11   0   128G  0 part
│ └─md2       9:2    0 127.9G  0 raid1 /var
├─nvme2n1p4 259:12   0     1K  0 part
└─nvme2n1p5 259:13   0 792.9G  0 part
  └─md3       9:3    0 792.7G  0 raid1 /

/dev/nvme1n1 and /dev/nvme2n1 are both in use for my OS, but /dev/nvme0n1 and /dev/nvme3n1 are leftover for my zfs mirror.

Create the mirror

zpool create postgres mirror -o ashift=12 /dev/nvme0n1 /dev/nvme3n1

It returned nice and fast!

Checking again with lsblk shows some usage

nvme0n1     259:6    0 953.9G  0 disk
├─nvme0n1p1 259:14   0 953.9G  0 part
└─nvme0n1p9 259:15   0     8M  0 part
nvme3n1     259:7    0 953.9G  0 disk
├─nvme3n1p1 259:18   0 953.9G  0 part
└─nvme3n1p9 259:19   0     8M  0 part

Confirmed with zpool status

  pool: postgres
 state: ONLINE
config:

	NAME         STATE     READ WRITE CKSUM
	postgres     ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    nvme0n1  ONLINE       0     0     0
	    nvme3n1  ONLINE       0     0     0

errors: No known data errors

The pool is mounted by default at /postgres. Nice!

Postgres

Before we can setup datasets, we need to setup postgres. Postgres needs to run its init before we move it to zfs. Annoying, but ok.

Ubuntu 22.04 has a pretty old version in its repos, so install the postgres one

sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -

apt update

apt install postgresql-16 postgresql-contrib-16
systemctl enable postgresql
systemctl start postgresql

Then, stop postgres. Move its data to a temp location, create the datasets, and move things back again. Could probably rsync instead of mv/cp to keep permissions.

systemctl stop postgresql

# move postgres data to temp
mv /var/lib/postgresql/16/main/pg_wal /tmp/pg_wal
mv /var/lib/postgresql /tmp/postgresql

# create the datasets
zfs create postgres/data -o mountpoint=/var/lib/postgresql
zfs create postgres/wal -o mountpoint=/var/lib/postgresql/16/main/pg_wal

# switcharoo the data back
cp -r /tmp/postgresql/* /var/lib/postgresql
cp -r /tmp/pg_wal/* /var/lib/postgresql/16/main/pg_wal

# sort perms
chmod -R 0700 /var/lib/postgresql
chmod -R 0700 /var/lib/postgresql/16/main/pg_wal
chown -R postgres: /var/lib/postgresql

# start postgres once more
systemctl start postgresql

Check all is ok:

zfs list

NAME                 USED  AVAIL     REFER  MOUNTPOINT
postgres             600K   922G       24K  /postgres
postgres/db           72K   922G       24K  /postgres/db
postgres/db/base      24K   922G       24K  /postgres/db/base
postgres/db/pg_wal    24K   922G       24K  /postgres/db/pg_wal

Configuring ZFS

I've read a bunch, including:

https://vadosware.io/post/everything-ive-seen-on-optimizing-postgres-on-zfs-on-linux/
https://bun.uptrace.dev/postgres/tuning-zfs-aws-ebs.html

One of the above articles had issues where some nvme hardware was reporting a successful write when using fdatasync, but not actually writing ok. Many drives will report a successful write once the data has been stored in the volatile write cache, but not actually stored. Disable this with:

apt install nvme-cli
nvme set-feature -f 6 -v 0 /dev/nvme0n1
nvme set-feature -f 6 -v 0 /dev/nvme3n1

I spend a while thinking about the optimal recordsize. While I'd get a higher tps running recordsize = postgres block size = 8kb, Atuin is largely sequential + reads/writes large amounts of data. I could change the postgres block size (it's a compilation option), but I'll consider trying that in the future.

To begin with, I'll try the default of 128k and see how it goes. Lower numbers are potentially faster, but higher numbers tend to get better compression. It depends on a lot of factors though, so I'll measure things and see.

# enable compression
zfs set compression=zstd-3 postgres

# disable access time (so, so many writes...)
zfs set atime=off postgres

# enable improved extended attributes
zfs set xattr=sa postgres

# zfs set recordsize=16k postgres

I then set full_page_writes = off on the postgres side - zfs cannot write partial pages so it's pretty redundant.

Next up, we'll give the ARC (ZFS page cache) 75% of the system memory. The remainder will be used by postgres shared_buffers.

echo 51539607552 >> /sys/module/zfs/parameters/zfs_arc_max

To persist across reboots, set this in /etc/modprobe.d/zfs.conf

options zfs zfs_arc_max=51539607552

Postgres config

Not specific to ZFS, but some general postgres tuning

# 25% of 64GB
shared_buffers = 16GB

work_mem = 8MB

# make vaccuums/etc faster
maintenance_work_mem = 1GB

# tell the planner how much the ZFS ARC will likely cache
effective_cache_size = 48GB

There's a bunch more tuning we can do, but realistically this will help the most. Postgres has a pretty tiny amount of memory configured by default!

At this point, I rebooted to ensure everything was OK + persisted properly.

Chuck load at it

I'd like to ensure my system performs at least mostly ok, so I ran pgbench and got

scaling factor: 50
query mode: simple number of clients: 20 
number of threads: 4 
maximum number of tries: 1 
number of transactions per client: 100000 
number of transactions actually processed: 2000000/2000000 
number of failed transactions: 0 (0.000%) 
latency average = 2.863 ms 
initial connection time = 10.287 ms 
tps = 6984.804317 (without initial connection time)

Not bad! Not that representative of my workload, but at least an indication it's not totally broken. I'll tune this some more in the future.

Next I'll be configuring backups with pgbackrest, and setting up a hot standby we can failover to should things go wrong!

Then copying the dataset across, and making this new database production 🚀

Exploring Pytest Fixtures: Notes and Examples

vladsiv — Tue, 31 Oct 2023 02:00:00 GMT

Here, I present a compilation of notes and practical scenarios drawn from my experiences, demonstrating the effective utilization of pytest fixtures. These examples provide valuable insights into leveraging fixtures to refine and improve the architecture of your testing module.

SDXL LoRA Training

smcleod.net — Mon, 30 Oct 2023 22:00:00 GMT

A talk I gave to some peers on creating your own SDXL LoRA models from my tinkering around over the last few weeks.

Note 121

Qubyte Codes — Mon, 30 Oct 2023 17:01:36 GMT

Big skies over Brighton these days.

Paper Hoard

Love, Blood & Rhetoric — Mon, 30 Oct 2023 07:05:19 GMT

Harold had expected the last dragon would be … larger. “Disappointing, I know.” The dragon, barely three feet long, spoken in a voice like a bamboo flute—high, thin and melancholy. “Hardly a challenge at all. Go ahead, kill me. Get it over with.” “I’m not here to kill you,” Harold said. “You’re a link to […]

How I smash burgers

xenodium.com @alvaro — Mon, 30 Oct 2023 02:00:00 GMT

I'm neither a burger expert nor a connoisseur of any kind, yet I sure have a lot of fun smashing burgers at home. Needless to say, I shamelessly enjoy gobbling them too!

my smash burger

I'll share details on how I smash my burgers, but here's a quick ingredient list, if that's all you need.

Mince beef (20%-30% fat).
Streaky bacon.
Brioche burger buns.
American cheese slices (cheddar individual slices work too).
Lettuce.
Tomatoes.
Onions.
Pickles.
Jalapeños.
Garlic.
Chipotle powder.
Mayonnaise.
Salt.
Pepper.
Oil.
Greaseproof paper.
Butter.

The calling

My quest to smash burgers at home didn't start until earlier this year, while watching the The Menu. I just could't stop craving the burger from that scene, so I set out to start smashing my own.

The gear

Don't rush to buy anything fancy. Your existing gear will likely do the job just fine. I'd say try a few things out and only upgrade when needed. I'll share the gear I use and where I felt I needed tweaking.

Skillet

While I didn't have a griddle at home, I did have a couple of trusty Lodge skillets (cast iron and carbon steel). Both work great for burgers, though I have a slight preference for the carbon steel one, as it's the bigger of the two and gives a little more room for manoeuvring, specially when smashing two burgers at a time.

Heat the skillet up and add a little oil. If the oil starts smoking, be quick to drop the patties and start smashing.

Grill Spatula (too big/stiff for skillet)

Somewhat inspired by the film, I got myself a wide spatula so I could firmly press those patties against the skillet, and to flip of course.

While this kind of spatula may work well on a spacious griddle, I felt constrained on a relatively small cast iron. Specially when flipping. I went looking for an alternative.

Spatula + smasher (my winning combo)

Over at the r/castiron subreddit, I discovered fish spatulas. They are fairly agile on cast irons but also work great for loosening burger patties before flipping.

While effective for flipping, fish spatulas are obviously no good for smashing. So I got myself a burger smasher. This combo worked well for me.

When smashing, use greaseproof paper to prevent the patties from sticking to the smasher.

Ingredients

While I've drawn inspiration from others, I've landed on my own preferred ingredients. I'm sure that will continue changing over time. Pick and choose as your heart desires.

Minced/ground beef

Minced beef with higher fat content (around 20-30%) is often recommended for a couple of reasons:

Flavour: Fat equals flavour in cooking. The higher fat content will melt during cooking and become 'self-basting', resulting in a juicier and more flavourful burger.
Texture: The fat in the beef melts under heat, helping the burger achieve a crispy, caramelized exterior known as the Maillard reaction, which contrasts nicely against the soft, juicy interior.

In the UK, I can typically find minced beef with 15%-20% fat content at the main supermarkets.

Be sure to salt and pepper to taste (as in picture) on one side. Once flipped on pan, salt and pepper the other side.

Bacon

I tend to prefer smoked streaky bacon, but hey these will be your burgers. Your burgers, your rules.

Buns (brioche)

I hear potato buns are great for burgers. I've yet to try them. So far, I've settled for brioche. I happen to find these near me, so I've gone with them.

Butter the buns and brown on the skillet for a minute. Check the buns often. Brioche buns can burn quickly.

American cheese

American cheese is often the burger cheese of choice.

While American cheese isn't widely available in the UK, the individually wrapped orange-looking cheddar cheese slices work just fine.

Toppings

I like my burgers with lettuce, tomatoes, onions, pickles, and occasionally jalapeños. For pickles, I typically just take cornichons and slice them up.

Burger sauce (chipotle/garlic/mayo)

While classic burger sauce is often made with mayo, ketchup, pickles, and mustard, I've gone fairly rogue here.

You see, I love chipotle mayo. I'm also a fan of garlic mayo, so I figured why not both? Turns out these three ingredients work great together.

I like to draw out the flavours by first mixing the garlic and chipotle with a little hot water.

1 garlic clove.
2 teaspoons of chipotle powder.
1 tablespoon of hot water.
Pinch of salt.

…and then thicken with mayo.

1/4 cup of mayo.

These are very rough measurements, tweak to your preference. Make more garlicky, spicier, or soften things by adding garlic, chipotle, or mayo.

Assembling

I like to assemble in the following order from the bottom bun up.

Sauce on bottom bun.
Lettuce.
Tomatoes.
Onions.
2 patties (melted cheese on both).
Bacon.
Pickles.
Jalapeños.
Sauce on top bun (oops, I forgot in the picture).

…and here's the final product.

If you gave smashing burgers a go, I'd love to hear about it. Also any tips are very much welcome. Get in touch (Mastodon / Twitter / Reddit / Email).

Partitioning The Index

Weblog on marginalia.nu — Mon, 30 Oct 2023 02:00:00 GMT

So a bit of an update on what I’ve been working on. This will be adapted into release notes in a while, but I haven’t quite wrapped a bow on the change set yet.

Still, it has certainly been a few weeks. Didn’t quite land how busy I’ve been until I set down to draft this post. Them’s some changes, and I’m skipping a few to keep this meandering post at a sane length.

Python, C, Assembly - 2'500x Faster Cosine Similarity 📐

Ash's Blog — Mon, 30 Oct 2023 02:00:00 GMT

From pure Python to AVX-512 assembly, optimizing cosine similarity reveals a 2,500x speedup through SIMD, FP16, and VNNI instructions on modern Intel CPUs.

Build The Custom Feature

rrich.io — Mon, 30 Oct 2023 02:00:00 GMT

Investors, mentors, and fellow founders often advise against tailoring features or functionality for just a handful of customers (let alone a single customer!).

That time I wrote malware and got caught

ntietz.com blog - technically a blog — Mon, 30 Oct 2023 02:00:00 GMT

Most of us make some bad decisions in high school. While other people were drinking, going to parties, and who knows what else, I was doing some experimentation of my own. I was writing my first (and only) piece of malware.

From as early as I can remember, I've had a fascination with security. In games, I would play rogues and try to pickpocket people or pick open locks. This came from two inner drives.

The first one is the obvious one: getting access to things you're not supposed to. I was a curious kid¹ and wanted to know everything. Nothing seemed like it should not be my business. So playing as a character who could get into any room was just my cup of tea: unfettered access to know what was going on.

The second one was less obvious and took me a while to realize in myself: a deep desire to know how things work. More precisely, a deep desire to figure out how things work. If I read a book about something, that's fun and I learn something. But if I poke at a system enough to figure out how it works and why it works that way, that's deeply satisfying and such a thrill.

Our high school had computers in most classrooms, and we had a few computer labs² . Like is common with computer labs, these required logging in with your school credentials³ . Once you logged in, you had access to your personal drive (mounted at P:\\, I assume for "personal") as well as a few shared drives. Students could read files on some of these drives, and could read and write to one of them. Is this foreshadowing? Is it ever.

Another thing that our computers had was ~~spyware~~ "monitoring software" so that the lab supervisor could see what we were doing. On the one hand, high school students do many unwise things so this is probably a reasonable practice. But on the other hand, it inures people to being spied on, and it definitely didn't prevent me from doing naughty things, soooo... it wasn't very effective. To prevent us from killing the process that monitored us, we had no access to Task Manager.

My junior year, I was in a programming class, and we used .NET languages (VB.NET and C#) in our classes. Since we were using Visual Studio, we had access to a fun drag-and-drop builder, and we also had hooks into Windows APIs to do convenient things. You could capture keystrokes, like Ctrl-C for copy if you want to do something different with it.

Naturally, I wanted to explore the limts of these APIs. What would it let me capture, and what would it not? Unfortunately, they let me capture almost everything. From here, I created my malware: Fluffy, Destroyer of Worlds.

Fluffy was a simple program. When you ran it, it would expand itself to full screen and display a picture of a kitten jumping through a field, labeled with "Fluffy, Destroyer of Worlds"—in Comic Sans, of course. Below that was a loading bar which started out quick but would slow down exponentially, so you would get to 90% quickly but then would never get to 100%.

Users would sit there and wait expectantly for this program that Nicole wrote to do something cool, presumably. But eventually they'd pick up on the gag, maybe because I was giggling. So they would try to close it. But I was able to capture Alt-Tab and prevent the user from changing windows. And I was able to capture Ctrl-Q and Alt-F4 and prevent the user from closing the program. I was not able to capture Ctrl-Alt-Delete.. but that took you to a login screen that only had options to resume, log out, or restart the computer (no admin controls could even override this, shocking to me to this day).

They had no choice but to log out or restart, which would make them lose any work they had open.

Like any good hacker, I developed my malware in my parents' basement. And like any good hacker, I tested it on my friend first. Shoutout to Andrew for running something I sent him without really questioning it. He got off easy because his home computer did not have Ctrl-Alt-Delete blocked. We tried to transfer it to him by AIM or email, but .exe files were blocked, so naturally we transferred it by pretending it was a .zip file.

Once Andrew had confirmed that it did work as expected, I carried Fluffy, Destroyer of Worlds to school with me on a flash drive. Our computers didn't prevent running arbitrary executables, so I was able to just copy it onto my personal drive and run it. But it was more fun if someone else ran it, so I put it on that shared drive. (It returned!)

Then I told my friends to run it. They thought it was funny. I had my Latin teacher run it, and she lost half a period of notes; I felt slightly bad about that. My English teacher ran it, and he thought it was hilarious even though he lost notes too. I thought that was the end of it, I'd had my fun.

The next day my programming teacher asked me about the program. Apparently, some other people had run it, because they found it on the shared drive. And some of them had our librarian run it, and hoo boy she did not find it funny in the slightest. She wanted me to immediately lose all computer privileges which, honestly, fair.

My programming teacher went to bat for me, and struck a deal with IT to keep my computer privileges⁴ . The deal was that I had to get rid of the program and monitor for it coming back, and make sure (as far as I could) no one else was affected by it. I deleted that copy from the shared drive but people kept putting it back. Why??? So I kept deleting it over and over, until the novelty wore off and we all forgot about it.

Side note, can we just say how shocking it is that everyone ran a random executable? That we just ran things we found? Security understanding sure has changed over the last two decades.

High school was a weird time.

By writing some very unsophisticated malware, I learned quite a bit.

I was able to explore the bounds of a system and what it was able to do. But more importantly, I learned that writing malware wasn't harmless and could hurt other people and could also put my own activities at risk. It scared me out of doing any sort of security work for a while.

It taught me how much privilege I had. What I did was not legal and violated school rules, and some people may have had the book thrown at them. Instead, I had a teacher and mentor go to bat for me and ensure I could keep on learning.

It also taught me about the boundaries of systems, and the ways that security features can be abused. The ways that the systems we put in place can be exploited. Exploring systems, boundaries, what you can and cannot do—such a great way to learn. Just, do it with consent.

Taming Your Infinite Queues

taylor.town — Mon, 30 Oct 2023 02:00:00 GMT

Strategy is the best weapon you wield against the infinite. Bad algorithms invite anxiety, overwhelmedness, and frustration. Good algorithms create pride, peace, and accomplishment.

Why Don't More Languages Have a call/cc Operator?

Lambda Land — Mon, 30 Oct 2023 02:00:00 GMT

Something I’ve wondered about for a little while: why don’t more languages have a call/cc operator? Having first-class continuations in your programming language gives your programmers a powerful construct. So why do only a handful of languages have it?

The short answer is: it’s tricky to implement efficiently. One way to get call/cc is to convert your code into continuation-passing style. Then, call/cc simply takes the continuation in that representation and binds it to a variable. Most languages don’t seem to go through a continuation-passing style conversion pass though, so there’s no continuation to grab.

I asked Matthew Flatt about this today, and his answer was that most languages use the C model of functions: when you call a function, you push the arguments to the function onto a stack along with the return address. Then, when you return, you pop those element back off the stack. To get call/cc, you’ve have to copy the entire stack and pass that around.

October retro: Accessibility, Dev Events, Sustainability

Fabian's public notepad — Sun, 29 Oct 2023 23:10:00 GMT

I've recently made a few accessibility improvements for the site. For one, I finally got around to add a skip-to-content link to the navigation, mostly following along an article from CSS Tricks, with very few tweaks. And I replaced the sorttable script for the tables of content with a custom-element I wrote, in order to make the table columns also sortable via keyboard navigation. On the same note: some time ago I wrote that small things matter for accessibility, which as a general principle is true, but I recently learned that the example which triggered the article is not correct anymore, because bold and italics actually aren't read by screen readers after all. That was a feature of the NVDA screen reader, but not a very popular one, therefore it got disabled by default. Generally there is no audible distinction between i/em or b/strong tags for screen reader users.

Also three weeks ago, I had the pleasure to participate in my first post-pandemic in-person developer event (which incidentally has been my last pre-pandemic conference as well...), the unKonf in Mannheim. The unKonf is basically a barcamp, but it is traditionally opened with a keynote, which this year was given by Uwe Friedrichsen on the topic of sustainability patterns for IT. One fact that suprised me was that the global IT landscape causes more greenhouse gas emissions than the traffic of the worldwide logistics industry. I can only recommend the slide deck, which is filled with tons of interesting and important pointers. One take-away was that measuring and visualizing resource and energy consumption of a system is a necessary first step to move towards a more sustainable IT. Apart from that, there were many other great sessions and the "hallway track" made me realize how much I actually have missed to talk shop (outside of my actual shop).

Firefox Tabs Design

tbolt.space — Sun, 29 Oct 2023 17:20:42 GMT

Over two years ago Firefox launched a new design with a change to how tabs look: Inspired tab design: Floating tabs neatly contain information and offer cues only when you need them, like visual indicators for audio controls. The rounded design of the active tab signals the ability to easily grab and move tabs as needed.

How Acrylic Painting Makes Me a Better Remote Worker

The Pittsburgh Roboticist Blog - Matt Puchalski - Pittsburgh Roboticist — Sun, 29 Oct 2023 17:16:00 GMT

Exploring the unexpected intersection of acrylic painting and remote work during the pandemic. Dive into a personal journey that reveals how a canvas and brush can transform the digital workspace. From mindful breaks amidst screen chaos to celebrating small wins, discover how art offers refreshing insights for the remote worker.

Two-factor login

dcz's posts — Sun, 29 Oct 2023 16:00:00 GMT

Two-factor login

Do you still type a password to get into your laptop? What about in the office? Or when you're on the train? What if there are cameras above your head? Have you ever worried that a snooper might get your password and do bad things with it?

I'm sure you haven't. Like a properly security-aware hacker, you disabled password-based SSH login already. You also don't reuse your laptop's password for anything else, especially not for your backups. Even if someone snoops the password, all they get is access to the laptop itself. Right?

Haha, I betcha. Let's face it: opsec is hard. Upgrading your opsec skill takes effort. Good opsec reduces the likelihood that secrets will be spilled.

That's why I decided to augment my laptop with two-factor login.

Now, I can whip out my laptop on the train, and log in to it without worrying too much whether the password gets intercepted. It's unique, so the attacker won't hack any of my other properties (and I'm not leaving my laptop sitting around). I see you grumble: "another unique password is hard to remember! What's the point if you could just change your login password?" The hardware token makes short and memorable passwords possible, because it bricks itself after 3 failed guesses! So I can get away with a password that's really easy to remember, without losing resistance to brute forcing.

Now I can also leave the laptop in a public space for 5 minutes. Even the security guard who used the ceiling camera to record me typing the password won't get into my laptop undetected. After all, I took the hardware token with half of my login credentials with me!

WARNING: I only cover the user login procedure in this guide. Not unlocking the disk encryption. The evil security guard can still compromise the laptop by watching me enter the decryption password and then rebooting the system. Stealing the laptop is not in my threat model for this blog post.

Second factor

Two-factor authentication splits your secret into pieces of two different types (factors), which have different security properties. This is to reduce the likelihood of the entire secret getting intercepted. Each factor may get intercepted easily on its own, but both at the same time is going to be much more difficult because of how they differ.

There are two most common factors: "something you know", like a password, and "something you have", like a hardware token. Passwords can be intercepted by watching someone type them, or by key logging, or by retrieving it from badly secured storage. A hardware token cannot be duplicated, and its secret can only be intercepted by physically stealing the token.

While hardware tokens contain some data constituting the secret, it's not possible to get that data out. If you can duplicate the secret, then the token not doing its basic job as a "something you have", and stops being a second factor to a password. It basically becomes a password with extra steps.

Login setup

This guide will show you how to enable 2-factor authentication with a password and a smart card (Nitrokey Pro) on Fedora 37. It does not disable the original password, so that locking yourself out is harder.

First, install some packages:

sudo dnf install sssd opensc p11-kit gnutls-utils nitrokey-app easy-rsa

Let's play around with the smart card itself. Plug it into the USB port and give it a go. First, run nitrokey-app and set up the user and administrator passwords (if you have trouble finding the options, check out the menu bar).

Now, play around with the key.

$ p11tool --list-tokens
[...]

Token 2:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=ZeitControl;serial=xxxxxx;token=OpenPGP%20card%20%28User%20PIN%29
        Label: OpenPGP card (User PIN)
        Type: Hardware token
        Flags: RNG, Requires login
        Manufacturer: ZeitControl
        Model: PKCS#15 emulated
        Serial: 000500005c61
        Module: opensc-pkcs11.so


Token 3:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=ZeitControl;serial=xxxxxx;token=OpenPGP%20card%20%28User%20PIN%20%28sig%29%29
        Label: OpenPGP card (User PIN (sig))
        Type: Hardware token
        Flags: RNG, Requires login
        Manufacturer: ZeitControl
        Model: PKCS#15 emulated
        Serial: 000500005c61
        Module: opensc-pkcs11.so

Remember this command, cause the URL field will be needed later.

Certificates

The login procedure involves checking PKCS12 certificates, so let's set up one using easy-rsa. First, create a file called "vars" in some directory, and put this inside:

set_var EASYRSA_KEY_SIZE 4096

Now, let's create the entire certificate chain for the user "thatsme". It will ask you for different passwords: one for the certificate authority (you only need to use it when adding another certificate), and one for the private key (you'll need it once, when uploading the private key to the smart card), so pay attention to them.

ln -s /usr/share/easy-rsa/3.0.8/ easyrsa
EASYRSA_VARS_FILE=`pwd`/vars ./easyrsa/easyrsa init-pki
EASYRSA_VARS_FILE=`pwd`/vars ./easyrsa/easyrsa build-ca
EASYRSA_VARS_FILE=`pwd`/vars ./easyrsa/easyrsa build-client-full thatsme

Whew, this wasn't as difficult as the last time I tried to set up a new certificate authority. Good job, easy-rsa folks!

Now, store the generated private key and the corresponding certificate (approved by the newly minted certificate authority) into the smart card.

WARNING: we generated a private key on the host computer. It can be intercepted by a rogue application on this computer, especially if you don't delete it fully (and deleting is harder than it seems). The interceptor will be able to follow the next steps to duplicate your smart card. If you want to be super careful, you should generate the key directly on the smart card, and use Certificate Signing Requests (CSRs) with easy-rsa to approve the key. This is not relevant to me, because I not will use the smart card to log in to any other computer. If someone snooped the key from the laptop, I'm already hosed. So I chose the easy way to generate the key.

pkcs15-init --delete-objects privkey,pubkey --id 3 --store-private-key ./pki/private/thatsme.key --format pem --auth-id 3 --verify-pin
pkcs15-init --id 3 --store-certificate ./pki/issued/thatsme.crt

See how snugly the key and the certificate both sit on your smart card now? Use the URL you noted earlier to show them.

$ p11tool --provider /usr/lib64/opensc-pkcs11.so --list-all pkcs11:model=PKCS%2315%20emulated;manufacturer=ZeitControl;serial=xxxx;token=OpenPGP%20card%20%28User%20PIN%29
Object 0:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=ZeitControl;serial=xxxx;token=OpenPGP%20card%20%28User%20PIN%29;id=%03;object=Authentication%20key;type=public
        Type: Public key (RSA-4096)
        Label: Authentication key
        Flags: CKA_WRAP/UNWRAP; CKA_EXTRACTABLE; 
        ID: 03

Object 1:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=ZeitControl;serial=xxxx;token=OpenPGP%20card%20%28User%20PIN%29;id=%03;object=Cardholder%20certificate;type=cert
        Type: X.509 Certificate (RSA-4096)
        Expires: Sun Sep 28 09:07:36 2025
        Label: Cardholder certificate
        ID: 03

I tried listing the private keys, but for some reason, p11tool won't display any.

Login

Now that the card is prepared, let's set up the login. First, copy the certificate authority to where sssd can find it. Any key signed with this certificate will be candidate for login.

sudo cp pki/ca.crt /etc/sssd/pki/sssd_auth_ca_db.pem
sudo chmod ugo-w /etc/sssd/pki/sssd_auth_ca_db.pem

Then, set up sssd itself by creating the file /etc/sssd/conf.d/base.conf with the following contents:

[sssd]
services = nss, pam
domains = files
disable_netlink = true

[domain/files]
id_provider = files

[pam]
pam_cert_auth = True
pam_p11_allowed_services = +kde, +kscreensaver, +sddm

[certmap/files/thatsme]
matchrule = <SUBJECT>^CN=thatsme$

NOTE: The common name (file name) given to easy-rsa must match the username on this system. If anyone knows how to break that relationship, please tell me! I know it has to do with sss-certmap, but the docs are incomprehensible to me.

Finally, enable sssd. Do this as root:

systemctl enable sssd
authselect select sssd with-smartcard --force
systemctl restart sssd

And that's it. When the smart card is plugged in, you should be able to see this:

$ sudo echo "ok"
PIN for OpenPGP card (User PIN):

rather than the normal

$ sudo echo "ok"
Password:

Similarly, the KDE screen locker will accept the smart card password rather than the user password. It doesn't indicate that this is happening though, other than being really slow with the card in.

Debugging

Enable logging by adding the debug_level entry:

[pam]
pam_cert_auth = True
debug_level = 9

and restarting sssd.

The two relevant log files are here:

/var/log/sssd/sssd_pam.log

and

/var/log/sssd/p11_child.log

but they are not that helpful, because sssd documentation doesn't really explain a lot.

Good luck upgrading your opsec!

Actually, no login

I did all of that, I rebooted my computer, I plugged in the Nitrokey, and I entered my password at the SDDM login screen. The prompt takes some long seconds to resolve – good, it's working with the card.

Login refused

Wait, what? I try again, same result. I try my regular password, and that goes through.

The logs in /var/log/sssd/sssd_pam.log show:

[pam_reply] (0x4000): [CID#3] pam_reply initially called with result [9]: Authentication service cannot retrieve authentication info. this result might be changed during processing
[pam_reply] (0x0200): [CID#3] blen: 22
[pam_reply] (0x0200): [CID#3] Returning [9]: Authentication service cannot retrieve authentication info to the client
[client_recv] (0x0200): [CID#3] Client disconnected!

Meanwhile, a successful use in sudo:

[pam_reply] (0x4000): [CID#9] pam_reply initially called with result [9]: Authentication service cannot retrieve authentication info. this result might be changed during processing
[pam_reply] (0x0040): [CID#9] Assuming offline authentication setting status for pam call 249 to PAM_SUCCESS.
[pam_eval_prompting_config] (0x4000): [CID#9] No prompting configuration found.
[may_do_passkey_auth] (0x0400): [CID#9] Passkey auth not possible, SSSD built without passkey support!
[pam_reply] (0x0200): [CID#9] blen: 22
[pam_reply] (0x0200): [CID#9] Returning [0]: Success to the client

I don't know if this difference is relevant. My search engine skills failed me, and I eventually gave up.

I guess I'll just have to use the main password for login. Thankfully, when traveling, I resume the laptop from sleep, and the screen locker accepts the smart card password, so it's not that bad.

Either way, if you know how to solve this problem (any sddm devs around?), please let me know! I'll update this post with your info.

Thanks to one of my coworkers at Purism, for showing me that not every password must be guarded jealously.

Career advice, 10 years in

Ernest Oppetit — Sun, 29 Oct 2023 14:32:59 GMT

It just occurred to me that I’ve been working professionally for 10 years 👴 Here are a few aphorisms and tidbits I’ve picked up along the way, shared in the form of Kevin Kelly’s. Your mileage may vary! Also sadly I’ve found that the best way to internalise advice is to actually forget it when […]

Note 120

Qubyte Codes — Sun, 29 Oct 2023 14:30:06 GMT

I made a cheesecake the other day.

Light at the end of the tunnel

deadlime — Sun, 29 Oct 2023 12:13:38 GMT

Accessing your local machine from the Internet through SSH

Analyzing Data 170,000x Faster with Python

sidsite — Sun, 29 Oct 2023 03:00:00 GMT

The article, Analyzing Data 180,000x Faster with Rust, first presents some unoptimized Python code, and then shows the process of rewriting and optimizing the code in Rust, resulting in a 180,000x speed-up. The author notes:

Daufuskie and Hunting Island 2023

Sam Solomon — Sun, 29 Oct 2023 03:00:00 GMT

A road trip through the South Carolina lowcountry.

How to Get Total Views Statistic on Substack

Chris' Full Stack Blog RSS Feed — Sun, 29 Oct 2023 03:00:00 GMT

A simple script gets the job done!

My Lolcow & Me

foreverliketh.is — Sun, 29 Oct 2023 03:00:00 GMT

Image Details & Source

man in red jacket standing beside brown cow on green grass field during daytime by Frank Holleman on Unsplash

Click Here for the Original, High-Quality Image

This page was last modified: October 31st, 2023

🎶 Well, Ali Baba had them forty thieves 🎶 #

Back in my college days, towards the start, I made an acquaintance that I’ll never forget. Over time, I’d grow to know him. I want to imagine that there was a period in that relationship where we would have sincerely considered one another “friend”. For the longest time, I certainly would have called him that. But, now, years later, I sit here at my keyboard with the realization that what I actually treated him like, was a “Lolcow”.

I’m not using that word lightly either. I’m well aware of its connotations online. I link specifically to that Urban Dictionary definition because it excludes the elements of harassment, stalking, bullying and more, associated with the term. Sadly, even without those aspects, it’s still not a pretty picture. The dynamic is about extracting (“milking”) comedy (“lols”) from a human being (“a cow”). Almost undoubtedly at their expense, and irrespective of their awareness of it.

Right now, online, there’s a bit of a mainstream-adjacent zeitgeist skirting around the label and the individuals, and groups, (unfortunate enough to be) caught within its orbit. While I reckon this growth in popularity is primarily a consequence of the paradoxically unquenchable thirst for “content” that our oversaturated, ad-centered, corporatized, modern internet possesses. I also think there’s a little more to it.

🎶 Scheherazade had a thousand tales 🎶 #

Alonso was… really something else, yo. From joining cults and renouncing loving family members to running with gangs and taking first dates on creepy, orgy-esque outings. My most insane life stories feature Alonso. And at some moment in our time together, I began seeing him as less of a person and more of a generator of wild, comical experiences. The decisions I’d make regarding him warped to the degree of prioritizing concepts like “entertainment potential” over more sensible things like “personal wellbeing” or “troublesome inconvenience”. Take the cult story for example: I drove him to the meetings! Cause it was funny. It felt sitcom-ish.

But it didn’t start out that way… And while I’m not going to excuse or justify my actions, I didn’t change without reason. Alonso constantly made senseless choices. Well-meaning intent, timely interventions, deep discussions, seemingly transformative revelations, it didn’t matter. He’d pick the “wrong” option, and then, even more confounding was, he could not reconcile their (usually) logical outcomes. To a cartoonish degree! While his real friends grew increasingly frustrated with him, I just kinda checked-out.

Ironically, it was that very apathy that led us to hanging out the most we ever would. Was it because I stopped treating him like a “problem” needing to be “fixed” and it allowed us to grow closer? No, dear reader, I’m pretty sure he just enjoyed having an overt enabler. And, so… there we were: my ignorant, home-brew Don Quixote, and I, his twisted, off-brand Sancho Panza. Doing whatever I could to oblige the fighting of windmills. Chapter by chapter, off we went, filling the pages of our lowly “epic”.

🎶 But, master, you in luck ‘cause, up your sleeves 🎶 #

What made me that way? What was the attraction? What kept me fascinated? I’m no angel, but it’d be superficial to just write me off as a “heartless, manipulative jerk”. Likewise, it would be just as shallow to place the blame squarely on Alonso and his village idiot–like behavior. Here’s what I suspect:

Morbid curiosity might be evolutionarily ingrained in us. Left to its own devices, traffic WILL slow for a train wreck. “The News” IS biased toward the negative. Surely you’re familiar with freak shows, traveler? The fascination with the unusual, isn’t so unusual. But I’d say this still only addresses about half of the “lolcow phenomena”.

The science of laughter is surprisingly vast, nuanced and unfinished. But as far back as Ancient Greece there have existed theories linking humor and superiority. And while it’s obvious that there’s more at play here, my personal feelings on the matter hint at this being a significant element.

🎶 You got a brand of magic never fails 🎶 #

In my opinion, Alonso rarely cared about anything but himself and our time together was nothing short of reciprocal parasitism. But he never needed me as much as I needed him. I used him to feel good about myself. I used him as a key to wacky, dreamlike affairs. And I used him as a vicarious substitute for failure, to avoid the criticism that I both hypocritically dished him and secretly, childishly feared.

Alonso was a fool, in more ways than one. But even fools bear wisdom. He was eternally blind (and deaf) to it mind you, but it was there. Unbelievable were the moments where things just… worked out for him. Frequent too, beyond that of a broken clock. A “man of action”, and that’s not meant to be a compliment; if he had thoughts I can practically guarantee they were worthless. But fortune favors the bold, and it never audits the source.

That walking, talking joke… That absolute circus clown of a man! For all his faults, he still, somehow, managed to embody a valuable truth. One more priceless than even the lifetime of laughs he left me with:

“To live is to lol.”

So, visitor, I ask you, truly–

got milk?

Contributors: Plato | Miguel de Cervantes –> Don Quixote [#1 & #2] | One Thousand and One Nights –> Aladdin –> Aladdin (1992) –> Robin Williams –> Friend Like Me [#1 & #2] –> Ali Baba and the Forty Thieves –> Scheherazade | Elbert Hubbard –> Little journeys to the homes of American statesmen | Coltan Scrivner | Jonathan Sandling | Vox –> Dylan Matthews | Scientific American –> Giovanni Sabato | Geno Samuel | Hiromu Arakawa –> Fullmetal Alchemist –> Equivalent Exchange | Akira Toriyama –> Dragon Ball Z –> Vegeta’s Pride | Trey Parker & Matt Stone –> South Park –> Britney’s New Look

Syndication: Agora Road, Mastodon & basement community

This blog post is part of Agora Road Travelogue - Oct [#1 & #2]. A project attempting to encourage individuals to both write and exercise ownership over their content.

Comments #

Email: hi@foreverliketh.is

IndieWeb h-entry

My Lolcow & Me

Published by foreverliketh.is on 29^th October 2023

Equivalent Exchange

🎶 Well, Ali Baba had them forty thieves 🎶

My Lolcow & Me

Agora Road, Mastodon & basement community

Psychology

Knowing Your Workflow Tools: pnpm-only

The Opinionated Software Architect - Medium — Sun, 29 Oct 2023 00:38:56 GMT

Assure pnpm is the only package manager used in your node projects.

Photo by Álvaro Serrano.

The Opinionated TLDR

The pnpm-only-template repository on GitHub includes a shell script that configures your node project to exclusively use pnpm as its package manager. Also consider:

Mastering the tools you employ in software development.
Favor deep expertise in a focused set of tools over a cursory understanding of many.
Favor constraining the workflow over documentation.

Introduction

I enjoy coding and software architecture, but dealing with the tools that facilitate that? Not so much. I used to get by knowing just the basics of each tool; just enough to “get the job done.”

However, I’ve learned much too late, that to have an effective workflow to do the things I like, I need to really understand the tools I use.

Taking a step back, it’s been a transformative experience to deep-dive into the tools that shape my coding process, starting with setting up and configuring node packages. The package manager I use for node projects is pnpm.

Assuring that only pnpm is used as the package manger for a given project goes beyond just documentation. Constraining the workflow itself is a more effective way to guide new developers, reducing the likelihood of them accidentally using the wrong package manager.

Enforcing pnpm as the Exclusive Package Manager

Ensuring that pnpm is the sole package manager for our projects required some research and community contributions. Special thanks to those who have shared insights on this topic.

Disabling yarn

Disabling yarn as a package installer was easiest.

Running only-allow in the preinstall script stops yarn, yarn add and yarn install: even with an un-initialized project. The results can be seen below.

$ yarn add {package}, yarn install or yarn
yarn add v1.22.19
info No lockfile found.
$ npx only-allow pnpm
╔═════════════════════════════════════════════════════════════╗
║                                                             ║
║   Use "pnpm install" for installation in this project.      ║
║                                                             ║
║   If you don't have pnpm, install it via "npm i -g pnpm".   ║
║   For more details, go to https://pnpm.js.org/              ║
║                                                             ║
╚═════════════════════════════════════════════════════════════╝
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation
about this command.

Disabling npm

It takes a little more work to disable npm from installing packages because npm installs packages before calling the preinstall script.

To stop npm from installing packages, we needed to add to package.json:

// package.json
{
  "engines": {
    "npm": "Use pnpm instead of npm."
  }
}

and create a .npmrc files with engine-strict=true:

echo "# Stop npm from being used as the installer.
# The following was added to package.json
# { \"engines\": { \"npm\": \"Use pnpm instead of npm.\" } }
engine-strict=true" > .npmrc

Running npm install and npm add now shows the following error:

$ npm install
npm ERR! code EBADENGINE
npm ERR! engine Unsupported engine

All of this is done for you using the following shell script.

What Sort of Works and Other Options

Pnpm Used to Initialize a Project

If a project has already been initialized by running pnpm install, running npm results in an error:

$ npm install
npm ERR! Cannot set properties of null (setting 'peer')
npm ERR! A complete log of this run can be found in: ....

Remove npm (not always possible)

Removing npm from the developers machine is an option.

Script a Replacement (not recommended)

# create the file

echo -e '#!/bin/bash
echo "Please use pnpm."
exit 1' > npm && chmod +x npm

# you can run it as follows but calling

$ ./npm

# this still runs npm
$ npm

# WARNING: Adding the current directory as part of the PATH can
# be dangerous.
# prepend the current working directory to your path
# MUST be prepended so the npm script will override npm
echo 'export PATH=".:$PATH"' >> ~/.zshrc && source ~/.zshrc

What Doesn’t Work

Using Package.json packageManager Option

At the time of writing, there is an experimental packageManager option in package.json which doesn't seem to work yet.

Conclusion

Mastering your workflow tools is not just a nicety; it’s a necessity for efficient and effective software development. While documentation has its place, favor a workflow that’s designed to guide developers in the right direction. This is what the pnpm-only-template aims to achieve.

Knowing Your Workflow Tools: pnpm-only was originally published in The Opinionated Software Architect on Medium, where people are continuing the conversation by highlighting and responding to this story.

HN500 Updates

Brenton Cleeland — Sat, 28 Oct 2023 22:49:31 GMT

HN500 is a tiny site that lists posts on Hacker News that receive over 500 points. Unfortunately, it was down for a few days this week. I migrated servers and had a mistake in the cronjob that fetches posts.

Fixed now!

The good news is that I think it's only really used by me (of course, there are no analytics, and I'm keeping it up for me, so I don't mind not knowing).

There are two features that I've added in the last few months that might be valuable to folks:

/summaries/ shows a summary of all posts from the last 7 days. The summaries are generated using the Kagi Universal Summarizer API.
The JSON Feed includes the summary and has been updated with better formatting for the story/comments links.

I rarely visit the HN500 homepage now and instead use the feed as a way to quickly catch up on the popular stories through my RSS reader.

[Computing] Expect-CT Lite: A humble proposal for minimal CT enforcement in TLS certificates

devever.net/~hl — Sat, 28 Oct 2023 17:07:00 GMT

Last week we witnessed the discovery of the compromise of an XMPP service, most likely by a Western nation-state. While it's not directly related to any attack which was carried out in this incident, one thing that has been brought to my attention in the aftermath of this is that Certificate Transparency logging remains optional for CAs.

[Computing] Mitigating the Hetzner/Linode XMPP.ru MitM interception incident, part 2: XMPP-specific mitigations

devever.net/~hl — Sat, 28 Oct 2023 17:06:00 GMT

I've had a few comments in response to my previous article, Mitigating the Hetzner/Linode XMPP.ru MitM interception incident which have provided XMPP-specific suggestions for how this could have been mitigated.

Practical Deep Learning

Vít Baisa — Sat, 28 Oct 2023 03:00:00 GMT

notes from the course by fast.ai

Hyperdimension Neptunia Re;Birth1 review

ブラック — Sat, 28 Oct 2023 03:00:00 GMT

I can't believe it's not kusoge!

The story of Hyperdimension Neptunia Re;Birth1 isn't really anything special. It's ultimately just kind of a generic save-the-world adventure story and the actual substance is in the (comedic) character interactions more than anything else. I don't think I can claim to have loved the story, but I did find it charming in its own way. At least I don't regret reading through it. The English translation seems a bit liberal, but not to a degree that it'd completely ruin the entire thing either.

The combat is surprisingly good. There's kind of a good rhythm to the whole thing, where you first try to get a sneak attack in the overworld, then take turns in the battle to position yourself optimally, chip away at the enemy's shields and then deal heavy damage to them. During boss fights you would also look to optimise the mileage you get out of your powerful EX skills too. It's also fairly fast-paced, with the possibility to skip most of the animations, so you can really charge through your battles.

And boy is it a good thing the combat feels as good as it does since this game leans very heavily on grinding to pad out the game. I don't think there's any way to get through the story without grinding. You just won't get the levels to clear all the level check bosses. I had trouble beating a couple of the bosses, both early in the game and late in the game, and when I searched for tips in case I missed something, most of the advice on the Internet was "grind more". So if there's a way to beat the game without grinding, it went unnoticed by me and seemingly the rest of the Internet too.

The grinding also extends to the achievements. About half of the achievements for this game require grinding, be it for character levels or money. The amount of grinding is pretty mental. Really has this "low budget, must stretch out the game somehow" thing written all over it.

Not to say that there's no aspect for battle tactics and gameplay. For example, I lost to a boss, restarted the fight and then won the second bout. No grinding needed in the between the attempts. However, I'm not sure if the victory game down to skills or just luck. I feel like there's a limited amount of things you can do in regards to your tactics when the boss has the ability to insta-kill two members from your three-member party. It's quite possible that the boss just unleashed less of its wave of death area-of-effect attacks on my party on the latter attempt.

The rest of the game mechanics are kinda lame. The quests feel like they were added to check a checkbox, same as with the colosseum. And the share balancing act that you need to do in order to unlock the true ending or to unlock the little sisters is also not amazing, especially since it'll involve those quests and colosseum. Not massively time consuming thankfully, but still a bit of a mindless grind, doing the same things again and again to shift some percentages around.

Technically the game isn't the best. While I do like the art, despite the fact that the game reuses assets like it's going out of fashion, the graphics felt so bad and blurry at the beginning that I felt the need to install the Neptastic Mod on my Steam Deck. That did certainly help the game look nicer, although the FPS would not necessarily always be a stable 60 FPS. It also felt like the FPS would gradually get worse as I kept the game open as the Steam Deck was asleep. I think I went from around 40 FPS to 60 FPS in the same dungeon by just simply restarting the game. Deck users beware.

But in general, Hyperdimension Neptunia Re;Birth1 does run pretty well on a Steam Deck. Maybe not as well as you'd expect considering that this is originally a PlayStation Vita game, but well enough even with the modding. Combined with the fast-paced combat and heavy focus on grinding, it makes for a pretty decent portable time killer when you have a couple of minutes to do something. Not really sure if I'd ever want to actually sit in front of my gaming PC and play through this game. It just doesn't feel like that kind of a game (quite possibly due to its Vita roots), which is why I played it from start to finish on the Steam Deck. Fairly certain even that I beat most of the game while sitting on the toilet.

Hyperdimension Neptunia Re;Birth1 is a bit of a messy game with good core gameplay loop and enough charm to keep you around. For the couple of euro I paid for it, it did make for some good portable entertainment. Just don't really expect much beyond that from it.

ActionMailer attachments in Ruby on Rails

RailsNotes, the Ruby on Rails guides you wished you had. — Sat, 28 Oct 2023 03:00:00 GMT

ActionMailer makes it easy to attach files to your emails. In this article, I show you how to attach single or multiple files, set custom encodings and mime_types, and attach images as inline attachments to display in your email body.

Serverless, GPU Accelerated OpenAI API Using Podman, Llama.cpp, and systemd - No OpenFaaS Here!

9μm Pixels — Fri, 27 Oct 2023 20:45:24 GMT

Recently, I began putting together a new personal server for large language model training and inference. I really was interested in experimenting more with llama models and wanted the faster iterations having a local development machine would bring.

TLDR: Script here that creates a podman container, systemd service, and socket

The Attention Mal-Economy and My Brush with Terence Tao

Synystron Synlogica — Fri, 27 Oct 2023 19:54:21 GMT

I've been shocked, repeatedly, now in 2023 with how darn hard it is, anymore, to get any kind of praise (or even engagement) from others anymore, online.

2023-10-27-001

Srijan Choudhary, all posts — Fri, 27 Oct 2023 19:05:00 GMT

#Patterns

Sofa consensus

Blogs on Michail Rybakov — Fri, 27 Oct 2023 17:07:46 GMT

The contest of wills begins

--Mike-- — Fri, 27 Oct 2023 14:53:03 GMT

So, this morning when I click on my morning link to Peter Zeihan, I've been greeted with this, instead of him talking from some new location.

We've entered the contest of wills. Google/YouTube thinks that they can force ads down my throat in service of their customers(the advertisers), and is willing to block my ad blocker in an attempt to force my compliance.

I'm sure there will eventually be a different platform to watch Peter's videos on... and the wait begins. We programmers/technicians have a superpower, we're willing to put up with a ton of grief to bend things to our will, it's how we route around the normal rules and get shit done.

I curate my sources, and Peter is one of them, I'm not willing to pollute that stream of information with the Ads that Google wants to force feed me. It'll be interesting to see how this all works out.

FatturaElettronica for .NET v3.4.11

Nicola Iarocci — Fri, 27 Oct 2023 09:05:25 GMT

Fattura Elettronica for .NET v3.4.11 was released on NuGet today. The Fattura Elettronica project allows for the fixes a missing validation point. See the changelog for details (Italian). validation and de/serialization of electronic invoices following the Italian Revenue Agency standards.

RetroFit

Love, Blood & Rhetoric — Fri, 27 Oct 2023 07:13:31 GMT

“So, how does this work, exactly?” “First step is intake. That’s what we’re doing now. We’ll determine your goals and map out a strategy for how to achieve them. You’ll train in our bubble—the entire facility is temporally isolated—and we’ll reinsert you into the timestream when you’re ready. Guaranteed return within ten minutes of pickup. […]

Prototyping a PDF Chatbot from Scratch